[arin-ppml] Draft Policy ARIN-2019-2: Waiting List Block Size Restriction

[Ronald F. Guilmette]

In message <FB276AD4-B5B5-4570-9FB1-B85897B79F6F at netconsonance.com>, 
Jo Rhett <jrhett at netconsonance.com> wrote:

>Robert,

It's Ron, actually.  But you can call me rfg.  Everyone else does.

>I'd first like to ask that you back off on your tone. I
>have treated you with respect...

As I have done for you.  Disagreement, even vigorous disagreement, on
purely technical grounds is not equal to disrespect.   I have not
questioned either your motives or your ancestry, nor will I.  I am
completely sure that both are entirely beyond reproach.  And I have
not called you "ignorant" as you have done to me.  If anyone has a
basis for indignation, that would be me.  But I prefer not to dwell
on such small slights, to turn the other cheek, and to continue to
engage with you on the technical points of your arguments and assertions,
seeing where we may have points of common agreement.  There appear
to be many.

>> I _am_ familiar with the IPv4 16-bit limit on port numbers, but I fail
>> to see why that should affect the allocation size choice.  How many
>> IPv4 port numbers does it take to support 10,000 unique and distinct
>> web sites?
>
>I have production services implemented by a small cluster of nodes which
>burn through more outbound TCP ports than can be accommodated for by
>less than a /26.

I'll take your word for it.  But I'm not sure how what you just said in
any way negates the part of what I said that you quoted.

You pretty obviously have some rather unique and distinctive stuff going
on if your web *servers* need to make so many *outbound* connections that
you need a whole /26 just to accomodate them all.

Or were you seriously intending to assert that having web *servers* that
make lots of lots of outbound (client) TCP connections is in some sense
"typical" for the average Joe on the Internet?

I'm in complete agreement if you say that one may need lots and lots and
LOTS of IPv4 ports, possibly even distributed across lots and lots and LOTs
of individual IPv4 addresses.  Heck!  Just ask Comcast!  But the assertion
of mine that you quoted, and that you appear to want to take issue with,
intended to refer only to *typical* server-side stuff.  Thus, I am inclined
to wonder if you and I are even really disagreeing at all.  (It would
appear not, but I'm perfecvtly happty to let you be the judge of that.)

>> How many IPv4 port numbers does it take to support mail service for
>10,000 dmains?  Answer: One.
>
>Only if all 10,00 domains only send mail to 6 different locations within
>the TIME_WAIT period.

That's a *very* interesting point, and I am very glad that you raised it.

I infer from what you just said that you are assuming... quite correctly,
I think... that each -outbound- (client-side) connection, e.g. from my mail
server to your mail server, typically makes use of its own unique and
(at least during the transaction) dedicated outbound TCP port number.
Thus, if I am supporting 10,000 domains for email on my single little
IPv4 address, and if -all- of them were to try, simultaneously, to send
mail to 6 or more different other/remote places, then yes, I can see that
this would promptly exhaust all 65536 potential outbound TCP port numbers
and that thus, email havoc might then possibly ensue.  (Or maybe the mail
server would just wait until some of the outbound ports got freed up.)

I think that you are correct that this is indeed the way that most or all
existing mail servers handle their outbound flow.  I mean it's just a lot
simpler, programatically, to do things this way, i.e. using each local port
number in a "dedicated" fashion for each outbound (client) SMTP connection/
transaction.

But just because this is, or may be, both traditional and programatically
much easier than the alternative, I'm not sure that it *must* be that way.

I may not be the world's biggest or best hotshot sockets programmer, but
I've done a bit of it in my time, and my recollection is that each TCP
connection is denoted, on each end, by a unique tuple consisting of (at
least) the source IP and source port number, *and* also the destination
IP and destination port number, and that the only requirement is that
the entire tuple, when considered as a whole, be sufficiently unique so
as to allow the OS to reliably sort out which packet should go to which
process on each end on the wire.

If so, then it would seem to me that, in theory at least, a mail server,
when acting in its role as an SMTP -client-, could carry on a mutitude of
simultaneous connections / SMTP sessions, all using only a single TCP port
on the local machine, provided that each of these outbound / client
connections were to a unique and different remote IP address.

In short, even when a given mail server hosted on a single IPv4 address was
asked to make, say, a million simultaneous outbound TCP/SMTP connections
to a million remote mail servers elsewhere, it could most certainly do
so, I believe, by simply using each of the outbound TCP port numbers
available to it for multiple simultaneous connections to mutiple other
mail servers elsewhere.

If someone were to code up (or if someone already _has_ coded up) a mail
server which supported this rather trivial port reuse trick, then that
might be reasonably expected to aleviate the kind of port number exhaustion
(specifically for mail servers acting in heir capacity as mail -clients-)
that you have quite reasonably noted.

I'll have to ask around and find out if any mail servers/clients are
already doing this, and if not why not.  It may perhaps just simply be
the case that nobody saw it as being worthwhile to implement, but
certainly, if one were in fact trying to support, say, mail for a VERY
large number of domains, using just, say, a 32-core Ryzen with a big pipe
on a single IPv4 address, then it's possible that a failure to make fully
efficient use of the available IPv4 outbound port numbers could create
unfortunately low hard limits where using the port numbers efficiently
might allow millions of domains to be supported, for both inbound and
outbound email, on a single IPv4 address.

As IPv4 addresses become more scarce and more expensive, it may become
worthwhile (for mail server authors) to make better use of the available
IPv4 port numbers on the outbound (client) side of things.  In any case
it is definitely worth me asking about whether or not mail servers currently
can make multiple simultaneous uses of a single otbound port number or not,
and I intend to do just that (and then report back).

>And before you try and do the math, recall that needs to include reverse
>auth, DKIM auth, SPF auth, domain callbacks, greylisting/mandatory
>retrys  a single message delivery can utilize as many as 24
>tcp/udp ports these days.

It's an intersting assertion, although I'm not sure that I see how either
DKIM or SPF enter into the equation at all.  As I understand it, those are
verified via DNS (udp/53) and thus should not have an effect on port usage.

Callbacks mean your mail server needs to make an outbound SMTP connection
for each inbound one, and thus, I must conceed that you have a point that
this might impact total outbound TCP port usage.  But as noted above, this
likely only becomes an issue when you are already near saturation of the
available 64k - 2K outbound port numbers, and even in such cases, it seems
like it should be possible to use single outbound TCP ports for multiple
outbound SMTP connections, which, if it's actually possible, would pretty
much solve even that extreme port exhaustion problem.  But I need to make
some inquiries about that.

Greylisting just means that your mail server, whan acting in its capacity
as an SMTP client, may sometimes needs to drop the connection (thus giving
back the outbound port number it was using) and then try again later, so
I'm not sure that I see how that affects the calculation either, other
than that fact that, at scale, it mans your mail server is likely to
spend a bit more time being a client than it outwise would, i.e. in the
absence of greylisting.

>> How many IPv4 port numbers does it take to support DNS service for
>10,000 domains?
>
>Only if those domains only request or forward queries for 6 outside
>references a second.

Here again, you are starting from the assumption that each -outbound-
(client side) DNS query needs to have its own unique port number.  I could
be wrong about this, but I don't believe that that is actually the case,
even for current gen (or even prior gen) name servers that are already widely
deployed, and that have been, for many years already.  In fact I seem to
vaguely recall there being some setting in BIND 9 where the operator could
instruct the thing to originate -all- outbound queries from a single local
port number, e.g. UDP port 53.

Or maybe I'm mis-remembering.

On the other hand, if I am actually remembering that correctly, then I think
that it is possible to operate a name server on a single IPv4 address which
not only provides authoritative service (on a single UDP port, 53) for
essentially unlimited numbers of local zones, but one which can also (and
simultaneously) act as a general resolver, providing answers pertaining
to non-local zones, and that it can serve in this (resolver) role as well
while still only using only a single IPv4 address -and- only a single local
port number (e.g. udp/53).

But I'm sure that someone will correct me if I'm wrong about that.

>And that's only on
>the server side: modern clients can burn 100 query ports loading a
>single web page.

I have already conceeded the point that _clients_, generally speaking,
and with the exceptions noted above, may need a LOT of both IP ports
and a lot of IP addresses.  And I cited Comcast as the quintessential
example.  But it is still my contention that most or all support for
end-luser client stuff should move to IPv6, leaving IPv4 for use as
servers.  It is also my contention that when and if IPv4 addresses
begin to be used only and exclusively for servers, then even a lowly
/24 block will be vast overkill for the vast majority of organizations.

If the addresses and ports are used with proper efficiency, then giving
someone even just a whole /24, even at the present moment, is arguably
analogous to giving them the entire contents of Lake Superior when really,
all they needed was to fill their bathtub.

Such uncareful and profligate use of IPv4 addresses, although common and
routine since the beginning of the Internet, is provably unsustainable,
going forward.

>> Given these facts, I'm not persuaded that the portion of your comment
>> relating to IPv4 port numbers either does have or should have any
>> relevance to the selection of an appropriate initial IPv4 allocation size.
>
>You aren't making a valid argument, simply demonstrating your
>own lack of knowledge of how IP works in today's services.

I can only encourage you to try, as best as you are able, to stick to
addressing the technical points, and leave the ad hominems for some other
day and for some other and more appropriate forum.

I believe that I have addressed above, point by point, all of the various
(mostly obvious) ways in which organizations of virtually all stripes
could make vastly more efficient use of each and every one of their IPv4
addresses.  (Your organization, Mr. Rhett, may perhaps be the exception
That proves the general rule.) If there is a clear and compelling case
to be made that even very large (and *typical*) hosting organizations
could not support even up to, say, a million customer domains, ON THE
SERVER SIDE, using a single IPv4 address, then I, for one, don't believe
that you have yet persuasively made that case, arm waving and personal
attacks aside.

>You appear to be thinking of the 1980s when a single TCP connect was the
>session ;-)

I would turn this around and say that you appear to be thinking in 1990s
terms, where each web site needed its own dedicated IP address, where
each mail server only supported a single domain, an where each name server
did likewise.  But we have come a long way from that, and nowadays we have
better tools, better software, and much better ideas of how to support
lots and lots of server-side stuff on each single IP address.

Which set of notions about the current best way to maximize the efficient
use of each individual IPv4 address, your's or mine, are more apropos to
the current era and the current situation will, of course, be judged by
the rest of the folks here.  I can only restat what I believe to be my
basic claim, which is that profligate and INefficent use of IPv4 addresses...
by snowshoe spammers, by horders, by speculators, and even by a majority
of run-of-the mill organizations... is at present far more the rule than 
is is th execption, and that it is appropriate that policy should be guided
by that undeniable reality.  (If fact, we are only even discussing these
issues because, as reported by ARIN itself, the recent activities of the
speculators has now, finally, gotten rather entirely out of hand.)

>Further, very few services
>return the entire content themselves, but instead build it from hundreds
>of other sources.

Now THAT is an extraordinaily sweeping assertion!

Can you support it with actual facts, as opposed to just the assertion,
standing by itself?

I certainly know that *my* little personal web site(s), when they were up,
quite certainly did not need to "build content from hundreds of other 
sources".

I understand that you may resonably assert that me and my little web sites
don't count as being "typical" in the modern era, but I would simply turn
that around and ask you what makes you so sure that these complex things...
whatever they are... that you allege are "built from hundreds of other
sources" might not also be rather entirely atypical and not at all
representative of the norm on the modern Internet, and that thus, it is
your experinnce, perhaps in addition to mine, that is far out on one
extreme of the bell curve r the other, and that thus, neither should be
used as the last word or deciding factor in any decision intended to
address the most common usage scenarios.

>> Your concern about the inherent port number limitations of
>> IPv4 seems to arise exclusively with respect to the use of IPv4
>> addresses as _clients_
>
>I think I've already addressed this. I haven't worked
>anywhere that a single connection to the public server didn't
>require hundreds of related connections to other public services to
>deliver the page you see. Nor have I seen many web pages which don't
>contain content loaded from less than 50 places. The world is more
>complex these days.

Again, we seem to be talking past one another.  I assert yet again that
I've already conceeded that ON THE CLIENT SIDE lots and lots of IPs and
also, perhaps, lots and lots of port numbers may be required, and I believe
that I can count on Comacast and other such "eyeball providers" to support
assertion/concession.  But I also reiterate that -clients- and -eyeballs-
should be migrated to IPv6, immediately if not sooner, leaving IPv4 to be
used for the -efficient- (and highly shared) hosting of servers.

>and ARIN issues IPs to clients, so their needs must be
>considered in IP policy.

I agree.  But this is 2019, not 2003.  If entities need to support gobs
and gobs of eyeballs/clients, then ARIN should be giving them nice juicy
blobs of of IPv6 that they can swim around in, all the live long day.

Clients/eyeballs can live, breath, and be happy on IPv6.  Unfortunately
however, -servers- still need to be on IPv4 in order to be accessible
to everyone.  But as I have elaborated above, these days you can pack
server-side support for one hell of a lot of domains... perhaps several
million... onto even just a -single- IPv4 address, *if* you do it properly.
So giving any new applicant even a whole IPv4 /24 is arguably ridiculous
overkill (by a factor or 256) which is only even made necessary by the
fact that it is hard or impossible to get anything smaller than that routed.

>Yes, yes
>we should be embracing IPv6, and I personally am doing everything I can
>to drag organizations to that conclusion. But the overwhelming evidence
>is that enterprises and many regional networks are not playing ball.

Fortunately, the solution to this longstanding problem is finally at hand!
We now have ARIN itself telling us all, point blank, that they have caught
some folks red handed, gaming the curent IPv4 allocation system for private
profit.  This is a rather radical new development!

To quote again from one of my favorite authors "As our situation is new,
so me must think anew and act anew."

The present circumstance gives the entire ARIN community exactly the kind
of golden opportunity that it has never yet had, and that it may never have
again.  It may only be at this specific moment in history when ARIN might
have, in hand, a compellling justification to begin insisting that -new-
"eyeballs" (i.e. clients) be pushed onto IPv6 whether they like it or not.

Everyone who has been advocating and evanglizing for IPv6 adoption since
its arrival on the scene should be relishing and reveling in this moment,
and should be supporting the limitation of IPv4 allocations, going forward
to no more than a /24, *except* in cases where a compelling case can be
be made that EVEN WITH the most modern address and port reuse techniques
presently available, there is a demonstratable need for something in
excess of 256 unique and independent *servers* by the requesting party.

>Ronald, in summary: I get that you have only faced certain of the
>limitations of IP. And I get that you want a simple world.

Likewise and conversely, I get that your unique and atypical usage may
compel yu to create "servers" which themselves also must act as clients
to a highly unusual level, and that you thus have reasonable concerns
about being able to support your unique usage patterns, going forward.
I get that.  I just don't think that your unique requirements are or
should properly be the one and only basis for a generalized allocation
policy, intended for all, going forward,

>Understand
>that (A) it's not as simple as we might all want it and (B)
>being rude and disrespectful to people while discarding the possibility
>of usage or needs outside your own experience does not help us create
>good policy.

I agree with both assertions and merely hope that just because you've never
before been asked to consider how you migh further minimize your usage of
IPv4 addresses ON THE SERVER SIDE, you will not allow that lack of
experience on your part to cause you to carelessly or frivolously discard
the possibility that what I have suggested in the way of optimizing usage
of IPv4 addresss may in fact be do-able, even if it may also be outside
of your personal past experience and/or outside of your organization's
current capacities, based on the unique complexities of your operations.

In short, I don't believe that it is either radical or disrespectful to
state the obvious, which is that IPv4 addresses are not, by and large,
used efficiently at the present time.  Your organization may be, and is,
for all I know, a shining exception to this general rule.  But that doesn't
mean that the general observation is not valid.

>I spent an hour writing this message just to get you to
>calm down a bit, and stop writing to us like we're all idiots.

I am and have been perfectly calm.  I have no idea why you would suggest
otherwise.

I spent more that an hour writing this reply, which attempts to address
your technical points, one by one, and I don't think I've treated anyone
like an idiot.  But if there is consensus on that point, I will concede
it.  I ask for other opinions on the following simple proposition:  Has
rfg treated or addressed anyone as an idiot?  If there is unanimity here
that I have done, then I will apologize forthwith a review my manner of
communication with an eye towward improvement.

I believe however that I've stuck to the technical issues, and that the
points I've made have technical merit.  You obviously believe otherwise,
and consider *me* an idiot, apparently based primarily or exclusively on
the fact that I simply disagree with your view, technically.  So be it.
I can only encourage you to try to stick to discussion of the technical
points, and leave the personally directed critiques for another time and
and for some other more appropriate forum.


Regards,
rfg