[arin-ppml] debunking the myth that Moore's law helps

[Matthew Petach]

On Wed, Dec 16, 2009 at 10:42 AM, Ted Mittelstaedt <tedm at ipinc.net> wrote:
...
> Internet routing works the same way in the macro scale.  You wanna pass
> a gig of data between point A and point B?  Well you can get a single
> gig link or you can get 10 100mb links, you just need 10 routers to do it.

If each data flow is larger than 100mb, that doesn't work quite so well; but
for many small flows, it's an entirely viable option.

> The problem I think is that the high power routers are so incredibly
> expensive and multiple circuits are more expensive that network
> administrators are still thinking more in terms of getting 1 big hulking
> router and a high speed link rather than multiple smaller routers and
> multiple links, when they need more throughput.  It simplifies network
> design to think like this because you then don't have to deal with the
> ickyness of load balancing.

Actually, the problem tends to be limited slot space on routers.  If put
all your long haul links into one router, you can use 100% of the slots
for long-haul links; traffic comes in, traffic goes out, all is good.

Now, if you want to split your traffic up across two routers, you don't
get 2x the number of long-haul ports, because you have to burn a
certain number of cross-link ports to get traffic between the two
routers; in fact, the number of cross-links needed between the
two routers is potentially equal to the number of inbound links
into either one of the two routers (since worst-case scenario,
every packet entering into router A has to cross over to router B
to egress towards the destination).
Thus, in your worst-case traffic flow scenario, you've actually
gained zero additional long-haul links, because half of the ports
you freed up on each router (by adding the second router) immediately
get consumed to provide cross-links between the two routers.

"Fine", you may think.  "I'll just add another router into the mix...and
*then* I'll be able to get more long haul ports."  But you again have
to come up with some way to get packets from routers A and B over
to router C, so you have to take away some of the long-haul ports on
routers A and B so that you can plumb up crosslinks from A to C, and
from B to C.  The good news is that since you now have even fewer
long-haul links on A and B, you don't need quite as many cross links
to each of the other routers; the bad news is, you now have twice as
many routers for each router to cross-link to.

Ultimately, it ends up being a losing proposition, unless you can
gamble on better than worst-case traffic scenarios, and build less
crosslinks between routers than would be needed to handle the
full worst-case traffic load.

> But, Moore had an answer to that, because the other part of Moores law
> has to do with cost - meaning that if you keep the same transistor count
> on a chip, the chip cost falls.

Only if your production costs drop for some reason; otherwise, keeping
the same transistor count on the chip will continue to cost you the same
amount of money.

> I suspect if the Cisco CRS-1 was selling for $500 USD, rather than $500K
> USD, the answer would be pretty obvious to most people.

You can get old 7k-class Cisco routers for $500 on eBay; why aren't
more people just running large clusters of $500 routers, then?  See
the above cross-linking problem, and look at the rack space cost
to put them in; at your typical facility like Equinix, at $1500/rack/month,
if you can only fit 3 7k-class routers per rack, if the router only costs
$500, you end up spending as much on space for the router as your
hardware cost *every month*.  Logic says that if you can get the
same interface count in a smaller package, and thus pay for less
rack space each month, it's worth paying *more* for the hardware
to reduce those ongoing monthly expenses.

> So yeah, I think Moore's Law applies.  I also think that the router vendors
> believe that most backbone providers have deep pockets and will
> continue to pay lots of money for high power routers, thus they are
> not competing on price for the high-end routers, and thus not exercising
> that aspect of Moore's law.  Otherwise you would see yearly price drops
> on CRS-1 line cards.  Also the carriers themselves have a vested
> interest in keeping high speed link prices high, too.  They would
> rather charge the same money every year for an increasing amount of
> fiber capacity than lower their price every year for the same amount
> of fiber capacity.

Fiber costs are generally going down year over year, in areas where it
is relatively open and plentiful (eg, most of the ARIN region).  But do
keep in mind that data services over fiber cost a pathetically small
amount compared to voice services, so if you're a telco, you have
a massive incentive to reserve capacity for voice services as much
as possible, and only sell off capacity for data when there's absolutely
no possibility of voice demand rising to require the circuit.

Doing the math, if your voice service calls a penny a minute, for example,
and uses a 64k DS0 channel, that means on an OC192 you can pack in
129,024 DS0 channels (24 DS0 per DS1, 28 DS1 per DS3, 192 DS3
equivalents in an OC192).  So, that's 129024 cents, or $1,290.24 per
minute if you sell that circuit for voice service.  There's generally about
43,200 minutes in an average length month; if we figure the circuit gets
about 10% utilization for voice calls, that's about 4,320 minutes of voice
calls going through it in a month (if you prefer, you can say 10% of the
DS0s are in use for the full 43,200 minutes; the math comes out the
same either way) -- you end up with potentially $1,290.24*4,320, or
about $5,573,836.80/month revenue for an OC192 running at 10%
capacity carrying voice services.

Compare that to the actual amount generally charged for a data service
OC192 in the ARIN region, which is maybe $50,000/month, and you start
to realize that the financial pressure is really to use as many circuits
as possible for voice services, if you're a telco, and *only* sell data
circuits when you just can't generate any additional voice demand,
since you only make 1/100th the revenue on the data circuits that
you can make off voice services running along the same circuit.

There's definitely pressures at work in the market; it's just somewhat
dangerous to oversimplify what those pressures are.

> Whether this situation will continue is anyone's guess.  It definitely has
> not continued in the low end SOHO router and switch market.  There,
> everything is commodity, now.
> Ted

The low end SOHO router and switch devices can't handle even close to
a full set of internet routes...that's a completely false comparison, it's
like trying to compare the cost for a huffy one speed beach cruiser bike
with a 78 passenger bus.  Two completely different sets of requirements
for the two, and you can't substitute one for the other, no matter how
creative you try to get.

Matt