[arin-ppml] debunking the myth that Moore's law helps

[Ted Mittelstaedt]

Matthew Petach wrote:
> On Wed, Dec 16, 2009 at 10:42 AM, Ted Mittelstaedt <tedm at ipinc.net> wrote:
> ...
>> Internet routing works the same way in the macro scale.  You wanna pass
>> a gig of data between point A and point B?  Well you can get a single
>> gig link or you can get 10 100mb links, you just need 10 routers to do it.
> 
> If each data flow is larger than 100mb, that doesn't work quite so well; but
> for many small flows, it's an entirely viable option.
> 
>> The problem I think is that the high power routers are so incredibly
>> expensive and multiple circuits are more expensive that network
>> administrators are still thinking more in terms of getting 1 big hulking
>> router and a high speed link rather than multiple smaller routers and
>> multiple links, when they need more throughput.  It simplifies network
>> design to think like this because you then don't have to deal with the
>> ickyness of load balancing.
> 
> Actually, the problem tends to be limited slot space on routers.  If put
> all your long haul links into one router, you can use 100% of the slots
> for long-haul links; traffic comes in, traffic goes out, all is good.
> 
> Now, if you want to split your traffic up across two routers, you don't
> get 2x the number of long-haul ports, because you have to burn a
> certain number of cross-link ports to get traffic between the two
> routers; in fact, the number of cross-links needed between the
> two routers is potentially equal to the number of inbound links
> into either one of the two routers (since worst-case scenario,
> every packet entering into router A has to cross over to router B
> to egress towards the destination).
> Thus, in your worst-case traffic flow scenario, you've actually
> gained zero additional long-haul links, because half of the ports
> you freed up on each router (by adding the second router) immediately
> get consumed to provide cross-links between the two routers.
> 
> "Fine", you may think.  "I'll just add another router into the mix...and
> *then* I'll be able to get more long haul ports."  But you again have
> to come up with some way to get packets from routers A and B over
> to router C, so you have to take away some of the long-haul ports on
> routers A and B so that you can plumb up crosslinks from A to C, and
> from B to C.  The good news is that since you now have even fewer
> long-haul links on A and B, you don't need quite as many cross links
> to each of the other routers; the bad news is, you now have twice as
> many routers for each router to cross-link to.
> 
> Ultimately, it ends up being a losing proposition, unless you can
> gamble on better than worst-case traffic scenarios, and build less
> crosslinks between routers than would be needed to handle the
> full worst-case traffic load.
> 
>> But, Moore had an answer to that, because the other part of Moores law
>> has to do with cost - meaning that if you keep the same transistor count
>> on a chip, the chip cost falls.
> 
> Only if your production costs drop for some reason; otherwise, keeping
> the same transistor count on the chip will continue to cost you the same
> amount of money.
> 
>> I suspect if the Cisco CRS-1 was selling for $500 USD, rather than $500K
>> USD, the answer would be pretty obvious to most people.
> 
> You can get old 7k-class Cisco routers for $500 on eBay; why aren't
> more people just running large clusters of $500 routers, then?  See
> the above cross-linking problem, and look at the rack space cost
> to put them in; at your typical facility like Equinix, at $1500/rack/month,
> if you can only fit 3 7k-class routers per rack, if the router only costs
> $500, you end up spending as much on space for the router as your
> hardware cost *every month*.  Logic says that if you can get the
> same interface count in a smaller package, and thus pay for less
> rack space each month, it's worth paying *more* for the hardware
> to reduce those ongoing monthly expenses.
> 
>> So yeah, I think Moore's Law applies.  I also think that the router vendors
>> believe that most backbone providers have deep pockets and will
>> continue to pay lots of money for high power routers, thus they are
>> not competing on price for the high-end routers, and thus not exercising
>> that aspect of Moore's law.  Otherwise you would see yearly price drops
>> on CRS-1 line cards.  Also the carriers themselves have a vested
>> interest in keeping high speed link prices high, too.  They would
>> rather charge the same money every year for an increasing amount of
>> fiber capacity than lower their price every year for the same amount
>> of fiber capacity.
> 
> Fiber costs are generally going down year over year, in areas where it
> is relatively open and plentiful (eg, most of the ARIN region).  But do
> keep in mind that data services over fiber cost a pathetically small
> amount compared to voice services, so if you're a telco, you have
> a massive incentive to reserve capacity for voice services as much
> as possible, and only sell off capacity for data when there's absolutely
> no possibility of voice demand rising to require the circuit.
> 
> Doing the math, if your voice service calls a penny a minute, for example,
> and uses a 64k DS0 channel, that means on an OC192 you can pack in
> 129,024 DS0 channels (24 DS0 per DS1, 28 DS1 per DS3, 192 DS3
> equivalents in an OC192).  So, that's 129024 cents, or $1,290.24 per
> minute if you sell that circuit for voice service.  There's generally about
> 43,200 minutes in an average length month; if we figure the circuit gets
> about 10% utilization for voice calls, that's about 4,320 minutes of voice
> calls going through it in a month (if you prefer, you can say 10% of the
> DS0s are in use for the full 43,200 minutes; the math comes out the
> same either way) -- you end up with potentially $1,290.24*4,320, or
> about $5,573,836.80/month revenue for an OC192 running at 10%
> capacity carrying voice services.
> 
> Compare that to the actual amount generally charged for a data service
> OC192 in the ARIN region, which is maybe $50,000/month, and you start
> to realize that the financial pressure is really to use as many circuits
> as possible for voice services, if you're a telco, and *only* sell data
> circuits when you just can't generate any additional voice demand,
> since you only make 1/100th the revenue on the data circuits that
> you can make off voice services running along the same circuit.
> 
> There's definitely pressures at work in the market; it's just somewhat
> dangerous to oversimplify what those pressures are.
> 
>> Whether this situation will continue is anyone's guess.  It definitely has
>> not continued in the low end SOHO router and switch market.  There,
>> everything is commodity, now.
>> Ted
> 
> The low end SOHO router and switch devices can't handle even close to
> a full set of internet routes...that's a completely false comparison, it's
> like trying to compare the cost for a huffy one speed beach cruiser bike
> with a 78 passenger bus.  Two completely different sets of requirements
> for the two, and you can't substitute one for the other, no matter how
> creative you try to get.
> 

I wasn't meaning substitution in that, but more history.

Up until Linksys came out with their BEF series, and charged $49.95 for
them, the typical SOHO router was in the $300-$500 range.  The Linksys
devices were game-changers.  Linksys could afford to do it this way 
because they worked hand-in-hand with one of the chip foundries, 
Broadcom probably, to come out with a single-ASIC design that put the
CPU and ethernet ports all on one chip.  That was only possible because
of Moore's Law.

Production costs for all of the other vendors still did remain the
same for their designs, and they didn't lower their prices - with
the result that I saw quite a lot of NetScreens and stuff like that
get replaced with the Linksys devices. - and the end result that
companies like SMC and 3com became non-players in that market.

Right now a small handful of router vendors, ie: Cisco and Juniper,
have a lock on the mid-to-high end router market.  But it's anybodies
guess that this lock will remain since it could be
broken at any time with another game-changer appearing out of nowhere.

Ted
> Matt
>