[arin-ppml] Policy Proposal: A Modest Proposal for an Alternate IPv6 Allocation Process

[Leo Bicknell]

In a message written on Fri, Jun 05, 2009 at 11:06:32AM -0700, Ted Mittelstaedt wrote:
> Is this an accurate assessment?  Or is there really some reason that a
> router cannot be built with more ram than a half gig?

It is not an accurate assessment.

Plenty of routers have more than half a gig of ram.  Cisco 7301's
and 7200's with NPE-G[12]'s take 1G.  GSR's take 4G.  Juniper M20's
have 2G in them, and they are an old platform!

The memory to store BGP tables is not a huge problem.  There is a wall
at 4G, in that some of the router OS's aren't 64 bit; but quite frankly
even a very busy peering router is unlikely to be using more than around
1.5G for BGP information.  "Show ip bgp sum" on your favorite Cisco will
tell you in the header block how much DRAM you are using.

The problem is that those BGP routes (the RIB, routing information base)
have to be distilled into the set of forwarding routes.  You've done
this manually, route xyz comes from providers a, b and c, we like b, the
next hop is from a static route which points to the far end of connected
interface foo, so xyz->foo.  That distilled information is generally
called the FIB (forwarding information base).

Now consider a core router with 10 10GE interfaces (100Gbits/sec through
the box).  DRAM does not have enough bandwidth to look up that packet
rate of routes in the FIB, and even if it did the latency to do the
lookup would cause you to buffer hundreds of megabytes of data on the
line cards.  It just can't be done with off the shelf parts.

Folks build things with off the shelf parts, look at Vyatta for an
example.  You'll see how far "standard PC's" can be pushed.  It's a long
way, but not even remotely close to what core boxes need.

So virtually all vendors use something called TCAM.  Conceptually think
of it as hash memory. Rather than telling the memory you want address
1234 and getting back a value, you tell the memory you want the
corresponding pair to 10.0.0.0/8 and get back a value.  It's also
blisteringly fast, way faster than DRAM.

Here's some good summaries:

http://www.enterprisenetworkingplanet.com/nethub/article.php/3527301
http://en.wikipedia.org/wiki/Content-addressable_memory#Ternary_CAMs

Of course, this is also only part of the story.  It's not good enough to
look up the route and forward; you also need to look up things like
filters to see if the packet should be filtered, or ARP entries, or MPLS
lables.  Many boxes have multiple fast TCAM's and SRAM banks connected
by custom designed hardware ASICs to be able to do this at line rate.

This comes back to why some boxes are DRAM limited.  If you only have
enough TCAM to store, say, 256k distilled routes, why would you put
enough DRAM on the box to store 20 million routes?

Anyway, this isn't a router architecture list.  Google TCAM, join
Cisco-nsp or Juniper-nsp, and/or have your hardware vendor send out
an engineer.  While I'm not sure I would say in all instances these
large boxes are offered at a fair price, it's also not the case
that $100 of DDR2-800 would fix the problem.  There are real,
serious, expensive engineering challenges moving more than a couple
of Gigabits per second.

-- 
       Leo Bicknell - bicknell at ufp.org - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 825 bytes
Desc: not available
URL: <https://lists.arin.net/pipermail/arin-ppml/attachments/20090605/cfc53a5f/attachment.sig>