small multihomed organizations

[Scott Richard Slater]

Jason - see below

----- Original Message -----
From: "Redisch, Jason" <jredisch at virtela.net>
To: "'Scott Richard Slater'" <scottslater77 at hotmail.com>
Cc: <ppml at arin.net>
Sent: Thursday, August 09, 2001 10:19 AM
Subject: RE: small multihomed organizations


> Scott,
>
> The problem is we are trying to solve is getting a small multihomed end
user
> 'global' redundant connectivity using the tools available to us at ARIN.
I
> do not think modifying filter lists or forcing all customers to ARIN are
> solutions we should be looking at.

I am NOT saying that "modifying filter lists or forcing all customers to
ARIN" are solutions.  I do not know if there is a specific solution that
will completely satisfy small multihomed customers.

>
> If a customer is allocated a /24 from their upstream provider today they
can
> announce this block from both circuits over both providers.  A majority of
> the Internet will reach them. Those upstream providers that filter will
> still reach them based on the larger aggregate announcement of the
provider
> that assigned the customer the IP space.

You say "majority of the Internet will reach them".  This is NOT necessarily
true.  Let's just say for an example that XYZ Corporation is multihomed to
ISP1 and ISP2.  (Keep in mind XYZ Corporation wants to control/balance
packet flow over both connections.)  With your statement ISP1 would delegate
a /24 (256 hosts) to XYZ Corporationn from their larger aggregate of
138.39/16  Let's say that is 138.39.1/24   Now ISP1 is also going to
delegate a /30 to XYZ Corpration for physical connectivity.  It doesn't
matter what that block is.  The /24 is back end network for the customer.
The network they want to Internet to accept.   ISP2 would then only delegate
XYZ corporation just a /30 for the physical connectivity to XYZ corporation.
It doesn't matter what that block is. OK, this is a common configuration.
Now, XYZ can announce 138.39.1/24 to ISP 1 and ISP 2.  ISP 1 would aggregate
the 138.39.1/24 into their 138.39/16 and announce 138.39/16 to the Internet.
ISP2 would receive the 138.39.1/24 from XYZ Corporation and not be able to
aggregate the block.  It would then announce the 138.39.1/24 to the
Internet.  Now, there is no guarantee that any of ISP2 peers will accept the
138.39.1/24.   As some ISPs reject routes for prefixes with lengths greater
than /19 or /20 for newly delegated address space.  Now, this is a major
problem for the XYZ Corporation.  This also includes the problem if the
ISP1-XYZ Corporation connection is not functional then there is no gurantee
that the Internet will accept the 138.29.1/24 route.

This is not a best solution.

>
> If a customer is allocated a /25 from their upstream a much larger
> percentage of the Internet (most) will filter them and the process breaks
> down.  Most traffic comes over the provider that allocated them the IP
> space. If that provider fails a variety of events can occur, many of which
> result in loss of connectivity for the customer.

Actually, its the opposite.  Normally, the ISP that allocates IP space to a
customer is the backup connection to the customer in a multihomed topology.
This can be fixed if the allocating ISP backdoors the routes the customer is
announcing to them.

>
> One option we might look at is option one described below.  Multihomed
> customers have a specific set of ARIN requirements already in order to
> justify an AS.
>
> "In order to be assigned an ASN, each requesting organization must provide
> ARIN with verification that it has one of the following:
>
> 1) A unique routing policy (its policy differs from its border gateway
> peers)
> 2) A multi-homed site. "
>
> We could recommend modifying the policy so that if you have justified an
AS
> number you also justify a /24 from one of your upstream providers
regardless
> of utilization.

I do not agree with this.  Where is it said that small organizations who
choose to multihome with two different ISPs and receive an ASN justify 256
hosts.  Remember "Conservation of IP address space".  Also,  I know that
ISPs will not be happy allocating, per ARIN, a /24 to every multihomed
customer with an ASN.  Maybe customers should just look at multhoming with
one provider.  I know this is still a single-point of failure but it solves
these problems.  The customer could get each connection from a different
router/switch to decrease a single point of failure.  The customer will also
probably get a better price on bandwidth and other services with the ISP.
No ISP likes multihomed customers to them and another ISP, all ISPs want all
the bandwidth.

>
> This will solve a problem where a small multihomed organization has an AS
> number and two providers but does not have enough hosts to justify a whole
> /24.

Where do the wasted IP addresses go?  Let's say the customer only needs 5
hosts.  Where does the other 251 hosts go?  Not a good solution.  I guess,
smaller organizations will just have to settle with getting two connections
from one ISP until they can grow to a larger organization and get a
connection to a second ISP and receive 'globally' routeable address blocks.

*Multihoming is a luxury for a company and needs to be seen that way.  Where
does it say anywhere that if you have one connection to the internet you
should deserve two, so there is no single point of failure ?  Hey, I have
cable modem in my home residence, I would love to also have a DSL, T1, T3,
et cetera... so I can have a redundant home network, BUT I can not justify
it.   The current policy is pretty good.  I would prefer to look at
addressing filtering policies or moreso creating one. :)

>
> This would be a policy change and does not match the current allocation
> guidelines. However, believe it is within the charter of ARIN to make such
a
> change if the membership feels this will help the community with this
> problem.
>
> I do not think it makes sense to discuss the second option of having ARIN
> make such allocations until we are on the same page with this first
option.
>
> Jason Redisch
>
>
> -----Original Message-----
> From: Scott Richard Slater [mailto:scottslater77 at hotmail.com]
> Sent: Wednesday, August 08, 2001 5:11 PM
> To: Redisch, Jason
> Cc: ppml at arin.net
> Subject: Re: small multihomed organizations
>
>
> see below.
>
> ----- Original Message -----
> From: "Redisch, Jason" <jredisch at virtela.net>
> To: "'Scott Richard Slater'" <scottslater77 at hotmail.com>; <ppml at arin.net>
> Cc: "Lee Howard" <lhoward at UU.NET>
> Sent: Wednesday, August 08, 2001 5:08 PM
> Subject: RE: small multihomed organizations
>
>
> > All,
> > Rather than writing filter lists for the ISP members of ARIN, I
> > think we should focus on things within the ARIN charter that can solve
the
> > issues described below.  I listed a few potential policy changes that it
> may
> > be worth discussing in relation to these issues.
> >
> > 1) ARIN could look into allowing upstream providers to automatically
> assign
> > a /24 to a downstream if the downstream meets the same criteria required
> to
> > justify an AS number.  This assignment could be made regardless of the
> > utilization of the block. It could be limited to one /24 per justified
AS
> > number.
>
> First, an upstream provider can already assign a /24 from their larger
> aggregate address space, well.. if the ISP(upstream provider) possesses
> aggregate address space greater than a /24, as long as the
> customer(downstream) provides justification to their ISP(upstream
provider)
> for 256 hosts.  Second, are the requirements to obtain an ASN the same as
> the requirements to obtain a /24 address block or any address space for
that
> matter ?   I don't think so.  You can be a multihomed network with two
> connections to ISPs(upstream providers) and require an ASN and not require
> 256 hosts (/24).  ASN justification
> (http://www.arin.net/regserv/templates/asntemplate.txt).  Let's just say
for
> a minute that ARIN assigned a /24 to every ASN.  There are approximately
> 11628 ASNs assigned currently (I think) so 11628 * 256 hosts = 2976768
> hosts.  That's 11628 /24s or about 6 /13s.
>
> Next,  does a /24 assigned to a customer(downstream) still solve the
> filtering problem the customer "may" face with some ISPs ?  Is there a
> policy anywhere that defines aggregate address space filtering by ISPs ?
>
> >
> > 2) ARIN could look into revisiting the micro-allocation policy which
would
> > encourage such downstream customers to go directly to ARIN for a /24.
They
> > could then apply the same criteria listed in policy change number one.
>
> First, let's just state that the micro-allocation policy limits the
address
> space allocation size from ARIN to a /24 and not greater.  So if the
> customer needs 257 hosts they will need to request address space from
their
> ISP(upstream provider) as well.  Second, does a small multi-homed
> organization meet that guidelines of the micro-allocation policy?  I guess
> that would be discussed when revisiting the policy.  Third, this is a
great
> policy to revisit, since ARINs current minimum allocation is a /20.
> ISPs(upstream providers) would end up retaining more IPs.  ISPs(upstream
> providers) would be happy if the micro-allocation policy justified an ARIN
> allocation of /24 blocks to their customers(downstream).  I think a good
> policy change directed towards the ISPs(upstream providers) would be in
> regards to their IP allocation policy.  Maybe a policy change such as
> allowing ISPs(upstream providers) to only break up their aggregate address
> space for customers(downstream) into blocks smaller than a /24.  So, maybe
> the policy change would only allow IPs(upstream providers) to create
> non-aggregate address blocks of /25,/26,/27,/28,/29,/30.  The only problem
> with this idea is that ARINs IP address allocation policy seems a bit more
> refined and stringent than the alocation policy of the ISPs(upstream
> providers) to the customer(downstream).   In addition, is our ficticious
> customer going to encounter difficulty with routing allocated ARIN address
> space ?  Just an idea.
>
> >
> > 3) ARIN could take no action and wait for filters to change or other
> > solutions to arrive.
>
> sounds fine for now.
>
> >
> > Jason Redisch
> >
> >
> >
> > -----Original Message-----
> > From: Scott Richard Slater [mailto:scottslater77 at hotmail.com]
> > Sent: Wednesday, August 08, 2001 2:30 PM
> > To: ppml at arin.net
> > Cc: Lee Howard
> > Subject: Re: small multihomed organizations
> >
> >
> > objective: "small multhomed organizations with the complete ability to
> > balance all inbound and outbound traffic over both connections".
> >
> > I think there are a few issues here.  The major issue is "filtering".
We
> > need to adjust filtering to accomodate smaller multihomed organizations
> and
> > enable these smaller multihomed organizations to control/balance their
> > inbound and outbound traffic over each connection.
> >
> > There are three methods of "filtering":
> > 1) filtering using route information (access-list, distribute-list)
> > 2) filtering using path information (ip as-path access-list,
filter-list)
> > 3) filtering using communities(ip community-list)
> >
> > Once we identify how to filter with each method then we next determine
if
> > any of the filtering methods accomodate our objective.  If none of the
> > filtering methods accomodates our objective then we should create a new
> > method for "filtering".
> >
> > note:  There is another option I just thought of.  If the customer is
> > delegated address space from ARIN, and the inbound and outbound traffic
is
> > not balanced.  The customer can use the "set as-path prepend" command to
> > balance the traffic over each connection.
> >
> > Also, please note, that the filtering methods above can be used
together.
> > You can filter using "community-list" and "ip as-path access-list" on
the
> > same router bgp configuration.
> >
> > Also, I was just thinking of what I wrote yesterday, about if one of the
> two
> > ISPs delegates address space to the customer then the delegating ISP
will
> > most probably be the backup connection for the customer.  This can be
> > corrected using the BGP "backdoor" command.
> >
> > Also, note that the customers configuration will absolutely be different
> > dependent on the number of BGP neighbors.  The more BGP neighbors
chances
> > are the customer will use some BGP metrics to choose packet flow.
> >
> > Just another thought,  its hard to develop an answer to adjust the
present
> > "filtering" configurations when we do not know how every provider
filters.
> >
> > Another solution for the customer's 24 bit address space to be accepted
by
> > providers is the following, if the provider uses prefix-lists.  This way
> the
> > customers specific address space is accepted, a mask length of up to 24
> bits
> > in routes is accepted  EXAMPLE:  " ip prefix-list abc permit 0.0.0.0/8
le
> > 24"  If the provider uses "le" they can specify the prefix length they
> will
> > permit or deny.
> >
> > I am working on some solutions and drawings.
> >
> > Scott
> >
> > ----- Original Message -----
> > From: "Scott Richard Slater" <scottslater77 at hotmail.com>
> > To: "Lee Howard" <lhoward at UU.NET>
> > Sent: Tuesday, August 07, 2001 7:52 PM
> > Subject: Re: small multihomed organizations
> >
> >
> > > some thoughts, see below.    I sort of said the same things you were
> > saying.
> > > I agree with you.  I am checking with some customers I worked with
that
> > are
> > > mutli-homed.  I can not remember exactly who allocated their address
> > space.
> > >
> > >
> > > ----- Original Message -----
> > > From: "Lee Howard" <lhoward at UU.NET>
> > > To: "Scott Richard Slater" <scottslater77 at hotmail.com>
> > > Sent: Tuesday, August 07, 2001 3:15 PM
> > > Subject: Re: small multihomed organizations
> > >
> > >
> > > > The justification for an ASN is a routing policy that differs from
> > > > adjacent Autonomous Systems.  Multihoming is accepted as sufficient
> > > > difference.
> > >
> > > I agree.  I basically stated the same thing below. :)
> > >
> > > >
> > > >
> > > > On Mon, 6 Aug 2001, Scott Richard Slater wrote:
> > > >
> > > > > Date: Mon, 6 Aug 2001 18:54:07 -0400
> > > > > From: Scott Richard Slater <scottslater77 at hotmail.com>
> > > > > To: Lee Howard <lhoward at UU.NET>
> > > > > Subject: Re: small multihomed organizations
> > > > >
> > > > >
> > > > > Let me reiterate your example.  You are talking about a customer
who
> > is
> > > > > multi-homed, that is connected to two ISPs.  First, the customer
> will
> > > > > require an Autonomous System(AS).   Then the customer will require
a
> > > unique
> > > > > Autonomous System Number(ASN) from ARIN.  This ASN identifies the
> > > customer's
> > > > > network to the rest of the Internet through the Border Gateway
> > > Protocol(BGP)
> > > > > protocol.  BGP is implemented on the customer's router as well as
> the
> > > two
> > > > > ISPs routers.  Since the customer's AS is talking to both ISPs ASs
> > using
> > > > > BGP, this implementation is called External BGP(eBGP).
> > > > >
> > > > > With the basic design stated above, we must identify the
customer's
> > > > > objective before we can determine more answers.
> > > > >
> > > > > More often than not, the customer is multi-homed to the two ISPs
to
> > > obtain
> > > > > "redundancy" for their network.  In addition, most probably one
ISP
> > > > > connection will be used strictly for backup.  If this is NOT the
> case
> > > and
> > > > > the customer wants to "balance" traffic over the two ISP
connections
> > > this
> > > > > may be tricky, dependent on the ISPs BGP policies.
> > > >
> > > > I'm not sure that most multi-homing orgs use one only as a backup.
> But
> > > > I agree that it's tricky to split use between two paths.  Outbound
> > > > balancing is easy (two equal static routes) but inbound balancing is
> > > > impossible between two differently-homed connections.  I find that
> most
> > > > customers are satisfied with "some" traffic going over each link.
> > >
> > > Both ISPs can statically insert the customers routes, but the customer
> > wants
> > > to use BGP to take full routes from both ISPs to make more intelligent
> > > routing decisions, and to be able to add routes without having to call
> the
> > > ISP.
> > >
> > > >
> > > >
> > > > > I do not think that a
> > > > > /24 (Class C) is justifiable solely due to a customer being
> > multi-homed.
> > > > > The customer should provide thorough justification of their AS. '
> > > >
> > > > Well, sure, I never said we should change the justification for
ASNs.
> > > >
> > > > >  Which
> > > > > includes their network table, their number of hosts specific to
> > hardware
> > > > > (routers, firewalls, et cetera..), and their subnetting should be
> > > detailed.
> > > > > This is important for the ISP to look at.  The ISP may be able to
> > > suggest
> > > > > different network designs for the customer to conserve address
space
> > > based
> > > > > on the network table the customer provides.  Normally, customer
> routes
> > > that
> > > > > are a /24 or greater are not filtered.
> > > >
> > > > Right, exactly.  So if a customer has 24 hosts, a T1 to me and a T1
to
> > > you,
> > > > what do we do?  If I give him a /27, nobody except you will take
your
> > > path.
> > > > If you give him a /27, nobody except me will take my path.
> > >
> > > Kinda,  let's look at it like this.  If the customer receives address
> > space
> > > from me(example 138.39.1/24), normally I would then be the customers
> > backup
> > > connection, and you would announce the more specific route to the
> > customers
> > > address space and you would attract more traffic.  This is due to the
> fact
> > > that I would normally announce my larger aggregate address
space(example
> > > 138.39/16).  Now, this is slightly fixable, if the delegating space
> > provider
> > > (example: me)announces the customers address space(example
138.39.1/24).
> > > Most delegating providers will announce their customers address space
to
> > > their peers.  Some delegating providers want to keep their routing
table
> > > small and not announce a customer prefix out of their aggregate
address
> > > space.  This is an individual-case-basis. The other problem is that
> there
> > is
> > > no gurantee that any of my peers will except the customers address
> space,
> > > due to the peers routing policies (this is an ever-so-changing policy
> > > between peers - some peers filter at /19,/20,/24).
> > >
> > > So, the customer will not be satisfied if they receive address space
> from
> > > either me or you. Now, even if me and you give the customer address
> space
> > > (example a /24 from me to the customer & a /24 from you to the
customer)
> > > there is still the problem of our peers filtering out our customers
> > specific
> > > /24 address space.
> > >
> > > Now, even if the customer is delegated a /24 from ARIN the customer
will
> > > still encounter the peers filtering.  The only advantage of the ARIN
> > > delegation of address space to the customer is the customer will never
> > need
> > > to renumber their domains only their physical interfaces with the
> provider
> > > giving them the connections.
> > >
> > > So,  there is only a few options.  It is understandable that smaller
> > > organizations will not and do not receive the same benefits as
> > medium-large
> > > organizations.    Let me first state that "multi-homing with true
> > > load-blancing" (the objective we are trying to achieve) is a WANT from
> any
> > > customer with an internet connection.  Who wouldn't want redundancy
and
> > more
> > > reliability and better performance.  BUT, every customer can not HAVE
> > > "multi-homing with true load-blancing".  It is the same as a customer
> who
> > is
> > > using a Cisco 2500 series router.  I am sure the customer would much
> > prefer
> > > using a Cisco 12000 series router but every customer can not justify.
> > >
> > > Maybe AS-based configurations will go away and community-based
> > > configurations with local-pref will be used throughout.   Though, I
know
> > > every provider does not currently use community tags.  And there is
> still
> > > the chance of corruption using community tags as with not using
filters.
> > >
> > > >
> > > >
> > > > > The key to finding a solution for this customer is to understand
the
> > > > > customers objective first, next understand how many hosts does the
> > > customer
> > > > > actually need, third to understand the two ISPs BGP policies.  The
> > > reason I
> > > > > say this is that ISPs have very different BGP policies.  Some ISPs
> > > adjust
> > > > > their local-pref values.   The customer must also get a fine look
at
> > the
> > > > > ISPs BGP 'import' AND 'export' policies.  Obviously, the customer
> will
> > > have
> > > > > their own routing decisions based on the ISPs performance and
cost.
> > > >
> > > > I understand well the differences between different ISPs' routing
> > > policies,
> > > > but I don't understand how that will change what the ARIN policy
> should
> > > be.
> > > >
> > > > Maybe we should consider what customers want. . .
> > > >
> > > > CUSTOMER WANTS PRIMARY AND BACKUP
> > > > Even if most people use one ISP as primary and one as backup, any
> > failure
> > > > catastrophic enough to cause primary ISP to fail will prevent
traffic
> > from
> > > > taking backup.  What I mean is, Customer connected to Primary
Networks
> > and
> > > > Backup ISP Inc., gets a /27 from Primary's /16.  Great, all traffic
> > takes
> > > > Primary (except maybe traffic originating in Backup's network).  But
> if
> > > > Primary goes down, or some failure causes the /27 route to be
> withdrawn,
> > > > Unrelated ISP's traffic will still try to go through Primary, and
get
> > > > blackholed.  If Customer is lucky enough that Backup peers with
> Primary,
> > > > and if Primary allows more-specifics of its own aggregates to be
> > announced
> > > > to it, then Primary may shuffle traffic to Backup in case of an
> outage,
> > > but
> > > > if case of Primary going completely down (or otherwise withdrawing
the
> > > /16)
> > > > nobody will be able to reach Customer.  No redundancy is achieved.
> > > >
> > > > Looked at the other way, let's say Customer gets IPs from Backup.
> Nope,
> > > > doesn't work--no Internet traffic will take Primary because nobody
> will
> > > hear
> > > > the /27.
> > >
> > > no static routing :)
> > >
> > > >
> > > > CUSTOMER WANTS LOAD BALANCING
> > > > Ain't no such animal.
> > > > OK, fine, you want some traffic to use each link, so you buy a
> > connection
> > > > from Big Mama Networking MegaTelco, and Joe Bob's Bait, Tackle &
ISP.
> > You
> > > > get a /27 from Joe Bob, and you're barely reachable.  Maybe Joe Bob
> has
> > > > transit from somebody, maybe he has a few peers, or maybe not.  If
Joe
> > > > Bob's aggregate block (/20 or greater) is globally routed, then
fine,
> > you
> > > > have access.  Big Mama MegaTelco will send locally originated
traffic
> to
> > > > you, too; it could be a significant amount of traffic.   But if Joe
> Bob
> > > > drops his pager while fishing, and there's an outage, Big Mama is
the
> > only
> > > > ISP that will be able to reach you; everybody else will filter your
> /27.
> > > > It works the other way, too.  Sure, you have some traffic using each
> > link,
> > > > but you have almost zero redundancy if Joe Bob gets eaten by a bear.
> > Our
> > > > goal was load sharing, and we've got it, but it's less than most
> > customers
> > > > are looking for, in my experience.
> > >
> > > Customers must be very specific which provider connections they want
and
> > > which provider connections they need.
> > >
> > > >
> > > > I think customers should reasonably expect transit to include global
> (or
> > > > at least nearly-global) routability.  The reason that doesn't exist
> for
> > > > longer prefixes now is that too many people make mistakes and leak
all
> > > > of their internal routes.  So other people filter, to avoid hearing
> > > > those routes--they assume that all routes smaller than a /24 are
> > leakage.
> > > > I'm saying that not all of them are, and we should make it possible
> for
> > > > smaller customers to multihome.  How do we do that?
> > >
> > > just can't right now.  Due to everyone's filtering policies.  Unless
> there
> > > was some sort of "ARIN approval system" for non-aggregate address
space.
> > > First we would need to determine what is aggregate address space. (Is
it
> > /19
> > > or /20 or /24) and then every provider would have to adjust their
> filters
> > > accordingly.  Second, we would then need this "ARIN approval system"
> that
> > > grants a non-aggregate address block the same routing privledges that
an
> > > aggregate address block has.  The only problem with this would be the
> > > adjustment on everyone's filters.  If filters did not exist then
> everyone
> > > would rely on faith of the customer.  AND we know customers make
> mistakes.
> > > The only solution I can think of is written above.
> > >
> > > >
> > > > Lee
> > > >
> > > >
> > >
> > > Scott
> > >
> > > > >
> > > > > Scott
> > > > >
> > > > > ----- Original Message -----
> > > > > From: "Lee Howard" <lhoward at UU.NET>
> > > > > To: <ppml at arin.net>
> > > > > Sent: Monday, August 06, 2001 3:42 PM
> > > > > Subject: small multihomed organizations
> > > > >
> > > > >
> > > > > >
> > > > > > We occasionally have customers who will be multihomed with us
and
> > > another
> > > > > > provider, and who need address space, but can't justify a /24
> based
> > on
> > > > > > usage and host count.  If we give them less than a /24, no
traffic
> > > will
> > > > > > come down their other line. [1]
> > > > >
> > > > > If no traffic is coming down their "other line" the customer
should
> > > check
> > > > > the BGP metrics set by both ISPs.  There are a few possibilities.
> The
> > > ISP
> > > > > with the line that is receiving all the traffic may have adjusted
> > their
> > > > > local-pref values to the customers advertisements on the "other
> line".
> > > I
> > > > > think I said that right. :)
> > > > >
> > > > > I would also be curious of both ISPs peering connections and their
> > > > > relationships with other "tier one" ISPs, especially the two ISPs
> > > involved.
> > > > >
> > > > > >
> > > > > > ARIN's current policy, as I understand it, is that multihoming
is
> > not
> > > > > > justification for a /24.  I would like to hear recommendations
and
> > > best
> > > > > > practices from others for how to address the customer (pun
> intended)
> > > > > > without violating ARIN policy.  Should ARIN accept multihoming
as
> > > > > > justification for a /24?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Lee Howard
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > [1] Here's why:  Most ISPs do not accept prefixes longer than 24
> > bits,
> > > so
> > > > > > they will ignore our propagation of our customer's /24, and they
> > will
> > > > > > ignore the route announced from the customer's other ISP.  Since
> we
> > > also
> > > > > > announce a large aggregate (say, a /13 from which that /24 was
> > > allocated)
> > > > > > those peers will send traffic to us, which we can then forward
to
> > our
> > > > > > customer.  But they'll never hear the other ISP's announcement,
> > unless
> > > our
> > > > > > network was completely cleared off the map and the aggregate
were
> > > > > withdrawn.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
>