From hcb at mail.clark.net Mon Oct 18 17:59:37 1999 From: hcb at mail.clark.net (Howard C. Berkowitz) Date: Mon, 18 Oct 1999 17:59:37 -0400 Subject: Multihoming document Message-ID: Here's an expired I-D that I was going to update for the IETF NAT WG, but that's not the ideal place for it. I presented it to the IDR WG at the last Washington IETF, but the consensus was that "it had a lot of things that need to be written down" but it wasn't quite appropriate for the standards track. For those who like pictures, I did a version of this at the NANOG Albuquerque meeting: http://www.academ.com/nanog/feb1998/multihoming INTERNET-DRAFT H. Berkowitz Expiration Date: August 1998 March 1998 To Be Multihomed: Requirements & Definitions draft-berkowitz-multirqmt-01.txt 1. Status of this Memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as ``work in progress.'' To learn the current status of any Internet-Draft, please check the ``1id-abstracts.txt'' listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast). 2. Abstract As organizations find their Internet connectivity increasingly critical to their mission, they seek ways of making that connectivity more robust. The term ''multi-homing'' often is used to describe means of fault-tolerant connection. Unfortunately, this term covers a variety of mechanisms, including naming/directory services, routing, and physical connectivity. The "home" may be identified by a DNS name, an IP address, or an IP address and transport-layer port identifier. Any of these identifiers may be translated in the path between source and destination. This memorandum presents a systematic way to define the requirement for resilience, and a taxonomy for describing mechanisms to achieve it. Its intended audience is primarily people concerned with planning and performing multihoming deployments, rather than protocol developers. It examines both requirements and applications, with the emphasis on the former. 3. Introduction As the Internet becomes more ubiquitous, more and more enterprises connect to it. Some of those enterprises, such as Web software vendors, have no effective business if their connectivity fails. Other enterprises do not have mission-critical Internet applications, but become so dependent on routine email, news, web, and similar access that a loss of connectivity becomes a crisis. As this Internet dependence becomes more critical, prudent management suggests there be no single point of failure that can break all Internet connectivity. The term "multihoming" has come into vogue to describe various means of enterprise-to-service provider connectivity that avoid a single point of failure. Multihoming also can describe connectivity between Internet Service Providers and "upstream" Network Service Providers. Several terms have become overloaded to the point of confusion, including multihoming, virtual private networks, and load balancing. This document attempts to bring some order to the definition of multihoming. It partially overlaps definitions of virtual private networks [Ferguson & Huston]. If we take the word "multihoming" in the broadest context, it implies there are multiple ways to reach a "home" destination. This "home" may be identified by a name, an IP address, or a combination of IP address and TCP/UDP port. In this document, TCP/UDP ports will be referred to as TU ports. There are other motivations for complex connectivity from enterprises to the Internet. Mergers and acquisitions, where the joined enterprises each had their own Internet access, often mean complex connectivity, at least for a transition period. Consolidation of separate divisional networks also creates this situation. A frequent case arises when a large enterprise decides that Internet access should be available corporate-wide, but their research labs have had Internet access for years -- and it works, as opposed to the new corporate connection that at best is untried. Many discussions of multihoming focus on the details of implementation, using such techniques as the Border Gateway Protocol (BGP) [RFC number of the Applicability Statement], multiple DNS entries for a server, etc. This document suggests that it is wise to look systematically at the requirements before selecting a means of resilient connectivity. One implementation technique is not appropriate for all requirements. There are special issues in implementing solutions in the general Internet, because poor implementations can jeopardize the proper function of global routing or DNS. An incorrect BGP route advertisement injected into the global routing system is a problem whether it originates in an ISP or in an enterprise. 4. Goals Requirements tend to be driven by one or more of several major goals for server availability and performance. Availability goals are realized with resiliency mechanisms, to avoid user-perceived failures from single failures in servers, routing systems, or media. Performance goals are realized by mechanisms that distribute the workload among multiple machines such that the load is distributed in an useful manner. Like multi-homing, the terms load-balancing and load-sharing have many definitions. In defining requirements, the servers themselves may either share or balance the load, there may be load-sharing or load-balancing routing paths to them, or the routed traffic may be carried over load-shared or load-balanced media. 4.1 Analyzing Application Requirements Several questions need to be answered in the process of refining goals: -- the administrative model and administrative awareness of endpoints -- availability requirements -- the security model -- addressing requirements -- scope of multihoming 4.1.1 Administrative Model A key question is: are endpoints predefined in the multihoming process, or will either the client or server end be arbitrary? The servers of interest may be inside the enterprise, or outside it. If they are outside, their names or addresses may or may not be preconfigured into multihoming mechanisms. In this document, intranet clients and servers are inside the enterprise and intendedprimarily for enterprise use. The existence of both can be preconfigured. Intranet clients have access only to machines on the intranet. Multinet clients servers are inside the enterprise, but there is pre-authorized access by known external partners. Internet servers are operated by the enterprise but intended to be accessible to the general Internet. Internet clients have general Internet access that may be mediated by a firewall. The client administrator will know the prior identity of clients, but not of servers. The server administrator will know the prior identity of servers, but not of clients. 4.1.2 Availability Requirements There are major implications between defining a requirement for high availability of initial access, and making the connection stay up once access has been achieved. The latter tends to require transport layer awareness. In the terminology of RFC1775, "To be 'on' the Internet," servers described here have "full" or a subset of "client" access. Client servers may not directly respond to specific IP packet from an arbitrary host, but a system such as a firewall MUST respond for them unless a security policy precludes that. Some valid security policies, for example, suppress the response of ICMP Destination Administratively Prohibited responses, because that would reveal there is an information resource being protected. RFC1775 defines full access as " a permanent (full-time) Internet attachment running TCP/IP, primarily appropriate for allowing the Internet community to access application servers, operated by Internet service providers. Machines with Full access are directly visible to others attached to the Internet, such as through the Internet Protocol's ICMP Echo (ping) facility. The core of the Internet comprises those machines with Full access." This definition is extended here to allow full firewalls or screening routers always to be present. If a proxy or address translation service exists between the real machine and the Internet, if this service is available on a full-time basis, and consistently responds to requests sent to a DNS name of the server, it is considered to be full-time. In this discussion, we generalize the definition beyond machines primarily appropriate for the Internet community as a whole, to include in-house and authorized partner machines that use the Internet for connectivity. Both the cases described in 4.2.3 and 4.2.4 have been termed "Virtual Private Networks." 4.1.3 Security Model Security requirements can include various cryptographic schemes, as well as mechanisms to hinder denial of service attacks. The requirements analyst must determine whether cryptography is needed, and, if so, whether cryptographic trust must be between end hosts or between end hosts and a trusted gateway. Such gateways can be routers or multiported application servers. 4.1.4. Addressing Refinements and Issues It is arguable that addressing used to support multihoming is a routing deployment issue, beyond the scope of this document. Rationales for including it here is that addressing MAY affect application behavior. There also may be administrative requirements for addressing, such as a service provider that contracts to run a multinet may require addresses to be registered, possibly from the provider's address space. If the enterprise runs applications that embed network layer addresses in higher-level data fields, solutions that employ address translation, at the packet or virtual connection level, MAY NOT work. Use of such applications inherently is a requirement for the eventual multihoming solution. Consideration also needs to be given to application caches in addition to those of DNS. Firewall proxy servers are a good example where multiple addresses associated with a given destination may not be supported. RFC1918 internal, static network address translation (NAT) to outside RFC1918 internal, dynamic port address translation (PAT) to outside Registered internal, Provider Assigned (PA) Registered internal, Provider Independent (PI) 4.1.5 Scope of multihoming Multihoming may be defined between an end host and a router or application gateway, on an end-to-end basis possibly involving virtual servers, among routers, or among elements in a transmission system. Different multihoming scopes may support the same application requirement. 4.2. Application Goals These goals need to be agreed to by the people or organization responsible for the applications. Not to reach fairly formal agreement here can lead to problems of inappropriate expectations. The term "service level agreement" often refers to expectations of performance, such as throughput or response time. Ideas here extend the performance-based model to include availability. 4.2.1 Specific server availability The first goal involves well-defined applications that run on specific servers visible to the Internet at large. This will be termed "endpoint multihoming", emphasizing the need for resilience of connectivity to well-defined endpoints. Solutions here often involve DNS mechanisms. There are both availability and performance goals here. Availability goals arise when there are multiple routing paths that can reach the server, protecting it from single routing failures. Other availability goals involve replicated servers, so that the client will reach a server regardless of single server failures. Performance goals include balancing client requests over multiple servers, so that one or more servers do not become overloaded and provide poor service. Requests can be distributed among servers in a round-robin fashion, or more sophisticated distribution mechanisms can be employed. Such mechanisms can consider actual real-time workload on the server, routing metric from the client to the server, known server capacity, etc. >From an application standpoint, this is either a many-to-one topology, many clients to one server, or a many-to-many topology when multiple servers are involved. It can be worthwhile to consider a many-to-few case, when the few are multiple instances of a server function, which may appear as a single server to the general Internet. The idea of many-to-few topology allows for a local optimization of inter-server communications, without affecting the global many-to-one model. Addresses on interfaces that connect to the general Internet need to be unique in the global Internet routing system, although they may be translated, at the network address or port level, from public to internal space.. 4.2.2 General Internet connectivity from the enterprise The second is high availability of general Internet connectivity for arbitrary enterprise users to the outside. This will be called "internetwork multihoming". Solutions here tend to involve routing mechanisms. This can be viewed as a few-to-many application topology. Addresses on interfaces that connect to the general Internet need to be unique in the global Internet routing system, although they may be translated, at the network address or port level, from internal private address to public space. 4.2.3 Use of Internet services to interconnect "intranet" enterprise campuses The third involves the growing number of situations where Internet services are used to interconnect parts of an enterprise. This is "intranetwork multihoming". This will usually involve dedicated or virtual circuits, or some sort of tunneling mechanisms. This case may be termed a "virtual private network." The "multihoming" aspect of this case is associated with high availability to the connectivity network that underlies the tunneling system. In this case, addresses only need to be unique within the enterprise. 4.2.4 Use of Internet services to connect to "multinet" partners A fourth category involves use of the Internet to connect with strategic partners. True, this does deal with endpoints, but the emphasis is different than the first case. In the first case, the emphasis is on connectivity from arbitrary points outside the enterprise to points within it. This second case deals with pairs of well-known endpoints. These endpoints may be linked with dedicated or virtual circuits defined at the physical or data link layer. Tunneling or other virtual private networks may be relevant here as well. There will be coordination issues that do not exist for the third case, where all resources are under common control. Addresses need to be unique in the different enterprises, but do not need to be unique in the global Internet. 5. Planning and Budgeting In each of these scenarios, organization managers need to assign some economic cost to outages. Typically, there will be an incident cost and an incremental cost based on the length or scope of the connectivity loss. Ideally, this cost is then weighted by the probability of outage. A weighted exposure cost results when the outage cost is multiplied by the probability of the outage. Resiliency measures modify the probability, but increase the cost of operation. Operational costs obviously include the costs of redundant mechanisms (i.e., the addititional multihomed paths), but also the incremental costs of personnel to administer the more complex mechanisms -- their training and salaries. 6. Issues 6.1 Performance vs. Robustness: the Cache Conundrum Goals of many forms of "multi-homing" conflict with goals of improving local performance. For example, DNS queries normally are cached in DNS servers, and in the requesting host. From the performance standpoint, this is a perfectly reasonable thing to do, reducing the need to send out queries. >From the multihoming standpoint, it is far less desirable, as application-level multihoming may be based on rapid changes of the DNS master files. The binding of a given IP address to a DNS name can change rapidly. 6.2 Service Level Agreements Enterprise networks, especially mainframe-based, are accustomed to building and enforcing service level agreements for application performance. A key to being able to do this is total control of the end-to-end communications path. In the current Internet, the enterprise(s) at one or both ends control their local environments, and have contractual control over connections to their direct service providers. If service level control is a requirement, and both ends of the path are not under control (i.e., cases 1 and 2), the general Internet cannot now provide service level guarantees. The need for control should be reexamined, and, if it still exists, the underlying structure will need to be dedicated resources at the network layer or below. A network service provider may be able to engineer this so that some facilities are shared to reduce cost, but the sharing is planned and controlled. 6.3 Symmetry One of the reasons service level agreements are not enforceable in the general Internet is the reality that global routing cannot be guaranteed to be symmetrical. Symmetrical routing assumes the path to a destination is simply reversed to return a response from that destination. Both legs of a symmetrical path are assumed to have the same performance characteristics. Global Internet routing is not necessarily optimized for best end-to-end routing, but for efficient handling in the Autonomous Systems along the path. Many service providers use "closest exit" routing, where they will go to the closest exit point from their perspective to get to the next hop AS. The return path, however, is not necessarily of a mirror image of the path from the original source to the destination. Closest exit routing is, in fact, a "feature" rather than a "bug" in some multihoming schemes [Peterson] [Friedman]. Especially when the enterprise network has multiple points of attachment to the Internet, either to a single ISP AS or to multiple ISPs, it becomes likely that the response to a given packet will not come back at the same entry point in which it left the enterprise. This is probably not avoidable, and troubleshooting procedures and traffic engineering have to consider this characteristic of multi-exit routing. 6.4 Security ISPs may be reluctant to let user routing advertisements or DNS zone information flow directly into their routing or naming systems. Users should understand that BGP is not intended to be a plug-and-play mechanism; manual configuration often is considered an important part of maintaining integrity. Supplemental mechanisms may be used for additional control, such as registering policies in a registry [RPS, RA documents] or egress/ingress filtering [Ferguson draft] Challenges may arise when client security mechanisms interact with fault tolerance mechanisms associated with servers. For example, if a server address changes to that of a backup server, a stateful packet screening firewall might not accept a valid return. Similarly, unless servers back one another up in a full mirroring mode, if one end of a TCP-based application connection fails, the user will need to reconnect. As long as another server is ready to accept that connection, there may not be major user impact, and the goal of high availability is realized. High availability and user transparent high availability are not synonymous. 6.5 Load Balancing vs. Load Sharing These terms are often interchanged, but they really mean different things. Load balancing is deterministic, and at a finer level of control than load sharing, which is statistical. Load balancing is generally not something that can be realized in general Internet routing, other than in special and local cases between adjacent AS. A degree of load sharing is achievable in routing, but it may introduce significant resource demands and operational complexity. Paul Ferguson defines load-balancing as "a true "50/50" sharing of equal paths. This can be done by either (a) round robin per- packet transmission, (b) binding pipes a the lower layers such that bits are either 'bit-striped' across all parallel paths (like the etherchannel stuff), or binding pipes so that SAR functions are done in a method such as multilink PPP. These are fundamentally the same. "Load-sharing is quite different. It simply implies that no link is sitting idle -- that at least all links get utilized in some fashion. Usually in closest exit routing. The equity of utilization may be massively skewed. It may also resemble something along the lines of 60/40, which is reasonable." 6.6 Application Compatibility Some deployment mechanisms involve network address, or network address and TCP/UDP port, translation (NAT and NAPT). If the application protocols embed IP addresses in their protocol fields, NAT or NAPT may cause protocol failures. Translation mechanisms for such cases may require knowledge of the application protocol, as typified by application proxies in firewalls, or in application gateways with multiple interfaces. 7. Multihoming Deployment Technologies A basic way to tell which technology(ies) is applicable is to ask oneself whether the functional requirement is defined in terms of multihoming to specific hosts, or to specific networks. If the former, some type of application or transport technology is needed, because only these technologies have awareness of specific hosts. A given multihoming implementation may draw on several of these technologies. For example, the Cisco Distributed Director does DNS name-level redirection based in part on routing metrics. 7.1 Application/Name Based Techologies in this category may involve referring a client request to different instances of the endpoint represented by a single name. Another aspect of application/name multihoming may work at the level of IP addressing, but specifically is constrained to endpoint (i.e., server) activities that redirect the client request to a different endpoint. Application-level firewall proxy services can provide this functionality, although their application protocol modification emphasizes security while a multihoming application service emphasizes availability and quality of service. 7.2 Transport Based Transport based technologies are based on maintaining tunnels through an underlying network or transmission system. The tunnels may be true end to end, connecting a client host with a server host, or may interconnect between proxy servers or other gateways. 7.3 Network Based Network based approaches to multihoming are router-based. They involve having alternate next-hops, or complete routes, to alternate destinations. 7.4 Data Link Based Data link layer methods may involve coordinated use of multiple physical paths, as in multilink PPP or X.25. If the underlying WAN service has a virtual circuit mechanism, as in frame relay or ATM, the service provider may have multihomed paths provided as part of the service. Such functions blur between data link and physical layers. Other data link methods may manipulate MAC addresses to provide virtual server functions. 7.5 Physically-based Physical multihoming strategies can use diverse media, often of different types such as a wire backed up with a wireless data link. They also involve transmission media which have internal redundancy, such as SONET. 8. Application/Name Multihoming While many people look at the multihoming problem as one of routing, the goal is often multihoming to endpoints. Finding an endpoint usually begins in DNS. Once an endpoint address is found, some application protocols, notably HTTP and FTP, may redirect the request to a different endpoint. The basic idea here is that arbitrary clients will first request access to a resource by its DNS name, and certain DNS servers will resolve the same name to different addresses based on conditions of which DNS is aware, or using some statistical load-distribution mechanism. There are some general DNS issues here. DNS was not really designed to do this. A key issue is that of DNS cacheing. Cacheing and frequent changes in name resolution are opposite goals. Traditional DNS schemes emphasize performance over resiliency. 8.1 DNS Caching DNS standards do provide the capability for short cache lifetimes, which in principle support name based multihoming. "The meaning of the TTL field is a time limit on how long an RR can be kept in a cache. This limit does not apply to authoritative data in zones; it is also timed out, but by the refreshing policies for the zone. The TTL is assigned by the administrator for the zone where the data originates. While short TTLs can be used to minimize caching, and a zero TTL prohibits caching, the realities of Internet performance suggest that these times should be on the order of days for the typical host. If a change can be anticipated, the TTL can be reduced prior to the change to minimize inconsistency during the change, and then increased back to its former value following the change. [RFC1034] " Several real-world factors limit the utility of simply shortening the cache time. Widely used BIND, the most widely used DNS implementation, does not accept cache lifetimes less than 5 minutes. Dynamic DNS may be a long-term solution here. In the short term, setting very short TTL values may be help in some cases, but is not likely to help elow a granularity of 5 minutes. Remember that the name normally is resolved when an application session first is established, and the decisions are made over a longer time base than per-packet routing decisions. 8.2 DNS Multiple Hosts and Round Robin Response The DNS protocol allows it to return multiple host addresses in respondse to a single query. At the first level of DNS-based multihoming, this can provide additional reliability. A DNS server knows three IP addresses for the server function identified by server.example.com, 10.0.1.1, 10.0.2.1, and 10.0.3.1. A simple response to a query for server.example.com returns all three addresses. Assume the response provides server addresses in the order 10.0.1.1, 10.0.2.1, and 10.0.3.1. Whether this will provide multihoming now depends on the DNS client. Not all host client implementations will, if the first address returned (i.e., 10.0.1.1) does not respond, try the additional addresses. In this example, 10.0.2.1 might be operating perfectly, A variant suggested by Kent England is to have the addresses returned in the DNS response come from the CIDR blocks of different ISPs that provide connectivity to the server function [England]. This approach combines aspects of name and network multihoming. Again, this will work when intelligent clients try every IP address returned until a server responds. 8.3 Replication One approach [Peterson] uses DNS as the main method, but makes assumptions about the underlying routing. The two assumptions it makes, which appears generally valid for the Internet, are that connectivity providers use closest exit routing, and once a packet reaches a provider network, that provider knows best how to reach the specific server inside that network. One implementation of this approach is Cisco's DistributedDirector product. It involves communication between DNS servers and routers in the multihomed domain. When a DNS request is received by a DistributedDirector DNS server, the DD server selects the IP address to be returned based on information on routing cost from the client entry point to closest server, on administrative weight, or other generally routing-associated factors. 8.4 Cache Servers and Application Multihoming The cache server is the primary home for servicing the client request, but the true server backs it up. 9. Transport Multihoming Transport layer functions are conceptually end-to-end. There are two broad classes of transport multihoming function, those maintained by the endpoints and those that involve intermediate translation devices. 9.1 End-to-end Tunnel Maintenance Basic point-to-point tunneling mechanisms include GRE, PPTP and L2TP. DVMRP is a special case. Choices here will depend in part on the security policy and the administrative model by which multihoming is provided. GRE, for example, does not itself provide encryption, while PPTP and L2TP do. "The differences between PPTP and L2TP are more of where one wishes the PPP session to terminate, and one of control. It really depends on who you are, and where you are, in the scheme of the control process. If your desire is to control where the PPP session terminates (as an ISP might wish to control), then L2TP might be the right approach. On the other hand, however, if you are a subscriber, and you wish to control where the PPP session terminates (to, say, a PPTP server somewhere across the cloud), then PPTP might be the right approach -- and it would be transparent to the service provider). It really depends on what problem one is trying to solve, and if you are in the business of trying to create "services." [Ferguson-1998-2] 9.2 Tunneling with Translation Various proxy and address translation mechanisms can play a role in multihoming. They can be divided into several levels of topological constraints: -- all servers are colocated in a single address domain "behind" the translator -- servers are in different parts of a single address domain. These parts are connected by tunnels. -- the servers are at arbitrary network addresses, but the translator knows how to reach them. Application-aware proxies can have even more knowledge of application load. A variant of NAT, called load sharing NAT (LS-NAT), offers a load sharing mechanism at the transport/network level rather than the DNS level [Srishuresh]. When considering LSNAT-style load sharing, remember that it emphasizes providing a pool of servers capable of servicing requests. In its "local" form, it does not easily provide mechanisms for increasing reliability by mapping the user request to geographically distributed servers. More advanced variants can combine with DNS- and routing-aware mechanisms to increase reliability as well as performance. The LSNAT function is visible globally as a server address. It is actually a virtual server. When a client request arrives at the LSNAT, the LSNAT translates the destination address, transparently to the client, and passes it to a server in the LSNAT's server pool. LSNATs, in their basic form, do have topological restriction. It has been suggested that all request/response traffic in a single session must go from the real client, to one specific LSNAT, to the server. It is conceivable that multiple routers could be used, but they would need to be tightly synchronized. LSNAT builds on NAPT, but adds intelligence to the port mapping function. Also, general NAPT is oriented toward outgoing requests from the stub domain to the outside, while LSNAT emphasizes incoming requests to a virtual server address. As currently conceived, LSNATs operate at the TCP/UDP transport level, so they could not easily change server hosts during a session. There are potential workarounds to this, including using a multicast address as the server pool destination, with coordination between the servers as to which actually answers the request. For highly fault-tolerant applications, more than one server conceivably could answer the NAT request, and the LSNAT decide which to pass to the client. Typically, if servers are identical, it would be the first response received by the server pool side of the LSNAT. This general topology restriction suggests LSNAT functionality belongs on a single NAT-capable border router, with the server pool inside the stub domain. A LSNAT violates the Internet end-to-end model in the same way that basic NAT does. There is no requirement that the addressing in the stub domain be private, only that all access to the servers go through the NAT. In basic LSNAT, an arbitrary external client attempts to establish a session with what it believes to be a server. In reality, it is attempting to establish a session with the virtual server address of the "outside" interface of the LSNAT. The LSNAT, based on internal criteria, redirects the external request to a specific internal server in a server pool. Unique connections are established from the LSNAT to the pool server for each request, translating addresses and ports as needed. 9.2.2. Load Shared Network Address and Port Translation (LS-NAPT) Adding the complexity of port as well as address translation gives additional flexibility. In particular, adding port translation removes many topological limitations between the real servers and the NAT. In a LS-NAT router, inbound sessions identified by the tuple From a typical server room, analog and digital signals physically flow to a wiring closet, where they join a riser cable. Depending on the specific building, the closet and riser may be the responsibility of the enterprise or ISP, the building management, or a telecommunications carrier. The riser cable joins with other riser cables in a cable vault, from which a cable leaves the building and goes to the end switching office of the local telecommunications provider. Most buildings have a single cable vault, possibly with multiple cables following a single physical route back to the end office. A single error by construction excavators can cut multiple cables on a single path. A failure in carrier systems can isolate a single end office. Highly robust systems have physical connectivity to two or more POPs reached through two or more end offices. Alternatives here can become creative. On a campus, it can be feasible to use some type of existing ductwork to run additional cables to another building that has a physically diverse path to the end office. Direct wire burial, fiber optic cables run in the air between buildings, etc., are all possible. In a non-campus environment, it is possible, in many urban areas, to find alternate means of running physical media to other buildings with alternate paths to end offices. Electrical power utilities may have empty ducts which they will lease, and through which privately owned fiber can be run. 11.2 Provider Core As demonstrated by a rash of fiber cuts in early 1997, carriers lease bandwidth from one another, so a cut to one carrier-owned facility may affect connectivity in several carriers. This reality makes some traditional diverse media strategies questionable. Many organizations consciously obtain WAN connectivity from multiple carriers, with the notion that a failure in carrier will not affect another. This is not a valid assumption. If the goal is to obtain diversity/resiliency among WAN circuits, it may be best to deal with a single service provider. The contract with this provider should require physical diversity among facilities, so the provider's engineering staff will be aware of requirements not to put multiple circuits into the same physical facility, owned by the carrier or leased from other carriers. 12. Security Considerations 13. Acknowledgments 14. References [RFC1775] D. Crocker. To Be "On" the Internet. March 1995. [RFC1930] Hawkinson, J., and T. Bates. Guidelines for creation, selection, and registration of an Autonomous System (AS). March 1996. (Also BCP0006) [RFC1034] Mockapetris, P.V. Domain names - concepts and facilities. Nov-01-1987. [RFC1998] An Application of the BGP Community Attribute in Multi-home Routing. E. Chen & T. Bates. August 1996 [RFC2071] Ferguson, P., and H. Berkowitz, "Network Renumbering Overview: Why would I want it and what is it anyway?", RFC 2071, January 1997. [RFC2050] Hubbard, K., Kosters, M., Conrad, D., Karrenberg, D., and J. Postel, "INTERNET REGISTRY IP ALLOCATION GUIDELINES", BCP 12, RFC 2050, November 1996. [RFC1631] Egevang,, K., and P. Francis, "The IP Network Address Translator (NAT)", RFC 1631, May 1994. [Srishuresh] Srishuresh, P., and D. Gan, "Load Sharing using IP Network Address Translation (LSNAT)", work in progress, ftp://ftp.ietf.org/internet-drafts/draft-srisuresh-lsnat-02.txt [RFC1918] Rekhter, Y., Moskowitz, R., Karrenberg, D., de Groot, G-J., and E. Lear, "Address Allocation for Private Internets", RFC 1918, February 1996. [RFC1900] Carpenter, B., and Y. Rekhter, "Renumbering Needs Work", RFC 1900, February 1996. [RFC2280] Routing Policy Specification Language (RPSL). C. Alaettinoglu, T. Bates, E. Gerich, D. Karrenberg, D. Meyer, M. Terpstra, C. Villamizar. January 1998. [RFC1812] Baker, F., "Requirements for IP Version 4 Routers", RFC 1812, June 1995. [Peterson] Peterson, A. "Dynamic Selection of Geographically Distributed Servers," presentation at NANOG October 1997 meeting, notes at http://www.academ.com/nanog/october1997/dynamic-selection.html [Freedman] Freedman, A. "Dynamic Selection of Geographically Distributed Servers," presentation at NANOG October 1997 meeting, notes at http://www.academ.com/nanog/october1997/dynamic-selection.html [Ferguson-1998-1] Ferguson, P. "Re: Comments on "What is a VPN?"" Message to IETF VPN mailing list, 08 Mar 1998 19:52:29 15. Author's Address Howard C. Berkowitz PO Box 6897 Arlington VA 22206 Phone: +1 703 998 5819 EMail: hcb at clark.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 48150 bytes Desc: not available URL: From hcb at mail.clark.net Mon Oct 18 17:59:58 1999 From: hcb at mail.clark.net (Howard C. Berkowitz) Date: Mon, 18 Oct 1999 17:59:58 -0400 Subject: Another deployment document Message-ID: Internet Engineering Task Force Howard C. Berkowitz INTERNET-DRAFT Gett Communications Expires August 1999 Requirements Taxonomy for Virtual Private Networks Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as ``work in progress''. To learn the current status of any Internet-Draft, please check the ``1id-abstracts.txt'' listing contained in the Internet-Drafts Shadow Directories on ftp.ietf.org (US East Coast), nic.nordu.net (Europe), ftp.isi.edu (US West Coast), or munnari.oz.au (Pacific Rim). Abstract Most useful networking ideas involve some level of abstraction. Virtual private networks (VPN) map an abstraction of an user-defined interface onto an infrastructure. Multihoming strategies are virtualizations in the infrastructure [Berkowitz multihome] that should be invisible to the VPN. Proprietary definitions of VPNs proliferate in the marketplace, to the confusion of end users. This memorandum proposes a taxonomy which can describe a variety of VPN models. It does not recommend any specific model, but suggests a consistent way of describing specific VPNs. This document differs from some other working papers that go more deeply into the VPN mechanisms. In contrast, this document is focused on user requirments. 1. Introduction Previous working papers [Gleeson, Muthukrishnan, Rosen] have dealt with fairly specific models where specific underlying technologies are considered at length. This document takes a different approach, trying to focus on requirements rather than technology. It may very well be appropriate to merge several of these proposals. Any VPN will have a minimal set of core capabilities, which, alone, are unlikely to satisfy any real-world user requirements. The taxonomy here provides a systematic way for extending the core capabilities to meet those requirements. It also provides a way of describing requirements for the shared infrastructure over which the VPN runs. A description of a specific VPN requirement will state the core capabilities optional user services, and the administrative model. A response to this requirement will state the infrastructure technologies and how the user requirements will map to them. The taxonomy consists of: Core capabilities Optional capabilities Administrative model Mapping of user services to different infrastructures Infrastructures For a VPN to work, it must be possible to map the user services to corresponding capabilities in the infrastructure. Mechanisms for these mappings are outside the scope of this taxonomy. A quality of service user requirement, for example, could map to ATM QoS facilities, to RSVP or differentiated services in an IP network, or to priorities in an 802.1p LAN. 2. Core Capabilities To define a VPN, it is necessary to be able to define an administrative mechanism for designating membership in the VPN. A given host may belong to one or more VPNs, as well as the global Internet. Useful definitions of VPNs include several policy statements beyond the administrative policies that define membership. An operational policy needs to be specified, which defines the organizations, inside or outside the enterprise, which are responsible for moves and changes, troubleshooting, performance monitoring, audit, etc. There should be a security policy and a QoS policy, even if these policy statements state there are no specific security or QoS requirements. If there are security requirements for the VPN, the owning enterprise should define a security policy that states the allowable connectivity over multiple VPNs and public networks. The set of members of a VPN will have an identifier [Fox], although that identifier might be null. 3. Optional Capabilities VPNs of practical use will have one or more optional capabilities in addition to the core set. Not all VPNs will need every capability. 3.1 Quality of Service The VPN may provide quality of service support. It either may accept QoS requests from end users signaled with RSVP, IP precedence bits, etc., or may internally assign quality of service requirements to be mapped to the transmission infrastructure. For quality of service to be effective, the infrastructure either must support explicit quality of service requests, or there must be a high level of confidence that the infrastructure consistently provides adequate QoS. Assumptions about QoS need to be stated as part of any VPN design. 3.2 Security User connectivity may be defined to include security using a variety of security mechanisms, including IPsec, L2TP, etc. These various mechanisms may provide other services, such as connectivity. L2TP can provide per-user authentication, or may have no security functions. Security may be requested on a discretionary basis by end user hosts, or the VPN may enforce a mandatory security policy. Cryptographic protections may be under the control of the enterprise, using host-to-host or host-to- security gateway methods, or the infrastructure may be trusted to provide encryption. The responsibilities for user authentication, device authentication, encryption, logging, and audit must be specified as part of the design of any practical VPN. 3.3 Addressing and load sharing The VPN may provide address assignment, presumably with DHCP. It also may provide network address translation (NAT), network address and port translation (NAPT), and load-shared network address translation (LSNAT). DNS services may be associated with the VPN, and operated by the enterprise or the service provider. While VPNs can appear as a single IP prefix (i.e., a single user domain), single prefixes will not scale to large size. The provider may set up multiple prefixes to serve user connectivity requirements. If there are multiple prefixes, it needs to be specified if routing among them is an enterprise or provider responsibility. 3.4 Frame Sequencing and MTU Support There may be requirements to deliver frames or packets in sequence. In addition, there may be a requirement to support, efficiently, larger MTUs that the provider might normally handle. 3.5 Non-IP Protocol Support While the emphasis of this taxonomy is on VPNs that support IP, the VPN may provide mechanisms for encapsulating non-IP protocols for transmission over an IP infrastructure. Techniques for doing so include NetBIOS over TCP [RFC1001/1002], IPX over IP [RFC1234], GRE [RFC1702], etc. 3.6 Multicast Support The IP VPN, as seen by end users, may support broadcast or multicast addressing. 3.7 Availability The enterprise may specify availability requirements for the infrastructure and for VPN gateway services. If redundancy of links or components is needed to provide the desired level of redundancy, these redundant components may either be visible to, or hidden from, the using enterprise. 3.8 Dynamic Provisioning The enterprise may need to be able to signal to the provider that new sites and/or users (especially dial users) have been added to the VPN. While it should generally be transparent to the VPN if new users are added to VPNs at existing sites, security requirements may make it necessary to inform the VPN, securely, that a new user has been added or an old user deleted. 4. Operational Model Several operational issues apply in VPN deployment: whether the enterprise is responsible for any customer premises equipment (CPE) that intelligently interoperates with components of the shared provider infrastructure, whether a service provider is contracted to operate the WAN infrastructure, and how any VPN client software in user hosts is managed and operated. Some organizations consider a VPN using all dial facilities, or other public facilities operated by a service provider that is not aware of the VPN, to have a customer-operated shared infrastructure. A service provider may place service-provider operated equipment at a customer site, and present a LAN or serial interface to the customer. Anything beyond the provider device is contractually a provider responsibility, but it cannot be directly controlled by the customer. There are two basic models of the administration and management of VPNs, although subcategories are perfectly viable. In the first category, the end user organization designs and operates the VPN, often with end user access through the public dial network. In the second, a service provider has contractual responsibility for designing and operating the VPN in response to specified user requirements. Another aspect of the model is whether clients are aware of the VPN, and if provider access components are aware of it. In principle, a client could attach to a generic ISP, establish an encrypted tunnel to a destination host, and operate transparently to the ISP. The VPN provider may be the ISP. In such cases, the VPN provider responsibility is to provided logins and connectivity. The login might specify a class of service to be used in the provider network. In an alternative model, the clients do not contain encryption software. Clients connect natively to the provider's access device through an administratively trusted link such as the dial telephone network. The client authenticates with the access device, and the access device(s) provide cryptographic services. Yet another model is traffic driven. Routers at customer sites sense when end user devices wish to send across the VPN, and either route them to predefined tunnels over dedicated infrastructure, or create appropriate dial calls to carry the traffic, encrypting if necessary. 5. Mapping to the Infrastructure The specific means by which end user views of the VPN are mapped onto the shared infrastructure generally involves tunneling, virtual circuit setup, or the establishment of a set of labels. When tunnels are used, they may provide no security (GRE), authentication (L2TP, L2F, and PPTP), or a wide range of security services (IPsec). Security services may also be provided by hosts, and a less secure tunnel mechanism used to carry host-encrypted data. Alternatively, the mapping of IP connectiviy may be to virtual circuits using Frame Relay or ATM, or to real circuits with ISDN or analog dial. When the VPN seen by the user appears to be multicast-capable, but the infrastructure is connection-oriented, provisions need to be made for supporting multicast. Techniques here might involve point-to-multipoint circuits, or the use of multicast replication servers. Details of these mappings will be in other, infrastructure-protocol-specific documents. Tunnel technologies need to be coordinated between the enterprise(s) and the provider(s). There may be single tunnels between sites, or possibly multiple tunnels for loadsharing and increased availability (e.g., with multilink PPP over IP). 6. Infrastructure Capabilities When different kinds of infrastructure are proposed, the main requiremment if for the infrastructure provider to take responsibility that all relevant user capabilities have matching capabilities in the infrastructure. The actual mappings are technology-specific and outside the scope of this document. This section does not attempt to define the specific infrastructure technologies that can be used for VPNs. Rather, it examines the capabilities that may be needed if a user capability is to be mapped successfully into one or more infrastructures. Specific VPNs may well be provisioned over one or more infrastructure types. In such cases, the designer needs to ensure the user capabilities map into each of the infrastructures. 6.1 Quality of Service When the user or the enterprise can request explicit QoS, either the infrastructure must be able to understand the explicit requests, or it must consistently supply a QoS that meets the most stringent user requirement. 6.2 Security Users may be responsible for cryptographic security, transparently to the provider. Alternatively, the VPN provider may offer encryption. If the user operates firewalls, VPN tunnels typically will terminate at the firewall. If the firewall is operated by the service provider, or if the user has stringent security requirements requiring end-to-end encryption, there may be compatibility issues of authenticated firewall traversal. 6.3 Addressing Providers can use registered or RFC1918 addresses internally in their networks. These may or may not be visible to the enterprise. When they are not, there should be a well-defined operational procedure that allows the user to request traceroutes through IP infrastructures. When the provider uses VPN identifiers to distinguish between routing tables for different VPNs, the same addresses, especially from the private address space, may be reused. Provider engineers should take care that 6.4 Non-IP Protocol Support Comnmonly, the enterprise will provide the tunneling necessary to carry non-IP protocols over the enterprise. When the VPN is offered as a service, however, the provider may offer appropriate encapsulation services. If the infrastructure is layer 2 and supports a protocol type field, it may be appropriate for the provider to encapsulate non-IP traffic with explicit protocol identification. 6.5 Availability When a specific availability requirement is defined for the enterprise VPN, it is a provider responsibility to ensure the infrastructure has the component reliability, diversity, etc., to meet these needs. It can be useful to distinguish between availability in the access part of a VPN, such as modem pools, and the backbone which carries the tunnels over the long-haul shared infrastructure. When the provider uses multihoming techniques inside the infrastructure to provide the availability agreed to by the providers, those multihomed services are not part of the VPN. When the enterprise uses application or transport layer multihoming to provide the availability required by the enterprise, and the VPN simply provides connectivity among the multihomed hosts, such multihoming is outside the scope of the VPN. 6.6 Interprovider Connectivity It may be necessary to provision the VPN infrastructure through multiple service providers. In such cases, the providers will need inter-provider provisioning and VPN identification. Much as a BGP confederation presents a single AS number to the outside but contains multiple internal ASNs, a multiprovider VPN identifier may map to a set of publicly visible ASNs. While BGP may be used to convey VPN reachability information among providers, the actual destinations may be prefixed with VPN IDs, and carried using the BGP-4 Multiprotocol Extensions. When VPN IDs are used in this manner, the routes carried need not be visible on the global Internet, but simply used to exchange information between ISPs with bilateral agreements. References [Berkowitz] Berkowitz, H. "To be Multihomed" Work in Progress, IETF, 1999. [Gleeson] Gleeson, Heinanen, Lin, Armitage, "A Framework for IP Based Virtual Private Networks", draft-gleeson-vpn-framework-00.txt. [Muthukrishnan] Muthukrishnan, Malis, "Core IP VPN Architecture", draft-muthukrishnan-corevpn-arch-00.txt. [RFC1001] Protocol standard for a NetBIOS service on a TCP/UDP transport: Concepts and methods. NetBIOS Working Group. Defense Advanced Research Projects Agency, Internet Activities Board, End-to-End Services Task Force. Mar-01-1987. [RFC1234] Tunneling IPX traffic through IP networks. D. Provan. Jun-01-1991 [RFC1702] Hanks, S., et al "Generic Routing Encapsulation over Ipv4 Networks", RFC 1702 Author Information Howard C. Berkowitz Gett Communications PO Box 6897 Arlington VA 22206 phone: +1-703-998-5819 email: hcb at clark.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 17474 bytes Desc: not available URL: From hcb at mail.clark.net Mon Oct 18 18:21:45 1999 From: hcb at mail.clark.net (Howard C. Berkowitz) Date: Mon, 18 Oct 1999 18:21:45 -0400 Subject: Multihoming document Message-ID: There's also a PowerPoint presentation on this topic at http://www.academ.com/nanog/feb1998/multihoming The document below is a later version of the material. INTERNET-DRAFT H. Berkowitz Expiration Date: August 1998 March 1998 To Be Multihomed: Requirements & Definitions draft-berkowitz-multirqmt-01.txt 1. Status of this Memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as ``work in progress.'' To learn the current status of any Internet-Draft, please check the ``1id-abstracts.txt'' listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast). 2. Abstract As organizations find their Internet connectivity increasingly critical to their mission, they seek ways of making that connectivity more robust. The term ''multi-homing'' often is used to describe means of fault-tolerant connection. Unfortunately, this term covers a variety of mechanisms, including naming/directory services, routing, and physical connectivity. The "home" may be identified by a DNS name, an IP address, or an IP address and transport-layer port identifier. Any of these identifiers may be translated in the path between source and destination. This memorandum presents a systematic way to define the requirement for resilience, and a taxonomy for describing mechanisms to achieve it. Its intended audience is primarily people concerned with planning and performing multihoming deployments, rather than protocol developers. It examines both requirements and applications, with the emphasis on the former. 3. Introduction As the Internet becomes more ubiquitous, more and more enterprises connect to it. Some of those enterprises, such as Web software vendors, have no effective business if their connectivity fails. Other enterprises do not have mission-critical Internet applications, but become so dependent on routine email, news, web, and similar access that a loss of connectivity becomes a crisis. As this Internet dependence becomes more critical, prudent management suggests there be no single point of failure that can break all Internet connectivity. The term "multihoming" has come into vogue to describe various means of enterprise-to-service provider connectivity that avoid a single point of failure. Multihoming also can describe connectivity between Internet Service Providers and "upstream" Network Service Providers. Several terms have become overloaded to the point of confusion, including multihoming, virtual private networks, and load balancing. This document attempts to bring some order to the definition of multihoming. It partially overlaps definitions of virtual private networks [Ferguson & Huston]. If we take the word "multihoming" in the broadest context, it implies there are multiple ways to reach a "home" destination. This "home" may be identified by a name, an IP address, or a combination of IP address and TCP/UDP port. In this document, TCP/UDP ports will be referred to as TU ports. There are other motivations for complex connectivity from enterprises to the Internet. Mergers and acquisitions, where the joined enterprises each had their own Internet access, often mean complex connectivity, at least for a transition period. Consolidation of separate divisional networks also creates this situation. A frequent case arises when a large enterprise decides that Internet access should be available corporate-wide, but their research labs have had Internet access for years -- and it works, as opposed to the new corporate connection that at best is untried. Many discussions of multihoming focus on the details of implementation, using such techniques as the Border Gateway Protocol (BGP) [RFC number of the Applicability Statement], multiple DNS entries for a server, etc. This document suggests that it is wise to look systematically at the requirements before selecting a means of resilient connectivity. One implementation technique is not appropriate for all requirements. There are special issues in implementing solutions in the general Internet, because poor implementations can jeopardize the proper function of global routing or DNS. An incorrect BGP route advertisement injected into the global routing system is a problem whether it originates in an ISP or in an enterprise. 4. Goals Requirements tend to be driven by one or more of several major goals for server availability and performance. Availability goals are realized with resiliency mechanisms, to avoid user-perceived failures from single failures in servers, routing systems, or media. Performance goals are realized by mechanisms that distribute the workload among multiple machines such that the load is distributed in an useful manner. Like multi-homing, the terms load-balancing and load-sharing have many definitions. In defining requirements, the servers themselves may either share or balance the load, there may be load-sharing or load-balancing routing paths to them, or the routed traffic may be carried over load-shared or load-balanced media. 4.1 Analyzing Application Requirements Several questions need to be answered in the process of refining goals: -- the administrative model and administrative awareness of endpoints -- availability requirements -- the security model -- addressing requirements -- scope of multihoming 4.1.1 Administrative Model A key question is: are endpoints predefined in the multihoming process, or will either the client or server end be arbitrary? The servers of interest may be inside the enterprise, or outside it. If they are outside, their names or addresses may or may not be preconfigured into multihoming mechanisms. In this document, intranet clients and servers are inside the enterprise and intendedprimarily for enterprise use. The existence of both can be preconfigured. Intranet clients have access only to machines on the intranet. Multinet clients servers are inside the enterprise, but there is pre-authorized access by known external partners. Internet servers are operated by the enterprise but intended to be accessible to the general Internet. Internet clients have general Internet access that may be mediated by a firewall. The client administrator will know the prior identity of clients, but not of servers. The server administrator will know the prior identity of servers, but not of clients. 4.1.2 Availability Requirements There are major implications between defining a requirement for high availability of initial access, and making the connection stay up once access has been achieved. The latter tends to require transport layer awareness. In the terminology of RFC1775, "To be 'on' the Internet," servers described here have "full" or a subset of "client" access. Client servers may not directly respond to specific IP packet from an arbitrary host, but a system such as a firewall MUST respond for them unless a security policy precludes that. Some valid security policies, for example, suppress the response of ICMP Destination Administratively Prohibited responses, because that would reveal there is an information resource being protected. RFC1775 defines full access as " a permanent (full-time) Internet attachment running TCP/IP, primarily appropriate for allowing the Internet community to access application servers, operated by Internet service providers. Machines with Full access are directly visible to others attached to the Internet, such as through the Internet Protocol's ICMP Echo (ping) facility. The core of the Internet comprises those machines with Full access." This definition is extended here to allow full firewalls or screening routers always to be present. If a proxy or address translation service exists between the real machine and the Internet, if this service is available on a full-time basis, and consistently responds to requests sent to a DNS name of the server, it is considered to be full-time. In this discussion, we generalize the definition beyond machines primarily appropriate for the Internet community as a whole, to include in-house and authorized partner machines that use the Internet for connectivity. Both the cases described in 4.2.3 and 4.2.4 have been termed "Virtual Private Networks." 4.1.3 Security Model Security requirements can include various cryptographic schemes, as well as mechanisms to hinder denial of service attacks. The requirements analyst must determine whether cryptography is needed, and, if so, whether cryptographic trust must be between end hosts or between end hosts and a trusted gateway. Such gateways can be routers or multiported application servers. 4.1.4. Addressing Refinements and Issues It is arguable that addressing used to support multihoming is a routing deployment issue, beyond the scope of this document. Rationales for including it here is that addressing MAY affect application behavior. There also may be administrative requirements for addressing, such as a service provider that contracts to run a multinet may require addresses to be registered, possibly from the provider's address space. If the enterprise runs applications that embed network layer addresses in higher-level data fields, solutions that employ address translation, at the packet or virtual connection level, MAY NOT work. Use of such applications inherently is a requirement for the eventual multihoming solution. Consideration also needs to be given to application caches in addition to those of DNS. Firewall proxy servers are a good example where multiple addresses associated with a given destination may not be supported. RFC1918 internal, static network address translation (NAT) to outside RFC1918 internal, dynamic port address translation (PAT) to outside Registered internal, Provider Assigned (PA) Registered internal, Provider Independent (PI) 4.1.5 Scope of multihoming Multihoming may be defined between an end host and a router or application gateway, on an end-to-end basis possibly involving virtual servers, among routers, or among elements in a transmission system. Different multihoming scopes may support the same application requirement. 4.2. Application Goals These goals need to be agreed to by the people or organization responsible for the applications. Not to reach fairly formal agreement here can lead to problems of inappropriate expectations. The term "service level agreement" often refers to expectations of performance, such as throughput or response time. Ideas here extend the performance-based model to include availability. 4.2.1 Specific server availability The first goal involves well-defined applications that run on specific servers visible to the Internet at large. This will be termed "endpoint multihoming", emphasizing the need for resilience of connectivity to well-defined endpoints. Solutions here often involve DNS mechanisms. There are both availability and performance goals here. Availability goals arise when there are multiple routing paths that can reach the server, protecting it from single routing failures. Other availability goals involve replicated servers, so that the client will reach a server regardless of single server failures. Performance goals include balancing client requests over multiple servers, so that one or more servers do not become overloaded and provide poor service. Requests can be distributed among servers in a round-robin fashion, or more sophisticated distribution mechanisms can be employed. Such mechanisms can consider actual real-time workload on the server, routing metric from the client to the server, known server capacity, etc. >From an application standpoint, this is either a many-to-one topology, many clients to one server, or a many-to-many topology when multiple servers are involved. It can be worthwhile to consider a many-to-few case, when the few are multiple instances of a server function, which may appear as a single server to the general Internet. The idea of many-to-few topology allows for a local optimization of inter-server communications, without affecting the global many-to-one model. Addresses on interfaces that connect to the general Internet need to be unique in the global Internet routing system, although they may be translated, at the network address or port level, from public to internal space.. 4.2.2 General Internet connectivity from the enterprise The second is high availability of general Internet connectivity for arbitrary enterprise users to the outside. This will be called "internetwork multihoming". Solutions here tend to involve routing mechanisms. This can be viewed as a few-to-many application topology. Addresses on interfaces that connect to the general Internet need to be unique in the global Internet routing system, although they may be translated, at the network address or port level, from internal private address to public space. 4.2.3 Use of Internet services to interconnect "intranet" enterprise campuses The third involves the growing number of situations where Internet services are used to interconnect parts of an enterprise. This is "intranetwork multihoming". This will usually involve dedicated or virtual circuits, or some sort of tunneling mechanisms. This case may be termed a "virtual private network." The "multihoming" aspect of this case is associated with high availability to the connectivity network that underlies the tunneling system. In this case, addresses only need to be unique within the enterprise. 4.2.4 Use of Internet services to connect to "multinet" partners A fourth category involves use of the Internet to connect with strategic partners. True, this does deal with endpoints, but the emphasis is different than the first case. In the first case, the emphasis is on connectivity from arbitrary points outside the enterprise to points within it. This second case deals with pairs of well-known endpoints. These endpoints may be linked with dedicated or virtual circuits defined at the physical or data link layer. Tunneling or other virtual private networks may be relevant here as well. There will be coordination issues that do not exist for the third case, where all resources are under common control. Addresses need to be unique in the different enterprises, but do not need to be unique in the global Internet. 5. Planning and Budgeting In each of these scenarios, organization managers need to assign some economic cost to outages. Typically, there will be an incident cost and an incremental cost based on the length or scope of the connectivity loss. Ideally, this cost is then weighted by the probability of outage. A weighted exposure cost results when the outage cost is multiplied by the probability of the outage. Resiliency measures modify the probability, but increase the cost of operation. Operational costs obviously include the costs of redundant mechanisms (i.e., the addititional multihomed paths), but also the incremental costs of personnel to administer the more complex mechanisms -- their training and salaries. 6. Issues 6.1 Performance vs. Robustness: the Cache Conundrum Goals of many forms of "multi-homing" conflict with goals of improving local performance. For example, DNS queries normally are cached in DNS servers, and in the requesting host. From the performance standpoint, this is a perfectly reasonable thing to do, reducing the need to send out queries. >From the multihoming standpoint, it is far less desirable, as application-level multihoming may be based on rapid changes of the DNS master files. The binding of a given IP address to a DNS name can change rapidly. 6.2 Service Level Agreements Enterprise networks, especially mainframe-based, are accustomed to building and enforcing service level agreements for application performance. A key to being able to do this is total control of the end-to-end communications path. In the current Internet, the enterprise(s) at one or both ends control their local environments, and have contractual control over connections to their direct service providers. If service level control is a requirement, and both ends of the path are not under control (i.e., cases 1 and 2), the general Internet cannot now provide service level guarantees. The need for control should be reexamined, and, if it still exists, the underlying structure will need to be dedicated resources at the network layer or below. A network service provider may be able to engineer this so that some facilities are shared to reduce cost, but the sharing is planned and controlled. 6.3 Symmetry One of the reasons service level agreements are not enforceable in the general Internet is the reality that global routing cannot be guaranteed to be symmetrical. Symmetrical routing assumes the path to a destination is simply reversed to return a response from that destination. Both legs of a symmetrical path are assumed to have the same performance characteristics. Global Internet routing is not necessarily optimized for best end-to-end routing, but for efficient handling in the Autonomous Systems along the path. Many service providers use "closest exit" routing, where they will go to the closest exit point from their perspective to get to the next hop AS. The return path, however, is not necessarily of a mirror image of the path from the original source to the destination. Closest exit routing is, in fact, a "feature" rather than a "bug" in some multihoming schemes [Peterson] [Friedman]. Especially when the enterprise network has multiple points of attachment to the Internet, either to a single ISP AS or to multiple ISPs, it becomes likely that the response to a given packet will not come back at the same entry point in which it left the enterprise. This is probably not avoidable, and troubleshooting procedures and traffic engineering have to consider this characteristic of multi-exit routing. 6.4 Security ISPs may be reluctant to let user routing advertisements or DNS zone information flow directly into their routing or naming systems. Users should understand that BGP is not intended to be a plug-and-play mechanism; manual configuration often is considered an important part of maintaining integrity. Supplemental mechanisms may be used for additional control, such as registering policies in a registry [RPS, RA documents] or egress/ingress filtering [Ferguson draft] Challenges may arise when client security mechanisms interact with fault tolerance mechanisms associated with servers. For example, if a server address changes to that of a backup server, a stateful packet screening firewall might not accept a valid return. Similarly, unless servers back one another up in a full mirroring mode, if one end of a TCP-based application connection fails, the user will need to reconnect. As long as another server is ready to accept that connection, there may not be major user impact, and the goal of high availability is realized. High availability and user transparent high availability are not synonymous. 6.5 Load Balancing vs. Load Sharing These terms are often interchanged, but they really mean different things. Load balancing is deterministic, and at a finer level of control than load sharing, which is statistical. Load balancing is generally not something that can be realized in general Internet routing, other than in special and local cases between adjacent AS. A degree of load sharing is achievable in routing, but it may introduce significant resource demands and operational complexity. Paul Ferguson defines load-balancing as "a true "50/50" sharing of equal paths. This can be done by either (a) round robin per- packet transmission, (b) binding pipes a the lower layers such that bits are either 'bit-striped' across all parallel paths (like the etherchannel stuff), or binding pipes so that SAR functions are done in a method such as multilink PPP. These are fundamentally the same. "Load-sharing is quite different. It simply implies that no link is sitting idle -- that at least all links get utilized in some fashion. Usually in closest exit routing. The equity of utilization may be massively skewed. It may also resemble something along the lines of 60/40, which is reasonable." 6.6 Application Compatibility Some deployment mechanisms involve network address, or network address and TCP/UDP port, translation (NAT and NAPT). If the application protocols embed IP addresses in their protocol fields, NAT or NAPT may cause protocol failures. Translation mechanisms for such cases may require knowledge of the application protocol, as typified by application proxies in firewalls, or in application gateways with multiple interfaces. 7. Multihoming Deployment Technologies A basic way to tell which technology(ies) is applicable is to ask oneself whether the functional requirement is defined in terms of multihoming to specific hosts, or to specific networks. If the former, some type of application or transport technology is needed, because only these technologies have awareness of specific hosts. A given multihoming implementation may draw on several of these technologies. For example, the Cisco Distributed Director does DNS name-level redirection based in part on routing metrics. 7.1 Application/Name Based Techologies in this category may involve referring a client request to different instances of the endpoint represented by a single name. Another aspect of application/name multihoming may work at the level of IP addressing, but specifically is constrained to endpoint (i.e., server) activities that redirect the client request to a different endpoint. Application-level firewall proxy services can provide this functionality, although their application protocol modification emphasizes security while a multihoming application service emphasizes availability and quality of service. 7.2 Transport Based Transport based technologies are based on maintaining tunnels through an underlying network or transmission system. The tunnels may be true end to end, connecting a client host with a server host, or may interconnect between proxy servers or other gateways. 7.3 Network Based Network based approaches to multihoming are router-based. They involve having alternate next-hops, or complete routes, to alternate destinations. 7.4 Data Link Based Data link layer methods may involve coordinated use of multiple physical paths, as in multilink PPP or X.25. If the underlying WAN service has a virtual circuit mechanism, as in frame relay or ATM, the service provider may have multihomed paths provided as part of the service. Such functions blur between data link and physical layers. Other data link methods may manipulate MAC addresses to provide virtual server functions. 7.5 Physically-based Physical multihoming strategies can use diverse media, often of different types such as a wire backed up with a wireless data link. They also involve transmission media which have internal redundancy, such as SONET. 8. Application/Name Multihoming While many people look at the multihoming problem as one of routing, the goal is often multihoming to endpoints. Finding an endpoint usually begins in DNS. Once an endpoint address is found, some application protocols, notably HTTP and FTP, may redirect the request to a different endpoint. The basic idea here is that arbitrary clients will first request access to a resource by its DNS name, and certain DNS servers will resolve the same name to different addresses based on conditions of which DNS is aware, or using some statistical load-distribution mechanism. There are some general DNS issues here. DNS was not really designed to do this. A key issue is that of DNS cacheing. Cacheing and frequent changes in name resolution are opposite goals. Traditional DNS schemes emphasize performance over resiliency. 8.1 DNS Caching DNS standards do provide the capability for short cache lifetimes, which in principle support name based multihoming. "The meaning of the TTL field is a time limit on how long an RR can be kept in a cache. This limit does not apply to authoritative data in zones; it is also timed out, but by the refreshing policies for the zone. The TTL is assigned by the administrator for the zone where the data originates. While short TTLs can be used to minimize caching, and a zero TTL prohibits caching, the realities of Internet performance suggest that these times should be on the order of days for the typical host. If a change can be anticipated, the TTL can be reduced prior to the change to minimize inconsistency during the change, and then increased back to its former value following the change. [RFC1034] " Several real-world factors limit the utility of simply shortening the cache time. Widely used BIND, the most widely used DNS implementation, does not accept cache lifetimes less than 5 minutes. Dynamic DNS may be a long-term solution here. In the short term, setting very short TTL values may be help in some cases, but is not likely to help elow a granularity of 5 minutes. Remember that the name normally is resolved when an application session first is established, and the decisions are made over a longer time base than per-packet routing decisions. 8.2 DNS Multiple Hosts and Round Robin Response The DNS protocol allows it to return multiple host addresses in respondse to a single query. At the first level of DNS-based multihoming, this can provide additional reliability. A DNS server knows three IP addresses for the server function identified by server.example.com, 10.0.1.1, 10.0.2.1, and 10.0.3.1. A simple response to a query for server.example.com returns all three addresses. Assume the response provides server addresses in the order 10.0.1.1, 10.0.2.1, and 10.0.3.1. Whether this will provide multihoming now depends on the DNS client. Not all host client implementations will, if the first address returned (i.e., 10.0.1.1) does not respond, try the additional addresses. In this example, 10.0.2.1 might be operating perfectly, A variant suggested by Kent England is to have the addresses returned in the DNS response come from the CIDR blocks of different ISPs that provide connectivity to the server function [England]. This approach combines aspects of name and network multihoming. Again, this will work when intelligent clients try every IP address returned until a server responds. 8.3 Replication One approach [Peterson] uses DNS as the main method, but makes assumptions about the underlying routing. The two assumptions it makes, which appears generally valid for the Internet, are that connectivity providers use closest exit routing, and once a packet reaches a provider network, that provider knows best how to reach the specific server inside that network. One implementation of this approach is Cisco's DistributedDirector product. It involves communication between DNS servers and routers in the multihomed domain. When a DNS request is received by a DistributedDirector DNS server, the DD server selects the IP address to be returned based on information on routing cost from the client entry point to closest server, on administrative weight, or other generally routing-associated factors. 8.4 Cache Servers and Application Multihoming The cache server is the primary home for servicing the client request, but the true server backs it up. 9. Transport Multihoming Transport layer functions are conceptually end-to-end. There are two broad classes of transport multihoming function, those maintained by the endpoints and those that involve intermediate translation devices. 9.1 End-to-end Tunnel Maintenance Basic point-to-point tunneling mechanisms include GRE, PPTP and L2TP. DVMRP is a special case. Choices here will depend in part on the security policy and the administrative model by which multihoming is provided. GRE, for example, does not itself provide encryption, while PPTP and L2TP do. "The differences between PPTP and L2TP are more of where one wishes the PPP session to terminate, and one of control. It really depends on who you are, and where you are, in the scheme of the control process. If your desire is to control where the PPP session terminates (as an ISP might wish to control), then L2TP might be the right approach. On the other hand, however, if you are a subscriber, and you wish to control where the PPP session terminates (to, say, a PPTP server somewhere across the cloud), then PPTP might be the right approach -- and it would be transparent to the service provider). It really depends on what problem one is trying to solve, and if you are in the business of trying to create "services." [Ferguson-1998-2] 9.2 Tunneling with Translation Various proxy and address translation mechanisms can play a role in multihoming. They can be divided into several levels of topological constraints: -- all servers are colocated in a single address domain "behind" the translator -- servers are in different parts of a single address domain. These parts are connected by tunnels. -- the servers are at arbitrary network addresses, but the translator knows how to reach them. Application-aware proxies can have even more knowledge of application load. A variant of NAT, called load sharing NAT (LS-NAT), offers a load sharing mechanism at the transport/network level rather than the DNS level [Srishuresh]. When considering LSNAT-style load sharing, remember that it emphasizes providing a pool of servers capable of servicing requests. In its "local" form, it does not easily provide mechanisms for increasing reliability by mapping the user request to geographically distributed servers. More advanced variants can combine with DNS- and routing-aware mechanisms to increase reliability as well as performance. The LSNAT function is visible globally as a server address. It is actually a virtual server. When a client request arrives at the LSNAT, the LSNAT translates the destination address, transparently to the client, and passes it to a server in the LSNAT's server pool. LSNATs, in their basic form, do have topological restriction. It has been suggested that all request/response traffic in a single session must go from the real client, to one specific LSNAT, to the server. It is conceivable that multiple routers could be used, but they would need to be tightly synchronized. LSNAT builds on NAPT, but adds intelligence to the port mapping function. Also, general NAPT is oriented toward outgoing requests from the stub domain to the outside, while LSNAT emphasizes incoming requests to a virtual server address. As currently conceived, LSNATs operate at the TCP/UDP transport level, so they could not easily change server hosts during a session. There are potential workarounds to this, including using a multicast address as the server pool destination, with coordination between the servers as to which actually answers the request. For highly fault-tolerant applications, more than one server conceivably could answer the NAT request, and the LSNAT decide which to pass to the client. Typically, if servers are identical, it would be the first response received by the server pool side of the LSNAT. This general topology restriction suggests LSNAT functionality belongs on a single NAT-capable border router, with the server pool inside the stub domain. A LSNAT violates the Internet end-to-end model in the same way that basic NAT does. There is no requirement that the addressing in the stub domain be private, only that all access to the servers go through the NAT. In basic LSNAT, an arbitrary external client attempts to establish a session with what it believes to be a server. In reality, it is attempting to establish a session with the virtual server address of the "outside" interface of the LSNAT. The LSNAT, based on internal criteria, redirects the external request to a specific internal server in a server pool. Unique connections are established from the LSNAT to the pool server for each request, translating addresses and ports as needed. 9.2.2. Load Shared Network Address and Port Translation (LS-NAPT) Adding the complexity of port as well as address translation gives additional flexibility. In particular, adding port translation removes many topological limitations between the real servers and the NAT. In a LS-NAT router, inbound sessions identified by the tuple From a typical server room, analog and digital signals physically flow to a wiring closet, where they join a riser cable. Depending on the specific building, the closet and riser may be the responsibility of the enterprise or ISP, the building management, or a telecommunications carrier. The riser cable joins with other riser cables in a cable vault, from which a cable leaves the building and goes to the end switching office of the local telecommunications provider. Most buildings have a single cable vault, possibly with multiple cables following a single physical route back to the end office. A single error by construction excavators can cut multiple cables on a single path. A failure in carrier systems can isolate a single end office. Highly robust systems have physical connectivity to two or more POPs reached through two or more end offices. Alternatives here can become creative. On a campus, it can be feasible to use some type of existing ductwork to run additional cables to another building that has a physically diverse path to the end office. Direct wire burial, fiber optic cables run in the air between buildings, etc., are all possible. In a non-campus environment, it is possible, in many urban areas, to find alternate means of running physical media to other buildings with alternate paths to end offices. Electrical power utilities may have empty ducts which they will lease, and through which privately owned fiber can be run. 11.2 Provider Core As demonstrated by a rash of fiber cuts in early 1997, carriers lease bandwidth from one another, so a cut to one carrier-owned facility may affect connectivity in several carriers. This reality makes some traditional diverse media strategies questionable. Many organizations consciously obtain WAN connectivity from multiple carriers, with the notion that a failure in carrier will not affect another. This is not a valid assumption. If the goal is to obtain diversity/resiliency among WAN circuits, it may be best to deal with a single service provider. The contract with this provider should require physical diversity among facilities, so the provider's engineering staff will be aware of requirements not to put multiple circuits into the same physical facility, owned by the carrier or leased from other carriers. 12. Security Considerations 13. Acknowledgments 14. References [RFC1775] D. Crocker. To Be "On" the Internet. March 1995. [RFC1930] Hawkinson, J., and T. Bates. Guidelines for creation, selection, and registration of an Autonomous System (AS). March 1996. (Also BCP0006) [RFC1034] Mockapetris, P.V. Domain names - concepts and facilities. Nov-01-1987. [RFC1998] An Application of the BGP Community Attribute in Multi-home Routing. E. Chen & T. Bates. August 1996 [RFC2071] Ferguson, P., and H. Berkowitz, "Network Renumbering Overview: Why would I want it and what is it anyway?", RFC 2071, January 1997. [RFC2050] Hubbard, K., Kosters, M., Conrad, D., Karrenberg, D., and J. Postel, "INTERNET REGISTRY IP ALLOCATION GUIDELINES", BCP 12, RFC 2050, November 1996. [RFC1631] Egevang,, K., and P. Francis, "The IP Network Address Translator (NAT)", RFC 1631, May 1994. [Srishuresh] Srishuresh, P., and D. Gan, "Load Sharing using IP Network Address Translation (LSNAT)", work in progress, ftp://ftp.ietf.org/internet-drafts/draft-srisuresh-lsnat-02.txt [RFC1918] Rekhter, Y., Moskowitz, R., Karrenberg, D., de Groot, G-J., and E. Lear, "Address Allocation for Private Internets", RFC 1918, February 1996. [RFC1900] Carpenter, B., and Y. Rekhter, "Renumbering Needs Work", RFC 1900, February 1996. [RFC2280] Routing Policy Specification Language (RPSL). C. Alaettinoglu, T. Bates, E. Gerich, D. Karrenberg, D. Meyer, M. Terpstra, C. Villamizar. January 1998. [RFC1812] Baker, F., "Requirements for IP Version 4 Routers", RFC 1812, June 1995. [Peterson] Peterson, A. "Dynamic Selection of Geographically Distributed Servers," presentation at NANOG October 1997 meeting, notes at http://www.academ.com/nanog/october1997/dynamic-selection.html [Freedman] Freedman, A. "Dynamic Selection of Geographically Distributed Servers," presentation at NANOG October 1997 meeting, notes at http://www.academ.com/nanog/october1997/dynamic-selection.html [Ferguson-1998-1] Ferguson, P. "Re: Comments on "What is a VPN?"" Message to IETF VPN mailing list, 08 Mar 1998 19:52:29 15. Author's Address Howard C. Berkowitz PO Box 6897 Arlington VA 22206 Phone: +1 703 998 5819 EMail: hcb at clark.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 47868 bytes Desc: not available URL: