One document matched: draft-villamizar-mpls-multipath-extn-00.xml


<?xml version="1.0" encoding="US-ASCII"?>
<!-- xml2rfc is available at http://xml.resource.org. -->
<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [

  <!ENTITY RFC2119 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.2119.xml">
  <!ENTITY RFC2475 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.2475.xml">
  <!ENTITY RFC2702 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.2702.xml">
  <!ENTITY RFC2991 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.2991.xml">
  <!ENTITY RFC3209 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.3209.xml">
  <!ENTITY RFC3260 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.3260.xml">
  <!ENTITY RFC3471 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.3471.xml">
  <!ENTITY RFC3945 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.3945.xml">
  <!ENTITY RFC4201 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.4201.xml">
  <!ENTITY RFC4206 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.4206.xml">
  <!ENTITY RFC4385 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.4385.xml">
  <!ENTITY RFC5420 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.5420.xml">
  <!ENTITY RFC5586 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.5586.xml">
  <!ENTITY RFC5786 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.5786.xml">
  <!ENTITY RFC5920 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.5920.xml">
  <!ENTITY RFC6107 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.6107.xml">
  <!ENTITY RFC6391 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.6391.xml">

  <!ENTITY I-D.ietf-mpls-tp-security-framework SYSTEM "http://xml.resource.org/public/rfc/bibxml3/reference.I-D.draft-ietf-mpls-tp-security-framework-04.xml">

  <!ENTITY I-D.villamizar-mpls-multipath-use SYSTEM "http://xml.resource.org/public/rfc/bibxml3/reference.I-D.draft-villamizar-mpls-multipath-use-00">

  <!ENTITY I-D.ietf-mpls-entropy-label SYSTEM "http://xml.resource.org/public/rfc/bibxml3/reference.I-D.draft-ietf-mpls-entropy-label-06">

  ]>

<?xml-stylesheet type='text/xsl' href='rfc2629.xslt' ?>
<?rfc strict="yes" ?>
<?rfc toc="yes"?>
<?rfc tocdepth="4"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes" ?>
<?rfc compact="yes" ?>
<?rfc subcompact="no" ?>
<?rfc comments="yes"?>
<?rfc inline="yes" ?>

<rfc category="std" ipr="trust200902"
     docName="draft-villamizar-mpls-multipath-extn-00">

  <front>
    <title abbrev="MPLS TE Multipath Extensions">
      Multipath Extensions for MPLS Traffic Engineering</title>

    <author role="editor"
	    fullname="Curtis Villamizar" initials="C." surname="Villamizar">
      <organization>Outer Cape Cod Network Consulting</organization>
      <address>
        <email>curtis@occnc.com</email>
      </address>
    </author>

    <date year="2012" />

    <area>Routing</area>
    <workgroup>MPLS</workgroup>

    <keyword>MPLS</keyword>
    <keyword>composite link</keyword>
    <keyword>link aggregation</keyword>
    <keyword>ECMP</keyword>
    <keyword>link bundling</keyword>
    <keyword>multipath</keyword>
    <keyword>MPLS-TP</keyword>

    <abstract>
      <t>
	Extensions to OSPF-TE, ISIS-TE, and RSVP-TE are defined in
	support of carrying LSP with strict packet ordering
	requirements over multipath and and carrying LSP with strict
	packet ordering requirements within LSP without violating
	requirements to maintain packet ordering.  LSP with strict
	packet ordering requirements include MPLS-TP LSP.
      </t>
      <t>
	OSPF-TE and ISIS-TE extensions defined here indicate node and
	link capability regarding support for ordered aggregates of
	traffic, multipath traffic distribution, and abilities to
	support multipath load distribution differently per LSP.
      </t>
      <t>
	RSVP-TE extensions either identifies an LSP as requiring
	strict packet order, or identifies an LSP as carrying one or
	more LSP that requires strict packet order further down in the
	label stack, or identifies an LSP as having no restrictions on
	packet ordering except the restriction to avoid reordering
	microflows.  In addition an extension indicates whether the
	first nibble of payload will reliably indicate whether payload
	is IPv4, IPv6, or other type of payload, most notably
	pseudowire using a pseudowire control word.
      </t>
    </abstract>
  </front>

  <middle>
    <section title="Introduction">
      <t>
	Today the requirement to handle large aggregations of traffic,
	can be handled by a number of techniques which we will
	collectively call multipath.  Multipath is very similar to
	composite link as defined in <xref target="ITU-T.G.800" />,
	except multipath specifically excludes inverse multiplexing.
	Some types of LSP, including but potentially not limited to
	MPLS-TP LSP, require strict packet ordering.
      </t>
      <t>
	A means to support a MPLS-TP client layer over a MPLS server
	layer using MPLS Entropy Label is defined in
	<xref target="I-D.villamizar-mpls-multipath-use" />.  It is
	noted in <xref target="I-D.villamizar-mpls-multipath-use" />
	that absent some protocol extensions, some limitations must be
	accepted.
      </t>
      <t>
	This document defines protocol extensions which better
	supports using MPLS with multipath as a server layer for
	MPLS-TP, or to carry MPLS-TP directly over a network which
	makes use of multipath.  Extensions are also applicable to
	MPLS when used in the presense of very large microflows or
	very large LSP which cannot be load split as a result of using
	MPLS Entropy Label
	<xref target="I-D.ietf-mpls-entropy-label" />.
      </t>

      <section title="Architecture Summary">

	<t>
	  Advertisements in a link state routing protocol, such as OSPF
	  or ISIS, support a topology map known as a link state database
	  (LSDB).  When traffic engineering information is included in
	  the LSDB the topology map is known as a TE-LSDB or traffic
	  engineering database (TED).
	</t>
	<t>
	  A common MPLS LSP path computation is known as a constrained
	  shortest path first computation (CSPF) (see <xref
	  target="RFC3945" />).  Other algorithms may be used for path
	  computation.  Constraint-based routing was first introduced in
	  <xref target="RFC2702" />).
	</t>
	<t>
	  OSPF-TE or ISIS-TE extensions are defined in <xref
	  target="node-capability" /> and <xref
	  target="link-capability" />.  OSPF-TE or ISIS-TE
	  advertisements serve to populate the TE-LSDB and provide the
	  basis for constraint-based routing path computation.  <xref
	  target="usage.adv" /> describes the use of OSPF-TE or
	  ISIS-TE multipath extensions in routing advertisements.
	</t>
	<t>
	  RSVP-TE extensions are defined in <xref target="rsvp_mpa_tlv" />.
	  <xref target="usage.rsvp" /> describes the use of RSVP-TE
	  extensions in setting up LSP including signaling constraints
	  on LSP which contain other LSP which specify RSVP-TE
	  extensions.
	</t>
	<t>
	  <xref target="usage.cspf" /> describes the
	  constraints on LSP path computation imposed by the
	  advertised ordered aggregate and multipath capabilities of
	  links.  <xref target="usage.ip-in-cspf" /> describes the
	  constraints on LSP path computation imposed by link
	  advertisements regarding use of IP headers in multipath
	  traffic distribution.  <xref target="usage.depth" />
	  describes the impact of label stack depth limitations.
	</t>

      </section>

      <section title="Requirements Language">
        <t>
	  The key words "MUST", "MUST NOT", "REQUIRED", "SHALL",
          "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY",
          and "OPTIONAL" in this document are to be interpreted as
          described in <xref target="RFC2119">RFC 2119</xref>.
	</t>
      </section>

      <section anchor="def" title="Definitions">

	<t>
	  Please refer to <xref target="I-D.villamizar-mpls-multipath-use" />.
	</t>
	<t><list style="hanging" hangIndent="3">
	  <t hangText="Ordered Aggregate (OA)">
	    <vspace blankLines="0" />
	    An ordered aggregate (OA) requires that packets be delivered
	    in the order in which they were received.  Please refer to
	    <xref target="RFC3260" />.
	  </t>
	  <t hangText="Microflow">
	    <vspace blankLines="0" />
	    A microflow is a single instance of an application-to-
	    application flow.  Please refer to <xref target="RFC2475"
	    />.  Reordering packets within a microflow can cause service
	    disruption.  Please refer to <xref target="RFC2991" />.
	  </t>
	  <t hangText="Multipath Traffic Distribution">
	    <vspace blankLines="0" />
	    Multipath traffic distribution refers to the mechanism which
	    distributes traffic among a set of component links or
	    component lower layer paths which together comprise a
	    multipath.  No assumptions are made about the algorithms
	    used in multipath traffic distribution.  This document only
	    discusses constraints of the type of information which can
	    be used as the basis for multipath traffic distribution in
	    specific circumstances.
	  </t>
	</list></t>
	<t>
	  The phrase "strict packet ordering requirements" refers to the
	  requirement to deliver all packet in the order that they were
	  received.  The absence of strict packet ordering requirements
	  does not imply total absence of packet ordering requirements.
	  The requirement to avoid reordering traffic within any given
	  microflow, as described in <xref target="RFC2991" /> applies
	  to all traffic aggregates including all MPLS LSP.
	</t>
	<t>
	  The abbreviations ELI and EL are Entropy Label Indicator and
	  Entropy Label, as defined in
	  <xref target="I-D.ietf-mpls-entropy-label" />.
	</t>

      </section>

    </section>

    <section anchor="encaps"
	     title="Protocol Extensions">

      <t>
	This section defined protocol extensions to OSPF-TE, ISIS-TE,
	and RSVP-TE.  Use of these extensions is described in
	<xref target="usage" /> and <xref target="compat" />.
      </t>
      <t>
	Two capability sub-TLV are added to two TLV that are used in
	both OSPF-TE and ISIS-TE.
	The Multipath Node Capability sub-TLV is added to the
	Node Attribute TLV ( see <xref target="node-capability" />).
	The Multipath Link Capability TLV is added to the
	Interface_ID (see <xref target="link-capability" />).
      </t>

      <t>
	One TLV is added to the LSP_ATTRIBUTES object defined in
	<xref target="RFC5420" />.
      </t>

      <section anchor="node-capability"
	       title="Multipath Node Capability sub-TLV">
	<t>
	  The Node Attribute TLV is defined in <xref target="RFC5786" />.

	  A new sub-TLV, the Multipath Node Capability sub-TLV,
	  is defined for inclusion in the Node Attribute TLV.
	</t>

	<figure anchor="tp-mp-node-capability-sub-tlv"
		title="Multipath Capability Sub-TLV">
	  <artwork>
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |              Type             |             Length            |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |              Flags            |   Max Depth   |   IP Depth    |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                  Largest Supportable Microflow                |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	  </artwork>
	</figure>

	<t>
	  The fields in the Multipath Capability sub-TLV are
	  defined as follows.
	</t>
	<t><list style="hanging" hangIndent="3">
	  <t hangText="Type">
	    <vspace blankLines="0" />
	    The Type field is assigned a value of IANA-TBD-1.  The Type
	    field is a two octet value.
	  </t>
	  <t hangText="Length">
	    <vspace blankLines="0" />
	    The Length field indicates the length of the sub-TLV in
	    octets, excluding the Type and Length fields.  The Length
	    field is a two octet value.
	  </t>
	  <t hangText="Flags">
	    <vspace blankLines="0" />
	    The Flags field is a two octet (16 bit) value.  The
	    following single bit fields are assigned within this
	    value, starting at the most significant bit, which is the
	    bit transmitted first.
	  <list style="hanging" hangIndent="3">
	    <t hangText="0x8000  Ordered Aggregate Enabled">
	      <vspace blankLines="0" />
	      Setting the Ordered Aggregate Enabled bit indicates that
	      an LSP can be carried as an Ordered Aggregate Enabled on
	      one or more links.
	    </t>
	    <t hangText="0x4000  Multipath Enabled">
 	      <vspace blankLines="0" />
	      Setting the Multipath Enabled bit indicates that an LSP
	      can be spread across component links at one or more
	      multipath links.
	    </t>
	    <t hangText="0x2000  IPv4 Enabled Multipath">
 	      <vspace blankLines="0" />
	      Setting the IPv4 Enabled Multipath bit indicates that
	      the IPv4 header information can be used in multipath
	      load balance.  The Multipath Enabled bit must be set if
	      the IPv4 Enabled Multipath bit is set.
	    </t>
	    <t hangText="0x1000  IPv6 Enabled Multipath">
 	      <vspace blankLines="0" />
	      Setting the IP bit indicates that the IPv6 header
	      information can be used in multipath load balance.  The
	      Multipath Enabled bit must be set if the IPv6 Enabled
	      Multipath bit is set.
	    </t>
	    <t hangText="0x0800  UDP/IPv4 Multipath">
 	      <vspace blankLines="0" />
	      Setting the UDP/IPv4 Multipath bit indicates that the UDP
	      port numbers carried in UDP over IPv4 can be used in
	      multipath load balance.  The IPv4 Enabled Multipath bit
	      must be set if UDP/IPv4 Multipath is set.  If the IPv4
	      Enabled Multipath bit is set and the UDP/IPv4 Multipath
	      bit is clear, then only source and destination IP
	      addresses are used.
	    </t>
	    <t hangText="0x0400  UDP/IPv6 Multipath">
 	      <vspace blankLines="0" />
	      Setting the UDP/IPv6 Multipath bit indicates that the
	      UDP port numbers carried in UDP over IPv6 can be used in
	      multipath load balance.  The IPv6 Enabled Multipath bit
	      must be set if UDP/IPv6 Multipath is set.  The IPv6
	      Enabled Multipath bit must be set if UDP/IPv6 Multipath
	      is set.  If the IPv6 Enabled Multipath bit is set and
	      the UDP/IPv6 Multipath bit is clear, then only source
	      and destination IP addresses are used.
	    </t>
	    <t hangText="0x0200  TCP/IPv4 Multipath">
 	      <vspace blankLines="0" />
	      Setting the TCP/IPv4 Multipath bit indicates that the TCP
	      port numbers carried in TCP over IPv4 can be used in
	      multipath load balance.  The IPv4 Enabled Multipath bit
	      must be set if TCP/IPv4 Multipath is set.  If the IPv4
	      Enabled Multipath bit is set and the TCP/IPv4 Multipath
	      bit is clear, then only source and destination IP
	      addresses are used.
	    </t>
	    <t hangText="0x0100  TCP/IPv6 Multipath">
 	      <vspace blankLines="0" />
	      Setting the TCP/IPv6 Multipath bit indicates that the
	      TCP port numbers carried in TCP over IPv6 can be used in
	      multipath load balance.  The IPv6 Enabled Multipath bit
	      must be set if TCP/IPv6 Multipath is set.  The IPv6
	      Enabled Multipath bit must be set if TCP/IPv6 Multipath
	      is set.  If the IPv6 Enabled Multipath bit is set and
	      the TCP/IPv6 Multipath bit is clear, then only source
	      and destination IP addresses are used.
	    </t>
	    <t hangText="0x0080  Default to Multipath">
 	      <vspace blankLines="0" />
	      Setting the Default to Multipath bit indicates that for
	      an LSP which does not signal a desired behavior the
	      traffic for that LSP will be spread across component
	      links at one or more multipath links.  If the Default to
	      Multipath bit is not set, then an LSP which does not
	      signal otherwise will be treated as an ordered
	      aggregate.
	    </t>
	    <t hangText="0x0040  Default to IP/MPLS Multipath">
 	      <vspace blankLines="0" />
	      Setting the Default to IP/MPLS Multipath indicates that
	      for an LSP which does not signal a desired behavior, the
	      IP header information will be used in the multipath load
	      distribution.  If the Default to IP/MPLS Multipath is
	      clear it indicates that the the IP header information
	      will not be used by default.
	    </t>
	    <t hangText="0x0020  Entropy Label Multipath">
 	      <vspace blankLines="0" />
	      Setting the Entropy Label Multipath bit indicates that
	      when multipath is enabled for a given LSP, if an Entropy
	      Label Indicator (ELI) is found in the label stack,
	      information below the Entropy Label (EL) will not be
	      used in multipath load distribution.
	    </t>
	    <t hangText="0x0010  IP Optional Multipath">
 	      <vspace blankLines="0" />
	      Setting the IP Optional Multipath bit indicates that when
	      multipath is enabled for a given LSP, whether the IP
	      header information is used in the multipath load
	      distribution can be set on a per LSP basis.
	    </t></list>
	    The remaining bits in the Flags field are reserved.
	  </t>
	  <t hangText="Max Depth">
	    <vspace blankLines="0" />
	    The Max Depth field is a one octet field indicating the
	    maximum label stack depth beyond which the multipath load
	    distribution cannot make use of further label stack
	    entries.
	  </t>
	  <t hangText="IP Depth">
	    <vspace blankLines="0" />
	    The IP Depth field is a one octet field indicating the
	    maximum label stack depth beyond which the multipath load
	    distribution cannot make use of IP information.
	  </t>
	  <t hangText="Largest Supportable Microflow">
	    <vspace blankLines="0" />
	    The Largest Supportable Microflow field is a IEEE 32 bit
	    floating point value expressing in bytes/second.  Any
	    microflow which exceeds this capacity may experience
	    either packet loss, or out-of-order delivery, or both.
	  </t>
	</list></t>
	<t>
	  The reserved bits in the Flags field MUST be set to zero and
	  MUST be ignored unless implementing an extension which
	  redefines one or more of the reserved bits.  Any further
	  extension which redefines one or more reserved Flags bit
	  should maintain backwards compatibility with prior
	  implementations.
	</t>	
      </section>

      <section anchor="link-capability"
	       title="Multipath Link Capability TLV">
	<t>
	  The Interface_ID is defined in <xref target="RFC3471" />.
	  The Interface_ID is updated in <xref target="RFC4201" /> to
	  support a form of multipath known as Link Bundling.
	</t>
	<t>
	  A new TLV, the Multipath Link Capability TLV, is defined
	  here.  The Multipath Link Capability TLV is optionally
	  included in the Interface_ID.  The format of the Multipath
	  Link Capability TLV is identical to the Multipath Node
	  Capability sub-TLV defined in <xref target="node-capability"
	  />, with one exception.  In the Multipath Link Capability
	  TLV the Type field value is IANA-TBD-2.
	</t>

	<t>
	  If a Multipath Link Capability TLV is advertised for any
	  link, then a Multipath Node Capability sub-TLV MUST be
	  advertised for the node.
	</t>

      </section>

      <section anchor="rsvp_mpa_tlv"
	       title="LSP Multipath Attributes TLV">

	<t>
	  The LSP_ATTRIBUTES object is defined in
	  <xref target="RFC5420" />.  A new LSP Multipath Attributes
	  TLV is defined for the LSP_ATTRIBUTES object.  The TLV Type
	  is IANA_TBD_3.  The format is described below.
	</t>

	<figure anchor="fig.rsvp_oa_tlv"
		title="LSP Multipath Attributes TLV">
	  <artwork>
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |              Type             |             Length            |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |     Flags     |    OA Depth   | LSP Min Depth | LSP IP Depth  |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                  Largest Microflow Capacities                 |
 |                        (variable length)                      |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	  </artwork>
	</figure>

	<t>
	  The fields in the LSP Multipath Attributes TLV are defined
	  as follows.
	</t>

	<t><list style="hanging" hangIndent="3">
	  <t hangText="Type">
	    <vspace blankLines="0" />
	    The Type field is assigned a value of IANA-TBD-3.  The Type
	    field is a two octet value.
	  </t>
	  <t hangText="Length">
	    <vspace blankLines="0" />
	    The Length field indicates the length of the sub-TLV in
	    octets, excluding the Type and Length fields.  The Length
	    field is a two octet value.
	  </t>
	  <t hangText="Flags">
	    <vspace blankLines="0" />
	    The Flags field is a one octet (8 bit) value.  The
	    following single bit fields are assigned within this value,
	    starting at the most significant bit, which is the bit
	    transmitted first.
	    <list style="hanging" hangIndent="3">
	      <t hangText="0x80 IP Multipath Allowed">
		<vspace blankLines="0" />
		Setting the IP Multipath Allowed bit indicates that it is
		safe to enable the use of a potential IP payload in the
		multipath traffic distribution.
	      </t>
	      <t hangText="0x40 May Contain IPv4">
		<vspace blankLines="0" />
		Setting the May Contain IPv4 bit indicates that IPv4
		traffic may be contained within this LSP.
	      </t>
	      <t hangText="0x20 May Contain IPv6">
		<vspace blankLines="0" />
		Setting the May Contain IPv6 bit indicates that IPv6
		traffic may be contained within this LSP.
	      </t>
	      <t hangText="0x02 Entropy Label Required">
		<vspace blankLines="0" />
		Setting the Entropy Label Used bit indicates that
		midpoint LSR MUST support ELI and EL in order to not
		violate packet ordering constraints of the LSP or of
		contained LSP.
	      </t>
	      <t hangText="0x01 Entropy Label Used">
		<vspace blankLines="0" />
		Setting the Entropy Label Used bit indicates that
		an ELI and EL is present in some or all label stack
		entries within this LSP.
	      </t>
	    </list>
	    The remaining bits in the Flags field are reserved.
	  </t>
	  <t hangText="OA Depth">
	    <vspace blankLines="0" />
	    The OA Depth field is set as follows
	    <list style="hanging" hangIndent="3">
	      <t hangText="0">
		An OA Depth value of zero indicates that no ordered
		aggregates are carried within the LSP, except those
		which are protected from out of order delivery using
		Entropy Label.
	      </t>
	      <t hangText="1">
		An OA Depth value of one indicates that the LSP is an
		ordered aggregate of traffic (the LSP requires strict
		ordering of packets) and has protected packet ordering
		using ELI and EL.
	      </t>
	      <t hangText=">1">
		An OA Depth value greater than one indicates that the
		LSP does not have strict packet ordering requirements
		but contains ordered aggregates at the label stack
		depth indicated or deeper and that the ordered
		aggregates are not protected using ELI and EL.
	      </t>
	    </list>
	  </t>
	  <t hangText="LSP Min Depth">
	    <vspace blankLines="0" />
	    The LSP Min Depth field indicates a minimal acceptable
	    number of label used in multipath traffic distribution for
	    the stated Largest Microflow Capacities field to be valid.
	    If the LSP Min Depth field is set to zero this value is
	    unknown.  See <xref target="usage.depth" />.
	  </t>
	  <t hangText="LSP IP Depth">
	    <vspace blankLines="0" />
	    The LSP IP Depth field indicates a minimal label stack
	    depth where using an IP header is necessary for the stated
	    Largest Microflow Capacities field to be valid.  If the
	    LSP IP Depth field is set to zero this value is unknown.
	    See <xref target="usage.depth" />.
	  </t>
	  <t hangText="Largest Microflow Capacities">
	    <vspace blankLines="0" />
	    The Largest Microflow Capacities field contains zero, one,
	    two, or three IEEE 32 bit floating point values.  Each
	    value is a capacity expressed in bytes per second.
	    <list style="hanging" hangIndent="3">
	      <t hangText="Largest LSE Microflow">
		<vspace blankLines="0" />
		The first value, the Largest LSE Microflow, is the
		capacity of the largest microflow if only the label
		stack entries are used in multipath traffic
		distribution.  If a Largest LSE Microflow is not
		included, the LSP bandwidth request MUST be used.
	      </t>
	      <t hangText="Largest IP Microflow">
		<vspace blankLines="0" />
		The second value, the Largest IP Microflow, if
		present, is the capacity of the largest microflow if
		the label stack entries and any potential IP source
		and destination address are used in multipath traffic
		distribution.  If the Largest IP Microflow is not
		included, the value of the Largest LSE Microflow MUST
		be used.
	      </t>
	      <t hangText="Largest L4 Microflow">
		<vspace blankLines="0" />
		The third, the Largest L4 Microflow, if present, is
		the capacity of the largest microflow if the label
		stack entries and any potential IP addresses and TCP
		or UDP port numbers are used in multipath traffic
		distribution.  If a Largest L4 Microflow is not
		included, the value of the Largest IP Microflow MUST
		be used.
	      </t>
	    </list>
	  </t>
	</list></t>

      </section>

    </section>

    <section anchor="usage"
	     title="Protocol Mechanisms">

      <section anchor="usage.adv"
	       title="OSPF-TE and ISIS-TE Advertisement">

	<t>
	  Every compliant node MUST advertise exactly one Multipath
	  Node Capability sub-TLV and MAY advertise zero of more
	  Multipath Link Capability sub-TLV as needed.
	</t>
	<t>
	  Procedures for accommodating legacy forwarding capabilities
	  and non-compliant nodes are discussed in
	  <xref target="compat" />.
	</t>

	<section anchor="usage.node"
		 title="Node Capability Advertisement">

	  <t>
	    Every LSR which is adjacent to one or more multipath link
	    MUST advertise a Multipath Node Capability sub-TLV (see
	    <xref target="node-capability" />).  The capabilities
	    advertised for the node SHOULD reflect the capabilities of
	    the majority of multipath links adjacent to the node.
	  </t>
	  <t>
	    Every LSR which is not adjacent to any multipath links
	    MUST advertise a Multipath Node Capability sub-TLV with
	    both the Ordered Aggregate Enabled bit in Flags set and
	    all other Flags bits clear.  Both Max Depth and IP Depth
	    can be set to zero.  This advertisement identifies the LSR
	    as one which is conformant but has no multipath links,
	    allowing it to be distinguished from a non-conformant LSR
	    with LAG or other multipath which may have to be avoided
	    when determining a path for some LSP.
	  </t>

	</section>

	<section anchor="usage.link"
		 title="Link Capability Advertisement">

	  <t>
	    For all of the links whose capability does not exactly
	    match the Multipath Node Capability sub-TLV advertised by
	    that same LSR, the LSR MUST advertise a Multipath Link
	    Capability sub-TLV
	    (see <xref target="link-capability" />).
	  </t>
	  <t>
	    For all of the links whose capability does exactly match
	    the Multipath Node Capability sub-TLV advertised by that
	    same LSR, the LSR SHOULD NOT advertise a Multipath Link
	    Capability sub-TLV (see <xref target="link-capability"
	    />).  In this case the Multipath Link Capability TLV
	    is redundant, but harmless.
	  </t>

	</section>

	<section anchor="usage.arch-limits"
		 title="Setting Max Depth and IP Depth">

	  <t>
	    The Max Depth and IP Depth field are intended to capture
	    architectural limits.  Most forwarding hardware will only
	    use a limited number of label entries in the multipath
	    traffic distribution.  This limit is reflected in the Max
	    Depth field.  Most forwarding hardware will limit the
	    number of label entries that it will look past before
	    looking for an IP header to be used in the multipath
	    traffic distribution.  This limit is reflected in the IP
	    Depth field.
	  </t>

	</section>

	<section anchor="usage.bundle"
		 title="Advertising Multipath as Link Bundling">

	  <t>
	    All multipath links and FA for PSC LSP which traverse
	    multipath links MAY be advertised as Link Bundles as
	    defined in <xref target="RFC4201" />.  The settings of the
	    Ordered Aggregate Enabled and Multipath Enabled fields
	    indicate key capabilities of the multipath.  Advertising
	    the multipath as a link bundle can provide additional
	    information, such as the capability to place LSP on
	    individual components.
	  </t>
	  <t>
	    If the Multipath Enabled bit is set in the Multipath Link
	    Capability TLV Flags, then the Maximum LSP Bandwidth in
	    the Interface Identification TLV can be set in one of two
	    ways.
	    <list style="numbers">
	      <t>
		If the desired behavior for legacy LSR acting as
		ingress is to limit LSP to the capacity of a single
		component link, then Maximum LSP Bandwidth SHOULD be
		set as specified in <xref target="RFC4201" />.
	      </t>
	      <t>
		If the desired behavior for legacy LSR acting as
		ingress is to allow LSP to exceed the capacity of a
		single component link, then Maximum LSP Bandwidth MAY
		be set to a higher value, up to the value of Maximum
		Reservable Bandwidth.  This would normally be done if
		the legacy LSR were known to be carrying traffic which
		is easily load split, such as typical Internet
		traffic.
	      </t>
	    </list>
	  </t>
	  <t>
	    LSR acting as ingress SHOULD ignore the Maximum LSP
	    Bandwidth and MAY set up LSP with capacity up to the
	    Maximum Reservable Bandwidth as long as microflows within
	    the LSP will not exceed the Largest Supportable Microflow
	    capacity.
	  </t>

	</section>

	<section anchor="usage.fa"
		 title="Hierarchical LSP Link Advertisement">

	  <t>
	    A PSC LSP, as defined in <xref target="RFC4206" /> and
	    updated in <xref target="RFC6107" />, may carry other LSP.
	    When signaling a PSC LSP expected characteristics of the
	    contained traffic must be estimated.  The FA advertised
	    for the PSC LSP MUST reflect the broadest set of
	    requirements the PSC LSP can carry.  If the setup of an
	    additional reservation would exceeded current capacity, a
	    PSC LSP may be resignaled using make-before-break
	    semantics (<xref target="RFC3209" />.
	  </t>
	  <t>
	    For example, if it is expected that a PSC LSP will carry
	    MPLS-TP LSP or other LSP with strict packet reordering
	    requirements at some label depth, the minimum label stack
	    depth at which an LSP with strict packet reordering
	    requirements can be carried must be signaled in the OA
	    Depth field of the LSP Multipath Attributes TLV (see
	    <xref target="rsvp_mpa_tlv" />.
	  </t>
	  <t>
	    When the Forwarding Adjacency (FA) is advertised, the
	    advertised Max Depth and IP Depth MUST be one less that
	    the minimum of the Max Depth and IP Depth of any link that
	    the PSC LSP traverses.  The Max Depth and IP Depth are
	    considered independently of each other.  The reduction by
	    one takes into account the PSC label.  The Flags
	    advertised for the FA MUST reflect the capabilities of the
	    links along the path.
	  </t>

	</section>

	<section anchor="usage.typical"
		 title="Advertisement of Legacy Multipath">

	  <t>
	    An Ethernet LAG with no support for Entropy Label MUST
	    have the Ordered Aggregate Enabled bit cleared and the
	    Multipath Enabled bit set in the Multipath Link Capability
	    TLV Flags.  This indicates that a MPLS-TP compliant server
	    layer cannot be supported and that LSP larger than the
	    component links (LAG members) can be supported.
	  </t>
	  <t>
	    A Link Bundle that does not support the all-ones component
	    defined in <xref target="RFC4201" /> MUST have the Ordered
	    Aggregate Enabled bit set and the Multipath Enabled bit
	    cleared in the Multipath Link Capability TLV Flags.  This
	    indicates that a MPLS-TP compliant server layer can be
	    supported and that LSP larger than the component links
	    cannot be supported.
	  </t>
	  <t>
	    A link bundle that can support either the pinning of a LSP
	    to a single component link or the spreading of traffic
	    across multiple component links MUST have the Ordered
	    Aggregate Enabled bit set and the Multipath Enabled bit
	    set in the Multipath Link Capability TLV Flags.  This
	    indicates that a MPLS-TP compliant server layer can be
	    supported and that LSP larger than the component links can
	    also be supported.
	  </t>
	  <t>
	    Any form of multipath that supports Entropy Label MUST
	    have the Ordered Aggregate Enabled bit set and the
	    Multipath Enabled bit set and the Entropy Label Multipath
	    bit set in the Multipath Link Capability TLV Flags.  Any
	    form of multipath that does not supports Entropy Label
	    MUST have the Entropy Label Multipath bit cleared in the
	    Multipath Link Capability TLV Flags.
	  </t>
	  <t>
	    The remaining bits in the Multipath Link Capability TLV
	    Flags MUST be set according to the capability and
	    configuration of the multipath or LSP.
	  </t>

	</section>

      </section>

      <section anchor="usage.rsvp"
	       title="RSVP-TE LSP Attributes">

	<t>
	  All LSR SHOULD advertise a LSP Multipath Attributes TLV with
	  the RSVP-TE signaling for each LSP for which it is serving
	  as ingress.  If any legacy LSR remain in the network, it is
	  easier to assign an acceptable default treatment for LSP
	  signaled by those legacy LSR if the conforming LSR always
	  send a LSP Multipath Attributes TLV.
	</t>
	<t>
	  There are two general cases, an LSP requires strict
	  ordering of packets, or it doesn't.  In the latter case the
	  LSP may contain other LSP that require strict ordering and
	  those must be protected from reordering using an Entropy
	  Label as described in
	  <xref target="I-D.villamizar-mpls-multipath-use" />.
	  These two cases are briefly described below.
	  <list style="hanging" hangIndent="3">
	    <t hangText="Ordered Aggregates">
	      <vspace blankLines="0" />
	      LSP with strict packet order requirements MUST set the
	      OA Depth field to one to indicate that the LSP MUST be
	      treated as ordered aggregate.
	      See <xref target="usage.rsvp-oa" />.
	    </t>
	    <t hangText="LSP without Packet Ordering">
	      <vspace blankLines="0" />
	      LSP which do not have strict packet order requirements
	      MUST only carry LSP whose requirements are reflected in
	      the containing LSP Multipath Attributes TLV.
	      See <xref target="usage.rsvp-mp" />.
	    </t>
	  </list>
	</t>
	<t>
	  If an attempt is made to signal a contained LSP whose
	  Ordered Aggregate Attributes TLV and LSP Multipath
	  Attributes TLV cannot be supported by the containing (PSC)
	  LSP, one of the two following actions MUST be taken.
	  <list style="numbers">
	    <t>
	      The containing (PSC) LSP MAY be resignaled with
	      appropriate TLVs to carry the new contained LSP using
	      make-before-break semantics, then continue to forward
	      the contained LSP PATH message if the containing (PSC)
	      LSP is successfully updated.
	    </t>
	    <t>
	      The LSR MAY reject the contained LSP signaling by
	      sending a PathErr message.
	    </t>
	  </list>
	</t>

	<section anchor="usage.rsvp-flags"
		 title="LSP Contained Ordered Aggregates Flags">

	  <t>
	    The Flags field in the LSP Multipath Attributes TLV MUST
	    be set as follows.
	  </t>
	  <t><list style="numbers">
	    <t>
	      If the LSP may directly contain IPv4 traffic, then the
	      May Contain IPv4 bit in the Flags field MUST be set.
	    </t>
	    <t>
	      If the LSP may directly contain IPv6 traffic, then the
	      May Contain IPv6 bit in the Flags field MUST be set.
	    </t>
	    <t>
	      If the LSP contains an LSP which has the May Contain
	      IPv4 bit in the Flags field then the May Contain IPv4
	      bit in the Flags field MUST be set in the containing LSP.
	    </t>
	    <t>
	      If the LSP contains an LSP which has the May Contain
	      IPv6 bit in the Flags field then the May Contain IPv6
	      bit in the Flags field MUST be set in the containing LSP.
	    </t>
	    <t>
	      If the LSP may contain pseudowires that do not use a
	      pseudowire control word <xref target="RFC4385" />, and
	      may contain IPv4 or IPv6 traffic, then the IP Multipath
	      Allowed bit in the Flags field MUST be cleared.
	    </t>
	    <t>
	      If the LSP is known to contain no pseudowires that do
	      not use a pseudowire control word, then the IP Multipath
	      Allowed bit in the Flags field SHOULD be set unless
	      disallowed due to a contained LSP.
	    </t>
	    <t>
	      If an Entropy Label is added below the LSP label, then
	      the Entropy Label Used bit MUST be set.
	    </t>
	    <t>
	      If the LSP contains any LSP with the IP Multipath
	      Allowed bit in the Flags field clear, then the IP
	      Multipath Allowed bit in the Flags field MUST be clear.
	    </t>
	  </list></t>
	  <t>
	    If the LSP does not contain other LSP, it may contain IP
	    traffic and/or pseudowire that terminate on that LSR.  If
	    the LSP does not contain other LSP.  The LER will know
	    whether the LSP is used in an IP LER capacity.  The LER
	    will also know whether it terminates any pseudowire for a
	    given LSP.  The LER will also know if it is using Entropy
	    Label for a given LSP and if it requires strict packet
	    ordering for a given LSP.  Therefore, when a LSP does not
	    contain other LSP, then it is possible to accurately set
	    the Flags field in the LSP Multipath Attributes TLV, as
	    well the OA Depth, and LSP IP Depth fields.
	  </t>
	  <t>
	    If an LSP contains other LSP, and all of the contained
	    include a LSP Multipath Attributes TLV, then it is still
	    possible to accurately set the Flags field in the LSP
	    Multipath Attributes TLV, as well the OA Depth, and LSP IP
	    Depth fields.  It is only when LSP contains other LSP that
	    do not have a LSP Multipath Attributes TLV where default
	    behavior has to be configured based on assumptions about
	    LSP originated by the legacy LSR where there is a
	    potential for those configured assumptions to be
	    inaccurate.
	  </t>
	  <t>
	    See <xref target="compat" /> for guidelines for handling
	    LSP which contain LSP that do not have a LSP Multipath
	    Attributes TLV.  The most conservative approach in this
	    case is to clear the IP Multipath Allowed bit and set the
	    May Contain IPv4 bit and the May Contain IPv6 bit, however
	    this may not always be necessary.
	  </t>

	</section>

	<section anchor="usage.rsvp-oa"
		 title="LSP Attributes for Ordered Aggregates">

	  <t>
	    An LSP with strict packet order requirements MUST always
	    include a LSP Multipath Attributes TLV.
	  </t>
	  <t>
	    Most of the Flags in the LSP Multipath Attributes TLV can
	    be set as described in <xref target="usage.rsvp-flags" />.
	    There are two cases which affect the setting of the
	    remaining Flags bits.
	    <list style="hanging" hangIndent="3">
	      <t hangText="Entropy Label not used">
		<vspace blankLines="0" />
		If an Entropy Label is not used, then the Entropy
		Label Used bit, the Entropy Label Required bit, and
		the IP Multipath Allowed bit MUST be cleared.
	      </t>
	      <t hangText="Entropy Label is used">
		If an Entropy Label is used, then the Entropy Label
		Used bit, and the Entropy Label Required bit, and the
		IP Multipath Allowed bit MUST be set.
	      </t>
	    </list>
	  </t>
	  <t>
	    The OA Depth field MUST be set to one.  The Min Depth MUST
	    be set to one.  The LSP IP Depth SHOULD be set to zero.
	    The Largest Microflow Capacities field SHOULD be empty.
	    The entire LSP is one microflow.  The Largest Microflow
	    Capacities field MAY contain one entry if there is some
	    reason to do so, such as an LSP which may peak at capacity
	    higher than the bandwidth reserved for the LSP.  The
	    Largest Microflow Capacities for an LSP SHOULD be
	    configurable independently of the LSP bandwidth
	    reservation.
	  </t>

	</section>

	<section anchor="usage.rsvp-mp"
		 title="Attributes for LSP without Packet Ordering">

	  <t>
	    If an LSP does not have strict packet order constraints,
	    then the LSR_ATTRIBUTE object SHOULD always include a
	    LSP Multipath Attributes TLV.
	  </t>
	  <t>
	    Most of the Flags in the LSP Multipath Attributes TLV can
	    be set as described in <xref target="usage.rsvp-flags" />.
	    There are two cases which affect the setting of the
	    remaining Flags bits, the OA Depth field, the LSP Min
	    Depth, and the LSP IP Depth field.
	    <list style="hanging" hangIndent="3">
	      <t hangText="Entropy Label not used">
		<vspace blankLines="0" />
		<list style="symbols">
		  <t>
		    The OA Depth MUST be either set to zero or set to
		    a configured value that is greater than one, or
		    set based on the contained LSP.
		  </t>
		  <t>
		    If the OA Depth is set to a configured value, then
		    any setup attempt for a contained LSP with a depth
		    greater than or equal to that value SHOULD be
		    rejected and a PathErr message sent.  Otherwise,
		    if a setup attempt for a contained LSP with a
		    depth greater that the current value included in
		    the containing LSP OA Depth field, then the
		    containing LSP MUST be rerouted with a OA Depth
		    field value greater than any of the contained OA
		    Depth field values.
		  </t>
		  <t>
		    The Entropy Label Used bit MUST be set if any
		    contained LSP has the Entropy Label Used bit set.
		  </t>
		  <t>
		    The Entropy Label Required bit MUST be set if any
		    contained LSP has the Entropy Label Required bit
		    set.
		  </t>
		</list>
	      </t>
	      <t hangText="Entropy Label is used">
		<list style="symbols">
		  <t>
		    The OA Depth MUST be set to zero.  
		  </t>
		  <t>
		    The Entropy Label Used bit MUST be set.
		  </t>
		  <t>
		    The Entropy Label Required bit MUST be set if any
		    contained LSP has the Entropy Label Required bit
		    set.
		  </t>
		  <t>
		    The Entropy Label Required bit MUST be set if any
		    contained LSP has the OA Depth field set to a
		    non-zero value.
		  </t>
		  <t>
		    The Entropy Label Required bit SHOULD be clear if
		    there are no contained LSP has the OA Depth field
		    set to a non-zero value and no contained LSP with
		    the Entropy Label Required bit set.  In this case
		    the Entropy Label Required bit MAY be set by
		    configuration, generally in anticipation of
		    needing it in the future to carry other LSP.
		  </t>
		  <t>
		    LSP Min Depth field MUST be set to three if the
		    Entropy Label Required bit is set.
		  </t>
		  <t>
		    If the Entropy Label Required bit is not set, then
		    the LSP Min Depth field and LSP IP Depth field
		    SHOULD be set to three if there are no contained
		    LSP.  The LSP Min Depth field and LSP IP Depth MAY
		    be configured to a minimum value greater than
		    three, generally in anticipation of needing it in
		    the future to carry other LSP.
		  </t>
		  <t>
		    If the Entropy Label Required bit is not set, and
		    there are contained LSP, then the LSP Min Depth
		    field MUST be set to a value greater than three.
		  </t>
		  <t>
		    If the Entropy Label Required bit is not set, the
		    LSP Min Depth field MUST be set to a value that is
		    at least the sum of three plus the LSP Min Depth
		    field in any contained LSP.
		  </t>
		  <t>
		    If the Entropy Label Required bit is not set, and
		    either the May Contain IPv4 bit or the May Contain
		    IPv6 bit is set, then the LSP IP Depth field MUST
		    be set to a value of one or greater.
		  </t>
		  <t>
		    If the Entropy Label Required bit is not set, and
		    any contained LSP has the May Contain IPv4 bit or
		    the May Contain IPv6 bit is set, then the LSP IP
		    Depth field MUST be set to a value of two or
		    greater.
		  </t>
		  <t>
		    If the Entropy Label Required bit is not set, and
		    any contained LSP has the LSP IP Depth field set
		    to a value greater than one, then the LSP IP Depth
		    field MUST be set to a value greater than the
		    highest LSP IP Depth value set in any contained LSP.
		  </t>
		</list>
	      </t>
	    </list>
	  </t>
	  <t>
	    The values of the LSP Min Depth field and the LSP IP Depth
	    field MAY be constrained to upper limits by configuration.
	    If an attempt to setup a contained LSP would result in
	    exceeding one of these limits, then the LSR SHOULD reject
	    the signaling attempt and send a PathErr message.
	  </t>
	  <t>
	    If Entropy Label is not used on the signaled LSP and there
	    are no contained LSP, then the Largest LSE Microflow in
	    the Largest Microflow Capacities field MUST be set to the
	    requested bandwidth of the LSP.  The optional Largest IP
	    Microflow and Largest L4 Microflow SHOULD be included and
	    MAY be set to configured minimum values.
	  </t>
	  <t>
	    If Entropy Label is not used on the signaled LSP an LSP
	    that does not have strict packet order constraints
	    contains other LSP, then the LSP Multipath Attributes TLV
	    advertised by the set of contained LSP MUST be used to set
	    the LSP Multipath Attributes TLV Largest Microflow
	    Capacities values for LSP Multipath Attributes TLV.  The
	    value of Largest LSE Microflow, Largest IP Microflow, and
	    Largest L4 Microflow in the LSP Multipath Attributes TLV
	    of the containing LSP cannot be less than the maximum
	    effective value of the same parameter for any contained
	    LSP Multipath Attributes TLV.
	  </t>
	  <t>
	    If Entropy Label is used on the signaled LSP then the
	    Largest LSE Microflow field is set according to the
	    largest microflow that can result from computing the
	    Entropy Label.  This value is the greatest of a set of
	    sources of entropy.  A configured value MAY be used for
	    IP, or it MAY be assumed that the largest interface
	    carrying IP could carry a single microflow.  For contained
	    LSP which have the Entropy Label Used bit clear, the value
	    in the Largest IP Microflow can be used if the IP
	    Multipath Allowed bit is set for that LSP and the LSR can
	    make use of the IP information in the label stack.  For
	    contained LSP which have the Entropy Label Used bit set,
	    the Largest LSE Microflow value already reflects any prior
	    hashing of IP fields.
	  </t>
	  <t>
	    If the Entropy Label Required bit is set and the LSP being
	    set up is using Entropy Label, then the Largest IP
	    Microflow and Largest L4 Microflow SHOULD be omitted.  All
	    midpoint LSR SHOULD not look for entropy beyond the
	    Entropy Label.
	  </t>
	  <t>
	    If the LSP being set up is not using Entropy Label, then
	    the Largest IP Microflow and Largest L4 Microflow SHOULD
	    be included unless the Entropy Label Used bit is set for
	    every contained LSP.  The Largest IP Microflow and Largest
	    L4 Microflow SHOULD be omitted if hashing on the IP
	    addresses or IP addresses and ports would yield no greater
	    entropy than hashing on the label stack alone.
	  </t>

	</section>

      </section>

      <section anchor="usage.cspf"
	       title="Path Computation Constraints">

	<t>
	  The RSVP-TE extensions provides a set of requirements to be
	  met by the links which the LSP is to traverse.  This set of
	  requirements also serves as the basis for path computation
	  constraints and for admission control constraints.
	</t>

	<section anchor="usage.mp-oa"
		 title="Link Multipath Capabilities and Path Computation">

	  <t>
	    Three cases are considered.  An LSP may have strict
	    ordering constraints.  An MPLS-TP LSP is an example of an
	    LSP with strict ordering constraints.  This first type of
	    LSP is covered in <xref target="usage.oa" />.  An LSP may
	    have no ordering constraints at all other than the
	    constraint that microflows cannot be reordered.  This
	    second case is covered in <xref target="usage.mp" />.  The
	    remaining case is where an LSP has no ordering constraints
	    but carries traffic for other LSP which do have ordering
	    constraints.  This third case is covered in <xref
	    target="usage.tpmp" />.
	  </t>

	  <section anchor="usage.oa"
		   title="Path Computation with Ordering Constraints">

	    <t>
	      For an MPLS-TP LSP or other LSP with a strict packet
	      ordering constraint, any link or FA for which the
	      Ordered Aggregate Enabled bit and Entropy Label
	      Multipath are both clear MUST be excluded from the path
	      computation.  If the Default to Multipath bit is set on
	      a link, then setting the OA Depth field to one will
	      override that default.
	    </t>
	    <t>
	      Link with the Ordered Aggregate Enabled bit clear and
	      the Entropy Label Multipath bit set MAY be included in
	      the path computation if the LSR is capable of
	      encapsulating an LSP with a strict packet ordering
	      constraint with a fixed Entropy Label.  If the LSR is
	      not capable of adding an ELI and EL, then these links
	      MUST be excluded from the path computation.
	    </t>

	  </section>

	  <section anchor="usage.mp"
		   title="Path Computation with No Ordering Constraint">

	    <t>
	      For an MPLS LSP which has no constraint on packet
	      ordering except that microflows must remain in order and
	      does not contain other LSP with ordering constraints,
	      any link for which the Multipath Enabled bit is set can
	      be used.  If s link is advertised as a Link Bundle and
	      the Multipath Enabled bit is set for the link, the
	      available bandwidth SHOULD be taken from the "Unreserved
	      Bandwidth" rather than the "Maximum LSP Bandwidth" (see
	      <xref target="RFC4201" />).
	    </t>
	    <t>
	      For most LSP, the bandwidth requirement of the largest
	      microflow is not known but an upper bound is known.  For
	      example if the LSP aggregates pseudowires or other LSP
	      of no more than some maximum capacity or LSP which have
	      signaled a microflow upper bound, then an upper bound on
	      the largest microflow is known.  If this upper bound
	      exceeds the "Maximum LSP Bandwidth" of a given link,
	      then that link MUST be excluded from the path
	      computation.
	    </t>

	  </section>

	  <section anchor="usage.tpmp"
		   title="Path Computation for MPLS containing MPLS-TP">

	    <t>
	      To carry LSP which have strict packet ordering
	      requirements and do not have the Entropy Label Used flag
	      set as a client within a server LSP that do not have
	      strict packet ordering requirements, Entropy Label must
	      be added at the server layer LSP to traverse any link or
	      FA that has the Multipath Enabled bit set.  For these
	      LSP links which have the Multipath Enabled bit set and
	      the Entropy Label Multipath bit clear MUST be excluded
	      from the path computation.
	    </t>
	    <t>
	      If the LSR is not capable of adding an Entropy Label,
	      then to carry any client LSP with the Entropy Label Used
	      clear and the OA Depth set to a non-zero value, the
	      server LSP SHOULD exclude any link or FA that has the
	      Multipath Enabled bit set.  For these LSP, any link or
	      FA that has the Multipath Enabled bit set and cannot
	      carry a microflow as large as the entire LSP MUST be
	      excluded from the path computation.  These LSP MAY be
	      signaled as having strict packet ordering requirements
	      if they can be carried as a single microflow, but this
	      practice is NOT RECOMMENDED.
	    </t>

	  </section>

	</section>

	<section anchor="usage.ip-in-cspf"
		 title="Link IP Capabilities and Path Computation">

	  <t>
	    An MPLS-TP LSP cannot be reordered.  There may be other
	    types of LSP with strict packet ordering requirements.  If
	    LSP with strict packet ordering requirements carry IP,
	    using IP headers in the multipath load distribution would
	    violate the packet ordering requirements.
	  </t>
	  <t>
	    Some LSP cannot be reordered but do not carry IP, and do
	    not carry payloads which could be mistaken as IP.  For
	    example, any LSP carrying only pseudowire traffic, where
	    all pseudowires are using a control word carries no
	    payloads which could be mistaken as IP.  These type of LSP
	    can be carried within MPLS LSP that allow use of IP header
	    information in multipath load distribution.
	  </t>
	  <t>
	    This section focuses on Cases in which links or FA must be
	    excluded from path computation based on the setings of the
	    IP related Flags bits in the Multipath Link Capability
	    TLV.
	  </t>

	  <section anchor="usage.ip-in-mpls"
		   title="LSP without Packet Ordering Requirements">

	    <t>
	      Many LSP carry only IP or predominantly IP, use no
	      hierarchy or have little diversity in the MPLS label
	      stack, and carry far more traffic than can be carried
	      over a single component link in a multipath.  Many LSP
	      due to their high capacity, must traverse only multipath
	      which will use IP header information in the multipath
	      traffic distribution.
	    </t>
	    <t>
	      For these LSP, links must be excluded from the path
	      computation which do not have the IPv4 Enabled Multipath
	      and IPv6 Enabled Multipath bit set (if carrying both
	      IPv4 and IPv6) and do not have either the Default to
	      IP/MPLS Multipath bit set or the IP Optional Multipath
	      bit set.
	    </t>
	    <t>
	      Hierarchical PSC LSP which require the use IP header
	      information in the multipath traffic distribution MUST
	      NOT set the Ordered Aggregate Enabled bit, MUST set the
	      Default to IP/MPLS Multipath bit, and MUST NOT set the
	      IP Optional Multipath bit in the FA advertisement.  The
	      IP Optional Multipath bit MUST be clear because the LSP
	      cannot change the behavior of midpoint LSR, except
	      perhaps in the case of single hop LSP.
	    </t>

	  </section>

	  <section anchor="usage.ip-in-tp"
		   title="LSP with Ordering Requirements">

	    <t>
	      The only time that links or FA with both the Ordered
	      Aggregate Enabled bit and the Entropy Label Multipath
	      bit clear can be used is a special case for MPLS-TP LSP
	      that carry only IP traffic.  This case does not apply if
	      the MPLS_TP LSP is carrying other LSP or if it is
	      carrying pseudowires.
	    </t>
	    <t>
	      Where MPLS-TP LSP are carrying only IP, any link or FA
	      with both the Ordered Aggregate Enabled bit and the
	      Entropy Label Multipath bit clear for which the use of
	      IP header information is not disabled or cannot be
	      disabled on a per LSP basis, that link MUST be excluded
	      from the path computation.
	    </t>
	    <t>
	      Where MPLS-TP LSP are carrying only IP, links MAY be
	      included in the path computation have the IPv4 Enabled
	      Multipath and IPv6 Enabled Multipath bits clear, or have
	      the Default to IP/MPLS Multipath bit clear, or have the
	      IP Optional Multipath bit set.  Links with the IP
	      Optional Multipath set, MUST disable use of IP in the
	      load balance for LSP with the IP Multipath Allowed bit
	      clear.
	    </t>
	    <t>
	      An MPLS-TP LSP are carrying only IP MUST have OA Depth
	      set to one and LSP Min Depth set to one and the IP
	      Multipath Allowed bit cleared.  Call admission SHOULD
	      NOT reject an LSP on the basis of OA Depth being set to
	      one if use of IP headers is always disabled, or can be
	      disabled for the specific LSP.  If an MPLS-TP is set up
	      this way end then does start to carry other LSP or carry
	      pseudowires, then reordering within the MPLS-TP LSP will
	      occur.
	    </t>

	  </section>

	</section>

	<section anchor="usage.depth"
		 title="Link Depth Limitations and Path Computation">

	  <t>
	    For any LSP which does not have strict packet ordering
	    constraints, LSP configuration SHOULD include the
	    following parameters.
	  </t>
	  <t><list style="hanging" hangIndent="3">
	    <t hangText="LSP Min Depth">
	      <vspace blankLines="0" />
	      a minimal acceptable number of label used in multipath
	      traffic distribution,
	    </t>
	    <t hangText="LSP IP Depth">
	      <vspace blankLines="0" />
	      a minimal label stack depth where the IP header can be
	      used in multipath traffic distribution
	    </t>
	  </list></t>
	  <t>
	    For example, if a PSC LSP will carry LSP which in turn
	    carry very high capacity pseudowires using the pseudowire
	    flow label (see <xref target="RFC6391" />),
	    the flow label is four labels deep.  In this case, LSP Min
	    Depth should be four or higher.
	  </t>
	  <t>
	    For example, if the same PSC LSP will carry LSP which
	    carry IP traffic with no additional labels, then the IP
	    header is two labels deep.  In this case, LSP IP Depth
	    should be two or higher.
	  </t>
	  <t>
	    For an LSP with non-zero LSP Min Depth, all links with
	    Max Depth set to a value below LSP Min Depth MUST be
	    excluded from the LSP Path Computation.
	  </t>
	  <t>
	    For an LSP with non-zero LSP IP Depth, all links with IP
	    Depth set to a value below LSP IP Depth MUST be excluded
	    from the LSP Path Computation.
	  </t>
	  <t>
	    If all LSP carried an accurate LSP Min Depth and LSP IP
	    Depth then neither of these parameters would ever have to
	    be configured.  In a network with legacy LSR, it may be
	    necessary to configure these parameters for some LSP.  A
	    per-LSP minimum value of each parameter SHOULD be
	    configurable.
	  </t>

	</section>

      </section>

    </section>

    <section anchor="compat"
	     title="Backwards Compatibility">

      <t>
	Networks today use three forms of multipath.
      </t>
      <t><list style="numbers">
	<t>
	  IP ECMP, including IP ECMP at LER using more than one LSP.
	</t>
	<t>
	  Ethernet Link Aggregation <xref target="IEEE-802.1AX" />.
	</t>
	<t>
	  MPLS Link Bundling <xref target="RFC4201" />.
	</t>
      </list></t>

      <section anchor="compat.legacy"
	       title="Legacy Multipath Behavior">

	<t>
	  IP ECMP and Ethernet Link Aggregation always distribute
	  traffic over the entire multipath either using information
	  in the MPLS label stack, or using information in a potential
	  IP header, or using both types of information.
	</t>
	<t>
	  One of two behaviors is assumed for link bundles.  Either
	  the link bundles place each LSP in its entirety on a single
	  link bundle component link for all LSP, or link bundles
	  distribute traffic over the entire link bundle using the
	  same techniques used for ECMP and Ethernet Link Aggregation.
	  This second behavior is known as the "all ones" component
	  link (see <xref target="RFC4201" />).
	</t>

      </section>

      <section anchor="compat.today"
	       title="Networks without Multipath Extensions">

	<t>
	  Networks exist that are comprised entirely of LSR which do
	  not support these multipath extensions.  In these networks
	  there is no way of telling how multipath links will behave.
	  Since an Ethernet Link Aggregation Goup (LAG) is advertised
	  as an ordinary link, there is no way to tell that it is a
	  LAG and not an ordinary link.
	</t>

	<section anchor="compat.all-mp"
		 title="Netowrks with MP Capability on all Multipath">

	  <t>
	    Most large core network today rely heavily on the use of
	    multipath.  Ethernet Link Aggregation is heavily used and
	    LSR are configured to use the "all ones" component link
	    for all LSP.  The "all ones" component link is the default
	    for many Link Bundling implementations used in core
	    networks.
	  </t>
	  <t>
	    This is equivalent to the following setting in the
	    Multipath Node Capabilities sub-TLV or Multipath Link
	    Capabilities sub-TLV.
	  </t>
	  <t><list style="numbers" hangIndent="3">
	    <t>
	      Clear the Ordered Aggregate Enabled bit and the IP
	      Optional Multipath bit.
	    </t>
	    <t>
	      Set the Multipath Enabled bit, set the Default to
	      Multipath bit, and clear the Entropy Label Multipath bit.
	    </t>
	    <t>
	      If the label stack is used in the multipath traffic
	      distribution, set Max Depth to the number of label stack
	      entries supported, otherwise set it to zero.
	    </t>
	    <t>
	      Since Entropy Label support is not yet widespread, most
	      LSR would behave as if Entropy Label Multipath were clear.
	    </t>
	    <t>
	      If an IP packet under the label stack can be used in the
	      multipath traffic distribution (very common, almost
	      universal in core LSR), set IPv4 Enabled Multipath, set
	      IPv6 Enabled Multipath, set Default to IP/MPLS
	      Multipath, and set IP Depth to the maximum number of
	      label stack entries which can be skipped over before
	      finding the IP stack.  Otherwise clear IPv4 Enabled
	      Multipath, clear IPv6 Enabled Multipath and clear
	      Default to IP/MPLS Multipath.
	    </t>
	    <t>
	      On core networks where UDP and TCP ports are rarely used
	      in multipath, clear all UDP and TCP related bits.  On
	      networks where multipath is configured to use TCP and
	      UDP port numbers, these bits would be set.
	    </t>
	  </list></t>
	  <t>
	    These networks can support very large LSP but cannot
	    support LSP which require strict packet ordering with
	    other labels below such an LSP, such as pseudowire labels.
	    They may also misroute OAM packet which use GAL (see <xref
	    target="RFC5586" />) if they use the GAL label in
	    determining the load balance.  Generally the Link Bundle
	    advertisements indicate a "Maximum LSP Bandwidth" that is
	    equal to the "Unreserved Bandwidth" if Link Bundling is
	    used with the all-ones component link.
	  </t>
	  <t>
	    Good or bad, if the behavior is consistent, defaults
	    can be configured in other LSR, such that an accurate
	    guess can be made when no Multipath Link Capability TLV is
	    available for a given link.
	  </t>
	  <t>
	    For example, in many networks, any link of 10 Gb/s or less
	    can be assumed to be a plain link (no multipath) while any
	    link with a capacity greater than 10 Gb/s can be assumed
	    to be a multipath These assumptions would hold if no 40
	    Gb/s or 100 Gb/s links are used.
	  </t>

	</section>

	<section anchor="compat.all-oa"
		 title="Netowrks with OA Capability on all Multipath">

	  <t>
	    Some networks, particularly edge networks which tend to be
	    lower capacity, do not use Link Aggregation, and if they
	    use Link Bundling at all, configure each LSR to place each
	    LSP in its entirety on a single link bundle component link
	    for all LSP.  Some edge equipment only supports this link
	    bundle behavior.
	  </t>
	  <t>
	    This is equivalent to the following setting in the
	    Multipath Node Capabilities sub-TLV or Multipath Link
	    Capabilities sub-TLV.
	  </t>
	  <t><list style="hanging" hangIndent="3">
	    <t>
	      Set the Ordered Aggregate Enabled bit,
	    </t>
	    <t>
	      Clear the Multipath Enabled bit.
	    </t>
	    <t>
	      All remaining bits in the Flags field should be clear.
	    </t>
	    <t>
	      The Max Depth and IP Depth should be set to zero.
	    </t>
	  </list></t>
	  <t>
	    These networks can support LSP which require strict packet
	    ordering, but cannot support very large LSP.
	  </t>
	  <t>
	    Like the behavior described in
	    <xref target="compat.all-mp" />, whether this behavior is
	    good or bad, defaults can be configured which accurately
	    guess the capabilities of links for which no Multipath
	    Link Capability TLV is available.
	  </t>

	</section>

        <section anchor="compat.all-mixed-up"
		 title="Legacy Netowrks with Mixed MP and OA Links">

	  <t>
	    Some network may support Ethernet Link Aggregation and all
	    or a subset of LSR which place each LSP in its entirety on
	    a single link bundle component link for all LSP.
	  </t>
	  <t>
	    If the "Maximum LSP Bandwidth" is set as described in
	    <xref target="compat.all-mp" />, then very large LSP can
	    be supported over link bundles.  Very large LSP cannot be
	    supported on LSR which place each LSP in its entirety on a
	    single link bundle component link for all LSP, but these
	    are clearly indicated in signaling,
	  </t>
	  <t>
	    In these mixed networks it may not be possible to reliably
	    support LSP which require strict packet ordering.  It is
	    not possible to know where Ethernet Link Aggregation is
	    used and it is not possible to accurately determine Link
	    Bundling behavior on link bundles where "Maximum LSP
	    Bandwidth" is equal to "Unreserved Bandwidth".
	  </t>
	  <t>
	    Upgrading LSR to support Entropy Label on both LAG and
	    Link Bundles would improve the ability to carry LSP which
	    require strict packet ordering.  To gain any benefit the
	    LSP ingress would have to add ELI and EL.
	  </t>
	  <t>
	    If not all LSR are upgraded, then the MPLS TE multipath
	    extensions identify those LSR and multipath that have been
	    upgraded.
	  </t>

	</section>

      </section>

      <section anchor="compat.transition"
	       title="Transition to Multipath Extension Support">

	<t>
	  If a Multipath Node Capability sub-TLV is not advertised
	  (see <xref target="node-capability" />), then the LSR does
	  not support these multipath extensions.  This allows
	  detection of such nodes and if necessary application of
	  defaults to cover legacy multipath such as typical Ethernet
	  Link Aggregation Behavior.
	</t>

	<section anchor="compat.simple"
		 title="Simple Transitions">

	  <t>
	    For networks with LSR that do not support multipath
	    extensions, transition is easiest if all legacy LSR
	    support and are configured with a common link bundling
	    behavior.  If Ethernet Link Aggregation is not used, a
	    single configured default is needed to cover LSR that do
	    not advertise a Multipath Node Capability sub-TLV.
	  </t>
	  <t>
	    If Ethernet Link Aggregation had been previously used on
	    Legacy LSR, if possible LAG should be disabled and the
	    members of the former LAG configured and advertised as a
	    link bundle which uses the equivalent "all ones" behavior.
	  </t>
	  <t>
	    If Ethernet Link Aggregation remains but can be identified
	    in some way, such as links with capacity in excess of some
	    value (for example: greater than 10 Gb/s), then defaults
	    can be set up for these LAG.  Alternately administrative
	    attributes could be used <xref target="RFC3209" />.
	  </t>
	  <t>
	    The transition network in this case lacks the ability to
	    determine the largest microflow that can pass through
	    legacy nodes, but this was the case prior to transition
	    for the entire network prior to transition.
	  </t>

	</section>

	<section anchor="compat.rough"
		 title="More Challenging Transitions">

	  <t>
	    Transition is made more difficult if legacy LSR in a
	    network support Ethernet Link Aggregation but do not
	    support Link Bundle and cannot be identified by simple
	    means, or the newer LSR lack sufficient configuration
	    capability to support conditional defaults.
	  </t>
	  <t>
	    This situation is most easily handled if a small upgrade
	    to such an LSR can advertise a fixed Multipath Node
	    Capability sub-TLV giving the characteristics of the
	    Ethernet Link Aggregation implementation on that node.
	    Absent of such cooperation, the problem can be solved by
	    configuration on newer LSR which allows association of a
	    Multipath Node Capability sub-TLV with a specific legacy
	    router ID and possibly a legacy router ID and link.
	  </t>
	  <t>
	    LSR supporting Multipath Extensions will need to assign
	    default values through configuration to these legacy LSR
	    running Ethernet Link Aggregation.  These default values
	    serve to allow LSP which require strict packet ordering to
	    avoid these legacy LSR.
	  </t>
	  <t>
	    LSR which do not support <xref target="RFC4201" /> may be
	    sufficiently rare that the ability to assign default
	    values per legacy LSR or per <xref target="RFC3209" />
	    administrative attribute may not be needed in practice.
	  </t>

	</section>

      </section>

    </section>

    <section anchor="iana" title="IANA Considerations">

      <t>
	[ ... to be completed ... ]
      </t>

      <t>
	The symbolic constants IANA-TBD-1 through IANA-TBD-3 need to
	be replaced.  Complete instructions, including identification
	of the number space for each of these will be added to a later
	version of this internet-draft.
      </t>

<!--

OSPF

   IANA maintains the "Open Shortest Path First (OSPF) Traffic
   Engineering TLVs" registries including the "Types for sub-TLVs of TE
   link TLV (Value 2)" registry.

   This document defines the following sub-TLV of TE link TLV (Value 2).

   Value  Sub-TLV
   -----  -------------------------------------------------
   25     Interface Adjustment Capability Descriptor (IACD)

ISIS

   This document defines the following new sub-TLV type of top-level TLV
   22 that has been reflected in the ISIS sub-TLV registry for TLV 22,
   141, and 222:

   Type  Description                                        Length
   ----  -------------------------------------------------  ------
   27    Interface Adjustment Capability Descriptor (IACD)  Var.

 -->

    </section>

    <section anchor="Security" title="Security Considerations">
      <t>
	The combination of MPLS, MPLS-TP, and multipath does not
	introduce any new security threats.  The security
	considerations for MPLS/GMPLS and for MPLS-TP are documented
	in <xref target="RFC5920" /> and
	<xref target="I-D.ietf-mpls-tp-security-framework" />.
      </t>
    </section>

  </middle>

  <back>

    <references title="Normative References">

      &RFC2119;
      &RFC3471;
      &RFC4201;
      &RFC5420;
      &RFC5786;

    </references>

    <references title="Informative References">

      &RFC2475;
      &RFC2702;
      &RFC2991;
      &RFC3209;
      &RFC3260;
      &RFC3945;
      &RFC4206;
      &RFC4385;
      &RFC5586;
      &RFC5920;
      &RFC6107;
      &RFC6391;

      &I-D.ietf-mpls-entropy-label;

      &I-D.villamizar-mpls-multipath-use;

      &I-D.ietf-mpls-tp-security-framework;

      <reference anchor="IEEE-802.1AX"
                 target="http://standards.ieee.org/getieee802/download/802.1AX-2008.pdf">
        <front>
          <title>IEEE Std 802.1AX-2008 IEEE Standard for
	    Local and Metropolitan Area Networks - Link Aggregation</title>

          <author>
            <organization>IEEE Standards Association</organization>
          </author>

          <date year="2006" />
        </front>
      </reference>

      <reference anchor="ITU-T.G.800"
                 target="http://www.itu.int/rec/T-REC-G/recommendation.asp?parent=T-REC-G.800">
        <front>
          <title>Unified functional architecture of transport
          networks</title>

          <author>
            <organization>ITU-T</organization>
          </author>

          <date year="2007" />
        </front>
      </reference>

    </references>

  </back>
</rfc>

PAFTECH AB 2003-20262026-04-24 03:18:57