One document matched: draft-briscoe-tsvwg-ecn-encap-guidelines-01.txt
Differences from draft-briscoe-tsvwg-ecn-encap-guidelines-00.txt
Transport Area Working Group B. Briscoe
Internet-Draft BT
Updates: 3819 (if approved) October 22, 2012
Intended status: BCP
Expires: April 25, 2013
Guidelines for Adding Congestion Notification to Protocols that
Encapsulate IP
draft-briscoe-tsvwg-ecn-encap-guidelines-01
Abstract
The purpose of this document is to guide the design of congestion
notification in any lower layer or tunnelling protocol that
encapsulates IP. The aim is for explicit congestion signals to
propagate consistently from lower layer protocols into IP. Then the
IP internetwork layer can act as a portability layer to carry
congestion notification from non-IP-aware congested nodes up to the
transport layer (L4). Following these guidelines should assure
interworking between new lower layer congestion notification
mechanisms, whether specified by the IETF or other standards bodies.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on April 25, 2013.
Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
Briscoe Expires April 25, 2013 [Page 1]
Internet-Draft ECN Encapsulation Guidelines October 2012
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Modes of Operation . . . . . . . . . . . . . . . . . . . . . . 6
3.1. Feed-Forward-and-Up Mode . . . . . . . . . . . . . . . . . 7
3.2. Feed-Up-and-Forward Mode . . . . . . . . . . . . . . . . . 8
3.3. Feed-Backward Mode . . . . . . . . . . . . . . . . . . . . 9
3.4. Null Mode . . . . . . . . . . . . . . . . . . . . . . . . 11
4. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion
Notification . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.1. Wire Protocol Design: Indication of ECN Support . . . . . 11
4.2. Encapsulation Guidelines . . . . . . . . . . . . . . . . . 13
4.3. Decapsulation Guidelines . . . . . . . . . . . . . . . . . 14
4.4. Reframing and Congestion Markings . . . . . . . . . . . . 15
5. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion
Notification . . . . . . . . . . . . . . . . . . . . . . . . . 16
6. Feed-Backward Mode: Guidelines for Adding Congestion
Notification . . . . . . . . . . . . . . . . . . . . . . . . . 17
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18
8. Security Considerations . . . . . . . . . . . . . . . . . . . 18
9. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 18
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 18
11. Comments Solicited . . . . . . . . . . . . . . . . . . . . . . 18
12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 19
12.1. Normative References . . . . . . . . . . . . . . . . . . . 19
12.2. Informative References . . . . . . . . . . . . . . . . . . 19
Appendix A. Outstanding Document Issues . . . . . . . . . . . . . 21
Appendix B. Changes in This Version (to be removed by RFC
Editor) . . . . . . . . . . . . . . . . . . . . . . . 22
Briscoe Expires April 25, 2013 [Page 2]
Internet-Draft ECN Encapsulation Guidelines October 2012
1. Introduction
Explicit Congestion Notification (ECN [RFC3168]) is defined in the IP
header (v4 & v6) to allow a resource to notify the onset of queue
build-up without having to drop packets, by explicitly marking a
proportion of packets with the congestion experienced (CE)
codepoint.ECN removes nearly all congestion loss and it cuts delays
for two main reasons: i) it avoids the delays recovering from
congestion losses, which particularly benefits small flows, making
their completion time predictably short [RFC2884]; and ii) as ECN is
used more widely by end-systems, it will gradually remove the need to
configure a degree of delay into buffers before they start to notify
congestion (the cause of bufferbloat). The latter delay is because
drop involves a trade-off between sending a timely signal and trying
to avoid impairment, whereas ECN is solely a signal so there is no
harm triggering it earlier.
Some lower layer technologies (e.g. MPLS, Ethernet) are used to form
large subnetworks with IP-aware nodes only at the edges.
Particularly now that end-system protocols are finally being deployed
without their earlier deficiencies, even the buffers of well-
provisioned interior switches will often need to signal episodes of
queuing. However, the above benefits of ECN can only be fully
realised if the relevant subnetwork technology supports it.
Propagation of ECN is defined for MPLS [RFC5129], and is being
defined for TRILL [trill-rbridge-options], but it remains to be
defined for a number of other subnetwork technologies.
Similarly, ECN propagation is yet to be defined for many tunnelling
protocols. [RFC6040] defines how ECN should be propagated for
IP-in-IP [RFC2003] and IPsec [RFC4301] tunnels. However, as Section
9.3 of RFC3168 pointed out, ECN support will need to be defined for
other tunnelling protocols, e.g. L2TP [RFC2661], GRE [RFC1701,
RFC2784], PPTP [RFC2637] and GTP [GTPv1, GTPv1-U, GTPv2-C].
The purpose of this document is to guide the addition of congestion
notification to any subnet technology or tunnelling protocol, so that
lower layer equipment can signal congestion explicitly and it will
propagate consistently into encapsulated (higher layer) headers,
otherwise the signals will not reach their ultimate destination.
Incremental deployment is the most tricky aspect when adding support
for ECN. The original ECN protocol in IP [RFC3168] was carefully
designed so that a congested buffer would not mark a packet (rather
than drop it) unless both source and destination hosts were ECN-
capable. Otherwise its congestion markings would never be detected
and congestion would just deteriorate further. However, to support
congestion marking below the IP layer, it is not sufficient to only
Briscoe Expires April 25, 2013 [Page 3]
Internet-Draft ECN Encapsulation Guidelines October 2012
check that the two end-points support ECN; correct operation also
depends on the decapsulator propagating congestion notifications
faithfully. Otherwise, a legacy decapsulator might silently fail to
propagate any ECN signals from the outer to the forwarded header.
Then the lost signals would never be detected and again congestion
would deteriorate further. The guidelines given later require
protocol designers to carefully consider incremental deployment, and
suggest various safe approaches for different circumstances.
Of course, the IETF does not have standards authority over every link
layer protocol. So this document gives guidelines for designing
propagation of congestion notification across the interface between
IP and protocols that may encapsulate IP (i.e. that can be layered
beneath IP). Each lower layer technology will exhibit different
issues and compromises, so the IETF or the relevant standards body
must be free to define the specifics of each lower layer congestion
notification scheme. Nonetheless, if the guidelines are followed,
congestion notification should interwork between different
technologies, using IP in its role as a 'portability layer'.
It has not been possible to give common guidelines for all lower
layer technologies, because they do not all fit a common pattern.
Instead they have been divided into a few distinct modes of
operation: Feed-Forward-and-Upward, Feed-Upward-and-Forward, Feed-
Backward and Null. These are described in Section 3, then in the
following sections separate guidelines are given for each mode.
This document updates the advice to subnetwork designers about ECN in
Section 13 of [RFC3819].
1.1. Scope
This document only concerns wire protocol processing of explicit
notification of congestion and makes no changes or recommendations
concerning algorithms for congestion marking or congestion response.
This document focuses on the congestion notification interface
between IP (v4 or v6) and lower layer protocols that can encapsulate
IP. However, it is likely that the guidelines will also be useful
when a lower layer protocol or tunnel encapsulates itself (e.g.
Ethernet MAC in MAC [IEEE802.1Qah]) or when it encapsulates other
protocols.
2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
Briscoe Expires April 25, 2013 [Page 4]
Internet-Draft ECN Encapsulation Guidelines October 2012
Further terminology used within this document:
Protocol data unit (PDU): Information that is delivered as a unit
among peer entities of a layered network consisting of protocol
control information (typically a header) and possibly user data
(payload) of that layer. The scope of this document includes
layer 2 and layer 3 networks, where the PDU is respectively termed
a frame or a packet (or a cell in ATM). PDU is a general term for
any of these. This definition also includes a payload with a shim
header lying somewhere between layer 2 & 3.
Transport: The end-to-end transmission control function,
conventionally considered at layer-4 in the OSI reference model.
Given the audience for this document will often use the word
transport to mean low level bit carriage, whenever the term is
used it will be qualified, e.g. 'L4 transport'.
Encapsulator: The link or tunnel endpoint function that adds an
outer header to a PDU (also termed the 'link ingress', the 'subnet
ingress', the 'ingress tunnel endpoint' or just the 'ingress'
where the context is clear).
Decapsulator: The link or tunnel endpoint function that removes an
outer header from a PDU (also termed the 'link egress', the
'subnet egress', the 'egress tunnel endpoint' or just the 'egress'
where the context is clear).
Incoming header: The header of an arriving PDU before encapsulation.
Outer header: The header added to encapsulate a PDU.
Inner header: The header encapsulated by the outer header.
Outgoing header: The header forwarded by the decapsulator.
CE: Congestion Experienced [RFC3168]
ECT: ECN-Capable Transport [RFC3168]
Not-ECT: Not ECN-Capable Transport [RFC3168]
ECN-PDU: A PDU that is part of a feedback loop within which the
nodes necessary to propagate explicit congestion notifications
back to the load regulator are ECN-capable. This is intended to
be a general term for a PDU at any layer, not just an IP PDU. An
IP packet with a non-zero ECN field would be an ECN-PDU, but the
term is intended to also be used to describe PDUs of protocols
that encapsulate IP packets, where it has been checked that the
Briscoe Expires April 25, 2013 [Page 5]
Internet-Draft ECN Encapsulation Guidelines October 2012
necessary egress nodes and endpoints in the feedback loop for that
PDU will propagate congestion notification.
Not-ECN-PDU: A PDU that is part of a feedback-loop within which some
nodes necessary to propagate explicit congestion notifications
back to the load regulator are not ECN-capable.
Load Regulator: For each flow of PDUs, the transport function that
is capable of controlling the data rate. Typically located at the
data source, but in-path nodes can regulate load in some
congestion control arrangements (e.g. admission control or
policing nodes). Note the term "a function capable of controlling
the load" deliberately includes a transport that doesn't actually
control the load but ought to (e.g. an application without
congestion control that uses UDP).
Congestion Baseline: The location of the function on the path that
initialised the values of all congestion notification fields in a
sequence of packets, before any are set to the congestion
experienced (CE) codepoint if they experience congestion further
downstream. Typically the original data source at layer-4.
3. Modes of Operation
This section sets down the different modes by which information is
passed between the lower layer and the higher one. It acts as a
reference framework for the following sections, which give normative
guidelines for designers of explicit congestion notification
protocols, taking each mode separately in turn:
Feed-Forward-and-Up: Nodes feed forward congestion notification
towards the destination within the lower layer then up the layers
(like IP does). The following local optimisation is possible:
Feed-Up-and-Forward: A lower layer switch feeds-up congestion
notification directly into the ECN field in the higher layer
(IP) header, irrespective of whether it is at the egress of a
subnet.
Feed-Backward: Nodes feed back congestion signals towards the
ingress of the lower layer and (optionally) attempt to control
congestion within their own layer.
Null: Nodes cannot experience congestion at the lower layer except
at ingress nodes that are also IP-aware (or equivalently higher-
layer-aware).
Briscoe Expires April 25, 2013 [Page 6]
Internet-Draft ECN Encapsulation Guidelines October 2012
3.1. Feed-Forward-and-Up Mode
Many subnet technologies are based on self-contained protocol data
units (PDUs) or frames sent unreliably. They provide no feedback
channel at the subnetwork layer, instead relying on higher layers
(e.g. TCP) to feed back loss signals.
In these cases, ECN may best be supported by standardising explicit
notification of congestion into the specific link layer protocol. It
will then also be necessary to define how the egress of the lower
layer subnet propagates this explicit signal into the forwarded upper
layer (IP) header. It can then continue forwards until it finally
reaches the destination transport (at L4). Then typically the
destination will feed this congestion notification back to the source
transport using an end-to-end protocol (e.g. TCP).
This mode is illustrated in Figure 1. Along the middle of the
figure, layers 2, 3 & 4 of the protocol stack are shown, and one
packet is shown along the bottom as it progresses across the network
from source to destination, crossing two subnets connected by a
router, and crossing two switches on the path across each subnet.
Congestion at the output of the first switch (shown as *) leads to a
congestion marking in the L2 header (shown as C in the illustration
of the packet). The chevrons show the progress of the resulting
congestion indication. It is propagated from link to link across the
subnet in the L2 header, then when the router removes the marked L2
header, it propagates the marking up into the L3 (IP) header. The
router forwards the marked L3 header into subnet 2, and when it adds
a new L2 header it copies the L3 marking into the L2 header as well,
as shown by the 'C's in both layers (assuming the technology of
subnet 2 also supports explicit congestion marking).
Note that there is no implication that each 'C' marking is encoded
the same; a different encoding might be used for the 'C' marking in
each protocol.
Finally, for completeness, we show the L3 marking arriving at the
destination, where the host transport protocol (e.g. TCP) feeds it
back to the source in the L4 acknowledgement (the 'C' at L4 in the
packet at the top of the diagram).
Briscoe Expires April 25, 2013 [Page 7]
Internet-Draft ECN Encapsulation Guidelines October 2012
_ _ _
/_______ | | |C| ACK Packet (V)
\ |_|_|_|
+---+ layer: 2 3 4 header +---+
| <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4
| | +---+ | ^ |
| | . . . . . . Packet U. . | >>|>>> Packet U >>>>>>>>>>>>|>^ |L3
| | +---+ +---+ | ^ | +---+ +---+ | |
| | | *|>>>>>|>>>|>>>>>|>^ | | | | | | |L2
|___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___|
source subnet A router subnet B dest
__ _ _ _ __ _ _ _ __ _ _ __ _ _ _
| | | | | | | | |C| | | |C| | | |C|C| Data________\
|__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| Packet (U) /
layer: 4 3 2A 4 3 2A 4 3 4 3 2B
header
Figure 1: Feed-Forward-and-Up Mode
Of course, modern networks are rarely as simple as this text-book
example, often involving multiple nested layers. Nonetheless, the
example illustrates the general idea of feeding congestion
notification forward then upward whenever a header is removed at the
egress of a subnet.
Note that the FECN (forward ECN) bit in Frame Relay and the explicit
forward congestion indication (EFCI [ITU-T.I.371]) bit in ATM user
data cells follow a feed-forward pattern. However, in ATM, this is
only as part of a feed-forward-and-backward pattern at the lower
layer, not feed-forward-and-up out of the lower layer--the intention
was never to interface to IP ECN at the subnet egress. To our
knowledge, Frame Relay FECN is solely used to detect where more
capacity should be provisioned [Buck00].
3.2. Feed-Up-and-Forward Mode
Ethernet is particularly difficult to extend incrementally to support
explicit congestion notification. One way to support ECN in such
cases has been to use so called 'layer-3 switches'. These are
Ethernet switches that bury into the Ethernet payload to find an IP
header and manipulate or act on certain IP fields (specifically
Diffserv & ECN). For instance, in Data Center TCP [DCTCP], layer-3
switches are configured to mark the ECN field of the IP header within
the Ethernet payload when their output buffer becomes congested.
With respect to switching, a layer-3 switch acts solely on the
addresses in the Ethernet header; it doesn't use IP addresses, and it
doesn't decrement the TTL field in the IP header.
Briscoe Expires April 25, 2013 [Page 8]
Internet-Draft ECN Encapsulation Guidelines October 2012
_ _ _
/_______ | | |C| ACK packet (V)
\ |_|_|_|
+---+ layer: 2 3 4 header +---+
| <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4
| | +---+ | ^ |
| | . . . >>>> Packet U >>>|>>>|>>> Packet U >>>>>>>>>>>>|>^ |L3
| | +--^+ +---+ | | +---+ +---+ | |
| | | *| | | | | | | | | | |L2
|___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___|
source subnet E router subnet F dest
__ _ _ _ __ _ _ _ __ _ _ __ _ _ _
| | | | | | | |C| | | | |C| | | |C|C| data________\
|__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) /
layer: 4 3 2 4 3 2 4 3 4 3 2
header
Figure 2: Feed-Up-and-Forward Mode
By comparing Figure 2 with Figure 1, it can be seen that subnet E
(perhaps a subnet of layer-3 Ethernet switches) works in feed-up-and-
forward mode by notifying congestion directly into L3 at the point of
congestion, even though the congested switch does not otherwise act
at L3. In this example, the technology in subnet F (e.g. MPLS) does
support ECN natively, so when the router adds the layer-2 header it
copies the ECN marking from L3 to L2 as well.
3.3. Feed-Backward Mode
In some layer 2 technologies, explicit congestion notification has
been defined for use internally within the subnet with its own
feedback and load regulation, but typically the interface with IP for
ECN has not been defined.
For instance, for the available bit-rate (ABR) service in ATM, the
relative rate mechanism was one of the more popular mechanisms for
managing traffic, tending to supersede earlier designs. In this
approach ATM switches send special resource management (RM) cells in
both the forward and backward directions to control the ingress rate
of user data into a virtual circuit. If a switch buffer is
approaching congestion or congested it sends an RM cell back towards
the ingress with respectively the No Increase (NI) or Congestion
Indication (CI) bit set in its message type field [ATM-TM-ABR]. The
ingress then holds or decreases its sending bit-rate accordingly.
Briscoe Expires April 25, 2013 [Page 9]
Internet-Draft ECN Encapsulation Guidelines October 2012
_ _ _
/_______ | | |C| ACK packet (X)
\ |_|_|_|
+---+ layer: 2 3 4 header +---+
| <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet X <<<<<<<<<<<<<|<< |L4
| | +---+ | ^ |
| | | *|>>> Packet W >>>>>>>>>>>>|>^ |L3
| | +---+ +---+ | | +---+ +---+ | |
| | | | | | | <|<<<<<|<<<|<(V)<|<<<| | |L2
| | . . | . |Packet U | . . | . | . . | . | . . | .*| . . | |L2
|___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___|
source subnet G router subnet H dest
__ _ _ _ __ _ _ _ __ _ _ __ _ _ _ later
| | | | | | | | | | | | | | | | |C| | data________\
|__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (W) /
4 3 2 4 3 2 4 3 4 3 2
_
/__ |C| Feedback control
\ |_| cell/frame (V)
2
__ _ _ _ __ _ _ _ __ _ _ __ _ _ _ earlier
| | | | | | | | | | | | | | | | | | | data________\
|__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) /
layer: 4 3 2 4 3 2 4 3 4 3 2
header
Figure 3: Feed-Backward Mode
ATM's feed-backward approach doesn't fit well when layered beneath
IP's feed-forward approach--unless the initial data source is the
same node as the ATM ingress. Figure 3 shows the feed-backward
approach being used in subnet H. If the final switch on the path is
congested (*), it doesn't feed-forward any congestion indications on
packet (U). Instead it sends a control cell (V) back to the router
at the ATM ingress.
However, the backward feedback doesn't reach the original data source
directly because IP doesn't support backward feedback (and subnet G
is independent of subnet H). Instead, the router in the middle
throttles down its sending rate but the original data source doesn't
reduce its rate. The resulting rate mismatch causes the middle
router's buffer at layer 3 to back up until it becomes congested,
which it signals forwards on later data packets at layer 3 (e.g.
packet W). Note that the forward signal from the middle router is
not triggered directly by the backward signal. Rather, it is
triggered by congestion resulting from the middle router's mismatched
rate response to the backward signal.
Briscoe Expires April 25, 2013 [Page 10]
Internet-Draft ECN Encapsulation Guidelines October 2012
In response to this later forward signalling, end-to-end feedback at
layer-4 finally completes the tortuous path of congestion indications
back to the origin data source, as before.
3.4. Null Mode
Often link and physical layer resources are 'non-blocking' by design.
In these cases congestion notification may be implemented but it does
not need to be deployed at the lower layer; ECN in IP would be
sufficient.
A degenerate example is a point-to-point Ethernet link. Excess
loading of the link merely causes the queue from the higher layer to
back up, while the lower layer remains immune to congestion. Even a
whole meshed subnetwork can be made immune to interior congestion by
limiting ingress capacity and careful sizing of links, particularly
if multi-path routing is used to ensure even worst-case patterns of
load cannot congest any link.
4. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion
Notification
These guidelines are consistent with the guidelines on the design of
alternate schemes for IP tunnelling of the ECN field [RFC6040] and
the more general best current practice for the design of alternate
ECN schemes given in [RFC4774].
The capitalised term 'SHOULD (NOT)' has often been used in preference
to 'MUST (NOT)' because it is difficult to know the compromises that
will be necessary in each protocol design. If a particular protocol
design chooses to contradict a 'SHOULD (NOT)' given in the advice
below, it MUST include a sound justification.
4.1. Wire Protocol Design: Indication of ECN Support
A lower layer (or subnet) congestion notification protocol
1. SHOULD NOT apply explicit congestion notifications to PDUs that
are destined for legacy layer-4 transport implementations that
will not understand ECN, and
2. SHOULD NOT apply explicit congestion notifications to PDUs that
are destined for a legacy subnet egress that will fail to
propagate them onward into the higher layer.
We use the term ECN-PDUs for a PDU on a feedback loop that will
propagate congestion notification properly because it meets both
these criteria. And a Not-ECN-PDU is a PDU on a feedback loop
Briscoe Expires April 25, 2013 [Page 11]
Internet-Draft ECN Encapsulation Guidelines October 2012
that does not meet both criteria, and will therefore not
propagate congestion notification properly. A corollary of the
above is that a lower layer congestion notification protocol:
3. SHOULD be able to distinguish ECN-PDUs from Not-ECN-PDUs.
In IP, if the ECN field in each PDU is cleared to the Not-ECT (not
ECN-capable transport) codepoint, it indicates that the L4 transport
will not understand congestion markings. A congested buffer must not
mark these Not-ECT PDUs, and therefore has to drop some. The
mechanism a lower layer uses to distinguish the ECN-capability of
PDUs need not mimic that of IP, but it should achieve the same
outcome. For instance, ECN-capable feedback loops might use PDUs
that are identified by a particular set of labels or tags.
Alternatively, logical link protocols that use flow state might
determine whether a PDU can be congestion marked by checking for ECN-
support in the flow state.
The per-domain checking of ECN support in MPLS [RFC5129] is a good
example of a way to avoid sending congestion markings to transports
that will not understand them--without using any header space in the
subnet protocol.
In MPLS, header space is extremely limited, therefore RFC5129 does
not provide a field in the MPLS header to indicate whether the PDU is
an ECN-PDU or a Not-ECN-PDU. Instead, interior nodes in a domain are
allowed to set explicit congestion indications without checking
whether the PDU is destined for a transport that will understand
them. Nonetheless, this is made safe by requiring that the network
operator upgrades all decapsulating edges of a whole domain at once,
if any switch within the domain is configured to mark rather than
drop during congestion. Therefore, there will be an implementation
of an ECN-capable decapsulator on any edge node that might
decapsulate a packet, which will check whether the higher layer
transport is ECN-capable. When decapsulating a CE-marked packet, if
the decapsulator discovers that the higher layer (inner header)
indicates the transport is not ECN-capable, it drops the packet on
behalf of the earlier congested node (see Decapsulation Guideline
Paragraph 1 in Section 4.3).
Note that it was only appropriate to define such an incremental
deployment strategy because MPLS is targeted solely at professional
operators, who can be expected to ensure that a whole subnetwork is
consistently configured. This strategy might not be appropriate for
other link technologies targeted at zero-configuration deployment or
deployment by the general public (e.g. Ethernet). For such 'plug-
and-play' environments it will be necessary to invent a failsafe
approach that ensures congestion markings will never fall into black
Briscoe Expires April 25, 2013 [Page 12]
Internet-Draft ECN Encapsulation Guidelines October 2012
holes, no matter how inconsistently a system is put together.
Alternatively, congestion notification relying on correct system
configuration could be confined to flavours of Ethernet intended only
for professional network operators, such as IEEE 802.1ah Provider
Backbone Bridges (PBB).
Note that these guidelines do not require the subnet wire protocol to
be changed at all to accommodate congestion notification. Another
way to add congestion notification without consuming header space in
the subnet protocol might be to use a control plane protocol in
parallel.
4.2. Encapsulation Guidelines
1. Egress Capability Check: A subnet ingress needs to be sure that
the corresponding egress of a subnet will propagate any
congestion notification added to the outer header across the
subnet. This is necessary in addition to checking that an
incoming PDU indicates an ECN-capable (L4) transport. Examples
of how this guarantee might be provided include:
* by configuration (e.g. if any label switches in a domain
support ECN marking, [RFC5129] requires all egress nodes to
have been configured to propagate ECN)
* by the ingress explicitly checking that the egress propagates
ECN (e.g. TRILL uses IS-IS to check path capabilities before
using critical options [trill-rbridge-options])
* by inherent design of the protocol (e.g. by encoding ECN
marking on the outer header in such a way that a legacy egress
that does not understand ECN will consider the PDU corrupt and
discard it, thus at least propagating a form of congestion
signal).
If the ingress cannot guarantee that the egress will propagate
congestion notification, the ingress SHOULD disable ECN when it
forwards the PDU at the lower layer. An example of how the
ingress might disable ECN at the lower layer would be by setting
the outer header of the PDU to identify it as a Not-ECN-PDU.
2. Standard Congestion Monitoring Baseline: Once the ingress to a
subnet has established that the egress will correctly propagate
ECN, on encapsulation it SHOULD encode the same level of
congestion in outer headers as is arriving in incoming headers.
For example it could copy any incoming congestion notification
into the outer header of the lower layer protocol.
Briscoe Expires April 25, 2013 [Page 13]
Internet-Draft ECN Encapsulation Guidelines October 2012
This ensures that all outer headers reflect congestion
accumulated along the whole upstream path, not just since the
ingress of the subnet. More precisely, congestion notifications
in outer headers SHOULD reflect congestion experienced along the
whole path since the node that regulates the load for that path
(the Load Regulator, typically the data source) and no other node
should re-initialise the amount of CE markings to zero along the
way.
This guideline is intended to ensure that any bulk congestion
monitoring of outer headers (e.g. by a network management node
monitoring ECN in passing frames) is most meaningful. For
instance, if an operator measures CE in 0.4% of passing packets,
this information is only useful if the operator knows where the
proportion of CE markings was last initialised to 0% (the
Congestion Baseline). Such monitoring information will not be
useful if some subnet ingress nodes reset all outer CE markings
while others copy incoming CE markings into the outer.
Most information can be extracted if the Congestion Baseline is
standardised at the node that is regulating the load (the Load
Regulator--typically the data source). Then the operator can
measure both congestion since the Load Regulator, and congestion
since the subnet ingress. The latter can be measured by
subtracting the level of CE markings on inner headers from that
on outer headers.
4.3. Decapsulation Guidelines
A subnet egress SHOULD NOT simply copy congestion notification from
outer headers to the forwarded header. It SHOULD calculate the
outgoing congestion notification field from the inner and outer
headers, using the following rules. If there is any conflict, rules
earlier in the list take precedence over rules later in the list:
1. If the arriving inner header is a Not-ECN-PDU it implies the L4
transport will not understand explicit congestion markings.
Then:
* If the outer header carries an explicit congestion marking,
the packet SHOULD be dropped--the only indication of
congestion that the L4 transport will understand.
* If the outer is an ECN-PDU that carries no indication of
congestion or a Not-ECN-PDU the PDU SHOULD be forwarded, but
still as a Not-ECN-PDU.
Briscoe Expires April 25, 2013 [Page 14]
Internet-Draft ECN Encapsulation Guidelines October 2012
2. If the outer header does not support explicit congestion
notification (a Not-ECN-PDU), but the inner header does (an ECN-
PDU), the inner header SHOULD be forwarded unchanged.
3. In some lower layer protocols congestion may be signalled as a
numerical level, such as in the control frames of quantised
congestion notification [IEEE802.1Qau]. If such an encoding
encapsulates an ECN-capable IP packet, a function will be needed
to convert the quantised congestion level into the frequency of
congestion markings in outgoing IP packets.
4. Congestion indications may be encoded by a severity level. For
instance increasing levels of congestion might be encoded by
numerically increasing indications, e.g. pre-congestion
notification (PCN) can be encoded in each PDU at three severity
levels in IP or MPLS [RFC6660].
If the arriving inner header is an ECN-PDU, where the inner and
outer headers carry indications of congestion of different
severity, the more severe indication SHOULD be forwarded in
preference to the less severe. Obviously, if the severities in
both inner and outer are the same, the same severity should be
forwarded.
5. The inner and outer headers might carry a combination of
congestion notification fields that should not be possible given
any currently used protocol transitions. For instance, if
Encapsulation Guideline Paragraph 2 in Section 4.2 had been
followed, it should not be possible to have a less severe
indication of congestion in the outer than in the inner. It MAY
be appropriate to log unexpected combinations of headers and
possibly raise an alarm. If a safe outgoing codepoint can be
defined for such a PDU, the PDU SHOULD be forwarded rather than
dropped.
Some implementers discard PDUs with currently unused combinations
of headers just in case they represent an attack. However, an
approach using alarms and policy-mediated drop is preferable to
hard-coded drop, so that operators can keep track of possible
attacks but currently unused combinations are not precluded from
future use through new standards actions.
4.4. Reframing and Congestion Markings
Where framing boundaries are different between two layers, congestion
indications SHOULD be propagated on the basis that a congestion
indication on a PDU applies to all the octets in the PDU. On
average, an encapsulator or decapsulator SHOULD approximately
Briscoe Expires April 25, 2013 [Page 15]
Internet-Draft ECN Encapsulation Guidelines October 2012
preserve the number of marked octets arriving and leaving (counting
the size of inner headers, but not added encapsulating headers).
The next departing frame SHOULD be immediately marked even if only
enough incoming marked octets have arrived for part of the departing
frame. This ensures that any outstanding congestion marked octets
are propagated immediately, rather than held back waiting for a frame
no bigger than the outstanding marked octets--which might involve a
long wait.
For instance, an algorithm for marking departing frames could
maintain a counter representing the balance of arriving marked octets
minus departing marked octets. It adds the size of every marked
frame that arrives and if the counter is positive it marks the next
frame to depart and subtracts its size from the counter. This will
often leave a negative remainder in the counter, which is deliberate.
5. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion
Notification
Marking the IP header while switching at layer-2 (by using a layer-3
switch) seems to represent a layering violation. However, it can be
considered as a benign optimisation if the guidelines below are
followed. Feed-up-and-forward is certainly not a general alternative
to implementing feed-forward congestion notification in the lower
layer, because:
o IPv4 and IPv6 are not the only layer-3 protocols that might be
encapsulated by lower layer protocols
o Link-layer encryption might be in use, making the layer-2 payload
inaccessible
o Many Ethernet switches do not have 'layer-3 switch' capabilities
so they cannot read and modify an IP payload
o It might be costly to find an IP header (v4 or v6) when it may be
encapsulated by more than one Ethernet header (e.g. when using
multiple encapsulations of MAC in MAC [IEEE802.1Qah]).
Nonetheless, configuring a layer-3 switch to look for an ECN field in
an encapsulated IP header is a useful optimisation. If the
implementation follows the guidelines below, this optimisation does
not have to be confined to a controlled environment such as within a
data centre; it could usefully be applied on any network--even if the
operator is not sure whether the above issues will never apply:
Briscoe Expires April 25, 2013 [Page 16]
Internet-Draft ECN Encapsulation Guidelines October 2012
1. If a native lower-layer congestion notification mechanism exists
for a subnet technology, it is safe to mix feed-up-and-forward
with feed-forward-and-up on other switches in the same subnet.
However, it will generally be more efficient to use the native
mechanism.
2. The depth of search for an IP header SHOULD be limited. If an IP
header is not found soon enough, or an unrecognised or unreadable
header is encountered, the switch SHOULD resort to an alternative
means of signalling congestion (e.g. drop, or the native lower
layer mechanism if available).
3. It is sufficient to use the first IP header found in the stack;
the egress of the relevant tunnel can propagate congestion
notification upwards to any more deeply encapsulated IP headers
later.
6. Feed-Backward Mode: Guidelines for Adding Congestion Notification
It can be seen from Section 3.3 that congestion notification in a
subnet using feed-backward mode has generally not been designed to
directly coupled with IP layer congestion notification. The subnet
attempts to minimise congestion internally, and if the incoming load
at the ingress exceeds capacity through the subnet, the layer 3
buffer into the ingress backs up. Thus, a feed-backward mode subnet
is in some sense similar to a null mode subnet, in that there is no
need for any direct interaction between the subnet and higher layer
congestion notification. Therefore no detailed protocol design
guidelines are appropriate. Nonetheless, a more general guideline is
appropriate:
1. A subnetwork technology intended to eventually interface to IP
SHOULD NOT be designed using only the feed-backward mode, which
is certainly best for a stand-alone subnet, but would need to be
modified to work efficiently as part of the wider Internet,
because IP uses feed-forward-and-up mode.
The feed-backward approach does at least work beneath IP, but it can
result in very inefficient and sluggish congestion control--except if
it is confined to the subnet directly connected to the original data
source, when it is faster than feed-forward. It would be possible to
design a protocol that could work in feed-backward mode for paths
that only cross one subnet, and in feed-forward-and-up mode for paths
that cross subnets.
In the early days of TCP/IP, a similar feed-backward approach was
tried for explicit congestion signalling, using source-quench (SQ)
ICMP control packets. However, SQ fell out of favour and is now
Briscoe Expires April 25, 2013 [Page 17]
Internet-Draft ECN Encapsulation Guidelines October 2012
formally deprecated [RFC6633]. The main problem was that it is hard
for a data source to tell the difference between a spoofed SQ message
and a quench request from a genuine buffer on the path. It is also
hard for a lower layer buffer to address an SQ message to the
original source, which may be buried within many layers of headers,
and possibly encrypted.
Quantised congestion notification (QCN--also known as backward
congestion notification or BCN) [IEEE802.1Qau] uses a feed-backward
mode very similar to ATM. However, QCN confines its applicability to
scenarios where all endpoints are directly attached by the same
Ethernet technology, and is used for example in server area networks
(SANs). If a QCN subnet were connected into a wider IP-based
internetwork (e.g. when attempting to interconnect SANs within
multiple data centres) it would suffer the same inefficiency as shown
in Figure 3.
7. IANA Considerations
This memo includes no request to IANA.
8. Security Considerations
{TBA}`
9. Conclusions
{TBA}
10. Acknowledgements
Thanks to Gorry Fairhurst for extensive initial review.
Bob Briscoe produced early drafts while partly funded by Trilogy, a
research project (ICT-216372) supported by the European Community
under its Seventh Framework Programme. The views expressed here are
those of the author only.
11. Comments Solicited
Comments and questions are encouraged and very welcome. They can be
addressed to the IETF Transport Area working group mailing list
<tsvwg@ietf.org>, and/or to the authors.
12. References
Briscoe Expires April 25, 2013 [Page 18]
Internet-Draft ECN Encapsulation Guidelines October 2012
12.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to
Indicate Requirement Levels", BCP 14,
RFC 2119, March 1997.
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black,
"The Addition of Explicit Congestion
Notification (ECN) to IP", RFC 3168,
September 2001.
[RFC3819] Karn, P., Bormann, C., Fairhurst, G.,
Grossman, D., Ludwig, R., Mahdavi, J.,
Montenegro, G., Touch, J., and L. Wood,
"Advice for Internet Subnetwork Designers",
BCP 89, RFC 3819, July 2004.
[RFC4774] Floyd, S., "Specifying Alternate Semantics
for the Explicit Congestion Notification
(ECN) Field", BCP 124, RFC 4774,
November 2006.
12.2. Informative References
[ATM-TM-ABR] Cisco, "Understanding the Available Bit Rate
(ABR) Service Category for ATM VCs", Design
Technote 10415, June 2005.
[Buck00] Buckwalter, J., "Frame Relay: Technology and
Practice", Pub. Addison Wesley ISBN-13: 978-
0201485240, 2000.
[DCTCP] Alizadeh, M., Greenberg, A., Maltz, D.,
Padhye, J., Patel, P., Prabhakar, B.,
Sengupta, S., and M. Sridharan, "Data Center
TCP (DCTCP)", ACM SIGCOMM CCR 40(4)63--74,
October 2010, <http://portal.acm.org/
citation.cfm?id=1851192>.
[GTPv1] 3GPP, "GPRS Tunnelling Protocol (GTP) across
the Gn and Gp interface", Technical
Specification TS 29.060.
[GTPv1-U] 3GPP, "General Packet Radio System (GPRS)
Tunnelling Protocol User Plane (GTPv1-U)",
Technical Specification TS 29.281.
[GTPv2-C] 3GPP, "Evolved General Packet Radio Service
Briscoe Expires April 25, 2013 [Page 19]
Internet-Draft ECN Encapsulation Guidelines October 2012
(GPRS) Tunnelling Protocol for Control plane
(GTPv2-C)", Technical Specification TS
29.274.
[IEEE802.1Qah] IEEE, "IEEE Standard for Local and
Metropolitan Area Networks--Virtual Bridged
Local Area Networks--Amendment 6: Provider
Backbone Bridges", IEEE Std 802.1Qah-2008,
August 2008, <http://www.ieee802.org/1/
pages/802.1ah.html>.
(Access Controlled link within page)
[IEEE802.1Qau] Finn, N., Ed., "IEEE Standard for Local and
Metropolitan Area Networks--Virtual Bridged
Local Area Networks - Amendment 13:
Congestion Notification", IEEE Std 802.1Qau-
2010, March 2010, <http://
ieeexplore.ieee.org/xpl/
mostRecentIssue.jsp?punumber=5454061>.
(Access Controlled link within page)
[ITU-T.I.371] ITU-T, "Traffic Control and Congestion
Control in B-ISDN", ITU-T Rec. I.371
(03/04), March 2004.
[RFC1701] Hanks, S., Li, T., Farinacci, D., and P.
Traina, "Generic Routing Encapsulation
(GRE)", RFC 1701, October 1994.
[RFC2003] Perkins, C., "IP Encapsulation within IP",
RFC 2003, October 1996.
[RFC2637] Hamzeh, K., Pall, G., Verthein, W., Taarud,
J., Little, W., and G. Zorn, "Point-to-Point
Tunneling Protocol", RFC 2637, July 1999.
[RFC2661] Townsley, W., Valencia, A., Rubens, A.,
Pall, G., Zorn, G., and B. Palter, "Layer
Two Tunneling Protocol "L2TP"", RFC 2661,
August 1999.
[RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D.,
and P. Traina, "Generic Routing
Encapsulation (GRE)", RFC 2784, March 2000.
[RFC2884] Hadi Salim, J. and U. Ahmed, "Performance
Briscoe Expires April 25, 2013 [Page 20]
Internet-Draft ECN Encapsulation Guidelines October 2012
Evaluation of Explicit Congestion
Notification (ECN) in IP Networks",
RFC 2884, July 2000.
[RFC4301] Kent, S. and K. Seo, "Security Architecture
for the Internet Protocol", RFC 4301,
December 2005.
[RFC5129] Davie, B., Briscoe, B., and J. Tay,
"Explicit Congestion Marking in MPLS",
RFC 5129, January 2008.
[RFC6040] Briscoe, B., "Tunnelling of Explicit
Congestion Notification", RFC 6040,
November 2010.
[RFC6633] Gont, F., "Deprecation of ICMP Source Quench
Messages", RFC 6633, May 2012.
[RFC6660] Briscoe, B., Moncaster, T., and M. Menth,
"Encoding Three Pre-Congestion Notification
(PCN) States in the IP Header Using a Single
Diffserv Codepoint (DSCP)", RFC 6660,
July 2012.
[trill-rbridge-options] Eastlake, D., Ghanwani, A., Manral, V., and
C. Bestler, "RBridges: Further TRILL Header
Extensions",
draft-ietf-trill-rbridge-options-07 (work in
progress), June 2012.
Appendix A. Outstanding Document Issues
1. [GF] Concern that certain guidelines warrant a MUST (NOT) rather
than a SHOULD (NOT). Given the guidelines say that if any SHOULD
(NOT)s are not followed, a strong justification will be needed,
they have been left as SHOULD (NOT) pending further list
discussion. In particular:
* If inner is a Not-ECN-PDU and Outer is CE (or highest severity
congestion level), MUST (not SHOULD) drop?
2. [GF] Impact of Diffserv on alternate marking schemes (referring
to RFC3168, RFC4774 & RFC2983)
3. Security Considerations
Briscoe Expires April 25, 2013 [Page 21]
Internet-Draft ECN Encapsulation Guidelines October 2012
Appendix B. Changes in This Version (to be removed by RFC Editor)
From briscoe-00 to 00:
* Intended status: BCP (was Informational) & updates 3819 added.
* Briefer Introduction: Introductory para justifying benefits of
ECN. Moved all but a brief enumeration of modes of operation
to their own new section (from both Intro & Scope). Introduced
incr. deployment as most tricky part.
* Tightened & added to terminology section
* Structured with Modes of Operation, then Guidelines section for
each mode.
* Tightened up guideline text to remove vagueness / passive voice
/ ambiguity and highlight main guidelines as numbered items.
* Added Outstanding Document Issues Appendix
* Updated references
Author's Address
Bob Briscoe
BT
B54/77, Adastral Park
Martlesham Heath
Ipswich IP5 3RE
UK
Phone: +44 1473 645196
EMail: bob.briscoe@bt.com
URI: http://bobbriscoe.net/
Briscoe Expires April 25, 2013 [Page 22]
| PAFTECH AB 2003-2026 | 2026-04-22 07:47:53 |