One document matched: draft-ietf-aqm-eval-guidelines-00.xml
<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?xml-stylesheet type='text/xsl' href='rfc2629.xslt' ?>
<?rfc strict="yes" ?>
<?rfc toc="yes"?>
<?rfc tocdepth="4"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes" ?>
<?rfc compact="yes" ?>
<?rfc subcompact="no" ?>
<rfc
category="info"
docName="draft-ietf-aqm-eval-guidelines-00"
ipr="trust200902">
<!-- category values: std, bcp, info, exp, and historic
ipr values: full3667, noModification3667, noDerivatives3667
you can add the attributes updates="NNNN" and obsoletes="NNNN"
they will automatically be output with "(if approved)" -->
<!-- ***** FRONT MATTER ***** -->
<front>
<!-- The abbreviated title is used in the page header - it is only necessary if the
full title is longer than 39 characters -->
<title abbrev="AQM Characterization Guidelines">AQM Characterization Guidelines</title>
<author role="editor" fullname="Nicolas Kuhn" initials="N." surname="Kuhn">
<organization>Telecom Bretagne</organization>
<address>
<postal>
<street>2 rue de la Chataigneraie</street>
<city>Cesson-Sevigne</city>
<region></region>
<code>35510</code>
<country>France</country>
</postal>
<phone>+33 2 99 12 70 46</phone>
<email>nicolas.kuhn@telecom-bretagne.eu</email>
</address>
</author>
<author role="editor" fullname="Preethi Natarajan" initials="P." surname="Natarajan">
<organization>Cisco Systems</organization>
<address>
<postal>
<street>510 McCarthy Blvd</street>
<city>Milpitas</city>
<region>California</region>
<code></code>
<country>United States</country>
</postal>
<phone></phone>
<email>prenatar@cisco.com</email>
</address>
</author>
<author fullname="David Ros" initials="D." surname="Ros">
<organization>Simula Research Laboratory AS</organization>
<address>
<postal>
<street>P.O. Box 134</street>
<city>Lysaker, 1325</city>
<region></region>
<code></code>
<country>Norway</country>
</postal>
<phone>+33 299 25 21 21</phone>
<email>dros@simula.no</email>
</address>
</author>
<author fullname="Naeem Khademi" initials="N." surname="Khademi">
<organization>University of Oslo</organization>
<address>
<postal>
<street>Department of Informatics, PO Box 1080 Blindern</street>
<city>N-0316 Oslo</city>
<region></region>
<code></code>
<country>Norway</country>
</postal>
<phone>+47 2285 24 93</phone>
<email>naeemk@ifi.uio.no</email>
</address>
</author>
<date month="September" year="2014" />
<!-- If the month and year are both specified and are the current ones, xml2rfc will fill
in the current day for you. If only the current year is specified, xml2rfc will fill
in the current day and month for you. If the year is not the current one, it is
necessary to specify at least a month (xml2rfc assumes day="1" if not specified for the
purpose of calculating the expiry date). With drafts it is normally sufficient to
specify just the year. -->
<!-- Meta-data Declarations -->
<area>Transport</area>
<workgroup>Internet Engineering Task Force</workgroup>
<!-- WG name at the upperleft corner of the doc,
IETF is fine for individual submissions.
If this element is not present, the default is "Network Working Group",
which is used by the RFC Editor as a nod to the history of the IETF. -->
<keyword>template</keyword>
<!-- Keywords will be incorporated into HTML output
files in a meta tag but they have no effect on text or nroff
output. If you submit your draft to the RFC Editor, the
keywords will be used for the search engine. -->
<!-- ######################################################-->
<!-- ######################################################-->
<!-- Head of the document -->
<!-- ######################################################-->
<!-- ######################################################-->
<abstract>
<t>Unmanaged large buffers in today's networks have given rise to a slew of performance issues. These performance issues can be addressed by some form of Active Queue Management (AQM), optionally in combination with a packet scheduling scheme such as fair queuing. The IETF AQM and packet scheduling working group was formed to standardize AQM schemes that are robust, easily implemented, and successfully deployed in today's networks. This document describes various criteria for performing precautionary characterizations of AQM proposals. This document also helps in ascertaining whether any given AQM proposal should be taken up for standardization by the AQM WG.</t>
</abstract>
</front>
<middle>
<section anchor="sec:introduction" title="Introduction">
<t>Active Queue Management (AQM) addresses the concerns arising from using unnecessarily large and unmanaged buffers in order to improve network and application performance. Several AQM algorithms have been proposed in the past years, most notable being Random Early Detection (RED), BLUE, and Proportional Integral controller (PI), and more recently CoDel <xref target="CODEL"></xref> and PIE <xref target="PIE"></xref>. In general, these algorithms actively interact with the Transmission Control Protocol (TCP) and any other transport protocol that deploys a congestion control scheme to manage the amount of data they keep in the network. The available buffer space in the routers and switches is large enough to accommodate the short-term buffering requirements. AQM schemes aim at reducing mean buffer occupancy, and therefore both end-to-end delay and jitter. Some of these algorithms, notably RED, have also been widely implemented in some network devices. However, any potential benefits of the RED AQM scheme have not been realized since RED is reported to be usually turned off. The main reason of this reluctance to use RED in today's deployments is its sensitivity to the operating conditions in the network and the difficulty of tuning its parameters.</t>
<t>A buffer is a physical volume of memory in which a queue or set of queues are stored. In real implementations of switches, a global memory is shared between the available devices: the size of the buffer for a given communication does not make sense, as its dedicated memory may vary over the time and real world buffering architectures are complex. For the sake of simplicity, when speaking of a specific queue in this document, "buffer size" refers to the maximum amount of data the buffer may store, which may be measured in bytes or packets. The rest of this memo therefore refers to the maximum queue depth as the size of the buffer for a given communication.</t>
<t>In order to meet mostly throughput-based SLA requirements and to avoid packet drops, many home gateway manufacturers resort to increasing the available memory beyond "reasonable values". This increase is also referred to as Bufferbloat <xref target="BB2011"></xref>. Deploying large unmanaged buffers on the Internet has lead to the increase in end-to-end delay, resulting in poor performance for latency sensitive applications such as real-time multimedia (e.g., voice, video, gaming, etc.). The degree to which this affects modern networking equipment, especially consumer-grade equipment, produces problems even with commonly used web services. Active queue management is thus essential to control queuing delay and decrease network latency.</t>
<t>The AQM and Packet Scheduling working group was recently formed within the TSV area to address the problems with large unmanaged buffers in the Internet. Specifically, the AQM WG is tasked with standardizing AQM schemes that not only address concerns with such buffers, but also that are robust under a wide variety of operating conditions. In order to ascertain whether the WG should undertake standardizing an AQM proposal, the WG requires guidelines for assessing AQM proposals. This document provides the necessary characterization guidelines.</t>
<section anchor="subsec:intro_guidelines" title="Guidelines for AQM designers">
<t>One of the key objectives behind formulating the guidelines is to help ascertain whether a specific AQM is not only better than drop-tail but also safe to deploy.<!--Thus, the evaluation of AQM performance can be divided into two categories: (1)--> The guidelines help to quantify AQM schemes' performance in terms of latency reduction, goodput maximization and the trade-off between the two. The guidelines also help to discuss AQM's safe deployment, including self adaptation, stability analysis, fairness, design/implementation complexity and robustness to different operating conditions.</t>
<t>This memo details generic characterization scenarios that any AQM proposal MUST be evaluated against. Irrespective of whether or not an AQM is standardized by the WG, we recommend the relevant scenarios and metrics discussed in this document to be considered. This document presents central aspects of an AQM algorithm that MUST be considered whatever the context is, such as burst absorption capacity, RTT fairness or resilience to fluctuating network conditions. These guidelines could not cover every possible aspect of a particular algorithm. In addition, it is worth noting that the proposed criteria are not bound to a particular evaluation toolset. These guidelines do not present context dependent scenarios (such as Wi-Fi, data-centers or rural broadband).</t>
<!-- Therefore, this document considers two different categories of evaluation scenarios: (1) generic scenarios that any AQM proposal SHOULD be evaluated against, and (2) evaluation scenarios specific to a network environment. Irrespective of whether or not an AQM is standardized by the WG, we recommend the relevant scenarios and metrics discussed in this document to be considered. Since a specific AQM scheme MAY NOT be applicable to all network environments, the specific evaluation scenarios enable to establish the environments where the AQM is applicable. These guidelines do not present every possible scenario and cannot cover every possible aspect of a particular algorithm. In addition, it is worth noting that the proposed criteria are not bound to a particular evaluation toolset. </t> -->
<t>This document details how an AQM designer can rate the feasibility of their proposal in different types of network devices (switches, routers, firewalls, hosts, drivers, etc.) where an AQM may be implemented.</t>
</section>
<section anchor="subsec:intro_tradeoff" title="Reducing the latency and maximizing the goodput">
<t>The trade-off between reducing the latency and maximizing the goodput is intrinsically linked to each AQM scheme and is key to evaluating its performance. This trade-off MUST be considered in various scenarios to ensure the safety of an AQM deployment. Whenever possible, solutions should aim at both maximizing goodput and minimizing latency. This document proposes guidelines that enable the reader to quantify (1) reduction of latency, (2) maximization of goodput and (3) the trade-off between the two.</t>
<t>Testers SHOULD discuss in a reference document the performance of their proposal in terms of performance and deployment in regards with those of drop-tail: basically, these guidelines provide the tools to understand the deployment costs versus the potential gain in performance of the introduction of the proposed scheme.</t>
</section>
<section anchor="subsec:intro_glossary" title="Glossary">
<t><list style="symbols">
<t>AQM: there may be confusion whether a scheduling scheme is added to an AQM or is a part of the AQM. The rest of this memo refers to AQM as a dropping policy that does not feature a scheduling scheme.</t>
<t>buffer: a physical volume of memory in which a queue or set of queues are stored.</t>
<t>buffer size: the maximum amount of data that may be stored in a buffer, measured in bytes or packets.</t>
</list></t>
</section>
<section anchor="subsec:intro_requi" title="Requirements Language">
<t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in <xref target="RFC2119">RFC 2119</xref>.</t>
</section>
</section>
<!-- ######################################################-->
<!-- ######################################################-->
<!-- Body of the document -->
<!-- ######################################################-->
<!-- ######################################################-->
<!-- ######################################################-->
<!-- New section -->
<!-- ######################################################-->
<section anchor="sec:e2e_metrics" title="End-to-end metrics">
<t>End-to-end delay is the result of propagation delay, serialization delay, service delay in a switch, medium-access delay and queuing delay, summed over the network elements in the path. AQM algorithms may reduce the queuing delay by providing signals to the sender on the emergence of congestion, but any impact on the goodput must be carefully considered. This section presents the metrics that COULD be used to better quantify (1) the reduction of latency, (2) maximization of goodput and (3) the trade-off between the two. These metrics SHOULD be considered to better assess the performance of an AQM scheme.</t>
<t>The metrics listed in this section are not necessarily suited to every type of traffic detailed in the rest of this document. It is therefore NOT REQUIRED to measure all of following metrics.</t>
<section anchor="subsec:e2e_metrics_complet_time" title="Flow Completion time">
<t>The flow completion time is an important performance metric for the end user. Considering the fact that an AQM scheme may drop packets, the flow completion time is directly linked to the dropping policy of the AQM scheme. This metric helps to better assess the performance of an AQM depending on the flow size.</t>
<!-- <t>To illustrate this metric: the x-axis show the size of the flow and the y-axis the flow completion time.</t> -->
</section>
<section anchor="subsec:e2e_metrics_loss" title="Packet loss">
<t>Packet losses, that may occur in a queue, impact on the end-to-end performance at the receiver's side.</t>
<t>The tester MUST evaluate, at the receiver:</t>
<t><list style="symbols">
<t>the packet loss probability: this metric should be frequently measured during the experiment as the long term loss probability is of interests for steady state scenarios only;</t>
<t>the interval between consecutive losses: the time between two losses should be measured. From the set of interval times, the tester should present the median value, the minimum and maximum values and the 10th and 90th percentiles.</t>
<!-- <t>the packet loss pattern.</t> -->
</list></t>
<!--<t>The guidelines advice that the tester SHOULD determine the minimum, average and maximum measurements of these metrics and the coefficient of variation for the average value as well.</t>-->
</section>
<section anchor="subsec:e2e_metrics_synch_loss" title="Packet loss synchronization">
<t>One goal of an AQM algorithm should be to help with avoiding global synchronization of flows going through the bottleneck buffer on which the AQM operates (<xref target="RFC2309"> </xref>). It is therefore important to assess the "degree" of packet-loss synchronization between flows, with and without the AQM under consideration.</t>
<t>As discussed e.g. in <xref target="LOSS-SYNCH-MET-08"></xref>, loss synchronization among flows may be quantified by several, slightly different, metrics that capture different aspects of the same issue. However, in real-world measurements the choice of metric may be imposed by practical considerations (e.g., is there fine-grained information on packet losses in the bottleneck available or not). For the purpose of AQM characterization, a good candidate metric is the global synchronization ratio, measuring the proportion of flows losing packets during a loss event. <xref target="YU06"></xref> used this metric in real-world experiments to characterize synchronization along arbitrary Internet paths; the full methodology is described in [YU06].</t>
<!--
<t>With the introduction of AQM schemes, the packet loss synchronization can be reduced. This is one original goal of AQMs, as explained in <xref target="RFC2309"> </xref>.</t>
<t>The synchronization ratio is defined as the degree of synchronization of loss events between two TCP flows on the same path: this metric is determined largely by the traffic mix on the congested link and by the AQM mechanism introduced <xref target="IRTF-TOOLS-5"></xref>.</t>
<t>The overall synchronization ratio (Sij) is defined for two flows i and j that lose packets in the same time slot. Sij=max(Si_j,Sj_i), where Sk_n denotes the fraction of loss events of flow k in which flow n (!=k) also suffers packet loss.</t>
<t>More details on the other metrics that can evaluate the packet loss synchronization can be found in <xref target="LOSS-SYNCH-MET-08"></xref>.</t>
-->
<!--
<t>It is important to evaluate this metric in order to check whether an AQM mechanism fairly drops packets of two flows or not. The introduction of AQM impacts on this metric has already been measured in <xref target="LOSS-SYNCH-AQM-08"></xref> and should be considered while evaluating an AQM proposal.</t>
<t>These guidelines propose to quantify the loss synchronization by the utilization of three possible metrics:</t>
<t><list style="symbols">
<t>overall synchronization ratio (Sij): this metric is defined for two flows i and j that lose packets in the same time slot. Sij=max(Si_j,Sj_i), where Sk_n denotes the fraction of loss events of flow k in which flow n (!=k) also suffers packet loss.</t>
<t>synchronization rate (Li): proportion of the total loss events at which i sees a packet loss.</t>
<t>global synchronization rate (Rl): proportion of flows losing packets during loss event l.</t>
</list></t>-->
</section>
<section anchor="subsec:e2e_metrics_goodput" title="Goodput">
<t>Measuring the goodput enables an end-to-end appreciation of how well the AQM improves transport and application performance. The measured end-to-end goodput is linked to the AQM scheme's dropping policy -- the smaller the packet drops, the fewer packets need retransmission, minimizing AQM's impact on transport and application performance. End-to-end goodput values help evaluate the AQM scheme's effectiveness in minimizing packet drops that impact application performance.</t>
<!-- Additionally, an AQM scheme may resort to Explicit Congestion Notification (ECN) marking as an initial means to control delay. Again, marking packets instead of dropping them reduces number of packet retransmissions and increases goodput. Overall, end-to-end goodput values help evaluate the AQM scheme's effectiveness in minimizing packet drops that impact application performance and estimate how well the AQM scheme works with ECN. </t> -->
<!-- <t>If scheduling comes into play, a measure of how individual queues are serviced may be necessary: the scheduling introduced on top of the AQM may starve some flows and boost others. The utilization of the link does not cover this, as the utilization would be the same, whereas the goodput lets the tester see if some flows are starved or not.</t> -->
<!--<t>The guidelines advice that the tester SHOULD determine the minimum, average and maximum measurements of the goodput and the coefficient of variation for the average value as well.</t>-->
<t>The measurement of the goodput let the tester evaluate to which extend the AQM is able to keep an high link utilization. This metric should be obtained frequently during the experiment: the long term goodput makes sense for steady-state scenarios only and may not reflect how the introduction of AQM actually impacts on the link utilization. It is worth pointing out that the fluctuations of this measurement may depend on other things than the introduction of an AQM, such as physical layer losses, fluctuating bandwidths (Wi-Fi), heavy congestion levels or transport layer congestion controls.</t>
</section>
<section anchor="subsec:e2e_metrics_latency" title="Latency and jitter">
<t>The end-to-end latency differs from the queuing delay: it is linked to the network topology and the path characteristics. Moreover, the jitter strongly depends on the traffic and the topology as well. The introduction of an AQM scheme would impact on these metrics and the end-to-end evaluation of performance SHOULD consider them to better assess the AQM schemes.</t>
<t>The guidelines advice that the tester SHOULD determine the minimum, average and maximum measurements for these metrics and the coefficient of variation for their average values as well.</t>
</section>
<section anchor="subsec:e2e_metrics_tradeoff" title="Discussion on the trade-off between latency and goodput">
<t>The metrics presented in this section MAY be considered, in order to discuss and quantify the trade-off between latency and goodput.</t>
<t>This trade-off can also be illustrated with figures following the recommendations of the section 5 of <xref target="TCPEVAL2013"></xref>. Each of the end-to-end delay and the goodput should be measured every second. From each of this sets of measurements, the 10th and 90th percentile and the median value should be computed. For each scenario, a graph can be generated, with the x-axis could show the end-to-end delay and the y-axis the goodput. This graph provides part of a better understanding (1) of the delay/goodput trade-off for a given congestion control mechanism, and (2) of how the goodput and average queue size vary as a function of the traffic load.</t>
<!-- <t>The end-to-end trade-off MUST be considered:</t>
<t><list style="symbols">
<t> end-to-end delay vs. goodput: the x-axis shows the end-to-end delay and the y-axis the average goodput;</t>
<t>drop rate vs. end-to-end delay: the x-axis shows the end-to-end delay and the y-axis the drop rate.</t>
</list></t>
<t>Each of the end-to-end delay, goodput and drop probability should be measured every second. From each of this sets of measurements, the 10th and 90th percentile and the median value should be computed. For each scenario case, an ellipse can be generated from the measurement of the percentiles and a point for the median value can be plotted.</t>
<t>This pair of graphs provide part of a better understanding (1) of the delay/goodput/drop-rate trade-off for a given congestion control mechanism, and (2) of how the goodput and average queue size vary as a function of the traffic load.</t> -->
</section>
</section>
<!-- ######################################################-->
<!-- New section -->
<!-- ######################################################-->
<section anchor="sec:discuss_setting" title="Generic set up for evaluations">
<t>This section presents the topology that can be used for each of the following scenarios, the corresponding notations and discuss various assumptions that have been made in the document.</t>
<section anchor="subsec:discuss_setting_topo_nota" title="Topology and notations">
<figure anchor="fig:topology" title="Topology and notations">
<artwork>
+---------+ +-----------+
|senders A| |receivers B|
+---------+ +-----------+
+--------------+ +--------------+
|traffic class1| |traffic class1|
|--------------| |--------------|
| SEN.Flow1.1 +---------+ +-----------+ REC.Flow1.1 |
| + | | | | + |
| | | | | | | |
| + | | | | + |
| SEN.Flow1.X +-----+ | | +--------+ REC.Flow1.X |
+--------------+ | | | | +--------------+
+ +-+---+---+ +--+--+---+ +
| |Router L | |Router R | |
| |---------| |---------| |
| | AQM | | | |
| | BuffSize| | | |
| | (Bsize) +-----+ | |
| +-----+--++ ++-+------+ |
+ | | | | +
+--------------+ | | | | +--------------+
|traffic classN| | | | | |traffic classN|
|--------------| | | | | |--------------|
| SEN.FlowN.1 +---------+ | | +-----------+ REC.FlowN.1 |
| + | | | | + |
| | | | | | | |
| + | | | | + |
| SEN.FlowN.Y +------------+ +-------------+ REC.FlowN.Y |
+--------------+ +--------------+
</artwork>
</figure>
<t><xref target="fig:topology"></xref> is a generic topology where:</t>
<t><list style="symbols">
<t>various classes of traffic can be introduced;</t>
<t>the timing of each flow (i.e., when does each flow start and stop) may be different;</t>
<t>each class of traffic can consider various number of flows;</t>
<t>each link is characterized by a couple (RTT,capacity);</t>
<t>Flows are generated between A and B, sharing a bottleneck (Routers L and R);</t>
<t>The links are supposed to be asymmetric in terms of bandwidth: the capacity from senders to receivers is higher than the one from receivers to senders.</t>
</list></t>
<t>This topology may not perfectly reflect actual topologies, however, this simple topology is commonly used in the world of simulations and small testbeds. This topology can be considered as adequate to evaluate AQM proposals, such as proposed in <xref target="TCPEVAL2013"></xref>. The tester should pay attention to the topology that has been used to evaluate the AQM scheme against which he compares his proposal.</t>
</section>
<section anchor="subsec:discuss_setting_buff_size" title="Buffer size">
<t>The size of the buffers MAY be carefully set considering the bandwidth-delay product. However, if the context or the application requires a specific buffer size, the tester MUST justify and detail the way the maximum queue depth is set while presenting the results of its evaluation. Indeed, the size of the buffer may impact on the AQM performance and is a dimensioning parameter that will be considered for a fair comparison between AQM proposals.</t>
</section>
<section anchor="subsec:discuss_setting_congestion_control" title="Congestion controls">
<t>This memo features three kind of congestion controls:</t>
<t><list style="symbols">
<t>TCP-friendly congestion controls: a base-line congestion control for this category is TCP New Reno, as explained in <xref target="RFC5681"> </xref>.</t>
<t>Aggressive congestion controls: a base-line congestion control for this category is TCP Cubic.</t>
<t>Less-than Best Effort (LBE) congestion controls: an LBE congestion control 'results in smaller bandwidth and/or delay impact on standard TCP than standard TCP itself, when sharing a bottleneck with it.' <xref target="RFC6297"> </xref></t>
</list></t>
<t>Recent transport layer protocols are not mentioned in the following sections, for the sake of simplicity.</t>
</section>
</section>
<!-- ######################################################-->
<!-- New section -->
<!-- ######################################################-->
<section anchor="sec:perf" title="Various TCP variants">
<!--<t>This section presents the set of scenarios that MUST be considered to evaluate the performance of an AQM scheme and quantify the trade-off between latency and goodput. For each selected scenario, the metrics presented in <xref target="sec:e2e_metrics"></xref> should be considered. While presenting the performance of an AQM algorithm for the selected scenarios, the tester MUST provide any parameter that had to be set beforehand. Moreover, the values for these parameters MUST be explained and justified as detailed in <xref target="subsec:stability_param_sensitivity"></xref>.</t>-->
<!--<t>The tester SHOULD compare its proposal's performance and deployment with those of drop-tail: basically, these guidelines provide the tools to understand the cost (in terms of deployment) versus the potential gain in performance of the introduction of the proposed scheme.</t>-->
<!--<t>This section does not present a large set of scenarios to evaluate the performance of an AQM in specific contexts, such as Wi-Fi, rural broadband or data-centers. These guidelines provide generic scenarios for performance evaluations that MUST be considered.</t>-->
<t>Network and end devices need to be configured with a reasonable amount of buffers in order to absorb transient bursts. In some situations, network providers configure devices with large buffers to avoid packet drops and increase goodput. Transmission Control Protocol (TCP) fills up these unmanaged buffers until the TCP sender receives a signal (packet drop) to cut down the sending rate. The larger the buffer, the higher the buffer occupancy, and therefore the queuing delay. On the other hand, an efficient AQM scheme sends out early congestion signals to TCP senders so that the queuing delay is brought under control.</t>
<t>Not all applications run over the same flavor of TCP. Variety of senders generate different classes of traffic which may not react to congestion signals (aka unresponsive flows) or may not cut down their sending rate as expected (aka aggressive flows): AQM schemes aim at maintaining the queuing delay under control, which is challenged if blasting traffics are present.</t>
<t>This section provides guidelines to assess the performance of an AQM proposal based on various metrics presented in <xref target="sec:e2e_metrics"></xref> irrespective of traffic profiles involved -- different senders (TCP variants, unresponsive, aggressive), traffic mix with different applications, etc.</t>
<!--
<section anchor="subsubsec:eval_generic_traff_profil_topo" title="Topology Description">
<t>The topology is presented in <xref target="fig:topology"></xref>. In this section, the capacities of the links MUST be set to 10Mbps and the RTT between the senders and the receivers to 100ms.</t>
</section>
-->
<section anchor="subsubsec:eval_generic_traff_profil_single_TCP" title="TCP-friendly Sender">
<t>This scenario helps to evaluate how an AQM scheme reacts to a TCP-friendly transport sender. A single long-lived, non application limited, TCP New Reno flow transmits data between sender A and receiver B.<!-- during 100s.--> Other TCP friendly congestion control schemes such as TCP-friendly rate control <xref target="RFC5348"> </xref> etc MAY also be considered.</t>
<!-- <t>For each TCP-friendly transport considered, the graphs described in <xref target="subsec:e2e_metrics_tradeoff"></xref> MUST be generated.</t> -->
<t>For each TCP-friendly transport considered, the graph described in <xref target="subsec:e2e_metrics_tradeoff"></xref> could be generated.</t>
<!--We expect that an AQM proposal exhibit similar behavior for all the TCP-friendly transports considered.</t>-->
</section>
<section anchor="subsubsec:eval_generic_traff_profil_aggress" title="Aggressive Transport Sender">
<t>This scenario helps to evaluate how an AQM scheme reacts to a transport sender whose sending rate is more aggressive than a single TCP-friendly sender. A single long-lived, non application limited, TCP Cubic flow transmits data between sender A and receiver B.<!-- during 100s--> Other aggressive congestion control schemes MAY also be considered. </t>
<!-- <t>For each flavor of aggressive transport, the graphs described in <xref target="subsec:e2e_metrics_tradeoff"></xref> MUST be generated.</t> -->
<t>For each flavor of aggressive transport, the graph described in <xref target="subsec:e2e_metrics_tradeoff"></xref> could be generated.</t>
</section>
<section anchor="subsubsec:eval_generic_traff_profil_unresp" title="Unresponsive Transport Sender">
<t>This scenario helps evaluate how an AQM scheme reacts to a transport sender who is not responsive to congestion signals (ECN marks and/or packet drops) from the AQM scheme. Note that faulty transport implementations on end hosts and/or faulty network elements en-route that "hide" congestion signals in packet headers <xref target="I-D.ietf-aqm-recommendation"></xref> may also lead to a similar situation, such that the AQM scheme needs to adapt to unresponsive traffic. To this end, these guidelines propose the two following scenarios.</t>
<t>The first scenario is the following. In order to create a test environment that results in queue build up, we consider unresponsive flow(s) whose sending rate is greater than the bottleneck link capacity between routers L and R. This scenario consists of a long-lived non application limited UDP flow transmits data <!--with an aggregate rate of 12Mbps--> between sender A and receiver B.<!--during 100s.--> Graphs described in <xref target="subsec:e2e_metrics_tradeoff"></xref> <!--MUST--> could be generated.</t>
<t>The second scenario is the following. In order to test to which extend the AQM scheme is able to keep responsive fraction under control, this scenario considers a mixture of TCP-friendly and unresponsive traffics. This scenario consists of a long-lived non application limited UDP flow and a single long-lived, non application limited, TCP New Reno flow that transmit data between sender A and receiver B. As opposed to the first scenario, the rate of the UDP traffic should not be greater than the bottleneck capacity, and should not be higher than half of the bottleneck capacity. For each type of traffic, the graph described in <xref target="subsec:e2e_metrics_tradeoff"></xref> COULD be generated.</t>
</section>
<section anchor="subsubsec:eval_generic_traff_profil_init_cwnd" title="TCP initial congestion window">
<t>This scenario helps evaluate how an AQM scheme adapts to a traffic mix consisting of TCP flows with different values for the initial congestion window (IW).</t>
<!-- <t><list style="symbols">
<t>TCP: Cubic and/or New Reno;</t>
<t>IW: 3 or 10 packets.</t>
</list></t> -->
<t>For this scenario, we consider two types of flow that MUST be generated between sender A and receiver B:</t>
<t><list style="symbols">
<t>a single long-lived non application limited TCP New Reno flow;</t>
<t>a single long-lived application limited TCP New Reno flow, with an IW set to 3 or 10 packets. The size of the data transmitted MUST be strictly higher than 10 packets and should be lower than 100 packets.</t>
</list></t>
<t>The transmission of both flows must not start simultaneously: a steady state must be achieved before the transmission of the application limited flow. As a result, the transmission of the non application limited flow MUST start before the transmission of the application limited flow.</t>
<t>For each of these scenarios, the graph described in <xref target="subsec:e2e_metrics_tradeoff"></xref> could be generated for each class of traffic. The completion time of the application limited TCP flow could be measured.</t>
</section>
<section anchor="subsubsec:eval_generic_traff_profil_mix" title="Traffic Mix">
<t>This scenario helps to evaluate how an AQM scheme reacts to a traffic mix consisting of different applications such as bulk transfer, web, voice, video traffic. These testing cases presented in this subsection have been inspired by the table 2 of <xref target="DOCSIS2013"></xref>:</t>
<t><list style="symbols">
<t>Bulk TCP transfer</t> <!--: (continuous file transmission (the tester may consider an LBE congestion control), or repeating 5MB file transmission);</t>-->
<t>Web traffic</t> <!--(repeated download of 700kB);</t>-->
<t>VoIP</t> <!-- (each of them 87kbps UDP stream);</t>-->
<t>Constant bit rate UDP traffic</t> <!-- (1Mbps UDP flow);</t>-->
<t>Adaptive video streaming</t> <!-- (2Mb/s and 4s chunks (1MB file size for each chunk), chunks can be sent at 4s intervals and their size may vary with standard deviation);</t>-->
</list></t>
<t><xref target="fig:traffic_mix"></xref> presents the various cases for the traffic that MUST be generated between sender A and receiver B.</t>
<figure anchor="fig:traffic_mix" title="Traffic Mix scenarios">
<artwork>
+----+-----------------------------+
|Case| Number of flows |
+ +----+----+----+---------+----+
| |VoIP|Webs|CBR |AdaptVid |FTP |
+----+----+----+----+---------+----+
|I | 1 | 1 | 0 | 0 | 0 |
| | | | | | |
|II | 1 | 1 | 0 | 0 | 1 |
| | | | | | |
|III | 1 | 1 | 0 | 0 | 5 |
| | | | | | |
|IV | 1 | 1 | 1 | 0 | 5 |
| | | | | | |
|V | 1 | 1 | 0 | 1 | 5 |
| | | | | | |
+----+----+----+----+---------+----+
</artwork>
</figure>
<t>For each of these scenarios, the graph described in <xref target="subsec:e2e_metrics_tradeoff"></xref> could be generated for each class of traffic. In addition, other metrics such as end-to-end latency, jitter and flow completion time MUST be generated.</t>
</section>
</section>
<!-- ######################################################-->
<!-- New section -->
<!-- ######################################################-->
<section anchor="sec:rtt_fairness" title="RTT fairness">
<section anchor="subsec:rtt_fairness_motivation" title="Motivation">
<t>The capability of AQM schemes to control the queuing delay highly depends on the way end-to-end protocols react to congestion signals. When the RTT varies, the behaviour of congestion controls is impacted and so the capability of AQM schemes to control the queue. It is therefore important to assess the AQM schemes against a set of RTTs (e.g., from 5 ms to 200 ms).</t>
<t>Also, asymmetry in terms of RTT between various paths SHOULD be considered so that the fairness between the flows can be discussed as one may react faster to congestion than another. The introduction of AQM schemes <!-- (featuring a scheduling scheme or not) --> may improve this fairness.</t>
<t>Moreover, introducing an AQM <!-- with/without a scheduling --> scheme may result in the absence of fairness between the flows, even when the RTTs are identical. This potential lack of fairness SHOULD be evaluated.</t>
</section>
<section anchor="subsec:rtt_fairness_tests" title="Required tests">
<t>The topology that SHOULD be used is detailed in <xref target="fig:topology"></xref>:</t>
<t><list style="symbols">
<t>to evaluate the inter-RTT fairness, for each run, ten flows divided into two categories. Category I (Flow1.1, ..., Flow1.5) which RTT between sender A and Router L SHOULD be 5ms. Category II (Flow2.1, ..., Flow 2.5) which RTT between sender A and Router L SHOULD be in [5ms;200ms].</t>
<t>to evaluate the impact of the RTT value on the AQM performance and the intra-protocol fairness, for each run, ten flows (Flow1.1, ..., Flow1.5 and Flow2.1, ..., Flow2.5) SHOULD be introduced. For each experiment, the set of RTT SHOULD be the same for all the flows and in [5ms;200ms].</t>
</list></t>
<t>These flows MUST use the same congestion control algorithm.</t>
</section>
<section anchor="subsubsec:rtt_fariness_metrics" title="Metrics to evaluate the RTT fairness">
<t>The output that MUST be measured is:</t>
<t><list style="symbols">
<t>for the inter-RTT fairness: (1) the cumulated average goodput of the flows from Category I, goodput_Cat_I (<xref target="subsec:e2e_metrics_goodput"></xref>); (2) the cumulated average goodput of the flows from Category II, goodput_Cat_II (<xref target="subsec:e2e_metrics_goodput"></xref>); (3) the ratio goodput_Cat_II/goodput_Cat_I; (4) the average packet drop rate for each category (<xref target="subsec:e2e_metrics_loss"></xref>).</t>
<t>for the intra-protocol RTT fairness: (1) the cumulated averga goodput of the ten flows (<xref target="subsec:e2e_metrics_goodput"></xref>); (2) the average packet drop rate for the ten flows(<xref target="subsec:e2e_metrics_loss"></xref>).</t>
</list></t>
</section>
</section>
<!-- ######################################################-->
<!-- New section -->
<!-- ######################################################-->
<section anchor="sec:burst_absorption" title="Burst absorption">
<section anchor="subsec:burst_absorption_motivation" title="Motivation">
<t>Packet arrivals can be bursty due to various reasons. Dropping one or more packets from a burst may result in performance penalties for the corresponding flows since the dropped packets have to be retransmitted. Performance penalties may turn into unmet SLAs and be disincentives to AQM adoption. Therefore, an AQM scheme SHOULD be designed to accommodate transient bursts. AQM schemes do not present the same tolerance to bursts of packets arriving in the buffer: this tolerance MUST be quantified.</t>
<t>Note that accommodating bursts translates to higher queue length and queuing delay. Naturally, it is important that the AQM scheme brings bursty traffic under control quickly. On the other hand, spiking packet drops in order to bring packet bursts quickly under control could result in multiple drops per flow and severely impact transport and application performance. Therefore, an AQM scheme SHOULD bring bursts under control by balancing both aspects -- (1) queuing delay spikes are minimized and (2) performance penalties for ongoing flows in terms of packet drops are minimized.</t>
<t>An AQM scheme maintains short queues to allow the remaining space in the queue for bursts of packets. The tolerance to bursts of packets depends on the number of packets in the queue, which is directly linked to the AQM algorithm. Moreover, one AQM scheme may implement a feature controlling the maximum size of accepted bursts, that may depend on the buffer occupancy or the currently estimated queuing delay. Also, the impact of the buffer size on the burst allowance MAY be evaluated.</t>
</section>
<section anchor="subsec:burst_absorption_tests" title="Required tests">
<!-- <t>The topology is presented in <xref target="fig:topology"></xref>. For this scenario, the capacities of the links MUST be set to 10Mbps and the RTT between senders and receivers to 100ms.</t> -->
<!--
<t>The required tests presented in this section can be divided into two scenarios: generic bursty traffic and realistic bursty traffic. One of this scenario MUST be considered.</t>
<section anchor="subsubsec:burst_absorption_tests_generic_burst" title="Generic bursty traffic">
<t>For this scenario, the three following traffic MUST be generated from sender A to receiver B in parallel:</t>
<t><list style="symbols">
<t>One Constant bit rate UDP traffic: 1Mbps UDP flow;</t>
<t>One TCP bulk transfer: repeating 5MB file transmission;</t>
<t>Burst of packets: size of the burst from 5 to MAX_BUFFER_SIZE packets.</t>
</list></t>
</section>
<section anchor="subsubsec:burst_absorption_tests_realistic_bursty" title="Realistic bursty traffic">
-->
<t>For this scenario, the following traffic MUST be generated from sender A to receiver B:</t>
<t><list style="symbols">
<t>IW10: TCP transfer with initial congestion window set to 10 of 5MB;</t>
<t>Bursty video frames;</t>
<t>Web traffic;</t>
<t>Constant bit rate UDP traffic.</t>
</list></t>
<t><xref target="fig:burst_traffic"></xref> presents the various cases for the traffic that MUST be generated between sender A and receiver B.</t>
<figure anchor="fig:burst_traffic" title="Bursty traffic scenarios">
<artwork>
+-----------------------------------------+
|Case| Number of traffic |
| +-----+----+----+--------------------+
| |Video|Webs| CBR| Bulk Traffic (IW10)|
+----|-----|----|----|--------------------|
|I | 0 | 1 | 1 | 0 |
|----|-----|----|----|--------------------|
|II | 0 | 1 | 1 | 1 |
|----|-----|----|----|--------------------|
|III | 1 | 1 | 0 | 0 |
+----|-----|----|----|--------------------|
|IV | 1 | 1 | 1 | 0 |
+----|-----|----|----|--------------------|
|V | 1 | 1 | 1 | 1 |
+----+-----+----+----+--------------------+
</artwork>
</figure>
<!-- <section anchor="subsubsec:burst_absorption_tests_metrics" title="Metrics to evaluate the burst absorption capacity"> -->
<t>For each of these scenarios, the graph described in <xref target="subsec:e2e_metrics_tradeoff"></xref> could be generated. In addition, other metrics such as end-to-end latency, jitter, flow completion time MUST be generated.</t>
<!--</section>-->
<!-- </section> -->
</section>
</section>
<!-- ######################################################-->
<!-- New section -->
<!-- ######################################################-->
<section anchor="sec:stability" title="Stability">
<section anchor="subsec:stability_motivation" title="Motivation">
<t>Network devices experience varying operating conditions depending on factors such as time of day, deployment scenario etc. For example:</t>
<t><list style="symbols">
<t>Traffic and congestion levels are higher during peak hours than off-peak hours.</t>
<t>In the presence of scheduler, a queue's draining rate may vary depending on other queues: a low load on a high priority queue implies higher draining rate for lower priority queues.</t>
<t>The available capacity on the physical layer may vary over time such as in the context of lossy channels.</t>
</list></t>
<t>Whether the target context is a not stable environment, the capability of an AQM scheme to actually maintain its control on the queuing delay and buffer occupancy is challenged. This document propose guidelines to assess the behaviour of AQM schemes under varying congestion levels and varying draining rates.</t>
</section>
<section anchor="subsec:stability_tests" title="Required tests">
<!-- <t>The topology is presented in <xref target="fig:topology"></xref>. For this scenario, the capacities of the links MUST be set to 10Mbps and the RTT between senders and receivers to 100ms.</t> -->
<section anchor="subsubsec:stability_tests_net_mild" title="Mild Congestion">
<t>This scenario helps to evaluate how an AQM scheme reacts to a light load of incoming traffic resulting in mild congestion -- packet drop rates less than 1%. <!-- The scenario consists of 4-5 TCP New Reno flows between sender A and receiver B. All TCP flows start at random times during the initial second of the experiment. -->Each single-lived non application limited TCP flow transfers data. <!-- during 100s.--></t>
<t>For this scenario, the graph described in <xref target="subsec:e2e_metrics_tradeoff"></xref> could be generated.</t>
</section>
<section anchor="subsubsec:stability_tests_net_medium" title="Medium Congestion">
<t>This scenario helps to evaluate how an AQM scheme reacts to incoming traffic resulting in medium congestion -- packet drop rates between 1%-3%. <!--The scenario consists of 10-20 TCP New Reno flows between sender A and receiver B. All TCP flows start at random times during the initial second of the experiment.-->Each single-lived non application limited TCP flow transfers data.<!-- during 100s.--></t>
<t>For this scenario, the graph described in <xref target="subsec:e2e_metrics_tradeoff"></xref> could be generated.</t>
</section>
<section anchor="subsubsec:stability_tests_net_heavy" title="Heavy Congestion">
<t>This scenario helps to evaluate how an AQM scheme reacts to incoming traffic resulting in heavy congestion -- packet drop rates between 5%-10%. <!--The scenario consists of 30-40 TCP New Reno flows between sender A and receiver B. All TCP flows start at random times during the initial second of the experiment. -->Each single lived non application limited TCP flow transfers data.<!-- during 100s.--></t>
<t>For this scenario, the graph described in <xref target="subsec:e2e_metrics_tradeoff"></xref> could be generated.</t>
</section>
<section anchor="subsubsec:stability_tests_net_varying" title="Varying congestion levels">
<t>This scenario helps to evaluate how an AQM scheme reacts to incoming traffic resulting in various level of congestions during the experiment. In this scenario, the congestion level varies according to a large time scale. The following phases may be considered: phase I - mild congestion during 0-5s; phase II - medium congestion during 5-10s; phase III - heavy congestion during 10-15s; phase I again, ... and so on. <!--The scenario consists of 30-40 TCP New Reno flows between sender A and receiver B. All TCP flows start at random times during the initial second of the experiment. -->Each single lived non application limited TCP flow transfers data.<!-- during 100s.--></t>
<t>For this scenario, the graph described in <xref target="subsec:e2e_metrics_tradeoff"></xref> could be generated. Moreover, one graph could be generated for each of the phases previously detailed.</t>
</section>
<section anchor="subsubsec:stability_tests_vary_dr_rate" title="Varying Available Bandwidth">
<t>This scenario helps evaluate how an AQM scheme adapts to varying available bandwidth on the outgoing link.</t>
<t>To simulate varying draining rates, the bottleneck bandwidth between nodes 'Router L' and 'Router R' varies over the course of the experiment as follows:</t>
<t><list style="symbols">
<t>Experiment 1: the capacity varies between two values according to a large time scale. As an example, the following phases may be considered: phase I - 100Mbps during 0-5s; phase II - 10Mbps during 5-10s: phase I again, ... and so on.</t>
<t>Experiment 2: the capacity varies between two values according to a short time scale. As an example, the following phases may be considered: phase I - 100Mbps during 100ms; phase II - 10Mbps during 100ms; phase I again during 100ms, ... and so on.</t>
</list></t>
<t>More realistic fluctuating bandwidth patterns MAY be considered.</t>
<t>The scenario consists of TCP New Reno flows between sender A and receiver B.<!-- All TCP flows start at random times during the initial second. Each TCP flow transfers a large file for a period of 150s. --> In order to better assess the impact of draining rates on the AQM behavior, the tester MUST compare its performance with those of drop-tail.</t>
<t> For this scenario, the graph described in <xref target="subsec:e2e_metrics_tradeoff"></xref> could be generated. Moreover, one graph SHOULD be generated for each of the phases previously detailed.</t>
</section>
</section>
<section anchor="subsec:stability_param_sensitivity" title="Parameter sensitivity and stability analysis">
<t>An AQM scheme's control law is the primary means by which the AQM controls queuing delay. Hence understanding the AQM control law is critical to understanding AQM behavior. The AQM's control law may include several input parameters whose values affect the AQM output behavior and stability. Additionally, AQM schemes may auto-tune parameter values in order to maintain stability under different network conditions (such as different congestion levels, draining rates or network environments). The stability of these auto-tuning techniques is also important to understand.</t>
<t>AQM proposals SHOULD provide background material showing control theoretic analysis of the AQM control law and the input parameter space within which the control law operates as expected; or could use other ways to discuss its stability. For parameters that are auto-tuned, the material SHOULD include stability analysis of the auto-tuning mechanism(s) as well. Such analysis helps to understand an AQM's <!-- and packet scheduling --> control law better and the network conditions/deployments under which the AQM is stable. </t>
<!--<t>The impact of every externally tuned parameter MUST be discussed. As an example, if an AQM proposal needs various external tuning to work on different scenarios, these external modifications MUST be clear for deployment issues. Also, the frequency at which some parameters are re-configured MUST be evaluated, as it may impact the capacity of the AQM to absorb incoming bursts of packets.</t>-->
</section>
</section>
<!-- ######################################################-->
<!-- New section -->
<!-- ######################################################-->
<section anchor="sec:imple_cost" title="Implementation cost">
<section anchor="subsec:imple_cost_motivation" title="Motivation">
<!-- NK: do we keep that section ?? It is difficult to have implementation cost evaluations in these guidelines: recommendations for evaluation guidelines ?-->
<t>An AQM's successful deployment is directly related to its ease of implementation. Network devices may need hardware or software implementations of the AQM. Depending on a device's capabilities and limitations, the device may or may not be able to implement some or all parts of the AQM logic.</t>
<t>AQM proposals SHOULD provide pseudo-code for the complete AQM scheme, highlighting generic implementation-specific aspects of the scheme such as "drop-tail" vs. "drop-head", inputs (current queuing delay, queue length), computations involved, need for timers etc. This helps identify costs associated with implementing the AQM on a particular hardware or software device. Also, it helps the WG understand which kind of devices can easily support the AQM <!-- and packet scheduling --> and which cannot.</t>
</section>
<section anchor="subsec:imple_cost_tests" title="Required discussion">
<t>AQM proposals SHOULD highlight parts of AQM logic that are device dependent and discuss if and how AQM behavior could be impacted by the device. For example, a queue-delay based AQM scheme requires current queuing delay as input from the device. If the device already maintains this value, then it is trivial to implement the AQM logic on the device. On the other hand, if the device provides indirect means to estimate queuing delay (for example: timestamps, dequeing rate etc.), then the AQM behavior is sensitive to how good the queuing delay estimate turns out on that device. Highlighting the AQM's sensitivity to queuing delay estimate helps implementers identify optimal means of implementing the AQM on a device.</t>
</section>
</section>
<!-- ######################################################-->
<!-- New section -->
<!-- ######################################################-->
<section anchor="sec:control_knobs" title="Operator control knobs and auto-tuning">
<t>One of the biggest hurdles for RED deployment was/is its parameter sensitivity to operating conditions -- how difficult it is to tune important RED parameters for a deployment in order to get maximum benefit from the RED implementation. Fluctuating congestion levels and network conditions add to the complexity. Incorrect parameter values lead to poor performance. This is one reason why RED is reported to be usually turned off.</t>
<t>Any AQM scheme is likely to have parameters whose values affect the AQM's control law and behavior. Exposing all these parameters as control knobs to a network operator (or user) can easily result in an unsafe AQM deployment. Unexpected AQM behavior ensues when parameter values are not set properly. A minimal number of control knobs minimizes the number of ways a, possible naive, user can break the AQM system. Fewer control knobs make the AQM scheme more user-friendly and easier to deploy and debug.</t>
<t>We recommend that an AQM scheme SHOULD minimize the number of control knobs exposed for operator tuning. An AQM scheme SHOULD expose only those knobs that control the macroscopic AQM behavior such as queue delay threshold, queue length threshold, etc.</t>
<t>Additionally, an AQM scheme's safety is directly related to its stability under varying operating conditions such as varying traffic profiles and fluctuating network conditions, as described in <xref target="sec:stability"></xref>. Operating conditions vary often and hence it is necessary that the AQM MUST remain stable under these conditions without the need for additional external tuning. If AQM parameters require tuning under these conditions, then the AQM MUST self-adapt necessary parameter values by employing auto-tuning techniques.</t>
</section>
<!-- ######################################################-->
<!-- New section -->
<!-- ######################################################-->
<section anchor="sec:interaction_ecn" title="Interaction with ECN">
<section anchor="subsec:interaction_ecn_motivation" title="Motivation">
<!-- NK : this is part of the recommendations and not evaluations-->
<t>Apart from packet drops, Explicit Congestion Notification (ECN) is an alternative means to signal data senders about network congestion. The AQM recommendation document <xref target="I-D.ietf-aqm-recommendation"></xref> describes some of the benefits of using ECN with AQM.</t> <!-- A network device explicitly marks specific bit(s) in packet headers to convey congestion information to data senders. A data sender implementing ECN treats the marked packet as if it were a packet drop and reacts the same way as it would to a packet drop. Note that ECN minimizes performance penalties, since packets do not have to be retransmitted.</t> -->
<t> </t>
</section>
<section anchor="subsec:interaction_ecn_discussion" title="Required discussion">
<t>An AQM scheme MAY support ECN, in which case testers MUST discuss and describe the support of ECN.</t><!-- If an AQM scheme supports ECN, an existence proof is needed. and SHOULD leverage ECN as an initial means to control queuing delay before resorting to packet drops. An AQM scheme SHOULD self-adapt and remain stable even with faulty and/or unresponsive ECN implementations en-route.</t> -->
</section>
</section>
<!-- ######################################################-->
<!-- New section -->
<!-- ######################################################-->
<section anchor="sec:interaction_scheduling" title="Interaction with scheduling">
<section anchor="subsec:interaction_scehduling_motivation" title="Motivation">
<t>Coupled with an AQM scheme, a router may schedule the transmission of packets in a specific manner by introducing a scheduling scheme. This algorithm may create sub-queues and integrate a dropping policy on each of these sub-queues. Another scheduling policy may modify the way packets are sequenced, modifying the timestamp of each packet.</t>
<!-- <t>Both schedulers and dropping policies can be introduced when packet are either enqueued or dequeued. If both schedulers and dropping policies are implemented when packet are enqueued, their interaction should not be a major issue. However, if one is introduced when packets are enqueued and the others when they are dequeued, there may be destructive interactions.</t> -->
</section>
<section anchor="subsec:interaction_scheduling_discussion" title="Required discussion">
<t>The scheduling and the AQM conjointly impact on the end-to-end performance. During the characterization process of a dropping policy, the tester MAY discuss the feasibility to add scheduling on top of its algorithm. This discussion MAY detail if the dropping policy is applied while packets are enqueued or dequeued.</t>
</section>
</section>
<!-- ######################################################-->
<!-- New section -->
<!-- ######################################################-->
<section anchor="sec:discussion" title="Discussion on methodology, metrics, AQM comparisons and packet sizes">
<section anchor="subsec:discussion_methodology" title="Methodology">
<t>A sufficiently detailed description of the test setup SHOULD be provided. Indeed, that would allow other to replicate the tests if needed. This test setup MAY include software and hardware versions. The tester MAY make its data available.</t>
<t>The proposals SHOULD be experimented on real systems, or they MAY be evaluated with event-driven simulations (such as NS-2, NS-3, OMNET, etc.). The proposed scenarios are not bound to a particular evaluation toolset.</t>
</section>
<section anchor="subsec:discussion_metrics" title="Comments on metrics measurement">
<t>In this document, we present the end-to-end metrics that SHOULD be evaluated to evaluate the trade-off between latency and goodput. The queue-related metrics enable a better understanding of the AQM behavior under tests and the impact of its internal parameters. Whenever it is possible, these guidelines advice to consider queue-related metrics, such as link utilization, queuing delay, queue size or packet loss.</t>
<t>These guidelines could hardly detail the way the metrics can be measured depends highly on the evaluation toolset.</t>
<!--NK: I am not sure whether we should refer to IPPM or not-->
</section>
<section anchor="subsec:discussion_comp_aqm" title="Comparing AQM schemes">
<t> This memo recognizes that the guidelines mentioned above may be used for comparing AQM schemes. This memo recommends that AQM schemes MUST be compared against both performance and deployment categories. In addition, this section details how best to achieve a fair comparison of AQM schemes by avoiding certain pitfalls.</t>
<section anchor="subsubsec:discussion_comp_aqm_perf" title="Performance comparison">
<t>AQM schemes MUST be compared against all the generic scenarios presented in this memo. AQM schemes MAY be compared for specific network environments such as data center, home networks etc. If an AQM scheme's parameter(s) were externally tuned for optimization or other purposes, these values MUST be disclosed.</t>
<t> Note that AQM schemes belong to different varieties such as queue-length based scheme (ex: RED) or queue-delay based scheme (ex: CoDel, PIE). Also, AQM schemes expose different control knobs associated with different semantics. For example, while both PIE and CoDel are queue-delay based schemes and each expose a knob to control the queueing delay -- PIE's "queueing delay reference" vs. CoDel's "queueing delay target", the two schemes' knobs have different semantics resulting in different control points. Such differences in AQM schemes can be easily overlooked while making comparisons.</t>
<t>This document recommends the following procedures for a fair performance comparison of two AQM schemes: </t>
<t> <list style="numbers">
<t>comparable control parameters and comparable input values: carefully identify the set of parameters that control similar behavior between the two AQM schemes and ensure these parameters have comparable input values. For example, while comparing how well a queue-length based AQM X controls queueing delay vs. queue-delay based AQM Y, identify the two schemes' parameters that control queue delay and ensure that their input values are comparable. Similarly, to compare two AQM schemes on how well they accommodate bursts, identify burst-related control parameters and ensure they are configured with similar values.</t>
<t>compare over a range of input configurations: there could be situations when the set of control parameters that affect a specific behavior have different semantics between the two AQM schemes. As mentioned above, PIE's knob to control queue delay has different semantics from CoDel's. In such situations, the schemes MUST be compared over a range of input configurations. For example, compare PIE vs. CoDel over the range of delay input configurations -- 5ms, 10ms, 15ms etc.</t>
</list> </t>
</section>
<section anchor="subsec:discussion_comp_aqm_deploy" title="Deployment comparison">
<t>AQM schemes MUST be compared against deployment criteria such as the parameter sensitivity (<xref target="subsec:stability_param_sensitivity"></xref>), the auto-tuning (<xref target="sec:control_knobs"></xref>) or the implementation cost (<xref target="sec:imple_cost"></xref>).</t>
</section>
</section>
<section anchor="subsec:discussion_packet_size" title="Packet sizes and congestion notification">
<t>An AQM scheme may be considering packet sizes while generating congestion signals. <xref target="RFC7141"></xref> discusses the motivations behind the same. For example, control packets such as DNS requests/responses, TCP SYNs/ACKs are small, and their loss can severely impact application performance. An AQM scheme may therefore be biased towards small packets by dropping them with smaller probability compared to larger packets. However, such an AQM scheme is unfair to data senders generating larger packets. Data senders, malicious or otherwise, are motivated to take advantage of the AQM scheme by transmitting smaller packets, and could result in unsafe deployments and unhealthy transport and/or application designs.</t>
<t> An AQM scheme SHOULD adhere to recommendations outlined in <xref target="RFC7141"></xref>, and SHOULD NOT provide undue advantage to flows with smaller packets.</t>
</section>
</section>
<!-- ######################################################-->
<!-- ######################################################-->
<!-- Tail of the document -->
<!-- ######################################################-->
<!-- ######################################################-->
<section anchor="sec:acknowledgements" title="Acknowledgements">
<t>This work has been partially supported by the European Community under its Seventh Framework Programme through the Reducing Internet Transport Latency (RITE) project (ICT-317700).</t>
</section>
<section anchor="sec:contributors" title="Contributors">
<t> Many thanks to S. Akhtar, A.B. Bagayoko, F. Baker, D. Collier-Brown, G. Fairhurst, T. Hoiland-Jorgensen, C. Kulatunga, W. Lautenschlager, R. Pan, D. Taht and M. Welzl for detailed and wise feedback on this document.</t>
</section>
<section anchor="sec:IANA" title="IANA Considerations">
<t>This memo includes no request to IANA.</t>
</section>
<section anchor="sec:ecurity" title="Security Considerations">
<t>This document, by itself, presents no new privacy nor security issues.<!--See <xref target="RFC3552">RFC 3552</xref> for a guide.--></t>
</section>
</middle>
<!-- *****BACK MATTER ***** -->
<back>
<!-- References split into informative and normative -->
<!-- There are 2 ways to insert reference entries from the citation libraries:
1. define an ENTITY at the top, and use "ampersand character"RFC2629; here (as shown)
2. simply use a PI "less than character"?rfc include="reference.RFC.2119.xml"?> here
(for I-Ds: include="reference.I-D.narten-iana-considerations-rfc2434bis.xml")
Both are cited textually in the same manner: by using xref elements.
If you use the PI option, xml2rfc will, by default, try to find included files in the same
directory as the including file. You can also define the XML_LIBRARY environment variable
with a value containing a set of directories to search. These can be either in the local
filing system or remote ones accessed by http (http://domain/dir/... ).-->
<references title="Normative References">
<?rfc include="reference.I-D.ietf-aqm-recommendation.xml"?>
<!-- <?rfc include="reference.I-D.ietf-tsvwg-byte-pkt-congest.xml"?> -->
<reference anchor="RFC7141">
<front>
<title>Byte and Packet Congestion Notification</title>
<author initials="B" surname="Briscoe">
</author>
<author initials="J" surname="Manner">
</author>
<date year="2014" />
</front>
<seriesInfo name="RFC" value="7141" />
</reference>
<!--
<reference anchor="IRTF-TOOLS-5">
<front>
<title>Tools for the Evaluation of Simulation and Testbed Scenarios</title>
<author initials="S" surname="Floyd">
<organization></organization>
</author>
<author initials="E" surname="Kohler">
<organization></organization>
</author>
<date year="2008" />
</front>
<seriesInfo name="TMRG-TOOLS" value="05" />
</reference>
-->
<reference anchor="RFC2119">
<front>
<title>Key words for use in RFCs to Indicate Requirement Levels</title>
<author initials="S" surname="Bradner">
<organization></organization>
</author>
<date year="1997" />
</front>
<seriesInfo name="RFC" value="2119" />
</reference>
<!--
<reference anchor="RFC5136">
<front>
<title>Defining Network Capacity</title>
<author initials="P" surname="Chimento">
<organization>JHU Applied Physics Lab</organization>
</author>
<author initials="J" surname="Ishac">
<organization>NASA Glenn Research Center</organization>
</author>
<date year="2008" />
</front>
<seriesInfo name="RFC" value="5136" />
</reference>
-->
</references>
<references title="Informative References">
<?rfc include="reference.RFC.5348.xml"?>
<?rfc include="reference.RFC.5681.xml"?>
<?rfc include="reference.RFC.6297.xml"?>
<reference anchor='RFC2309'>
<front>
<title abbrev='Internet Performance Recommendations'>Recommendations on Queue Management and Congestion Avoidance in the Internet</title>
<author initials='B.' surname='Braden' fullname='Bob Braden'>
<organization>USC Information Sciences Institute</organization>
<address>
<postal>
<street>4676 Admiralty Way</street>
<city>Marina del Rey</city>
<region>CA</region>
<code>90292</code></postal>
<phone>310-822-1511</phone>
<email>Braden@ISI.EDU</email></address></author>
<author initials='D.D.' surname='Clark' fullname='David D. Clark'>
<organization>MIT Laboratory for Computer Science</organization>
<address>
<postal>
<street>545 Technology Sq.</street>
<city>Cambridge</city>
<region>MA</region>
<code>02139</code></postal>
<phone>617-253-6003</phone>
<email>DDC@lcs.mit.edu</email></address></author>
<author initials='J.' surname='Crowcroft' fullname='Jon Crowcroft'>
<organization>University College London</organization>
<address>
<postal>
<street>Department of Computer Science</street>
<street>Gower Street</street>
<street>London, WC1E 6BT</street>
<street>ENGLAND</street></postal>
<phone>+44 171 380 7296</phone>
<email>Jon.Crowcroft@cs.ucl.ac.uk</email></address></author>
<author initials='B.' surname='Davie' fullname='Bruce Davie'>
<organization>Cisco Systems, Inc.</organization>
<address>
<postal>
<street>250 Apollo Drive</street>
<city>Chelmsford</city>
<region>MA</region>
<code>01824</code></postal>
<email>bdavie@cisco.com</email></address></author>
<author initials='S.' surname='Deering' fullname='Steve Deering'>
<organization>Cisco Systems, Inc.</organization>
<address>
<postal>
<street>170 West Tasman Drive</street>
<city>San Jose</city>
<region>CA</region>
<code>95134-1706</code></postal>
<phone>408-527-8213</phone>
<email>deering@cisco.com</email></address></author>
<author initials='D.' surname='Estrin' fullname='Deborah Estrin'>
<organization>USC Information Sciences Institute</organization>
<address>
<postal>
<street>4676 Admiralty Way</street>
<city>Marina del Rey</city>
<region>CA</region>
<code>90292</code></postal>
<phone>310-822-1511</phone>
<email>Estrin@usc.edu</email></address></author>
<author initials='S.' surname='Floyd' fullname='Sally Floyd'>
<organization>Lawrence Berkeley National Laboratory, MS 50B-2239, One Cyclotron Road, Berkeley CA 94720</organization>
<address>
<phone>510-486-7518</phone>
<email>Floyd@ee.lbl.gov</email></address></author>
<author initials='V.' surname='Jacobson' fullname='Van Jacobson'>
<organization>Lawrence Berkeley National Laboratory, MS 46A, One Cyclotron Road, Berkeley CA 94720</organization>
<address>
<phone>510-486-7519</phone>
<email>Van@ee.lbl.gov</email></address></author>
<author initials='G.' surname='Minshall' fullname='Greg Minshall'>
<organization>Fiberlane Communications</organization>
<address>
<postal>
<street>1399 Charleston Road</street>
<city>Mountain View</city>
<region>CA</region>
<code>94043</code></postal>
<phone>+1 650 237 3164</phone>
<email>Minshall@fiberlane.com</email></address></author>
<author initials='C.' surname='Partridge' fullname='Craig Partridge'>
<organization>BBN Technologies</organization>
<address>
<postal>
<street>10 Moulton St.</street>
<street>Cambridge MA 02138</street></postal>
<phone>510-558-8675</phone>
<email>craig@bbn.com</email></address></author>
<author initials='L.' surname='Peterson' fullname='Larry Peterson'>
<organization>Department of Computer Science</organization>
<address>
<postal>
<street>University of Arizona</street>
<city>Tucson</city>
<region>AZ</region>
<code>85721</code></postal>
<phone>520-621-4231</phone>
<email>LLP@cs.arizona.edu</email></address></author>
<author initials='K.K.' surname='Ramakrishnan' fullname='K.K. Ramakrishnan'>
<organization>AT&T Labs. Research</organization>
<address>
<postal>
<street>Rm. A155</street>
<street>180 Park Avenue</street>
<street>Florham Park, N.J. 07932</street></postal>
<phone>973-360-8766</phone>
<email>KKRama@research.att.com</email></address></author>
<author initials='S.' surname='Shenker' fullname='Scott Shenker'>
<organization>Xerox PARC</organization>
<address>
<postal>
<street>3333 Coyote Hill Road</street>
<city>Palo Alto</city>
<region>CA</region>
<code>94304</code></postal>
<phone>415-812-4840</phone>
<email>Shenker@parc.xerox.com</email></address></author>
<author initials='J.' surname='Wroclawski' fullname='John Wroclawski'>
<organization>MIT Laboratory for Computer Science</organization>
<address>
<postal>
<street>545 Technology Sq.</street>
<city>Cambridge</city>
<region>MA</region>
<code>02139</code></postal>
<phone>617-253-7885</phone>
<email>JTW@lcs.mit.edu</email></address></author>
<author initials='L.' surname='Zhang' fullname='Lixia Zhang'>
<organization>UCLA</organization>
<address>
<postal>
<street>4531G Boelter Hall</street>
<city>Los Angeles</city>
<region>CA</region>
<code>90024</code></postal>
<phone>310-825-2695</phone>
<email>Lixia@cs.ucla.edu</email></address></author>
<date year='1998' month='April' />
<area>Routing</area>
<keyword>congestion</keyword>
<abstract>
<t>
This memo presents two recommendations to the Internet community
concerning measures to improve and preserve Internet performance.
It presents a strong recommendation for testing, standardization,
and widespread deployment of active queue management in routers,
to improve the performance of today's Internet. It also urges a
concerted effort of research, measurement, and ultimate deployment
of router mechanisms to protect the Internet from flows that are
not sufficiently responsive to congestion notification.
</t></abstract></front>
<seriesInfo name='RFC' value='2309' />
<format type='TXT' octets='38079' target='http://www.rfc-editor.org/rfc/rfc2309.txt' />
<format type='XML' octets='42517' target='http://xml.resource.org/public/rfc/xml/rfc2309.xml' />
</reference>
<!--
<reference anchor="QOEVOICE2013">
<front>
<title>Voice quality prediction models and their application in VoIP networks</title>
<author initials="L." surname="Sun">
</author>
<author initials="E.C." surname="Ifeachor">
</author>
<date year="2006" />
</front>
<seriesInfo name="IEEE Transactions on Multimedia" value="" />
</reference>
<reference anchor="QOEVID2013">
<front>
<title>Model for estimating QoE of Video delivered using HTTP Adaptive Streaming</title>
<author initials="J." surname="De Vriendt">
</author>
<author initials="D." surname="Robinson">
</author>
<date year="2013" />
</front>
<seriesInfo name="IFIP/IEEE International Symposium on Integrated Network Management (IM 2013)" value="" />
</reference>
-->
<reference anchor="YU06">
<front>
<title>A preliminary analysis of loss synchronisation between concurrent TCP flows</title>
<author initials="P" surname="Jay">
</author>
<author initials="Q" surname="Fu">
</author>
<author initials="G" surname="Armitage">
</author>
<date year="2006" />
</front>
<seriesInfo name="Australian Telecommunication Networks and Application Conference (ATNAC)" value="" />
</reference>
<reference anchor="TCPEVAL2013">
<front>
<title>Common TCP Evaluation Suite</title>
<author initials="D" surname="Hayes">
</author>
<author initials="D" surname="Ros">
</author>
<author initials="L.L.H" surname="Andrew">
</author>
<author initials="S" surname="Floyd">
</author>
<date year="2013" />
</front>
<seriesInfo name="IRTF ICCRG" value="" />
</reference>
<!--
<reference anchor="LOSS-SYNCH-AQM-08">
<front>
<title>Loss synchronization, router buffer sizing and high-speed TCP versions: Adding RED to the mix</title>
<author initials="S" surname="Hassayoun">
</author>
<author initials="D" surname="Ros">
</author>
<date year="2008" />
</front>
<seriesInfo name="IEEE LCN" value="" />
</reference>-->
<reference anchor="PIE">
<front>
<title>PIE: A lightweight control scheme to address the bufferbloat problem</title>
<author initials="R" surname="Pan">
</author>
<author initials="P" surname="Natarajan">
</author>
<author initials="C" surname="Piglione">
</author>
<author initials="MS" surname="Prabhu">
</author>
<author initials="V" surname="Subramanian">
</author>
<author initials="F" surname="Baker">
</author>
<author initials="B" surname="VerSteeg">
</author>
<date year="2013" />
</front>
<seriesInfo name="IEEE HPSR" value="" />
</reference>
<reference anchor="CODEL">
<front>
<title>Controlling Queue Delay</title>
<author initials="K" surname="Nichols">
</author>
<author initials="V" surname="Jacobson">
</author>
<date year="2012" />
</front>
<seriesInfo name="ACM Queue" value="" />
</reference>
<reference anchor="LOSS-SYNCH-MET-08">
<front>
<title>Loss Synchronization and Router Buffer Sizing with High-Speed Versions of TCP</title>
<author initials="S" surname="Hassayoun">
</author>
<author initials="D" surname="Ros">
</author>
<date year="2008" />
</front>
<seriesInfo name="IEEE INFOCOM Workshops" value="" />
</reference>
<reference anchor="BB2011">
<front>
<title>BufferBloat: what's wrong with the internet?</title>
<author initials="" surname="">
</author>
<date year="2011" />
</front>
<seriesInfo name="ACM Queue" value="vol. 9" />
</reference>
<reference anchor="DOCSIS2013">
<front>
<title>Active Queue Management Algorithms for DOCSIS 3.0</title>
<author initials="G" surname="White">
</author>
<author initials="D" surname="Rice">
</author>
<date year="2013" />
</front>
<seriesInfo name="Technical report - Cable Television Laboratories" value="" />
</reference>
</references>
<!--
<section anchor="app-additional" title="Additional Stuff">
<t>This becomes an Appendix.</t>
</section>-->
<!-- Change Log
v00 2006-03-15 EBD Initial version
v01 2006-04-03 EBD Moved PI location back to position 1 -
v3.1 of XMLmind is better with them at this location.
v02 2007-03-07 AH removed extraneous nested_list attribute,
other minor corrections
v03 2007-03-09 EBD Added comments on null IANA sections and fixed heading capitalization.
Modified comments around figure to reflect non-implementation of
figure indent control. Put in reference using anchor="DOMINATION".
Fixed up the date specification comments to reflect current truth.
v04 2007-03-09 AH Major changes: shortened discussion of PIs,
added discussion of rfc include.
v05 2007-03-10 EBD Added preamble to C program example to tell about ABNF and alternative
images. Removed meta-characters from comments (causes problems). -->
</back>
</rfc>
| PAFTECH AB 2003-2026 | 2026-04-23 21:43:44 |