One document matched: draft-irtf-iccrg-tcpeval-01.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="rfc2629.xslt" ?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [
]>
<rfc ipr="trust200902" docName="draft-irtf-iccrg-tcpeval-01" category="info">
<?rfc toc="yes"?>
<?rfc sortrefs="yes"?>
<?rfc symrefs="yes"?>
<front>
<title>Common TCP Evaluation Suite</title>
<author initials="D." surname="Hayes" fullname="David Hayes">
<organization>University of Oslo</organization>
<address>
<postal>
<street>Department of Informatics, P.O. Box 1080 Blindern</street>
<city>Oslo</city>
<code>N-0316</code>
<country>Norway</country>
</postal>
<email>davihay@ifi.uio.no</email>
</address>
</author>
<author initials="D." surname="Ros" fullname="David Ros">
<organization>Simula Research Laboratory</organization>
<address>
<postal>
<street>P.O. Box 134</street>
<city>Lysaker</city>
<code>1325</code>
<country>Norway</country>
</postal>
<email>dros@simula.no</email>
</address>
</author>
<author initials="L. L. H." surname="Andrew" fullname="Lachlan L.H. Andrew">
<organization>Monash University</organization>
<address>
<postal>
<street>Clayton School of Information Technology</street>
<street>Ground Floor, Building 63</street>
<street>Monash University Clayton Campus, Wellington Road</street>
<city>Clayton</city>
<code>VIC 3800</code>
<country>Australia</country>
</postal>
<email>Lachlan.Andrew@monash.edu</email>
</address>
</author>
<author initials="S." surname="Floyd" fullname="Sally Floyd">
<organization>ICSI</organization>
<address>
<postal>
<street>1947 Center Street, Ste. 600</street>
<city>Berkeley</city>
<code>CA 94704</code>
<country>United States</country>
</postal>
<email>floyd@acm.org</email>
</address>
</author>
<date year="2014" month="July" day="4"/>
<area>Transport</area>
<keyword>Internet-Draft</keyword>
<abstract>
<t>This document presents an evaluation test suite for the initial
assessment of proposed TCP modifications. The goal of the test
suite is to allow researchers to quickly and easily evaluate
their proposed TCP extensions in simulators and testbeds using
a common set of well-defined, standard test cases, in order to
compare and contrast proposals against standard TCP as well as
other proposed modifications. This test suite is not intended to
result in an exhaustive evaluation of a proposed TCP modification
or new congestion control mechanism. Instead, the focus is on
quickly and easily generating an initial evaluation report that
allows the networking community to understand and discuss the
behavioral aspects of a new proposal, in order to guide further
experimentation that will be needed to fully investigate the
specific aspects of such proposal.</t>
</abstract>
</front>
<middle>
<section anchor="introduction" title="Introduction">
<t>This document describes a common test suite for the initial assessment of
new TCP extensions or modifications. It defines a small number of evaluation scenarios,
including traffic and delay distributions, network topologies, and
evaluation parameters and metrics. The motivation for such an evaluation
suite is to help researchers in evaluating their proposed modifications
to TCP. The evaluation suite will also enable independent duplication
and verification of reported results by others, which is an important
aspect of the scientific method that is not often put to use by the
networking community. A specific target is that the evaluations should
be able to be completed in a reasonable amount of time by simulation, or with a
reasonable amount of effort in a testbed.</t>
<t>It is not possible to provide TCP researchers with a complete set of
scenarios for an exhaustive evaluation of a new TCP extension;
especially because the characteristics of a new extension will often
require experiments with specific scenarios that highlight its
behavior. On the other hand, an exhaustive evaluation of a TCP
extension will need to include several standard scenarios, and it is
the focus of the test suite described in this document to define this
initial set of test cases.</t>
<t>These scenarios generalize current characteristics of the Internet
such as round-trip times (RTT), propagation delays, and buffer sizes. It is envisaged that as the Internet evolves these will need to be adjusted. In particular, we expect buffer sizes will need to be adjusted as latency becomes
increasingly important.</t>
<t>The scenarios specified here are intended to be as generic as possible, i.e., not tied to a particular simulation or emulation platform. However, when needed some details pertaining to implementation using a given tool are described.</t>
<t>This document has evolved from a “round-table” meeting on TCP evaluation,
held at Caltech on
November 8-9, 2007, reported in <xref target="TESTSUITE08"/>.
This document is the first step in constructing the evaluation suite;
the goal is for the evaluation suite to be adapted in response to
feedback from the networking community. It revises draft-irtf-tmrg-tests-02 <xref target="I-D-TMRG-TESTS"/>.</t>
<!--<t>Information related to the draft can be found at: <eref target="http://riteproject.eu/ietf-drafts">http://riteproject.eu/ietf-drafts</eref></t>-->
<!--<t>A sample implementation (including patched ns-2) is available at: <eref target="https://bitbucket.org/hayesd/tcp-evaluation-suite-public">https://bitbucket.org/hayesd/tcp-evaluation-suite-public</eref></t>-->
<t>The traces used and a sample implementation (including patched
ns-2) are available from: <eref target="http://trac.tools.ietf.org/group/irtf/trac/wiki/ICCRG">http://trac.tools.ietf.org/group/irtf/trac/wiki/ICCRG</eref></t>
</section>
<section anchor="traffic-generation" title="Traffic generation">
<t>Congestion control concerns the response of flows to bandwidth
limitations or to the presence of other flows. Cross-traffic and
reverse-path traffic are therefore important to the tests described in this suite.
Such traffic can have the
desirable effect of reducing the occurrence of pathological conditions,
such as global synchronization among competing flows, that might
otherwise be mis-interpreted as normal average behaviours of those
protocols <xref target="FLOYD03"/><xref target="MASCOLO06"/>. This traffic must be reasonably
realistic for the tests to predict the behaviour of congestion control
protocols in real networks, and also well-defined so that statistical
noise does not mask important effects.</t>
<section anchor="desirable-model-characteristics" title="Desirable model characteristics">
<t>Most scenarios use traffic produced by a traffic generator, with a
range of start times for user sessions, flow sizes, and the
like, mimicking the traffic patterns commonly observed in the
Internet. It is important that the same “amount” of congestion or
cross-traffic be used for the testing scenarios of different
congestion control algorithms. This is complicated by the fact that
packet arrivals and even flow arrivals are influenced by the behavior
of the algorithms. For this reason, a pure open-loop, packet-level
generation of traffic where generated traffic does not respond to the
behaviour of other present flows is not suitable. Instead, emulating
application or user behaviours at the end points using reactive
protocols such as TCP in a closed-loop fashion results in a closer
approximation of cross-traffic, where user behaviours are modeled by
well-defined parameters for source inputs (e.g., request sizes for
HTTP), destination inputs (e.g., response size), and think times
between pairs of source and destination inputs. By setting
appropriate parameters for the traffic generator, we can emulate
non-greedy user-interactive traffic (e.g., HTTP 1.1, SMTP and remote login),
greedy traffic (e.g., P2P and long file downloads), as well as
long-lived but non-greedy, non-interactive flows (or thin
streams).</t>
<t>This approach models protocol reactions to the congestion caused by
other flows in the common paths, although it fails to model the
reactions of users themselves to the presence of congestion. A
model that includes end-users’ reaction to congestion is beyond the
scope of this draft, but we invite researchers to explore how the user
behavior, as reflected in the flow sizes, user wait times, and
number of connections per session, might be affected by the level of
congestion experienced within a session <xref target="ROSSI03"/>.</t>
</section>
<section anchor="tmix" title="Tmix">
<t>There are several traffic generators available that implement a similar approach
to that discussed above. For now, we have chosen to use the
Tmix <xref target="WEIGLE06"/> traffic generator. Tmix is available for the NS2 and NS3 simulators,
and can generate traffic for testbeds (for example
GENI <xref target="GENITMIX"/>).</t>
<t>Tmix represents each TCP connection by a
connection vector (CV) consisting of a sequence of (request-size,
response-size, think-time) triples, thus representing bi-directional
traffic. Connection vectors used for traffic generation can be
obtained from Internet traffic traces.</t>
<section anchor="sec:base-tmix-trace" title="Base Tmix trace files for tests">
<t>The traces currently defined
for use in the test suite are based on campus traffic at the
University of North Carolina (see <xref target="TRACES"/> for a description of
construction methods and basic statistics).</t>
<t>The traces have an additional “m” field
added to each connection vector to provide each direction’s maximum segment size
for the connection. This is used to provide the packet size
distribution described in <xref target="sec:pack-size-distr"/>.</t>
<t>These traces contain a mixture of connections, from very short flows
that do not exist for long enough to be “congestion controlled”, to
long thin streams, to bulk file transfer like connections.</t>
<t>The traces are available at: <eref target="http://trac.tools.ietf.org/group/irtf/trac/wiki/ICCRG">http://trac.tools.ietf.org/group/irtf/trac/wiki/ICCRG</eref></t>
<t>Each of the nine bidirectional trace files are named with the following convention:</t>
<figure><artwork><![CDATA[
rAsI.org
]]></artwork></figure>
<t>where the I is the number of the Tmix initiator node, and A is the
number of the tmix acceptor node, when the traffic sources are set up
in the dumbbell configuration shown in <xref target="fig:dumbell"/>.</t>
</section>
</section>
<section anchor="loads" title="Loads">
<t>While the protocols being tested may differ, it is important that we
maintain the same “load” or level of congestion for the experimental
scenarios. For many of the scenarios, such
as the basic ones in <xref target="sec:basic-scenarios"/>, each scenario
is run for a range of loads, where the load is varied by varying the
rate of session arrivals.</t>
<section anchor="sec:varying-tmix-traffic" title="Varying the Tmix traffic load">
<t>To adjust the traffic load for a given scenario, the connection
start times for flows in a Tmix trace are scaled as follows. Connections are actually started
at:</t>
<figure><artwork><![CDATA[
experiment_cv_start_time = scale * cv_start_time (1)
]]></artwork></figure>
<t>where cv_start_time denotes the connection vector start
time in the Tmix traces and experiment_cv_start_time is the time the
connection starts in the experiment. Therefore, the smaller the scale
the higher (in general) the traffic load.</t>
<section anchor="notes" title="Notes">
<t>Changing the connection start times also changes the way the traffic
connections interact, potentially changing the “clumping” of traffic
bursts.</t>
<t>Very small changes in the scaling parameter can cause disproportionate
changes in the offered load. This is due to possibility of the small
change causing the exclusion or inclusion of a CV that will transfer a
very large amount of data.</t>
</section>
</section>
<section anchor="sec:dealing-with-non-stationarity" title="Dealing with non-stationarity">
<t>The Tmix traffic traces, as they are, offer a non-stationary load. This is
exacerbated for tests that do not require use of the full trace files,
but only a portion of them. While removing this non-stationarity does
also remove some of the “realism” of the traffic, it is necessary
for the test suite to produce reliable and consistent results.</t>
<t>A more stationary offered load is achieved by shuffling the start
times of connection vectors in the Tmix trace file. The trace file
is logically partitioned into n-second bins, which are then shuffled
using a Fisher-Yates shuffle <xref target="SHUFFLEWIKI"/>, and the required
portions written to shuffled trace files for the particular experiment
being conducted.</t>
<section anchor="bin-size" title="Bin size">
<t>The bin size is chosen so that there is enough shuffling with respect
to the test length. The offered traffic per test second from the Tmix
trace files depends on a scale factor (see
<xref target="sec:varying-tmix-traffic"/>), which is related to the
capacity of the bottleneck link. The shuffling bin size (in seconds)
is set at:</t>
<figure><artwork><![CDATA[
b = 500e6 / C (2)
]]></artwork></figure>
<t>where C is the bottleneck link’s capacity in bits per second, and
500e6 is a scaling factor (in bits).</t>
<t>Thus for the access link scenario described in
<xref target="sec:access-link"/>, the bin size for shuffling will be 5 seconds.</t>
</section>
<section anchor="ns2-implementation-specifics" title="NS2 implementation specifics">
<t>The tcl scripts for this process are distributed with the NS2 example
test suite implementation. Care must be taken when using this algorithm, so that the given random number generator and the same seed are employed, or else the resulting
experimental traces will be different.</t>
</section>
</section>
</section>
<section anchor="sec:pack-size-distr" title="Packet size distribution">
<t>For flows generated by the traffic generator, 10% of them use 536-byte
packets, and 90% 1500-byte packets. The base Tmix traces described
in <xref target="sec:base-tmix-trace"/> have been processed at the
<spanx style="emph">connection</spanx> level to have this characteristic. As a result, <spanx style="emph">packets</spanx> in a given test will
be roughly, but not be exactly, in
this proportion. However, the proportion of offered traffic will be
consistent for each experiment.</t>
<section anchor="potential-revision" title="Potential revision">
<t>As Tmix can now read and use a connection’s Maximum Segment Size (MSS) from the
trace file, it will be possible to produce Tmix connection vector
trace files where the packet sizes reflect actual measurements.</t>
</section>
</section>
</section>
<section anchor="sec:reliable-results" title="Achieving reliable results in minimum time">
<t>This section describes the techniques used to achieve reliable results
in the minimum test time.</t>
<section anchor="background" title="Background">
<t>Over a long time, because the session arrival times
are to a large extent independent of the transfer times, load could be
defined as:</t>
<figure><artwork><![CDATA[
A = E[f]/E[t],
]]></artwork></figure>
<t>where E[f] is the mean session (flow) size in bits transferred,
E[t] is the mean session inter-arrival time in seconds,
and A is the load in bps.</t>
<t>It is important to test congestion control protocols in “overloaded”
conditions. However, if A > C, where C is the capacity of the
bottleneck link, then the system has no equilibrium. In long-running
experiments with A > C, the expected number of flows would keep on
increasing with time (because as time passes, flows would tend to last
for longer and longer, thus “piling up” with newly-arriving ones).
This means that, in an overload scenario, some measures will be very sensitive to the
duration of the tests.</t>
</section>
<section anchor="sec:equil-or-steady" title="Equilibrium or Steady State">
<t>Ideally, experiments should be run until some sort of equilibrium
results can be obtained. Since every test algorithm can potentially
change how long this may take, the following approach
is adopted:</t>
<t><list style="numbers">
<t>Traces are shuffled to remove non-stationarity (see <xref target="sec:dealing-with-non-stationarity"/>.)</t>
<t>The experiment run time is determined from the traffic traces. The shuffled traces are compiled such that the estimate of traffic offered in the second third of the test is equal to the estimate of traffic offered in the final third of the test, to within a 5% tolerance. The length of the trace files becomes the total experiment run time (including the warm up time).</t>
<t>The warmup time until measurements start, as shown in <xref target="sec:basic-scenarios"/>, is calculated as the time at which the NS2 simulation of standard TCP achieves “steady state”. In this case, warmup time is determined as the time required so the measurements have statistically similar first and second half results. The metrics used as reference are: the bottleneck raw throughput, and the average bottleneck queue size. The latter is stable when A » C and A « C, but not when A ~= C. In this case the queue is not a stable measure, and just the raw bottleneck throughput is used. </t>
</list></t>
<section anchor="note-on-the-offered-load-in-ns2" title="Note on the offered load in NS2">
<t>The offered load in an NS2 simulation using one-way TCP will be higher
than the estimated load. One-way TCP uses fixed TCP segment sizes, so
all transmissions that would normally use a segment size less than the
maximum segment size (in this case 496B or 1460B), such as at the end
of a block of data, or for short queries or responses, will still be
sent as a maximum segment size packet.</t>
</section>
</section>
<section anchor="accelerated-test-start-up-time" title="Accelerated test start up time">
<t>Tmix traffic generation does not provide an instant constant load. It
can take quite a long time for the number of simultaneous TCP
connections, and thus the offered load, to build up. To accelerate the system start up, the system is
“prefilled” to a state close to “steady state”. This is done by
starting initial sessions over a shorter interval than they would
normally start, and biasing the sessions started to longer
sessions. Details of how this is achieved follow.</t>
<t>Connections that start before t=prefill_t in the Tmix traces, are selected with a bias
toward longer sessions (connections which are estimated to continue
past the long_flow_bias time (see <xref target="fig:prefilling"/>)).
These selected connections are then started at an accelerated rate by
starting them over the time interval prefill_si.</t>
<t>The prefill_t (in seconds) calculation is based on the following heuristic:</t>
<figure><artwork><![CDATA[
prefill_t = 1.5 * targetload * maxRTT (3)
]]></artwork></figure>
<t>where maxRTT is the median maximum RTT in the particular topology, and
targetload is given as a percentage. This generally works quite well,
but requires some adjustment for very high BDP scenarios. Experiment
tables specify the prefill_t value to be used in each experiment.</t>
<t>The long_flow_bias threshold is set at </t>
<figure><artwork><![CDATA[
long_flow_bias = prefill_t / 2 . (4)
]]></artwork></figure>
<t>These values are not optimal, but have been experimentally determined to give reasonable results.</t>
<t>The start up time interval, prefill_si, is calculated as follows:</t>
<figure><artwork><![CDATA[
prefill_si = total_pfcb / (C * TL / 100.0) (5)
]]></artwork></figure>
<t>where total_pfcb is the total number of bits estimated to be sent by the
prefill connections, C is the capacity of the bottleneck link, and TL
is the target offered load as a percentage.</t>
<t>This procedure has the effect of quickly bringing the system to a loaded
state. From this point the system runs until t = warmup (as calculated
in <xref target="sec:equil-or-steady"/>), after which moment statistics are computed.</t>
<figure title="Prefilling." anchor="fig:prefilling"><artwork><![CDATA[
|<----- test_duration ----->|
| |
prefill_si | |
|<-->| | |
|--------|------|----|-----------------|---------------------------|
t=0 | | |<---- warmup --->|
| | | |
| | t = prefill_t t = warmup + prefill_t
| |
| t = prefill_t - prefill_si
|
t = long_flow_bias
]]></artwork></figure>
</section>
</section>
<section anchor="sec:basic-scenarios" title="Basic scenarios">
<t>The purpose of the basic scenarios is to explore the behavior of a TCP
modification over different link types. These scenarios use the dumbbell
topology described in <xref target="sec:RTTs"/>.</t>
<section anchor="sec:RTTs" title="Basic topology">
<t>Most tests use a simple dumbbell
topology with a central link that connects two routers, as illustrated
in <xref target="fig:dumbell"/>.
Each router is also connected to three nodes by edge links.
In order to generate a typical range of round trip times,
edge links have different delays. Unless specified otherwise, such delays are as follows.
On one side, the one-way propagation delays are: 0ms, 12ms and 25ms;
on the other: 2ms, 37ms, and 75ms.
Traffic is uniformly shared among the nine source/destination pairs,
giving a distribution of per-flow RTTs in the absence of queueing delay
shown in <xref target="tab:RTTs"/>.
These RTTs are computed for a dumbbell topology assuming a delay of 0ms
for the central link.
The delay for the central link that is used in a specific scenario is given
in the next section.</t>
<figure title="A dumbbell topology." anchor="fig:dumbell"><artwork><![CDATA[
Node 1 Node 4
\_ _/
\_ _/
\_ __________ Central __________ _/
| | link | |
Node 2 ------| Router 1 |----------------| Router 2 |------ Node 5
_|__________| |__________|_
_/ \_
_/ \_
Node 3 / \ Node 6
]]></artwork></figure>
<t>For dummynet experiments, delays can be obtained by specifying the delay
of each flow. </t>
<texttable title="Minimum RTTs of the paths between two nodes, in
milliseconds." anchor="tab:RTTs">
<ttcol>Path</ttcol>
<ttcol>RTT</ttcol>
<ttcol>Path</ttcol>
<ttcol>RTT</ttcol>
<ttcol>Path</ttcol>
<ttcol>RTT</ttcol>
<c>1-4</c> <c>4</c> <c>1-5</c> <c> 74</c> <c>1-6</c> <c>150</c>
<c>2-4</c> <c>28</c> <c>2-5</c> <c> 98</c> <c>2-6</c> <c>174</c>
<c>3-4</c> <c>54</c> <c>3-5</c> <c>124</c> <c>3-6</c> <c>200</c>
</texttable>
</section>
<section anchor="sec:basic-traffic" title="Traffic">
<t>In all of the basic scenarios, <spanx style="emph">all</spanx> TCP flows use the TCP extension or modification under evaluation.</t>
<t>In general, the 9 bidirectional Tmix sources are connected to nodes 1
to 6 of <xref target="fig:dumbell"/> to create the paths tabulated in
<xref target="tab:RTTs"/>.</t>
<t>Offered loads are estimated directly from the shuffled and scaled Tmix
traces, as described in <xref target="sec:equil-or-steady"/>. The actual
measured loads will depend on the TCP variant and the scenario being
tested.</t>
<t>Buffer sizes are based on the Bandwidth Delay Product (BDP),
except for the Dial-up scenario where a BDP buffer does not provide enough
buffering.</t>
<t>The load generated by Tmix with the standard trace files is
asymmetric, with a higher load offered in the right to left direction
(refer to <xref target="fig:dumbell"/>) than in the left to right
direction. Loads are specified for the higher traffic right to left
direction. For each of the basic scenarios, three offered loads are
tested: moderate (60%), high (85%), and overload (110%). Loads are
for the bottleneck link, which is the central link in all scenarios
except the wireless LAN scenario.</t>
<t>The 9 Tmix traces are scaled using a single scaling factor in these
tests. This means that the traffic offered on each of the 9 paths
through the network is not equal, but combined at the bottleneck
produces the specified offered load.</t>
</section>
<section anchor="flows-under-test" title="Flows under test">
<t>For these basic scenarios, there is no differentiation between
“cross-traffic” and the “flows under test”. The aggregate traffic is
under test, with the metrics exploring both
aggregate traffic and distributions of flow-specific metrics.</t>
</section>
<section anchor="scenarios" title="Scenarios">
<section anchor="data-center" title="Data Center">
<t>The data center scenario models a case where bandwidth is plentiful
and link delays are generally low. All links have a capacity
of 1 Gbps. Links from
nodes 1, 2 and 4 have a one-way propagation delay of 10 us,
while those from nodes 3, 5 and 6 have 100 us <xref target="ALIZADEH10"/>, and the
central link has 0 ms delay. The central link has 10 ms buffers.</t>
<texttable title="Data center scenario parameters." anchor="tab:dcloads">
<ttcol>load</ttcol>
<ttcol>scale</ttcol>
<ttcol>warmup</ttcol>
<ttcol>test_duration</ttcol>
<ttcol>prefill_t</ttcol>
<ttcol>prefill_si</ttcol>
<c>60%</c> <c>0.4864</c> <c>63</c> <c> 69</c> <c> 9.0</c> <c>4.1</c>
<c>85%</c> <c>0.3707</c> <c>19</c> <c>328</c> <c>11.3</c> <c>5.1</c>
<c>110%</c> <c>0.3030</c> <c> 8</c> <c>663</c> <c>14.6</c> <c>6.9</c>
</texttable>
<section anchor="potential-revisions" title="Potential Revisions">
<t>The rate of 1 Gbps is chosen such that NS2 simulations can run in a
reasonable time. Higher values will become feasible (in simulation) as computing power
increases, however the current traces may not be long enough to drive
simulations or test bed experiments at higher rates.</t>
<t>The supplied Tmix traces
are used here to provide a standard comparison across scenarios. Data
Centers, however, have very specialised traffic which may not be
represented well in such traces. In the future, specialised Data
Center traffic traces may be needed to provide a more realistic test.</t>
</section>
</section>
<section anchor="sec:access-link" title="Access Link">
<t>The access link scenario models an access link connecting an
institution (e.g., a university or corporation) to an ISP. The
central and edge links are all 100 Mbps. The one-way propagation
delay of the central link is 2 ms, while the edge links have the
delays given in <xref target="sec:RTTs"/>. Our goal in assigning
delays to edge links is only to give a realistic distribution of
round-trip times for traffic on the central link. The Central link buffer
size is 100 ms, which is equivalent to the BDP (using the mean RTT).</t>
<texttable title="Access link scenario parameters (times in seconds)." anchor="tab:alloads">
<ttcol>load</ttcol>
<ttcol>scale</ttcol>
<ttcol>warmup</ttcol>
<ttcol>test_duration</ttcol>
<ttcol>prefill_t</ttcol>
<ttcol>prefill_si</ttcol>
<c>60%</c> <c>5.276</c> <c> 84</c> <c> 479</c> <c>36.72</c> <c>19.445</c>
<c>85%</c> <c>3.812</c> <c>179</c> <c> 829</c> <c>52.02</c> <c>30.745</c>
<c>110%</c> <c>2.947</c> <c> 34</c> <c>1423</c> <c>67.32</c> <c>38.078</c>
</texttable>
<section anchor="potential-revisions-1" title="Potential Revisions">
<t>As faster access links become common, the link speed for this scenario
will need to be updated accordingly. Also as access link buffer sizes
shrink to less than BDP sized buffers, this should be updated to
reflect these changes in the Internet.</t>
</section>
</section>
<section anchor="trans-oceanic-link" title="Trans-Oceanic Link">
<t>The trans-oceanic scenario models a test case where mostly lower-delay
edge links feed into a high-delay central link. Both the central and all
edge links are 1 Gbps. The central link has 100 ms buffers, and a one-way propagation delay of 65 ms.
65 ms is chosen as a “typical number”. The actual delay on real links depends, of
course, on their length. For example, Melbourne to Los Angeles is about 85 ms.</t>
<texttable title="Trans-Oceanic link scenario parameters." anchor="tab:toloads">
<ttcol>load</ttcol>
<ttcol>scale</ttcol>
<ttcol>warmup</ttcol>
<ttcol>test_duration</ttcol>
<ttcol>prefill_t</ttcol>
<ttcol>prefill_si</ttcol>
<c>60%</c> <c>0.5179</c> <c>140</c> <c> 82.5</c> <c> 89.1</c> <c> 30.4</c>
<c>85%</c> <c>0.3091</c> <c> 64</c> <c>252.0</c> <c>126.2</c> <c> 69.9</c>
<c>110%</c> <c>0.2 </c> <c> 82</c> <c>326.0</c> <c>163.4</c> <c>130.5</c>
</texttable>
</section>
<section anchor="sec:geost-satell" title="Geostationary Satellite">
<t>The geostationary satellite scenario models an asymmetric test case
with a high-bandwidth downlink and a low-bandwidth uplink <xref target="HENDERSON99"/><xref target="GURTOV04"/>.
The scenario modeled is that
of nodes connected to a satellite hub which has an asymmetric
satellite connection to the master base station which is connected to
the Internet. The capacity of the central link is asymmetric—40 Mbps down, and 4 Mbps up with a one-way propagation delay
of 300 ms. Edge links are all bidirectional 100 Mbps links
with one-way delays as given in <xref target="sec:RTTs"/>. The central
link buffer size is 100 ms for downlink and 1000 ms for
uplink. </t>
<t>Note that congestion in this case is often on the 4 Mbps uplink (left to right),
even though most of the traffic is in the downlink direction (right to left).</t>
<texttable title="Geostationary satellite link scenario parameters." anchor="tab:gsloads">
<ttcol>load</ttcol>
<ttcol>scale</ttcol>
<ttcol>warmup</ttcol>
<ttcol>test_duration</ttcol>
<ttcol>prefill_t</ttcol>
<ttcol>prefill_si</ttcol>
<c>60%</c> <c>15.0 </c> <c>163</c> <c>2513</c> <c>324.7</c> <c>126.2</c>
<c>85%</c> <c> 9.974</c> <c>230</c> <c>2184</c> <c>460.0</c> <c>219.1</c>
<c>110%</c> <c> 8.062</c> <c>298</c> <c>2481</c> <c>595.3</c> <c>339.5</c>
</texttable>
</section>
<section anchor="wireless-lan" title="Wireless LAN">
<t>The wireless LAN scenario models WiFi access to a wired
backbone, as depicted in <xref target="fig:wirelessdumbell"/>. </t>
<t>The capacity of the central link is 100 Mbps, with a one-way
delay of 2 ms. All links to Router 2 are wired. Router 1 acts as a
base station for a shared wireless IEEE 802.11g links. Although
802.11g has a peak bit rate of 54 Mbps, its typical throughput rate
is much lower, and decreases under high loads and bursty traffic.
The scales specified here are based on a nominal rate of 6 Mbps.</t>
<t>The Node_[123] to Wireless_[123] connections are to allow the same
RTT distribution as for the wired scenarios. This is in addition to
delays on the wireless link due to
CSMA. <xref target="fig:wirelessdumbell"/> shows how the topology should
look in a test bed.</t>
<figure title="Wireless dumbell topology for a test-bed. Wireless_n are wireless transceivers for connection to the base station." anchor="fig:wirelessdumbell"><artwork><![CDATA[
Node_1----Wireless_1.. Node_4
:. /
:... Base central link /
Node_2----Wireless_2 ....:..Station-------------- Router_2 --- Node_5
...: (Router 1) \
.: \
Node_3----Wireless_3.: Node_6
]]></artwork></figure>
<texttable title="Wireless LAN scenario parameters." anchor="tab:wloads">
<ttcol>load</ttcol>
<ttcol>scale</ttcol>
<ttcol>warmup</ttcol>
<ttcol>test_duration</ttcol>
<ttcol>prefill_t</ttcol>
<ttcol>prefill_si</ttcol>
<c>60%</c> <c>105.66</c> <c> 20</c> <c>4147</c> <c>0</c> <c>0</c>
<c>85%</c> <c> 85.93</c> <c> 20</c> <c>5397</c> <c>0</c> <c>0</c>
<c>110%</c> <c> 60.17</c> <c>620</c> <c>1797</c> <c>0</c> <c>0</c>
</texttable>
<t>The percentage load for this scenario is based on the sum of the
estimate of offered load in both directions since the wireless
bottleneck link is a shared media. Also, due to contention for the
bottleneck link, the accelerated start up using prefill is not used
for this scenario.</t>
<section anchor="ns2-implementation-specifics-1" title="NS2 implementation specifics">
<t>In NS2, this is implemented as depicted in <xref target="fig:dumbell"/>.
The delays between Node_1 and
Wireless_1 are implemented as delays through the Logical Link layer.</t>
<t>Since NS2 doesn’t have a simple way of measuring transport packet loss on the
wireless link, dropped packets are inferred based on flow arrivals and departures
(see <xref target="fig:wirelessmeasurements"/>). This gives a good
estimate of the average loss rate over a long enough period (long
compared with the transit delay of packets), which is the case here.</t>
<figure title="Wireless measurements in the ns2 simulator." anchor="fig:wirelessmeasurements"><artwork><![CDATA[
logical link
X--------------------X
| |
v |
n1--+-- . | _n4
: V /
n2--+-- .:.C0-------------C1---n5
: \_
n3--+-- . n6
]]></artwork></figure>
</section>
<section anchor="potential-revisions-2" title="Potential revisions">
<t>Wireless standards are continually evolving. This scenario may
need updating in the future to reflect these changes.</t>
<t>Wireless links have many other unique properties not
captured by delay and bitrate. In particular, the physical layer
might suffer from propagation effects that result in packet losses,
and the MAC layer might add high jitter under contention or large
steps in bandwidth due to adaptive modulation and coding. Specifying
these properties is beyond the scope of the current first version of
this test suite but may make useful additions in the future.</t>
<t>Latency in this scenario is very much affected by contention for the
media. It will be good to have end-to-end delay measurements to
quantify this characteristic. This could include per packet latency,
application burst completion times, and/or application session
completion times.</t>
</section>
</section>
<section anchor="dial-up-link" title="Dial-up Link">
<t>The dial-up link scenario models a network with a dial-up link of
64 kbps and a one-way delay of 5 ms for the central link. This could be
thought of as modeling a scenario reported as typical in Africa, with
many users sharing a single low-bandwidth dial-up link. Central link
buffer size is 1250 ms. Edge links are 100 Mbps.</t>
<texttable title="Dial-up link scenario parameters." anchor="tab:duloads">
<ttcol>load</ttcol>
<ttcol>scale</ttcol>
<ttcol>warmup</ttcol>
<ttcol>test_duration</ttcol>
<ttcol>prefill_t</ttcol>
<ttcol>prefill_si</ttcol>
<c>60%</c> <c>10981.7</c> <c>280</c> <c>168804</c> <c> 559</c> <c> 79</c>
<c>85%</c> <c> 7058.5</c> <c>400</c> <c> 88094</c> <c> 792</c> <c>297</c>
<c>110%</c> <c> 5753.1</c> <c>512</c> <c> 69891</c> <c>1025</c> <c>184</c>
</texttable>
<section anchor="note-on-parameters" title="Note on parameters">
<t>The traffic offered by Tmix over a low bandwidth link is very
bursty. It takes a long time to reach some sort of statistical
stability. For event based simulators, this is not too much of a problem, as the
number of packets transferred is not prohibitively high, however for
test beds these times are prohibitively long. This scenario needs
further investigation to address such issue. </t>
</section>
<section anchor="potential-revisions-3" title="Potential revisions">
<t>Modems often have asymmetric up and down link rates. Asymmetry is
tested in the Geostationary Satellite scenario
(<xref target="sec:geost-satell"/>), but the dial-up scenario could be modified to
model this as well.</t>
</section>
</section>
</section>
<section anchor="metrics-of-interest" title="Metrics of interest">
<t>For each run, the following metrics will be collected for the central
link in each direction:</t>
<t><list style="numbers">
<t>the aggregate link utilization,</t>
<t>the average packet drop rate, and</t>
<t>the average queueing delay.</t>
</list></t>
<t>These measures only provide a general overview of performance. The
goal of this draft is to produce a set of tests that can be “run” at
all levels of abstraction, from Grid500’s WAN, through WAN-in-Lab,
testbeds and simulations all the way to theory. Researchers may add
additional measures to illustrate other performance aspects as
required.</t>
<t>Other metrics of general interest include:</t>
<t><list style="numbers">
<t>end-to-end delay measurements</t>
<t>flow-centric: <vspace blankLines='1'/>
a. sending rate, <vspace blankLines='1'/>
b. goodput, <vspace blankLines='1'/>
c. cumulative loss and queueing delay trajectory for each flow, over time, <vspace blankLines='1'/>
d. the transfer time per flow versus file size</t>
<t>stability properties: <vspace blankLines='1'/>
a. standard deviation of the throughput and the queueing delay for the bottleneck link, <vspace blankLines='1'/>
b. worst case stability measures, especially proving (possibly theoretically) the stability of TCP.</t>
</list></t>
</section>
<section anchor="potential-revisions-4" title="Potential Revisions">
<t>As with all of the scenarios in this document, the basic scenarios
could benefit from more measurement studies about characteristics of
congested links in the current Internet, and about trends that could
help predict the characteristics of congested links in the future.
This would include more measurements on typical packet drop rates, and
on the range of round-trip times for traffic on congested links.</t>
</section>
</section>
<section anchor="latency-specific-experiments" title="Latency specific experiments">
<section anchor="sec-delay-thruput" title="Delay/throughput tradeoff as function of queue size">
<t>Performance in data communications is increasingly limited by
latency. Smaller and smarter buffers improve this measure, but often
at the expense of TCP throughput. The purpose of these tests is
to investigate delay-throughput tradeoffs,
<spanx style="emph">with and without the particular TCP extension under study</spanx>.</t>
<t>Different queue management mechanisms have different delay-throughput
tradeoffs. It is envisaged that the tests described here would be extended to explore and compare the performance of different Active Queue Management (AQM) techniques. However, this is an area of active research and beyond the scope of
this test suite at this time. For now, it may be better to have a
dedicated, separate test suite to look at AQM performance issues.</t>
<section anchor="topology" title="Topology">
<t>These tests use the topology of <xref target="sec:RTTs"/>. They are based on the
access link scenario (see <xref target="sec:access-link"/>) with the 85%
offered load used for this test.</t>
<t>For each Drop-Tail scenario set, five tests are run, with buffer sizes of
10%, 20%, 50%, 100%, and 200% of the Bandwidth Delay Product (BDP)
for a 100 ms base RTT flow (the average base RTT in the access link
dumbbell scenario is 100 ms).</t>
<section anchor="potential-revisions-5" title="Potential revisions">
<t>Buffer sizing is still an area of research. Results from this
research may necessitate changes to the test suite so that it models
these changes in the Internet.</t>
<t>AQM is currently an area of active research as well. It is envisaged that these tests could be extended to explore and compare the performance of key AQM techniques when it
becomes clear what these will be. For now a dedicated AQM test
suite would best serve such research efforts.</t>
</section>
</section>
<section anchor="flows-under-test-1" title="Flows under test">
<t>Two kinds of tests should be run: one where all TCP flows use the TCP modification under study, and another where no TCP flows use such modification, as a “baseline” version.</t>
<t>The level of traffic from the traffic generator is the same as that
described in <xref target="sec:access-link"/>.</t>
</section>
<section anchor="metrics-of-interest-1" title="Metrics of interest">
<t>For each test, three figures are kept: the average throughput, the
average packet drop rate, and the average queueing delay over the
measurement period.</t>
<t>Ideally it would be better to have more complete statistics,
especially for queueing delay where the delay distribution can be
important. It would also be good for this to be illustrated with a
delay/bandwidth graph, where the x-axis shows the average queueing delay,
and the y-axis shows the average throughput. For the drop-rate graph,
the x-axis shows the average queueing delay, and the y-axis shows the
average packet drop rate. Each pair of graphs illustrates the
delay/throughput/drop-rate tradeoffs with and without the TCP
mechanism under evaluation. For an AQM mechanism, each pair of graphs also illustrates
how the throughput and average queue size vary (or don’t vary) as a
function of the traffic load. Examples of delay/throughput tradeoffs
appear in Figures 1-3 of <xref target="FLOYD01"/> and Figures 4-5 of <xref target="ANDREW08"/>.</t>
</section>
</section>
<section anchor="sec:ramp-up-time" title="Ramp up time: completion time of one flow">
<t>These tests aim to determine how quickly existing flows make room for new flows.</t>
<section anchor="topology-and-background-traffic" title="Topology and background traffic">
<t>The ramp up time test uses the topology shown in
<xref target="fig:RampUpDumbell"/>. Two long-lived test TCP connections
are used in this experiment. Test TCP connection 1 is connected
between T_n1 and T_n3, with data flowing from T_n3 to T_n1, and
test TCP source 2 is connected between T_n2 and T_n4, with data
flowing from T_n4 to T_n2. The background traffic topology is
identical to that used in the basic scenarios (see
<xref target="sec:basic-scenarios"/> and <xref target="fig:dumbell"/>); i.e.,
background flows run between nodes B_n1 to B_n6.</t>
<figure title="Ramp up dumbbell test topology." anchor="fig:RampUpDumbell"><artwork><![CDATA[
T_n2 T_n4
| |
| |
T_n1 | | T_n3
\ | | /
\ | |/
B_n1--- R1--------------------------R2--- B_n4
/ | |\
/ | | \
B_n2 | | B_n5
| |
B_n3 B_n6
]]></artwork></figure>
<t>Experiments are conducted with capacities of 10 Mbps and
1 Gbps for the central link. Edge links are 1 Gbps.</t>
<t>For each capacity, three RTT scenarios should be tested, in which the
existing and newly arriving flow have RTTs of (80,80), (120,30), and
(30,120) respectively. This is achieved by having a central link with 2 ms delay in each
direction, and test link delays as shown in <xref target="tab-ramp-up"/>.</t>
<t>The buffers in R1 and R2 are sized at BDP (80ms worth of 1500B packet
buffering).</t>
<texttable title="Link delays for the test TCP source connections to the central link. Link delays are in milliseconds." anchor="tab-ramp-up">
<ttcol>RTT scenario</ttcol>
<ttcol>T_n1</ttcol>
<ttcol>T_n2</ttcol>
<ttcol>T_n3</ttcol>
<ttcol>T_n4</ttcol>
<c>1</c> <c> 0</c> <c> 0</c> <c>38</c> <c>38</c>
<c>2</c> <c>23</c> <c>12</c> <c>35</c> <c> 1</c>
<c>3</c> <c>12</c> <c>23</c> <c> 1</c> <c>35</c>
</texttable>
<texttable title="Ramp-up time scenario parameters (times in seconds)." anchor="tab:rutloads">
<ttcol>Test</ttcol>
<ttcol>Central link</ttcol>
<ttcol>Seed offset</ttcol>
<ttcol>scale</ttcol>
<ttcol>warmup</ttcol>
<ttcol>prefill_t</ttcol>
<ttcol>prefill_si</ttcol>
<c>1</c> <c>10 Mbps</c> <c> 1</c> <c>77.322</c> <c> 12</c> <c>500</c> <c>131.18</c>
<c>2</c> <c>10 Mbps</c> <c>11</c> <c>72.992</c> <c>114</c> <c>500</c> <c>187.14</c>
<c>3</c> <c>10 Mbps</c> <c>21</c> <c>68.326</c> <c> 12</c> <c>500</c> <c>246.13</c>
<c>1</c> <c> 1 Gbps</c> <c> 1</c> <c> 0.7 </c> <c>102</c> <c>200</c> <c>100.11</c>
<c>2</c> <c> 1 Gbps</c> <c>11</c> <c> 0.7 </c> <c>102</c> <c>200</c> <c>103.07</c>
<c>3</c> <c> 1 Gbps</c> <c>21</c> <c> 0.7 </c> <c>102</c>
<c>200</c> <c>101.02</c>
<postamble>For all tests: test_duration = 600 seconds.</postamble>
</texttable>
<t>For each RTT scenario, three tests are run with a different offset to
the random number generator’s base seed (see <xref target="tab:rutloads"/>).</t>
<t>Throughout the experiment, the offered load of the background (or
cross) traffic is 50% of the central link capacity in the right to
left direction. The background traffic is generated in the same manner
as for the basic scenarios (see <xref target="sec:basic-scenarios"/>)
except that the bin size for shuffling is set to 3 s for all scenarios.</t>
<t>All traffic for this scenario uses the TCP extension under test.</t>
</section>
<section anchor="flows-under-test-2" title="Flows under test">
<t>Traffic is dominated by the two long lived test flows, because we believe that
to be the worst case, in which convergence is slowest.</t>
<t>One flow starts in “equilibrium” (at least having finished normal
slow-start). A new flow then starts; slow-start is disabled by
setting the initial slow-start threshold to the initial CWND. Slow
start is disabled because this is the worst case, and could happen if
a loss occurred in the first RTT.</t>
<t>Both of the flows use 1500-byte packets. The test should be run both with
Standard TCP and with the TCP extension under test for comparison.</t>
<section anchor="potential-revisions-6" title="Potential Revisions">
<t>It may also be useful to conduct the tests with slow start
enabled too, if time permits.</t>
</section>
</section>
<section anchor="metrics-of-interest-2" title="Metrics of interest">
<t>The output of these experiments are the time until the (1500 * 10^n)-th
byte of the new flow is received, for n = 1,2,… . This measures how
quickly the existing flow releases capacity to the new flow, without
requiring a definition of when “fairness” has been achieved. By
leaving the upper limit on n unspecified, the test remains applicable
to very high-speed networks.</t>
<t>A single run of this test cannot achieve statistical reliability by
running for a long time. Instead, an average over at least three runs
should be taken. Different cross traffic is generated using the
standard Tmix trace files by changing the random number seed used to
shuffle the traces (as listed in <xref target="tab:rutloads"/>).</t>
</section>
</section>
<section anchor="sec:trans-rele-bandw" title="Transients: release of bandwidth, arrival of many flows">
<t>These tests investigate the impact of a sudden change of congestion level.
They differ from the “Ramp up time” test in that the congestion here is caused
by unresponsive traffic.</t>
<t>Note that this scenario has not yet been implemented in the NS2
example test suite.</t>
<section anchor="topology-and-background-traffic-1" title="Topology and background traffic">
<t>The network is a single bottleneck link (see <xref target="fig:Transient"/>), with bit
rate 100 Mbps, with a buffer of 1024 packets (i.e., 120% of the BDP
at 100 ms). Edge links are also 100 Mbps.</t>
<figure title="Transient test topology." anchor="fig:Transient"><artwork><![CDATA[
T T
\ /
\ /
R1--------------------------R2
/ \
/ \
U U
]]></artwork></figure>
<t>The transient traffic is generated using UDP, to avoid overlap with
the ramp-up time scenario (see <xref target="sec:ramp-up-time"/>) and
isolate the behavior of the flows under study.</t>
<t>Three transients are tested:</t>
<t><list style="numbers">
<t>step decrease from 75 Mbps to 0 Mbps,</t>
<t>step increase from 0 Mbps to 75 Mbps,</t>
<t>30 step increases of 2.5 Mbps at 1 s intervals.</t>
</list></t>
<t>These transients occur after the flow under test has exited
slow-start, and remain until the end of the experiment.</t>
<t>There is no TCP cross traffic in this experiment.</t>
</section>
<section anchor="flows-under-test-3" title="Flows under test">
<t>There is one flow under test: a long-lived flow in the same direction as the
transient traffic, with a 100 ms RTT. The test should be run both with
Standard TCP and with the TCP extension under test for comparison.</t>
</section>
<section anchor="metrics-of-interest-3" title="Metrics of interest">
<t>For the decrease in cross traffic, the metrics are</t>
<t><list style="numbers">
<t>the time taken for the TCP flow under test to increase its window to 60%, 80% and 90% of its BDP, and</t>
<t>the maximum change of the window in a single RTT while the window is increasing to that value.</t>
</list></t>
<t>For cases with an increase in cross traffic, the metric is
the number of <spanx style="emph">cross traffic</spanx> packets dropped from the start of the
transient until 100 s after the transient. This measures the harm caused by
algorithms which reduce their rates too slowly on congestion.</t>
</section>
</section>
</section>
<section anchor="throughput-and-fairness-related-experiments" title="Throughput- and fairness-related experiments">
<section anchor="impact-on-standard-tcp-traffic" title="Impact on standard TCP traffic">
<t>Many new TCP proposals achieve a gain, G, in their own throughput at the
expense of a loss, L, in the throughput of standard TCP flows sharing a
bottleneck, as well as by increasing the link utilization.
In this context a “standard TCP flow” is defined as a flow using SACK
TCP <xref target="RFC2883"/> but without ECN <xref target="RFC3168"/>.</t>
<t>The intention is for a “standard TCP flow” to correspond to
TCP as commonly deployed in the Internet today (with the notable exception of
CUBIC, which runs by default on the majority of web servers).
This scenario quantifies this trade off.</t>
<section anchor="topology-and-background-traffic-2" title="Topology and background traffic">
<t>The basic dumbbell topology of <xref target="sec:RTTs"/>
is used with the same capacities as for the
ramp-up time tests in <xref target="sec:ramp-up-time"/>.
All traffic in this scenario comes from the flows under test.</t>
<figure title="Dumbbell Topology for Assessing Impact on Standard TCP." anchor="fig:impact"><artwork><![CDATA[
A_1 A_4
B_1 B_4
\ /
\ central link /
A_2 --- Router_1 -------------- Router_2 --- A_5
B_2 / \ B_5
/ \
A_3 A_6
B_3 B_6
]]></artwork></figure>
</section>
<section anchor="flows-under-test-4" title="Flows under test">
<t>The scenario is performed by conducting pairs of experiments, with
identical flow arrival times and flow sizes. Within each experiment,
flows are divided into two camps. For every flow in camp A, there is
a flow with the same size, source and destination in camp B, and vice
versa.</t>
<t>These experiments use duplicate copies of the Tmix traces used in the
basic scenarios (see <xref target="sec:basic-scenarios"/>). Two offered
loads are tested: 50% and 100%. </t>
<t>Two experiments are conducted. A BASELINE experiment where both camp A and camp B use
standard TCP. In the second, called MIX, camp A uses standard TCP
and camp B uses the new TCP extension under evaluation.</t>
<t>The rationale for having paired camps is to remove the statistical
uncertainty which would come from randomly choosing half of the flows
to run each algorithm. This way, camp A and camp B have the same
loads.</t>
<texttable title="Impact on Standard TCP scenario parameters." anchor="tab:itloads">
<ttcol>load</ttcol>
<ttcol>scale</ttcol>
<ttcol>warmup</ttcol>
<ttcol>test_duration</ttcol>
<ttcol>prefill_t</ttcol>
<ttcol>prefill_si</ttcol>
<c>50%</c> <c>13.587</c> <c>26</c> <c>508</c> <c>45.90</c> <c>14.61</c>
<c>100%</c> <c> 5.780</c> <c>50</c> <c>498</c> <c>91.80</c> <c>22.97</c>
</texttable>
</section>
<section anchor="metrics-of-interest-4" title="Metrics of interest">
<t>The gain achieved by the new algorithm and loss incurred by standard
TCP are given, respectively, by G = T(B)_Mix/T(B)_Baseline and
L = T(A)_Mix/T(A)_Baseline where T(x) is the throughput obtained
by camp x, measured as the amount of data acknowledged by the
receivers (that is, “goodput”).</t>
<t>The loss, L, is analogous to the “bandwidth stolen from TCP” in <xref target="SOUZA03"/>
and “throughput degradation” in <xref target="SHIMONISHI07"/>.</t>
<t>A plot of G vs L represents the tradeoff between
efficiency and loss.</t>
<section anchor="suggestions" title="Suggestions">
<t>Other statistics of interest are the values of G and L for
each quartile of file sizes. This will reveal whether the new
proposal is more aggressive in starting up or more reluctant to
release its share of capacity.</t>
<t>As always, testing at other loads and averaging over multiple runs is
encouraged.</t>
</section>
</section>
</section>
<section anchor="sec:intra-protocol-inter-RTT" title="Intra-protocol and inter-RTT fairness">
<t>These tests aim to measure bottleneck bandwidth sharing
among flows of the same protocol with the same RTT, which represents
the flows going through the same routing path.
The tests also measure inter-RTT fairness, the bandwidth sharing among flows
of the same protocol where routing paths have a common
bottleneck segment but might have different overall paths
with different RTTs. </t>
<section anchor="topology-and-background-traffic-3" title="Topology and background traffic">
<t>The topology, the capacity and cross traffic conditions of these tests
are the same as in <xref target="sec:ramp-up-time"/>.
The bottleneck
buffer is varied from 25% to 200% of the BDP for a 100 ms base RTT
flow, increasing by factors of 2.</t>
</section>
<section anchor="flows-under-test-5" title="Flows under test">
<t>We use two flows of the same protocol variant for this experiment. The RTTs
of the flows range from 10 ms to 160 ms (10 ms, 20 ms, 40 ms,
80 ms, and 160 ms) such that the ratio of the minimum RTT over the
maximum RTT is at most 1/16.</t>
<section anchor="intra-protocol-fairness" title="Intra-protocol fairness">
<t>For each run, two flows with the
same RTT, taken from the range of RTTs above, start randomly within the
first 10% of the experiment duration. The order in which these flows start
doesn’t matter. An additional test of interest, but not part of this
suite, would involve two extreme cases – two flows with very short or
long RTTs (e.g., a delay less than 1-2 ms representing communication
happening in a data-center, and a delay larger than 600 ms representing
communication over a satellite link).</t>
</section>
<section anchor="inter-rtt-fairness" title="Inter-RTT fairness">
<t>For each run, one flow with a fixed RTT of 160 ms
starts first, and another flow with a different RTT taken from the
range of RTTs above, joins afterward. The starting times of both two
flows are randomly chosen within the first 10% of the experiment as
before.</t>
</section>
</section>
<section anchor="metrics-of-interest-5" title="Metrics of interest">
<t>The output of this experiment is the ratio of the average throughput
values of the two flows. The output also includes the packet drop rate
for the congested link.</t>
</section>
</section>
<section anchor="multiple-bottlenecks" title="Multiple bottlenecks">
<t>These experiments explore the relative bandwidth for a flow
that traverses multiple bottlenecks, with respect to that of flows that have the same round-trip time
but each traverse only one of the bottleneck links.</t>
<section anchor="topology-and-traffic" title="Topology and traffic">
<t>The topology is a “parking-lot” topology with
three (horizontal) bottleneck links and four (vertical) access links.
The bottleneck links have a rate of 100 Mbps,
and the access links have a rate of 1 Gbps. </t>
<t>All flows have a round-trip time of 60 ms, to enable the effect of
traversing multiple bottlenecks to be distinguished from that of
different round trip times.</t>
<t>This can be achieved in both a symmetric and asymmetric way (see
<xref target="fig:Asymmetricparkinglot"/> and <xref target="fig:Symmetricparkinglot"/>).
It is not clear whether there are
interesting performance differences between these two topologies, and
if so, which is more typical of the actual Internet.</t>
<figure title="Asymmetric parking lot topology." anchor="fig:Asymmetricparkinglot"><artwork><![CDATA[
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >
__________ 0ms _________________ 0ms __________________ 30ms ____
| ................ | ................ | ................ |
| : : | : : | : : |
| : : | : : | : : |
0ms : : 30ms : : 0ms : : 0ms
| ^ V | ^ V | ^ V |
]]></artwork></figure>
<figure title="Symmetric parking lot topology." anchor="fig:Symmetricparkinglot"><artwork><![CDATA[
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >
__________ 10ms _______________ 10ms ________________ 10ms ___
| ............... | ............... | ............... |
| : : | : : | : : |
| : : | : : | : : |
10ms : : 10ms : : 10ms : : 10ms
| ^ V | ^ V | ^ V |
]]></artwork></figure>
<t>The three hop topology used in the test suite is based on the
symmetric topology (see <xref target="fig:parkinglot"/>). Bidirectional traffic flows between
Nodes 1 and 8, 2 and 3, 4 and 5, and 6 and 7.</t>
<figure title="Test suite parking lot topology." anchor="fig:parkinglot"><artwork><![CDATA[
Node_1 Node_3 Node_5 Node_7
\ | | /
\ |10ms |10ms /10ms
0ms\ | | /
\ A | B | C /
Router1 ---Router2---Router3--- Router4
/ 10ms | 10ms | 10ms \
/ | | \
10ms/ |10ms |10ms \ 0ms
/ | | \
Node_2 Node_4 Node_6 Node_8
Flow 1: Node_1 <--> Node_8
Flow 2: Node_2 <--> Node_3
Flow 3: Node_4 <--> Node_5
Flow 4: Node_6 <--> Node_7
]]></artwork></figure>
<t>The r4s1.org Tmix trace file is used to generate the traffic. Each
Tmix source offers the same load for each experiment. Three
experiments are conducted at 30%, 40%, and 50% offered loads per
Tmix source. As two sources share each of the three bottlenecks
(A,B,C), the combined offered loads on the bottlenecks is 60%, 80%,
and 100% respectively.</t>
<t>All traffic uses the new TCP extension under test.</t>
<texttable title="Multiple bottleneck scenario parameters." anchor="tab:mbloads">
<ttcol>load</ttcol>
<ttcol>scale</ttcol>
<ttcol>warmup</ttcol>
<ttcol>test_duration</ttcol>
<ttcol>prefill_t</ttcol>
<ttcol>prefill_si</ttcol>
<c>60%</c> <c>1.1904</c> <c>173</c> <c> 470</c> <c>41.4</c> <c> 6.827</c>
<c>80%</c> <c>0.9867</c> <c> 37</c> <c>2052</c> <c>55.2</c> <c> 6.858</c>
<c>100%</c> <c>0.7222</c> <c> 38</c> <c>1338</c> <c>69.0</c> <c>13.740</c>
</texttable>
<section anchor="potential-revisions-7" title="Potential Revisions">
<t>Parking lot models with more hops may also be of interest.</t>
</section>
</section>
<section anchor="metrics-of-interest-6" title="Metrics of interest">
<t>The output for this experiment is the ratio between the average
throughput of the single-bottleneck flows and the
throughput of
the multiple-bottleneck flow, measured after the warmup period.
Output also includes the packet drop rate for the congested link.</t>
</section>
</section>
</section>
<section anchor="implementations" title="Implementations">
<t>At the moment the only implementation effort is using the NS2
simulator. It is still a work in progress, but contains the base to
most of the tests, as well as the algorithms that determined the test
parameters. It is being made available to the community for further
development and verification through
<eref target="https://bitbucket.org/hayesd/tcp-evaluation-suite-public">https://bitbucket.org/hayesd/tcp-evaluation-suite-public</eref>.</t>
<t>At the moment there are no ongoing test bed implementations. We invite
the community to initiate and contribute to the development of these
test beds.</t>
</section>
<section anchor="acknowledgements" title="Acknowledgements">
<t>This work is based on a paper by
Lachlan Andrew,
Cesar Marcondes,
Sally Floyd,
Lawrence Dunn,
Romaric Guillier,
Wang Gang,
Lars Eggert,
Sangtae Ha and
Injong Rhee <xref target="TESTSUITE08"/>.</t>
<t>The authors would also like to thank Roman Chertov, Doug Leith, Saverio Mascolo,
Ihsan Qazi, Bob Shorten, David Wei and Michele Weigle for valuable
feedback and acknowledge the work of Wang Gang to start the NS2 implementation.</t>
<t>This work has been partly funded by the European Community under its Seventh Framework
Programme through the Reducing Internet Transport Latency (RITE) project (ICT-317700),
by the Aurora–Hubert Curien Partnership program “ANT” (28844PD /
221629), and under
Australian Research Council’s Discovery Projects funding scheme
(project number 0985322).</t>
</section>
</middle>
<back>
<references title='Informative References'>
<reference anchor='RFC2883'>
<front>
<title>An Extension to the Selective Acknowledgement (SACK) Option for TCP</title>
<author initials='S.' surname='Floyd' fullname='S. Floyd'>
<organization /></author>
<author initials='J.' surname='Mahdavi' fullname='J. Mahdavi'>
<organization /></author>
<author initials='M.' surname='Mathis' fullname='M. Mathis'>
<organization /></author>
<author initials='M.' surname='Podolsky' fullname='M. Podolsky'>
<organization /></author>
<date year='2000' month='July' />
<abstract>
<t>This note defines an extension of the Selective Acknowledgement (SACK) Option for TCP. [STANDARDS-TRACK]</t></abstract></front>
<seriesInfo name='RFC' value='2883' />
<format type='TXT' octets='35794' target='http://www.rfc-editor.org/rfc/rfc2883.txt' />
</reference>
<reference anchor='RFC3168'>
<front>
<title>The Addition of Explicit Congestion Notification (ECN) to IP</title>
<author initials='K.' surname='Ramakrishnan' fullname='K. Ramakrishnan'>
<organization /></author>
<author initials='S.' surname='Floyd' fullname='S. Floyd'>
<organization /></author>
<author initials='D.' surname='Black' fullname='D. Black'>
<organization /></author>
<date year='2001' month='September' />
<abstract>
<t>This memo specifies the incorporation of ECN (Explicit Congestion Notification) to TCP and IP, including ECN's use of two bits in the IP header. [STANDARDS-TRACK]</t></abstract></front>
<seriesInfo name='RFC' value='3168' />
<format type='TXT' octets='170966' target='http://www.rfc-editor.org/rfc/rfc3168.txt' />
</reference>
<reference anchor="TESTSUITE08" target="http://www.caia.swin.edu.au/cv/landrew/pubs/TCP-suite-PFLDnet.pdf">
<front>
<title>Towards a Common TCP Evaluation Suite</title>
<author initials="L. L. H." surname="Andrew">
<organization></organization>
</author>
<author initials="C." surname="Marcondes">
<organization></organization>
</author>
<author initials="S." surname="Floyd">
<organization></organization>
</author>
<author initials="L." surname="Dunn">
<organization></organization>
</author>
<author initials="R." surname="Guillier">
<organization></organization>
</author>
<author initials="W." surname="Gang">
<organization></organization>
</author>
<author initials="L." surname="Eggert">
<organization></organization>
</author>
<author initials="S." surname="Ha">
<organization></organization>
</author>
<author initials="I." surname="Rhee">
<organization></organization>
</author>
<date year="2008" month="March"/>
</front>
<seriesInfo name="Protocols for Fast, Long Distance Networks (PFLDnet)" value=""/>
</reference>
<reference anchor="I-D-TMRG-TESTS" target="http://tools.ietf.org/html/draft-irtf-tmrg-tests">
<front>
<title>Common TCP Evaluation Suite</title>
<author initials="L. L. H." surname="Andrew">
<organization></organization>
</author>
<author initials="S." surname="Floyd">
<organization></organization>
</author>
<author initials="W." surname="Gang">
<organization></organization>
</author>
<date year="2009" month="July"/>
</front>
<seriesInfo name="Internet Draft draft-irtf-tmrg-tests-02, work in progress" value=""/>
</reference>
<reference anchor="FLOYD03" >
<front>
<title>Internet research needs better models</title>
<author initials="S." surname="Floyd">
<organization></organization>
</author>
<author initials="E." surname="Kohler">
<organization></organization>
</author>
<date year="2003" month="January"/>
</front>
<seriesInfo name="SIGCOMM Computer Communication Review" value=""/>
</reference>
<reference anchor="MASCOLO06" >
<front>
<title>The Effect of Reverse Traffic on the Performance of New TCP Congestion Control Algorithms for Gigabit Networks</title>
<author initials="S." surname="Mascolo">
<organization></organization>
</author>
<author initials="F." surname="Vacirca">
<organization></organization>
</author>
<date year="2006"/>
</front>
<seriesInfo name="Protocols for Fast, Long Distance Networks (PFLDnet)" value=""/>
</reference>
<reference anchor="ROSSI03" >
<front>
<title>User patience and the Web: a hands-on investigation</title>
<author initials="D." surname="Rossi">
<organization></organization>
</author>
<author initials="M." surname="Mellia">
<organization></organization>
</author>
<author initials="C." surname="Casetti">
<organization></organization>
</author>
<date year="2003"/>
</front>
<seriesInfo name="IEEE GLOBECOM" value=""/>
</reference>
<reference anchor="WEIGLE06" >
<front>
<title>Tmix: a tool for generating realistic TCP application workloads in ns-2</title>
<author initials="M. C." surname="Weigle">
<organization></organization>
</author>
<author initials="P." surname="Adurthi">
<organization></organization>
</author>
<author initials="F." surname="Hernandez-Campos">
<organization></organization>
</author>
<author initials="K." surname="Jeffay">
<organization></organization>
</author>
<author initials="F. D." surname="Smith">
<organization></organization>
</author>
<date year="2006" month="July"/>
</front>
<seriesInfo name="SIGCOMM Computer Communication Review" value=""/>
</reference>
<reference anchor="GENITMIX" target="http://groups.geni.net/geni/wiki/GeniTmix">
<front>
<title>Tmix on ProtoGENI</title>
<author>
<organization>GENI project</organization>
</author>
<date/>
</front>
</reference>
<reference anchor="TRACES" target="http://web.archive.org/web/20100711061914/http://wil-ns.cs.caltech.edu/~benchmark/traffic/">
<front>
<title>Tmix trace generation for the TCP evaluation suite</title>
<author>
<organization>Caltech</organization>
</author>
<date year="n.d."/>
</front>
</reference>
<reference anchor="SHUFFLEWIKI" target="http://en.wikipedia.org/wiki/Fisher-Yates_shuffle">
<front>
<title>Fisher-Yates shuffle</title>
<author/>
<date/>
</front>
</reference>
<reference anchor="ALIZADEH10" >
<front>
<title>Data center TCP (DCTCP)</title>
<author initials="M." surname="Alizadeh">
<organization></organization>
</author>
<author initials="A." surname="Greenberg">
<organization></organization>
</author>
<author initials="D. A." surname="Maltz">
<organization></organization>
</author>
<author initials="J." surname="Padhye">
<organization></organization>
</author>
<author initials="P." surname="Patel">
<organization></organization>
</author>
<author initials="B." surname="Prabhakar">
<organization></organization>
</author>
<author initials="S." surname="Sengupta">
<organization></organization>
</author>
<author initials="M." surname="Sridharan">
<organization></organization>
</author>
<date year="2010"/>
</front>
<seriesInfo name="ACM SIGCOMM 2010" value=""/>
</reference>
<reference anchor="HENDERSON99" >
<front>
<title>Transport protocols for Internet-compatible satellite networks</title>
<author initials="T. R." surname="Henderson">
<organization></organization>
</author>
<author initials="R. H." surname="Katz">
<organization></organization>
</author>
<date year="1999"/>
</front>
<seriesInfo name="IEEE Journal on Selected Areas in Communications" value=""/>
</reference>
<reference anchor="GURTOV04" >
<front>
<title>Modeling wireless links for transport protocols</title>
<author initials="A." surname="Gurtov">
<organization></organization>
</author>
<author initials="S." surname="Floyd">
<organization></organization>
</author>
<date year="2004" month="April"/>
</front>
<seriesInfo name="SIGCOMM Computer Communication Review" value=""/>
</reference>
<reference anchor="FLOYD01" target="http://www.icir.org/floyd/papers/adaptiveRed.pdf">
<front>
<title>Adaptive RED: An Algorithm for Increasing the Robustness of RED</title>
<author initials="S." surname="Floyd">
<organization></organization>
</author>
<author initials="R." surname="Gummadi">
<organization></organization>
</author>
<author initials="S." surname="Shenker">
<organization></organization>
</author>
<date year="2001"/>
</front>
<seriesInfo name="ICIR Technical Report" value=""/>
</reference>
<reference anchor="ANDREW08" >
<front>
<title>Active Queue Management for Fair Resource Allocation in Wireless Networks</title>
<author initials="L. L. H." surname="Andrew">
<organization></organization>
</author>
<author initials="S." surname="Hanly">
<organization></organization>
</author>
<author initials="R. G." surname="Mukhtar">
<organization></organization>
</author>
<date year="2008" month="February"/>
</front>
<seriesInfo name="IEEE Transactions on Mobile Computing" value=""/>
</reference>
<reference anchor="SOUZA03" >
<front>
<title>A HighSpeed TCP Study: Characteristics and Deployment Issues</title>
<author initials="E." surname="Souza">
<organization></organization>
</author>
<author initials="D. A." surname="Agarwal">
<organization></organization>
</author>
<date year="2003"/>
</front>
<seriesInfo name="LBNL Technical Report LBNL-53215" value=""/>
</reference>
<reference anchor="SHIMONISHI07" >
<front>
<title>Assessing Interactions among Legacy and High-Speed TCP Protocols</title>
<author initials="H." surname="Shimonishi">
<organization></organization>
</author>
<author initials="M." surname="Sanadidi">
<organization></organization>
</author>
<author initials="T." surname="Murase">
<organization></organization>
</author>
<date year="2007"/>
</front>
<seriesInfo name="Protocols for Fast, Long Distance Networks (PFLDnet)" value=""/>
</reference>
<reference anchor="HOHN03" >
<front>
<title>The impact of the flow arrival process in Internet traffic</title>
<author initials="N." surname="Hohn">
<organization></organization>
</author>
<author initials="D." surname="Veitch">
<organization></organization>
</author>
<author initials="P." surname="Abry">
<organization></organization>
</author>
<date year="2003"/>
</front>
<seriesInfo name="IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '03)" value=""/>
</reference>
<reference anchor="KELLY79" >
<front>
<title>Reversibility and stochastic networks</title>
<author initials="F. P." surname="Kelly">
<organization></organization>
</author>
<date year="1979"/>
</front>
<seriesInfo name="University of Cambridge Statistical Laboratory" value=""/>
</reference>
</references>
<section anchor="discussions-on-traffic" title="Discussions on Traffic">
<t>While the protocols being tested may differ, it is important that we
maintain the same “load” or level of congestion for the experimental
scenarios. To enable this, we use a hybrid of open-loop and
close-loop approaches. For this test suite, network traffic consists
of sessions corresponding to individual users. Because users are
independent, these session arrivals are well modeled by an open-loop
Poisson process. A session may consist of a single greedy TCP flow,
multiple greedy flows separated by user “think” times, a single
non-greedy flow with embedded think times, or many non-greedy “thin stream” flows.
The session arrival process forms a Poisson process <xref target="HOHN03"/>.
Both the
think times and burst sizes have heavy-tailed distributions, with the
exact distribution based on empirical studies. The think times and
burst sizes will be chosen independently. This is unlikely to be the
case in practice, but we have not been able to find any measurements
of the joint distribution. We invite researchers to study this joint
distribution, and future revisions of this test suite will use such
statistics when they are available.</t>
<t>For most current traffic generators, the traffic is specified by an
arrival rate for independent user sessions, along with specifications
of connection sizes, number of connections per sessions, user wait
times within sessions, and the like. Because the session arrival times
are specified independently of the transfer times, one way to specify the
load would be as</t>
<figure><artwork><![CDATA[
A = E[f]/E[t],
]]></artwork></figure>
<t>where E[f] is the mean session size (in bits transferred),
E[t] is the mean session inter-arrival time in seconds,
and A is the load in bps.</t>
<t>Instead, for equilibrium experiments, we measure the load as the
“mean number of jobs in an M/G/1 queue using processor sharing,”
where a job is a user session. This reflects the fact that TCP aims
at processor sharing of variable sized files. Because processor
sharing is a symmetric discipline <xref target="KELLY79"/>,
the mean number of flows is equal to that of an M/M/1 queue, namely
rho/(1-rho), where rho=lambda S/C, and
lambda is the arrival rate of
jobs/flows (in flows per second), S is the mean job size (in bits) and
C is the bottleneck capacity (in bits per second). For small
loads, say 10%,
this is essentially equal to the fraction of the capacity that is used. However,
for overloaded systems, the fraction of the bandwidth used will be
much less than this measure of load.</t>
<t>In order to minimize the dependence of the results on the experiment durations,
scenarios should be as stationary as possible. To this end, experiments will
start with rho/(1-rho) active cross-traffic flows,
with traffic of the specified load.</t>
</section>
</back>
</rfc>
| PAFTECH AB 2003-2026 | 2026-04-24 01:52:21 |