One document matched: draft-ietf-rmcat-eval-test-04.xml
<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [
<!ENTITY rfc2119 PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.2119.xml">
<!ENTITY rfc3550 PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.3550.xml">
<!ENTITY rfc3551 PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.3551.xml">
<!ENTITY rfc3611 PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.3611.xml">
<!ENTITY rfc4585 PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.4585.xml">
<!ENTITY rfc5506 PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.5506.xml">
<!ENTITY rfc5166 PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.5166.xml">
<!ENTITY rfc5033 PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.5033.xml">
<!ENTITY rfc5681 PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.5681.xml">
<!ENTITY I-D.ietf-rmcat-cc-requirements PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml3/reference.I-D.ietf-rmcat-cc-requirements.xml">
<!ENTITY I-D.ietf-avtcore-rtp-circuit-breakers PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml3/reference.I-D.ietf-avtcore-rtp-circuit-breakers.xml">
<!ENTITY I-D.ietf-rmcat-eval-criteria PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml3/reference.I-D.ietf-rmcat-eval-criteria.xml">
<!ENTITY I-D.ietf-rtcweb-use-cases-and-requirements PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml3/reference.I-D.ietf-rtcweb-use-cases-and-requirements.xml">
<!ENTITY I-D.ietf-rmcat-wireless-tests PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml3/reference.I-D.ietf-rmcat-wireless-tests.xml">
<!ENTITY I-D.ietf-rmcat-video-traffic-model PUBLIC "" "http://xml2rfc.tools.ietf.org/public/rfc/bibxml3/reference.I-D.ietf-rmcat-video-traffic-model.xml">
]>
<?xml-stylesheet type='text/xsl' href='rfc2629.xslt' ?>
<?rfc toc="yes" ?>
<?rfc compact="yes" ?>
<?rfc symrefs="yes" ?>
<rfc category="info" docName="draft-ietf-rmcat-eval-test-04" ipr="trust200902">
<!-- What is the category field value-->
<front>
<title abbrev="Test Scenarios for RMCAT">Test Cases for Evaluating RMCAT
Proposals</title>
<author fullname="Zaheduzzaman Sarker" initials="Z" surname="Sarker">
<organization>Ericsson AB</organization>
<address>
<postal>
<street></street>
<city>Luleå</city>
<region>SE</region>
<code>977 53</code>
<country>Sweden</country>
</postal>
<phone>+46 10 717 37 43</phone>
<email>zaheduzzaman.sarker@ericsson.com</email>
</address>
</author>
<author fullname="Varun Singh" initials="V" surname="Singh">
<organization abbrev="callstats.io">Nemu Dialogue Systems
Oy</organization>
<address>
<postal>
<street>Runeberginkatu 4c A 4</street>
<city>Helsinki</city>
<code>00100</code>
<country>Finland</country>
</postal>
<email>varun.singh@iki.fi</email>
<uri>http://www.callstats.io/</uri>
</address>
</author>
<author fullname="Xiaoqing Zhu" initials="X" surname="Zhu">
<organization>Cisco Systems</organization>
<address>
<postal>
<street>12515 Research Blvd</street>
<city>Austing</city>
<region>TX</region>
<code>78759</code>
<country>USA</country>
</postal>
<email>xiaoqzhu@cisco.com</email>
</address>
</author>
<author fullname="Michael A. Ramalho" initials="M. A." surname="Ramalho">
<organization abbrev="Cisco Systems">Cisco Systems, Inc.</organization>
<address>
<postal>
<street>6310 Watercrest Way Unit 203</street>
<city>Lakewood Ranch</city>
<region>FL</region>
<code>34202-5211</code>
<country>USA</country>
</postal>
<phone>+1 919 476 2038</phone>
<email>mramalho@cisco.com</email>
</address>
</author>
<date day="26" month="October" year="2016" />
<area>TSV</area>
<keyword>Multimedia</keyword>
<keyword>Test cases</keyword>
<keyword>Congestion Control</keyword>
<abstract>
<t>The Real-time Transport Protocol (RTP) is used to transmit media in
multimedia telephony applications, these applications are typically
required to implement congestion control. This document describes the
test cases to be used in the performance evaluation of such congestion
control algorithms.</t>
<!--The draft defines a structure on how to define test cases and uses
that stracture to define test case those are essensial to RMCAT to evaluate
the candidate algorithms. also the intention here is define the test cases
in such a way that people can customize by tunning the same test case to fit
their to be tested network characteristics. this is the theme and we need
more text here.-->
</abstract>
</front>
<middle>
<section title="Introduction">
<t>This memo describes a set of test cases for evaluating congestion
control algorithm proposals for real-time interactive media. It is based
on the guidelines enumerated in <xref
target="I-D.ietf-rmcat-eval-criteria"></xref> and the requirements
discussed in <xref target="I-D.ietf-rmcat-cc-requirements"></xref>. The
test cases cover basic usage scenarios and are described using a common
structure, which allows for additional test cases to be added to those
described herein to accommodate other topologies and/or the modeling of
different path characteristics. The described test cases in this memo
SHOULD be used to evaluate any proposed congestion control algorithm for
real-time interactive media.</t>
</section>
<section anchor="sec-terminology" title="Terminology">
<!--<t> The key words "MUST", "MUST NOT", "REQUIRED", "SHALL",
"SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described
in BCP 14, <xref target="RFC2119" /> and indicate requirement
levels for compliant implementations. </t> -->
<t>The terminology defined in <xref target="RFC3550">RTP</xref>, <xref
target="RFC3551">RTP Profile for Audio and Video Conferences with
Minimal Control</xref>, <xref target="RFC3611">RTCP Extended Report
(XR)</xref>, <xref target="RFC4585">Extended RTP Profile for RTCP-based
Feedback (RTP/AVPF)</xref>, and <xref target="RFC5506">Support for
Reduced-Size RTCP</xref> apply.</t>
</section>
<section anchor="TS" title="Structure of Test cases">
<t>All the test cases in this document follow a basic structure allowing
implementers to describe a new test scenario without repeatedly
explaining common attributes. The structure includes a general
description section that describes the test case and its motivation.
Additionally the test case defines a set of attributes that characterize
the testbed, for example, the network path between communicating peers
and the diverse traffic sources.</t>
<t><list style="symbols">
<t>Define the test case: <list style="symbols">
<t>General description: describes the motivation and the goals
of the test case.</t>
<t>Expected behavior: describes the desired rate adaptation
behavior.</t>
<t>Define a list of metrics to evaluate the desired behavior:
this indicates the minimum set of metrics (e.g., link
utilization, media sending rate) that a proposed algorithm needs
to measure to validate the expected rate adaptation behavior. It
should also indicate the time granularity (e.g., averaged over
10ms, 100ms, or 1s) for measuring certain metrics. Typical
measurement interval is 200ms.</t>
<!-- we agreed this should be 200ms or was it based on some IPPM doc? -->
</list></t>
<t>Define testbed topology: every test case needs to define an
evaluation testbed topology. <xref target="fig-eval-topo"></xref>
shows such an evaluation topology. In this evaluation topology,
S1..Sn are traffic sources. These sources generate media traffic and
use either congestion control algorithm under investigation. R1..Rn
are the corresponding receivers. A test case can have one or more
such traffic sources (S) and their corresponding receivers (R). The
path from the source to destination is denoted as "forward" and the
path from a destination to a source is denoted as "backward". The
following basic structure of the test case has been described from
the perspective of media generating endpoints attached on the
left-hand side of <xref target="fig-eval-topo"></xref>. In this
setup, the media flows are transported in forward direction and
corresponding feedback/control messages are transported in the
backward direction. However, it is also possible to set up the test
with media in both forward and backward directions. In that case,
unless otherwise specified by the test case, it is expected that the
backward path does not introduce any congestion related impairments
and has enough capacity to accommodate both media and
feedback/control messages. It should be noted that depending on the
test cases it is possible to have different path characteristics in
either of the directions.<figure anchor="fig-eval-topo"
title="Example of A Testbed Topology">
<artwork><![CDATA[
+---+ +---+
|S1 |====== \ Forward --> / =======|R1 |
+---+ \\ // +---+
\\ //
+---+ +-----+ +-----+ +---+
|S2 |=======| A |------------------------------>| B |=======|R2 |
+---+ | |<------------------------------| | +---+
+-----+ +-----+
(...) // \\ (...)
// <-- Backward \\
+---+ // \\ +---+
|Sn |====== / \ ======|Rn |
+---+ +---+
]]></artwork>
</figure>In a testbed environment where real equipments are used
to create a laboratory, there may exist a significant amount of
traffic on portions of the network path between the endpoints that
is not desired for the purposes of the tests described in the
document. Some of this traffic may be generated by other processes
on the endpoints themselves (e.g., discovery protocols) or by other
endpoints not presently under test. It is recommended not to route
traffic generated by endpoints that are not under test through the
test bed and route those traffic generated by the endpoints under
test around the bottleneck links specified herein.</t>
<t>Define testbed attributes: <list style="symbols">
<t>Duration: defines the duration of the test in seconds.</t>
<t>Path characteristics: defines the end-to-end transport level
path characteristics of the testbed for a particular test case.
Two sets of attributes describe the path characteristics, one
for the forward path and the other for the backward path. The
path characteristics for a particular path direction is
applicable to all the Sources "S" sending traffic on that path.
If only one attribute is specified, it is used for both path
directions, however, unless specified the reverse path has no
capacity restrictions and no path loss.<list style="symbols">
<t>Path direction: forward or backward.</t>
<t>Bottleneck-link capacity: defines minimum capacity of the
end-to-end path</t>
<t>Reference bottleneck capacity: defines a reference value
for the bottleneck capacity for test cases with time-varying
bottleneck capacities. All bottleneck capacities will be
specified as a ratio with respect to the reference capacity
value.</t>
<t>One-way propagation delay: describes the end-to-end
latency along the path when network queues are empty, i.e.,
the time it takes for a packet to go from the sender to the
receiver without encountering any queuing delay.</t>
<t>Maximum end-to-end jitter: defines the maximum jitter
that can be observed along the path.</t>
<t>Bottleneck queue type: for example, Droptail, FQ-CoDel,
or PIE.</t>
<t>Bottleneck queue size: defines the size of queue in terms
of queuing time when the queue is full (in
milliseconds).</t>
<t>Path loss ratio: characterizes the non-congested,
additive, losses to be generated on the end-to-end path.
MUST describe the loss pattern or loss model used to
generate the losses.</t>
</list></t>
<t>Application-related: defines the traffic source behavior for
implementing the test case <list style="symbols">
<t>Media traffic Source: defines the characteristics of the
media sources. When using more than one media source, the
different attributes are enumerated separately for each
different media source. <list style="symbols">
<t>Media type: Video/Voice</t>
<t>Media flow direction: forward, backward or both.</t>
<t>Number of media sources: defines the total number of
media sources</t>
<t>Media codec: Constant Bit Rate (CBR) or Variable Bit
Rate (VBR)</t>
<t>Media source behavior: describes the media encoder
behavior. It defines the main parameters that affect the
adaptation behavior. This may include but is not limited
to: <list style="symbols">
<t>Adaptability: describes the adaptation options.
For example, in the case of video it defines the
following ranges of adaptation: bit rate, frame
rate, video resolution. Similarly, in the case of
voice, it defines the range of bit rate adaptation,
the sampling rate variation, and the variation in
packetization interval.</t>
<t>Output variation : for a VBR encoder it defines
the encoder output variation from the average target
rate over a particular measurement interval. For
example, on average the encoder output may vary
between 5% to 15% above or below the average target
bit rate when measured over a 100 ms time window.
The time interval over which the variation is
specified must be provided.</t>
<!-- -->
<t>Responsiveness to a new bit rate request: the lag
in time between a new bit rate request from the
congestion control algorithm and actual rate changes
in encoder output. Depending on the encoder, this
value may be specified in absolute time (e.g. 10ms
to 1000ms) or other appropriate metric (e.g. next
frame interval time).</t>
</list> More detailed discussions on expected media
source behavior, including those from synthetic video
traffic sources, is at <xref
target="I-D.ietf-rmcat-video-traffic-model"></xref>.</t>
<t>Media content: describes the chosen media sequences;
For example, test sequences are available at: <xref
target="xiph-seq"></xref> and <xref
target="HEVC-seq"></xref>.</t>
<t>Media timeline: describes the point when the media
source is introduced and removed from the testbed. For
example, the media source may start transmitting
immediately when the test case begins, or after a few
seconds.</t>
<t>Startup behavior: the media starts at a defined bit
rate, which may be the minimum, maximum bit rate, or a
value in between (in Kbps).</t>
</list></t>
<t>Competing traffic source: describes the characteristics
of the competing traffic source, the different types of
competing flows are enumerated in <xref
target="I-D.ietf-rmcat-eval-criteria"></xref>. <list
style="symbols">
<t>Traffic direction: forward, backward or both.</t>
<t>Type of sources: defines the types of competing
traffic sources. Types of competing traffic flows are
listed in <xref
target="I-D.ietf-rmcat-eval-criteria"></xref>. For
example, the number of TCP flows connected to a web
browser, the mean size and distribution of the content
downloaded.</t>
<t>Number of sources: defines the total number of
competing sources of each media type per traffic
direction.</t>
<t>Congestion control: enumerates the congestion control
used by each type of competing traffic.</t>
<t>Traffic timeline: describes when the competing
traffic starts and ends in the test case.</t>
</list></t>
</list></t>
<t>Additional attributes: describes attributes essential for
implementing a test case which are not included in the above
structure. These attributes MUST be well defined, so that the
other implementers of that particular test case are able to
implement it easily.</t>
</list></t>
</list></t>
<t>Any attribute can have a set of values (enclosed within "[]"). Each
member value of such a set MUST be treated as different value for the
same attribute. It is desired to run separate tests for each such
attribute value.</t>
<t>The test cases described in this document follow the above
structure.</t>
</section>
<section anchor="sec-rec-eval" title="Recommended Evaluation Settings">
<t>This section describes recommended test case settings and could be
overwritten by the respective test cases.</t>
<section anchor="EM" title="Evaluation metrics">
<t>To evaluate the performance of the candidate algorithms the
implementers MUST log enough information to visualize the following
metrics at a fine enough time granularity: <list style="numbers">
<t>Flow level: <list style="letters">
<t>End-to-end delay for the congestion controlled media
flow.</t>
<t>Variation in sending bit rate and goodput. Mainly observing
the frequency and magnitude of oscillations.</t>
<t>Packet losses observed at the receiving endpoint.</t>
<t>Feedback message overhead.</t>
<t>Convergence time - time to reach steady state for the
congestion controlled media flow(s).</t>
</list></t>
<t>Transport level: <list style="letters">
<t>Bandwidth utilization.</t>
<t>Queue length (milliseconds at specified path capacity):
<list style="symbols">
<t>average over the length of the session.</t>
<t>5 and 95 percentile.</t>
<t>median, maximum, minimum.</t>
</list></t>
</list></t>
</list></t>
</section>
<section anchor="PC" title="Path characteristics">
<t>Each path between a sender and receiver as described in <xref
target="fig-eval-topo"></xref> have the following characteristics
unless otherwise specified in the test case. <list style="symbols">
<t>Path direction: forward and backward.</t>
<t>Reference bottleneck capacity: 1Mbps.</t>
<t>One-Way propagation delay: 50ms. Implementers are encouraged to
run the experiment with additional propagation delays mentioned in
<xref target="I-D.ietf-rmcat-eval-criteria"></xref></t>
<t>Maximum end-to-end jitter: 30ms. Jitter models are described in
<xref target="I-D.ietf-rmcat-eval-criteria"></xref></t>
<t>Bottleneck queue type: Drop tail. Implementers are encouraged
to run the experiment with other AQM schemes, such as FQ-CoDel and
PIE.</t>
<t>Bottleneck queue size: 300ms.</t>
<t>Path loss ratio: 0%.</t>
</list></t>
<t>Examples of additional network parameters are discussed in <xref
target="I-D.ietf-rmcat-eval-criteria"></xref>.</t>
<t>For test cases involving time-varying bottleneck capacity, all
capacity values are specified as a ratio with respect to a reference
capacity value, so as to allow flexible scaling of capacity values
along with media source rate range. There exist two different
mechanisms for inducing path capacity variation: a) by explicitly
modifying the value of physical link capacity; or b) by introducing
background non-adaptive UDP traffic with time-varying traffic rate.
Implementers are encouraged to run the experiments with both
mechanisms for test cases specified in <xref target="VACS"></xref>,
<xref target="VACM"></xref>, and <xref target="CFL"></xref>.</t>
</section>
<section anchor="MS" title="Media source">
<t>Unless otherwise specified, each test case will include one or more
media sources as described below. <list style="symbols">
<t>Media type: Video <list style="symbols">
<t>Media codec: VBR</t>
<t>Media source behavior: <list style="symbols">
<t>Adaptability: <list style="symbols">
<t>Bit rate range: 150 Kbps - 1.5 Mbps. In real-life
applications the bit rate range can vary a lot
depending on the provided service, for example, the
maximum bit rate can be up to 4Mbps. However, for
running tests to evaluate the congestion control
algorithms it is more important to have a look at how
they are reacting to certain amount of bandwidth
change. Also it is possible that the media traffic
generator used in a particular simulator or testbed is
not capable of generating higher bit rate. Hence we
have selected a suitable bit rate range typical of
consumer-grade video conferencing applications in
designing the test case. If a different bit rate range
is used in the test cases, then the end-to-end path
capacity values will also need to be scaled
accordingly.</t>
<t>Frame resolution: 144p - 720p (or 1080p). This
resolution range is selected based on the bit rate
range. If a different bit rate range is used in the
test cases then the frame resolution range also need
to be selected suitably.</t>
<t>Frame rate: 10fps - 30fps. This frame rate range is
selected based on the bit rate range. If a different
bit rate range is used in the test cases then the
frame rate range also need to be adjusted
suitably.</t>
</list></t>
<t>Variation from target bit rate: +/-5%. Unless otherwise
specified in the test case(s), bit rate variation SHOULD
be calculated over one (1) second period of time.</t>
<t>Responsiveness to new bit rate request: 100ms</t>
</list></t>
<t>Media content: The media content should represent a typical
video conversational scenario with head and shoulder movement.
We recommend to use Foreman video sequence.</t>
<t>Media startup behavior: 150Kbps. It should be noted that
applications can use smart ways to select an optimal startup
bit rate value for a certain network condition. In such cases
the candidate proposals MAY show the effectiveness of such
smart approach as an additional information for the evaluation
process.</t>
</list></t>
</list><list style="symbols">
<t>Media type: Audio<list style="symbols">
<t>Media codec: CBR</t>
<t>Media bit rate: 20Kbps</t>
</list></t>
</list></t>
</section>
</section>
<section anchor="TC" title="Basic Test Cases">
<section anchor="VACS"
title="Variable Available Capacity with a Single Flow">
<t>In this test case the bottleneck-link capacity between the two
endpoints varies over time. This test is designed to measure the
responsiveness of the candidate algorithm. This test tries to address
the requirements in <xref
target="I-D.ietf-rmcat-cc-requirements"></xref>, which requires the
algorithm to adapt the flow(s) and provide lower end-to-end latency
when there exists: <list style="symbols">
<t>an intermediate bottleneck</t>
<t>change in available capacity (e.g., due to interface change,
routing change, abrupt arrival/departure of background
non-adaptive traffic).</t>
<t>maximum media bit rate is greater than link capacity. In this
case, the application will attempt to ramp up to its maximum bit
rate, since the link capacity is limited to a value lower, the
congestion control scheme is expected to stabilize the sending bit
rate close to the available bottleneck capacity.</t>
<!-- -->
</list>It should be noted that the exact variation in available
capacity due to any of the above depends on the underlying
technologies. Hence, we describe a set of known factors, which may be
extended to devise a more specific test case targeting certain
behaviors in a certain network environment.</t>
<t>Expected behavior: the candidate algorithm is expected to detect
the path capacity constraint, converges to the bottleneck link's
capacity and adapt the flow to avoid unwanted oscillation when the
sending bit rate is approaching the bottleneck link's capacity. The
oscillations occur when the media flow(s) attempts to reach its
maximum bit rate but overshoots the usage of the available bottleneck
capacity then to rectify, it reduces the bit rate and starts to ramp
up again.</t>
<t>Evaluation metrics : as described in <xref target="EM"></xref>.</t>
<t>Testbed topology: One media source S1 is connected to the
corresponding R1. The media traffic is transported over the forward
path and corresponding feedback/control traffic is transported over
the backward path. <figure anchor="fig-eval-topo-4-2"
title="Testbed Topology for Limited Link Capacity">
<artwork><![CDATA[
Forward -->
+---+ +-----+ +-----+ +---+
|S1 |=======| A |------------------------------>| B |=======|R1 |
+---+ | |<------------------------------| | +---+
+-----+ +-----+
<-- Backward
]]></artwork>
</figure></t>
<t>Testbed attributes:</t>
<t><list style="symbols">
<t>Test duration: 100s</t>
<t>Path characteristics: as described in <xref
target="PC"></xref></t>
<t>Application-related: <list style="symbols">
<t>Media Traffic:<list style="symbols">
<t>Media type: Video<list style="symbols">
<t>Media direction: forward.</t>
<t>Number of media sources: one (1)</t>
<t>Media timeline: <list style="symbols">
<t>Start time: 0s.</t>
<t>End time: 99s.</t>
</list></t>
</list></t>
<t>Media type: Audio<list style="symbols">
<t>Media direction: forward.</t>
<t>Number of media sources: one (1)</t>
<t>Media timeline: <list style="symbols">
<t>Start time: 0s.</t>
<t>End time: 99s.</t>
</list></t>
</list></t>
</list></t>
<t>Competing traffic: <list style="symbols">
<t>Number of sources : zero (0)</t>
</list></t>
</list></t>
<t>Test Specific Information: <list style="symbols">
<t>One-way propagation delay: [ 50 ms, 100 ms]. on the forward
path direction</t>
<t>This test uses bottleneck path capacity variation as listed
in <xref target="VACS_US"></xref></t>
<t>When using background non-adaptive UDP traffic to induce
time-varying bottleneck , the physical path capacity remains
at 4Mbps and the UDP traffic source rate changes over time as
(4-x)Mbps, where x is the bottleneck capacity specified in
<xref target="VACS_US"></xref></t>
</list></t>
</list></t>
<texttable anchor="VACS_US"
title="Path capacity variation pattern for forward direction">
<ttcol>Variation pattern index</ttcol>
<ttcol>Path direction</ttcol>
<ttcol>Start time</ttcol>
<ttcol>Path capacity ratio</ttcol>
<c>One</c>
<c>Forward</c>
<c>0s</c>
<c>1.0</c>
<c>Two</c>
<c>Forward</c>
<c>40s</c>
<c>2.5</c>
<c>Three</c>
<c>Forward</c>
<c>60s</c>
<c>0.6</c>
<c>Four</c>
<c>Forward</c>
<c>80s</c>
<c>1.0</c>
<!-- <postamble>Table 1: Path capacity variation pattern for
forward direction</postamble> -->
</texttable>
<t></t>
</section>
<section anchor="VACM"
title="Variable Available Capacity with Multiple Flows">
<t>This test case is similar to <xref target="VACS"></xref>. However
in addition this test will also consider persistent network load due
to competing traffic.</t>
<t>Expected behavior: the candidate algorithm is expected to detect
the variation in available capacity and adapt the media stream(s)
accordingly. The flows stabilize around their maximum bit rate as the
maximum link capacity is large enough to accommodate the flows. When
the available capacity drops, the flows adapt by decreasing their
sending bit rate, and when congestion disappears, the flows are again
expected to ramp up.</t>
<t>Evaluation metrics : as described in <xref target="EM"></xref>.</t>
<t>Testbed Topology: Two (2) media sources S1 and S2 are connected to
their corresponding destinations R1 and R2. The media traffic is
transported over the forward path and corresponding feedback/control
traffic is transported over the backward path. <figure
anchor="fig-eval-topo-4-1"
title="Testbed Topology for Variable Available Capacity">
<artwork><![CDATA[
+---+ +---+
|S1 |===== \ / =======|R1 |
+---+ \\ Forward --> // +---+
\\ //
+-----+ +-----+
| A |------------------------------>| B |
| |<------------------------------| |
+-----+ +-----+
// \\
// <-- Backward \\
+---+ // \\ +---+
|S2 |====== / \ ======|R2 |
+---+ +---+
]]></artwork>
</figure></t>
<t>Testbed attributes:</t>
<t>Testbed attributes are similar as described in <xref
target="VACS"></xref> except the test specific capacity variation
setup.</t>
<t>Test Specific Information: This test uses path capacity variation
as listed in <xref target="VACM_US"></xref> with a corresponding end
time of 125 seconds. The reference bottleneck capacity is 2Mbps. When
using background non-adaptive UDP traffic to induce time-varying
bottleneck for congestion controlled media flows, the physical path
capacity is 4Mbps and the UDP traffic source rate changes over time as
(4-x)Mbps, where x is the bottleneck capacity specified in <xref
target="VACM_US"></xref>.</t>
<texttable anchor="VACM_US"
title="Path capacity variation pattern for forward direction">
<ttcol>Variation pattern index</ttcol>
<ttcol>Path direction</ttcol>
<ttcol>Start time</ttcol>
<ttcol>Path capacity ratio</ttcol>
<c>One</c>
<c>Forward</c>
<c>0s</c>
<c>2.0</c>
<c>Two</c>
<c>Forward</c>
<c>25s</c>
<c>1.0</c>
<c>Three</c>
<c>Forward</c>
<c>50s</c>
<c>1.75</c>
<c>Four</c>
<c>Forward</c>
<c>75s</c>
<c>0.5</c>
<c>Five</c>
<c>Forward</c>
<c>100s</c>
<c>1.0</c>
<!-- <postamble>Table 2: Path capacity variation pattern for
forward direction</postamble> -->
</texttable>
</section>
<section anchor="CFL"
title="Congested Feedback Link with Bi-directional Media Flows">
<t>Real-time interactive media uses RTP hence it is assumed that RTCP,
RTP header extension or such would be used by the congestion control
algorithm in the backchannel. Due to asymmetric nature of the link
between communicating peers it is possible for a participating peer to
not receive such feedback information due to an impaired or congested
backchannel (even when the forward channel might not be impaired).
This test case is designed to observe the candidate congestion control
behavior in such an event.</t>
<t>It is expected that the candidate algorithms are able to cope with
the lack of feedback information and adapt to minimize the performance
degradation of media flows in the forward channel.</t>
<t>It should be noted that for this test case: logs are compared with
the reference case, i.e, when the backward channel has no
impairments.</t>
<t>Evaluation metrics : as described in <xref target="EM"></xref>.</t>
<t>Testbed topology: One (1) media source S1 is connected to
corresponding R1, but both endpoints are additionally receiving and
sending data, respectively. The media traffic (S1->R1) is
transported over the forward path and corresponding feedback/control
traffic is transported over the backward path. Likewise media traffic
(S2->R2) is transported over the backward path and corresponding
feedback/control traffic is transported over the forward path.</t>
<t><figure anchor="fig-eval-topo-4-6"
title="Testbed Topology for Congested Feedback Link">
<artwork><![CDATA[
+---+ +---+
|S1 |===== \ Forward --> / =======|R1 |
+---+ \\ // +---+
\\ //
+-----+ +-----+
| A |------------------------------>| B |
| |<------------------------------| |
+-----+ +-----+
// \\
// <-- Backward \\
+---+ // \\ +---+
|R2 |===== / \ ======|S2 |
+---+ +---+
]]></artwork>
</figure></t>
<t>Testbed attributes:</t>
<t><list style="symbols">
<t>Test duration: 100s</t>
<t>Path characteristics: <list style="symbols">
<t>Reference bottleneck capacity: 1Mbps.</t>
</list></t>
<t>Application-related: <list style="symbols">
<t>Media Source: <list style="symbols">
<t>Media type: Video<list style="symbols">
<t>Media direction: forward and backward</t>
<t>Number of media sources: two (2)</t>
<t>Media timeline: <list style="symbols">
<t>Start time: 0s.</t>
<t>End time: 99s.</t>
</list></t>
</list></t>
<t>Media type: Audio<list style="symbols">
<t>Media direction: forward and backward</t>
<t>Number of media sources: two (2)</t>
<t>Media timeline: <list style="symbols">
<t>Start time: 0s.</t>
<t>End time: 99s.</t>
</list></t>
</list></t>
</list></t>
<t>Competing traffic: <list style="symbols">
<t>Number of sources : zero (0)</t>
</list></t>
</list></t>
<t>Test Specific Information: this test uses path capacity
variations to create congested feedback link. <xref
target="CFL_US"> </xref> lists the variation patterns applied to
the forward path and <xref target="CFL_DS"></xref> lists the
variation patterns applied to the backward path. When using
background non-adaptive UDP traffic to induce time-varying
bottleneck for congestion controlled media flows, the physical
path capacity is 4Mbps for both directions and the UDP traffic
source rate changes over time as (4-x)Mbps in each direction,
where x is the bottleneck capacity specified in <xref
target="CFL_DS"></xref>.</t>
</list></t>
<texttable anchor="CFL_US"
title="Path capacity variation pattern for forward direction">
<ttcol>Variation pattern index</ttcol>
<ttcol>Path direction</ttcol>
<ttcol>Start time</ttcol>
<ttcol>Path capacity ratio</ttcol>
<c>One</c>
<c>Forward</c>
<c>0s</c>
<c>2.0</c>
<c>Two</c>
<c>Forward</c>
<c>20s</c>
<c>1.0</c>
<c>Three</c>
<c>Forward</c>
<c>40s</c>
<c>0.5</c>
<c>Four</c>
<c>Forward</c>
<c>60s</c>
<c>2.0</c>
<!-- <postamble>Table 3: Path capacity variation pattern for
forward direction</postamble> -->
</texttable>
<t></t>
<texttable anchor="CFL_DS"
title="Path capacity variation pattern for backward direction">
<ttcol>Variation pattern index</ttcol>
<ttcol>Path direction</ttcol>
<ttcol>Start time</ttcol>
<ttcol>Path capacity ratio</ttcol>
<c>One</c>
<c>Backward</c>
<c>0s</c>
<c>2.0</c>
<c>Two</c>
<c>Backward</c>
<c>35s</c>
<c>0.8</c>
<c>Three</c>
<c>Backward</c>
<c>70s</c>
<c>2.0</c>
<!-- <postamble>Table 4: Path capacity variation pattern for
backward direction</postamble> -->
</texttable>
</section>
<section anchor="competing-rmcat-flow"
title="Competing Media Flows with same Congestion Control Algorithm">
<t>In this test case, more than one media flows share the bottleneck
link and each of them uses the same congestion control algorithm. This
is a typical scenario where a real-time interactive application sends
more than one media flow to the same destination and these flows are
multiplexed over the same port. In such a scenario it is likely that
the flows will be routed via the same path and need to share the
available bandwidth amongst themselves. For the sake of simplicity it
is assumed that there are no other competing traffic sources in the
bottleneck link and that there is sufficient capacity to accommodate
all the flows individually. While this appears to be a variant of the
test case defined in <xref target="VACM"></xref>, it focuses on the
capacity sharing aspect of the candidate algorithm. The previous test
case, on the other hand, measures adaptability, stability, and
responsiveness of the candidate algorithm.</t>
<t>Expected behavior: It is expected that the competing flows will
converge to an optimum bit rate to accommodate all the flows with
minimum possible latency and loss. Specifically, the test introduces
three media flows at different time instances, when the second flow
appears there should still be room to accommodate another flow on the
bottleneck link. Lastly, when the third flow appears the bottleneck
link should be saturated.</t>
<t>Evaluation metrics : as described in <xref target="EM"></xref>.</t>
<t>Testbed topology: Three media sources S1, S2, S3 are connected to
R1, R2, R3 respectively. The media traffic is transported over the
forward path and corresponding feedback/control traffic is transported
over the backward path. <figure anchor="fig-eval-topo-4-3"
title="Testbed Topology for Multiple congestion controlled media Flows">
<artwork><![CDATA[
+---+ +---+
|S1 |===== \ Forward --> / =======|R1 |
+---+ \\ // +---+
\\ //
+---+ +-----+ +-----+ +---+
|S2 |=======| A |------------------------------>| B |=======|R2 |
+---+ | |<------------------------------| | +---+
+-----+ +-----+
// \\
// <-- Backward \\
+---+ // \\ +---+
|S3 |====== / \ ======|R3 |
+---+ +---+
]]></artwork>
</figure></t>
<t>Testbed attributes:</t>
<t><list style="symbols">
<t>Test duration: 120s</t>
<t>Path characteristics: <list style="symbols">
<t>Reference bottleneck capacity: 3.5Mbps</t>
<t>Path capacity ratio: 1.0</t>
<!-- <t>One-Way propagation delay: [10ms, 50ms]</t> -->
</list></t>
<t>Application-related: <list style="symbols">
<t>Media Source: <list style="symbols">
<t>Media type: Video<list style="symbols">
<t>Media direction: forward.</t>
<t>Number of media sources: three (3)</t>
<t>Media timeline: new media flows are added
sequentially, at short time intervals. See test
specific setup below.</t>
</list></t>
<t>Media type: Audio<list style="symbols">
<t>Media direction: forward.</t>
<t>Number of media sources: three (3)</t>
<t>Media timeline: new media flows are added
sequentially, at short time intervals. See test
specific setup below.</t>
</list></t>
</list></t>
<t>Competing traffic: <list style="symbols">
<t>Number of sources : zero (0)</t>
</list></t>
</list></t>
<t>Test Specific Information: <xref target="MTL_CF"></xref>
defines the media timeline for both media type.</t>
</list></t>
<texttable anchor="MTL_CF"
title="Media Timeline for Video and Audio media sources">
<ttcol>Flow ID</ttcol>
<ttcol>Media type</ttcol>
<ttcol>Start time</ttcol>
<ttcol>End time</ttcol>
<c>1</c>
<c>Video</c>
<c>0s</c>
<c>119s</c>
<c>2</c>
<c>Video</c>
<c>20s</c>
<c>119s</c>
<c>3</c>
<c>Video</c>
<c>40s</c>
<c>119s</c>
<c>4</c>
<c>Audio</c>
<c>0s</c>
<c>119s</c>
<c>5</c>
<c>Audio</c>
<c>20s</c>
<c>119s</c>
<c>6</c>
<c>Audio</c>
<c>40s</c>
<c>119s</c>
</texttable>
</section>
<section title="Round Trip Time Fairness">
<t>In this test case, multiple media flows share the bottleneck link,
but the end-to-end path latency for each flow is different. For the
sake of simplicity it is assumed that there are no other competing
traffic sources in the bottleneck link and that there is sufficient
capacity to accommodate all the flows. While this appears to be a
variant of test case 5.2, it focuses on the capacity sharing aspect of
the candidate algorithm under different RTTs.</t>
<t>It is expected that the competing flows will converge to bit rates
to accommodate all the flows with minimum possible latency and loss.
Specifically, the test introduces five media flows at the same time
instance.</t>
<t>Evaluation metrics : as described in <xref target="EM"></xref>.</t>
<t>Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected
to their corresponding media sinks R1,R2,..,R5. The media traffic is
transported over the forward path and corresponding feedback/control
traffic is transported over the backward path. The topology is the
same as in <xref target="competing-rmcat-flow"></xref>. The end-to-end
path delays are: 10ms for S1-R1, 25ms for S2-R2, 50ms for S3-R3, 100ms
for S4-R4, and 150ms S5-R5, respectively.</t>
<t>Testbed attributes:</t>
<t><list style="symbols">
<t>Test duration: 300s</t>
<t>Path characteristics: <list style="symbols">
<t>One-Way propagation delay for each flow: 10ms, 25ms, 50ms,
100ms, 150ms.</t>
</list></t>
<t>Application-related: <list style="symbols">
<t>Media Source: <list style="symbols">
<t>Media type: Video<list style="symbols">
<t>Media direction: forward</t>
<t>Number of media sources: five (5)</t>
<t>Media timeline: new media flows are added
sequentially, at short time intervals. See test
specific setup below.</t>
</list></t>
<t>Media type: Audio<list style="symbols">
<t>Media direction: forward.</t>
<t>Number of media sources: five (5)</t>
<t>Media timeline: new media flows are added
sequentially, at short time intervals. See test
specific setup below.</t>
</list></t>
</list></t>
<t>Competing traffic: <list style="symbols">
<t>Number of sources : zero (0)</t>
</list></t>
</list></t>
<t>Test Specific Information: <xref target="MTL_RTT"></xref>
defines the media timeline for both media type.</t>
</list></t>
<texttable anchor="MTL_RTT"
title="Media Timeline for Video and Audio media sources">
<ttcol>Flow IF</ttcol>
<ttcol>Media type</ttcol>
<ttcol>Start time</ttcol>
<ttcol>End time</ttcol>
<c>1</c>
<c>Video</c>
<c>0s</c>
<c>299s</c>
<c>2</c>
<c>Video</c>
<c>10s</c>
<c>299s</c>
<c>3</c>
<c>Video</c>
<c>20s</c>
<c>299s</c>
<c>4</c>
<c>Video</c>
<c>30s</c>
<c>299s</c>
<c>5</c>
<c>Video</c>
<c>40s</c>
<c>299s</c>
<c>6</c>
<c>Audio</c>
<c>0</c>
<c>299s</c>
<c>7</c>
<c>Audio</c>
<c>10s</c>
<c>299s</c>
<c>8</c>
<c>Audio</c>
<c>20s</c>
<c>299s</c>
<c>9</c>
<c>Audio</c>
<c>30s</c>
<c>299s</c>
<c>10</c>
<c>Audio</c>
<c>40s</c>
<c>299s</c>
</texttable>
</section>
<section title="Media Flow Competing with a Long TCP Flow">
<t>In this test case, one or more media flows share the bottleneck
link with at least one long lived TCP flow. Long lived TCP flows
download data throughout the session and are expected to have infinite
amount of data to send and receive. This is a scenario where a
multimedia application co-exists with a large file download. The test
case measures the adaptivity of the candidate algorithm to competing
traffic. It addresses the requirement 3 in <xref
target="I-D.ietf-rmcat-cc-requirements"></xref>.</t>
<t>Expected behavior: depending on the convergence observed in test
case 5.1 and 5.2, the candidate algorithm may be able to avoid
congestion collapse. In the worst case, the media stream will fall to
the minimum media bit rate.</t>
<t>Evaluation metrics : following metrics in addition to as described
in <xref target="EM"></xref>. <list style="numbers">
<t>Flow level: <list style="letters">
<t>TCP throughput.</t>
<t>Loss for the TCP flow</t>
</list></t>
</list></t>
<t>Testbed topology: One (1) media source S1 is connected to the
corresponding media sink, R1. In addition, there is a long-live TCP
flow sharing the same bottleneck link. The media traffic is
transported over the forward path and corresponding feedback/control
traffic is transported over the backward path. The TCP traffic goes
over the forward path from, S_tcp with acknowledgment packets go over
the backward path from, R_tcp.</t>
<t><figure anchor="fig-eval-topo-4-4"
title="Testbed Topology for TCP vs congestion controlled media Flows">
<artwork><![CDATA[
+--+ +--+
|S1|===== \ Forward --> / =======|R1|
+--+ \\ // +--+
\\ //
+-----+ +-----+
| A |---------------------------->| B |
| |<----------------------------| |
+-----+ +-----+
// \\
// <-- Backward \\
+-----+ // \\ +-----+
|S_tcp|=== / \ ===|R_tcp|
+-----+ +-----+
]]></artwork>
</figure></t>
<t>Testbed attributes:</t>
<t><list style="symbols">
<t>Test duration: 120s</t>
<t>Path characteristics: <list style="symbols">
<t>Reference bottleneck capacity: 2Mbps</t>
<t>Path capacity ratio: 1.0</t>
<!-- <t>One-Way propagation delay: [10ms, 150ms]</t> -->
<t>Bottleneck queue size: [300ms, 1000ms]</t>
</list></t>
<t>Application-related: <list style="symbols">
<t>Media Source: <list style="symbols">
<t>Media type: Video<list style="symbols">
<t>Media direction: forward</t>
<t>Number of media sources: one (1)</t>
<t>Media timeline: <list style="symbols">
<t>Start time: 5s.</t>
<t>End time: 119s.</t>
</list></t>
</list></t>
<t>Media type: Audio<list style="symbols">
<t>Media direction: forward</t>
<t>Number of media sources: one (1)</t>
<t>Media timeline: <list style="symbols">
<t>Start time: 5s.</t>
<t>End time: 119s.</t>
</list></t>
</list></t>
</list></t>
<t>Additionally, implementers are encouraged to run the
experiment with multiple media sources.</t>
<t>Competing traffic: <list style="symbols">
<t>Number and Types of sources : one (1) and long-lived
TCP</t>
<t>Traffic direction : forward</t>
<t>Congestion control: default TCP congestion control<xref
target="RFC5681"></xref>.<!--Zs: Should we mention particual TCP congestion control? Cubic likely most dominat by usage, but WebRTC still uses Reno for data channel. Hence pointing to 5681--></t>
<t>Traffic timeline: <list style="symbols">
<t>Start time: 0s.</t>
<!--ZS: now a days TCP implementation uses
different techniques like auto tuning, hence if media and data traffic
start at the same the receiver window may be tuned in a way that it does
not grow more than a certain value. Hence this test case will require a
wamp-up time where TCP settings are settled down. However, this also
means we need to make sure the warm-up session does not sature the link
capacity when the media flows are inserted.-->
<!-- VS: I think we can split into 3 cases, TCP starts before, TCP starts
together with media, TCP starts after media. How does that sound? -->
<t>End time: 119s.</t>
</list></t>
</list></t>
<!--Zs: should we not indicate the amount of data download TCP should do?-->
</list></t>
<t>Test Specific Information: none</t>
</list></t>
</section>
<section title="Media Flow Competing with Short TCP Flows">
<t>In this test case, one or more congestion controlled media flow
shares the bottleneck link with multiple short-lived TCP flows.
Short-lived TCP flows resemble the on/off pattern observed in the web
traffic, wherein clients (browsers) connect to a server and download a
resource (typically a web page, few images, text files, etc.) using
several TCP connections (up to 4). This scenario shows the performance
of a multimedia application when several browser windows are active.
The test case measures the adaptivity of the candidate algorithm to
competing web traffic, it addresses the requirements 1.E in <xref
target="I-D.ietf-rmcat-cc-requirements"></xref>.</t>
<t>Depending on the number of short TCP flows, the cross-traffic
either appears as a short burst flow or resembles a long TCP flow. The
intention of this test is to observe the impact of short-term burst on
the behavior of the candidate algorithm.</t>
<t>Evaluation metrics : following metrics in addition to as described
in <xref target="EM"></xref>.<list style="numbers">
<t>Flow level: <list style="letters">
<t>Variation in the sending rate of the TCP flow.</t>
<t>TCP throughput.</t>
</list></t>
</list></t>
<t>Testbed topology: The topology described here is same as the one
described in <xref target="fig-eval-topo-4-4"></xref>.</t>
<t>Testbed attributes:</t>
<t><list style="symbols">
<t>Test duration: 300s</t>
<t>Path characteristics: <list style="symbols">
<t>Reference bottleneck capacity: 2.0Mbps</t>
<t>Path capacity ratio: 1.0</t>
<!-- <t>One-Way propagation delay: [10ms, 150ms]</t> -->
</list></t>
<t>Application-related: <list style="symbols">
<t>Media source: <list style="symbols">
<t>Media type: Video<list style="symbols">
<t>Media direction: forward</t>
<t>Number of media sources: two (2)</t>
<t>Media timeline: <list style="symbols">
<t>Start time: 5s.</t>
<t>End time: 299s.</t>
</list></t>
</list></t>
<t>Media type: Audio<list style="symbols">
<t>Media direction: forward</t>
<t>Number of media sources: two (2)</t>
<t>Media timeline: <list style="symbols">
<t>Start time: 5s.</t>
<t>End time: 299s.</t>
</list></t>
</list></t>
</list></t>
<t>Competing traffic: <list style="symbols">
<t>Number and Types of sources : ten (10), short-lived TCP
flows.</t>
<t>Traffic direction : forward</t>
<t>Congestion algorithm: default TCP Congestion control
<xref target="RFC5681"></xref>.</t>
<t>Traffic timeline: each short TCP flow is modeled as a
sequence of file downloads interleaved with idle periods.
Not all short TCP flows start at the same time, 2 of them
start in the ON state while rest of the 8 flows start in
an OFF stats. For description of short TCP flow model see
test specific information below.</t>
</list></t>
</list></t>
<t>Test Specific Information: <list style="symbols">
<t>Short-TCP traffic model: The short TCP model to be used in
this test is described in <xref
target="I-D.ietf-rmcat-eval-criteria"></xref>.</t>
</list></t>
</list></t>
</section>
<section title="Media Pause and Resume">
<t>In this test case, more than one real-time interactive media flows
share the link bandwidth and all flows reach to a steady state by
utilizing the link capacity in an optimum way. At this stage one of
the media flows is paused for a moment. This event will result in more
available bandwidth for the rest of the flows as they are on a shared
link. When the paused media flow resumes it would no longer have the
same bandwidth share on the link. It has to make it's way through the
other existing flows in the link to achieve a fair share of the link
capacity. This test case is important specially for real-time
interactive media which consists of more than one media flows and can
pause/resume media flows at any point of time during the session. This
test case directly addresses the requirement number 5 in <xref
target="I-D.ietf-rmcat-cc-requirements"></xref>. One can think it as a
variation of test case defined in <xref
target="competing-rmcat-flow"></xref>. However, it is different as the
candidate algorithms can use different strategies to increase its
efficiency, for example in terms of fairness, convergence time, reduce
oscillation etc, by capitalizing the fact that they have previous
information of the link.</t>
<t>Evaluation metrics : following metrics in addition to as described
in <xref target="EM"></xref>.<list style="numbers">
<t>Flow level: <list style="letters">
<t>Variation in sending bit rate and goodput. Mainly observing
the frequency and magnitude of oscillations.</t>
</list></t>
</list></t>
<t>Testbed Topology: Same as test case defined in <xref
target="competing-rmcat-flow"></xref></t>
<t>Testbed attributes: The general description of the testbed
parameters are same as <xref target="competing-rmcat-flow"></xref>
with changes in the test specific setup as below-</t>
<t><list style="symbols">
<t>Other test specific setup: <list style="symbols">
<t>Media flow timeline: <list style="symbols">
<t>Flow ID: one (1)</t>
<t>Start time: 0s</t>
<t>Flow duration: 119s</t>
<t>Pause time: not required</t>
<t>Resume time: not required</t>
</list></t>
<t>Media flow timeline: <list style="symbols">
<t>Flow ID: two (2)</t>
<t>Start time: 0s</t>
<t>Flow duration: 119s</t>
<t>Pause time: at 40s</t>
<t>Resume time: at 60s</t>
</list></t>
<t>Media flow timeline: <list style="symbols">
<t>Flow ID: three (3)</t>
<t>Start time: 0s</t>
<t>Flow duration:119s</t>
<t>Pause time: not required</t>
<t>Resume time: not required</t>
</list></t>
</list></t>
</list></t>
</section>
</section>
<section title="Other potential test cases">
<t>It has been noticed that there are other interesting test cases
besides the basic test cases listed above. In many aspects, these
additional test cases can help further evaluation of the candidate
algorithm. They are listed as below.</t>
<section title="Media Flows with Priority">
<t>In this test case media flows will have different priority levels.
This will be an extension of <xref
target="competing-rmcat-flow"></xref> where the same test will be run
with different priority levels imposed on each of the media flows. For
example, the first flow (S1) is assigned a priority of 2 whereas the
remaining two flows (S2 and S3) are assigned a priority of 1. The
candidate algorithm MUST reflect the relative priorities assigned to
each media flow. In the previous example, the first flow (S1) MUST
arrive at a steady-state rate approximately twice of that of the other
two flows (S2 and S3).</t>
<t>The candidate algorithm can use a coupled congestion control
mechanism for the bandwidth distribution according to the respective
media flow priority.</t>
</section>
<section title="Explicit Congestion Notification Usage">
<t>This test case requires to run all the basic test cases with the
availability of Explicit Congestion Notification (ECN) <xref
target="RFC6679"></xref> feature enabled. The goal of this test is to
exhibit that the candidate algorithms do not fail when ECN signals are
available. With ECN signals enabled the algorithms are expected to
perform better than their delay based variants.</t>
</section>
<section title="Multiple Bottlenecks">
<t>In this test case one congestion controlled media flow, S1->R2,
traverses a path with multiple bottlenecks. As illustrated in <xref
target="fig-eval-topo-6-2"></xref>, the first flow (S1->R1)
competes with the second congestion controlled media flow (S2->R2)
over the link between A and B which is close to the sender side;
again, that flow (S1->R1) competes with the third congestion
controlled media flow (S3->R3) over the link between C and D which
is close to the receiver side. The goal of this test is to ensure that
the candidate algorithms work properly in the presence of multiple
bottleneck links on the end to end path.</t>
<t>Expected behavior: the candidate algorithm is expected to achieve
full utilization at both bottleneck links without starving any of the
three congestion controlled media flows.</t>
<t><figure anchor="fig-eval-topo-6-2"
title="Testbed Topology for Multiple Bottlenecks">
<artwork><![CDATA[
Forward ---->
+---+ +---+ +---+ +---+
|S2 | |R2 | |S3 | |R3 |
+---+ +---+ +---+ +---+
| | | |
| | | |
+---+ +-----+ +-----+ +-----+ +-----+ +---+
|S1 |=======| A |------>| B |----->| C |---->| D |=======|R1 |
+---+ | |<------| |<-----| |<----| | +---+
+-----+ +-----+ +-----+ +-----+
1st 2nd
Bottleneck (A->B) Bottleneck (C->D)
<------ Backward
]]></artwork>
</figure></t>
<t>Testbed topology: Three media sources S1, S2, and S3 are connected
to respective destinations R1, R2, and R3. For all three flows the
media traffic is transported over the forward path and corresponding
feedback/control traffic is transported over the backward path.</t>
<t>Testbed attributes:</t>
<t><list style="symbols">
<t>Test duration: 300s</t>
<t>Path characteristics: <list style="symbols">
<t>Reference bottleneck capacity: 2Mbps.</t>
<t>Path capacity ratio between A and B: 1.0</t>
<t>Path capacity ratio between B and C: 4.0.</t>
<t>Path capacity ratio between C and D: 0.75.</t>
<t>One-Way propagation delay: <list style="numbers">
<t>Between S1 and R1: 100ms</t>
<t>Between S2 and R2: 40ms</t>
<t>Between S3 and R3: 40ms</t>
</list></t>
</list></t>
<t>Application-related: <list style="symbols">
<t>Media Source: <list style="symbols">
<t>Media type: Video <list style="symbols">
<t>Media direction: Forward</t>
<t>Number of media sources: Three (3)</t>
<t>Media timeline: <list style="symbols">
<t>Start time: 0s.</t>
<t>End time: 299s.</t>
</list></t>
</list></t>
<t>Media type: Audio <list style="symbols">
<t>Media direction: Forward</t>
<t>Number of media sources: Three (3)</t>
<t>Media timeline: <list style="symbols">
<t>Start time: 0s.</t>
<t>End time: 299s.</t>
</list></t>
</list></t>
</list></t>
<t>Competing traffic: <list style="symbols">
<t>Number of sources : Zero (0)</t>
</list></t>
</list></t>
</list></t>
</section>
</section>
<section title="Wireless Access Links">
<!-- keep the comments out of the tags.-->
<t>Additional wireless network (both cellular network and WiFi network)
specific test cases are defined in <xref
target="I-D.ietf-rmcat-wireless-tests"></xref>.</t>
</section>
<section title="Security Considerations">
<t>Security issues have not been discussed in this memo.</t>
<!-- Congestion Collapse, Denial of Service -->
</section>
<section title="IANA Considerations">
<t>There are no IANA impacts in this memo.</t>
</section>
<section title="Acknowledgements">
<t>Much of this document is derived from previous work on congestion
control at the IETF.</t>
<t>The content and concepts within this document are a product of the
discussion carried out in the Design Team.</t>
</section>
</middle>
<back>
<references title="Normative References">
<!--&rfc2119;-->
<?rfc include='reference.RFC.6679.xml'?>
<!-- RTP related -->
&rfc3550;
&rfc3551;
&rfc3611;
&rfc4585;
&rfc5506;
<!--RMCAT related -->
&I-D.ietf-rmcat-eval-criteria;
&I-D.ietf-rmcat-wireless-tests;
&I-D.ietf-rmcat-video-traffic-model;
</references>
<references title="Informative References">
<!--
&I-D.ietf-rtcweb-use-cases-and-requirements; -->
<?rfc include='reference.RFC.5681.xml'?>
<!-- &rfc5033; -->
<!-- CC Evaluation -->
<!-- &rfc5166; -->
<!-- CC Metrics -->
<!-- &rfc5681; -->
<!-- Standard TCP -->
<!-- <reference anchor="SA4-EVAL">
<front>
<title>LTE Link Level Throughput Data for SA4 Evaluation
Framework</title>
<author fullname="3GPP R1-081955" initials="3GPP"
surname="R1-081955">
<organization></organization>
</author>
<date month="5" year="2008" />
<abstract>
<t>In R1-081720, 3GPP SA4 has requested RAN1 and RAN2 for link
level throughput traces to be used in an evaluation framework they
are developing for dynamic video rate adaptation.</t>
</abstract>
</front>
<seriesInfo name="3GPP" value="R1-081955" />
<format octets="3459875"
target="http://www.3gpp.net/ftp/tsg_ran/WG1_RL1/TSGR1_53/Docs/R1-081955.zip"
type="ZIP" />
</reference> -->
<!-- <reference anchor="SA4-LR">
<front>
<title>Error Patterns for MBMS Streaming over UTRAN and
GERAN</title>
<author fullname="3GPP S4-050560" initials="3GPP"
surname="S4-050560">
<organization></organization>
</author>
<date month="5" year="2008" />
</front>
<seriesInfo name="3GPP" value="S4-050560" />
<format octets="335322"
target="http://www.3gpp.org/FTP/tsg_sa/WG4_CODEC/TSGS4_36/Docs/S4-050560.zip"
type="ZIP" />
</reference> -->
&I-D.ietf-rmcat-cc-requirements;
<reference anchor="xiph-seq">
<front>
<title>Video Test Media</title>
<author fullname="" initials="" surname="Xiph.org"></author>
<date month="" year="" />
</front>
<seriesInfo name="http://media.xiph.org/video/derf/" value="" />
</reference>
<reference anchor="HEVC-seq">
<front>
<title>Test Sequences</title>
<author fullname="" initials="" surname="HEVC"></author>
<date month="" year="" />
</front>
<seriesInfo name="http://www.netlab.tkk.fi/~varun/test_sequences/"
value="" />
</reference>
<!-- <reference anchor="TCP-eval-suite">
<front>
<title>Towards a Common TCP Evaluation Suite</title>
<author fullname="Andrew Lachlan" initials="A." surname="Lachlan"></author>
<author fullname="Cesar Marcondes" initials="C." surname="Marcondes"></author>
<author fullname="Sally Floyd" initials="S." surname="Floyd"></author>
<author fullname="Lawrence Dunn" initials="L." surname="Dunn"></author>
<author fullname="Romeric Guillier" initials="R." surname="Guillier"></author>
<author fullname="Wang Gang" initials="W." surname="Gang"></author>
<author fullname="Lars Eggert" initials="L." surname="Eggert"></author>
<author fullname="Sangtae Ha" initials="S." surname="Ha"></author>
<author fullname="Injong Rhee" initials="I." surname="Rhee"></author>
<date month="August" year="2008" />
</front>
<seriesInfo name="Proc. PFLDnet." value="2008" />
</reference> -->
</references>
<!-- <section anchor="App-cl" title="Change Log">
<t>Note to the RFC-Editor: please remove this section prior to
publication as an RFC.</t>
<section title="Changes in draft-singh-rmcat-cc-eval-01">
<t><list style="symbols">
<t></t>
</list></t>
</section>
</section>
-->
</back>
</rfc>
| PAFTECH AB 2003-2026 | 2026-04-23 14:19:32 |