One document matched: draft-irtf-tmrg-ns2-tcp-tool-00.txt
Internet Engineering Task Force Gang Wang
Yong Xia
NEC Labs China
David Harrison
Internet Draft BitTorrent
Intended status: Informational April 25, 2007
Expires: October 2007
An NS2 TCP Evaluation Tool
draft-irtf-tmrg-ns2-tcp-tool-00.txt
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
This Internet-Draft will expire on October 25, 2007.
Copyright Notice
Copyright (C) The IETF Trust (2007).
Abstract
This document introduces a TCP performance evaluation tool for the
network simulator NS2. This tool is motivated by the observation
that there is significant overlap among (but lack of an agreed set
of) the topologies, traffic, and metrics used by many researchers
in the evaluation of TCP alternatives: effort could be saved by
Wang, Xia, and Harrison [Page 1]
Internet-Draft TMRG, Evaluation, Tool April 2007
starting research from an existing framework. As such, our tool
includes several typical topologies and traffic models; it measures
some of the most important metrics commonly used in TCP evaluation;
and it can automatically generate simulation statistics and graphs
ready for inclusion in latex and html documents. The tool also
contains an extendable open-source framework. With community effort,
we hope the tool evolves into a widely accepted, well-defined set of
TCP performance evaluation benchmarks.
Table of Contents
1. Introduction.................................................3
2. Tool Components..............................................5
2.1. Network Topologies......................................5
2.1.1. A Single-Bottleneck Dumb-Bell Topology.............5
2.1.2. A Multiple-Bottleneck Parking-Lot Topology.........5
2.1.3. A Simple Network Topology..........................6
2.2. Traffic Models..........................................7
2.2.1. Long-lived FTP Traffic.............................7
2.2.2. Short-lived Web Traffic............................7
2.2.3. Streaming Video Traffic............................7
2.2.4. Interactive Voice Traffic..........................7
2.3. Performance Metrics.....................................8
2.3.1. Throughput, Delay, Jitter and Loss Rate............8
2.3.1.1. Throughput....................................8
2.3.1.2. Delay.........................................8
2.3.1.3. Jitter........................................9
2.3.1.4. Loss Rate.....................................9
2.3.2. Response Times and Oscillations....................9
2.3.3. Fairness and Convergence..........................10
2.3.4. Robustness in Challenging Environments............10
2.4. Simulation Results.....................................10
3. Usage Description...........................................10
4. Security Considerations.....................................11
5. IANA Considerations.........................................12
6. Acknowledgments.............................................12
7. Informative References......................................12
Author's Addresses.............................................14
Intellectual Property Statement................................14
Wang, Xia and Harrison [Page 2]
Internet-Draft TMRG, Evaluation, Tool April 2007
1. Introduction
Issues regarding the behavior of TCP in high-speed, long-distance
networks have recently attracted wide attention. The Additive-
Increase-Multiplicative-Decrease (AIMD) congestion control algorithm
employed by TCP [1] is designed to share network resources among
competing users fairly and efficiently. However, research results
show that TCP does not perform very well in high-speed, long-distance
networks [2]. To solve this problem, many TCP variants have been
proposed. These include HighSpeed TCP (HSTCP) [3], Scalable TCP
(STCP) [4], FAST [10], Binary Increase Control (BIC) TCP [5], CUBIC
TCP [6], H-TCP [7], XCP [8], and VCP [9], etc. The proliferation of
these many options logically brings the following question: what is
the most effective high speed transport protocol for general use?
In order to answer this question, a number of performance evaluations
for the high speed TCP variants have been conducted. These include
simulations [11, 12] using the network simulator NS2 [13], test-bed
analysis [14] and real world experiments [15]. Although test-beds
and real world experiments can produce much more realistic results
than simulations, implementing a large and complex experimental
scenario is challenging. Therefore, many researchers employ NS2
simulations to test their ideas in the early stage of protocol
design. Typically, each researcher writes his/her own NS2 scripts,
and much time is spent duplicating similar network topologies,
traffic models and performance metrics. On the other hand, the
community still lacks a set of ready-made tests to reveal common
flaws during early development, and a set of automated benchmarks to
serve as a meaningful comparison between schemes proposed by
different research groups. One might ask: why don't we avoid wheel
reinvention and form a community project to this end?
This document introduces a project [22, 21] which implements an
extendable tool that automates the NS2 TCP simulation process as much
as possible. Using this tool, one just needs to simply define
simulation scenarios (including network topology, traffic model and
performance metrics); the simulation results (statistics and graphs)
are automatically generated. The tool is very easy to use and
extend. It can save a lot of time spent in writing NS2 simulation
scripts. More important, we expect that, over the time, the
scenarios collectively defined by the users of this tool will
converge onto a set of community-acceptable TCP performance
evaluation benchmarks, which will in turn be used by each individual
researcher.
Wang, Xia and Harrison [Page 3]
Internet-Draft TMRG, Evaluation, Tool April 2007
This tool includes a suite of NS2 simulation scripts that:
1. Automate simulation and post-processing procedures;
2. Define a set of commonly used network topologies, traffic models
and performance evaluation metrics.
This tool can be used not only for high-speed TCP protocols, but for
other proposed changes to congestion control mechanisms as well, such
as ECN added to SYN/ACK packets, changes to make small transfers more
robust, changes in RTO estimation, and proposals to distinguish
between loss due to congestion or corruption, etc.
This simulation tool does not attempt to be final. Instead, it
intends to serve as a starting point. We invite community members to
contribute to the project by helping extend this tool toward a
comprehensive set of NS2 TCP evaluation benchmarks.
Network Topology Traffic Model Performance Metrics
+----------------+ +-----------------+ +-----------------+
| Dumb-bell | |Long-lived FTP | |Network Level |
| Parking-lot | |Short-Lived Web | | |
| Simple-network | |Streaming Video | |Application Level|
+----------------+ |Interactive Voice| +-----------------+
| +-----------------+ |
| | |
| | |
| | |
+---------------------------------------------+
|
|
V
+------------------+
| Results in |
| Statistics |
| and Graphs |
+------------------+
Figure 1 The architecture of our tool.
Wang, Xia and Harrison [Page 4]
Internet-Draft TMRG, Evaluation, Tool April 2007
2. Tool Components
The architecture of our tool is shown in Fig. 1, which is primarily
composed of the following components: network topologies, traffic
models, performance evaluation metrics, and, after a simulation is
done, a set of result statistics and graphs generated.
2.1. Network Topologies
The tool includes three commonly used topologies in TCP performance
evaluations. They are single-bottleneck dumb-bell, multiple-
bottleneck parking-lot, and a simple network topology. More
realistic and complex topologies can be added to the tool easily.
2.1.1. A Single-Bottleneck Dumb-Bell Topology
This is shown in Fig. 2, in which source nodes and sink nodes connect
to router 1 or router 2. The bandwidth between the two routers is
much lower than the other links, which causes the link between the
routers to be a bottleneck. (Traffic can be either uni-directional
or bi-directional.)
Src_1 Sink_1
\ /
\ bottleneck /
Src_2 --- Router1 -------------- Router2 --- Sink_2
/ \
/ \
Src_N Sink_N
Figure 2 A dumb-bell topology.
2.1.2. A Multiple-Bottleneck Parking-Lot Topology
The parking-lot topology shown in Fig. 3 is similar to the dumb-bell
topology except that it introduces cross traffic traversing the
intermediate routers.
Wang, Xia and Harrison [Page 5]
Internet-Draft TMRG, Evaluation, Tool April 2007
Src_1 CrossSrc_1 CrossSrc_2 ... Sink_1
\ | | /
\ | | /
Src_2 --- Router1 --- Router2 --- ... RouterN --- Sink_2
/ | | \
/ | | \
Src_N CrossSink_1 ... CrossSink_N Sink_N
Figure 3 A parking-lot topology.
2.1.3. A Simple Network Topology
A simple network topology is illustrated in Fig. 4. In this
configuration, the core routers represent the backbone of the network
with the access routers responsible for sender or receiver nodes to
connect to the network. It is similar to the transit and stub
domains in GT-ITM [16]. Static routing is employed as the default
routing protocol.
CR1
PC1 / \ PC5
\ / \ /
--- AR2 --- CR2 CR4 --- AR4 ---
/ \ / \
PC2 \ / PC6
CR3
|
/ \
PC3 PC4
PC: Personal Computer
AR: Access Router
CR: Core Router
Figure 4 A simple network topology.
All the links in the above topologies have settable parameters such
as link capacity, propagation delay, queue discipline, and so on.
Wang, Xia and Harrison [Page 6]
Internet-Draft TMRG, Evaluation, Tool April 2007
2.2. Traffic Models
The tool attempts to apply the typical traffic settings. The
applications involved include four common traffic types.
2.2.1. Long-lived FTP Traffic
FTP traffic uses infinite, non-stop file transmission, which begins
at a random time and runs on the top of TCP. Implementation details
and choice of TCP variants are decided by users, which is not in the
scope of this tool.
2.2.2. Short-lived Web Traffic
The web traffic module employs the PackMime HTTP traffic generator
[17], which is available in the recent NS2 releases.
2.2.3. Streaming Video Traffic
Streaming traffic is modeled using CBR traffic over UDP. Both
sending rate and packet size are settable.
2.2.4. Interactive Voice Traffic
There are currently two synthetic voice traffic generation methods
available in this tool. One is based on CBR-like streaming traffic.
The other is generated according to a two-state ON/OFF model, in
which ON and OFF states are exponentially distributed. The mean ON
period is 1.0 sec, and the mean OFF duration is 1.35 sec. These
values are set in accordance with ITU-T recommendations [18], but are
changeable if needed.
The voice packet size is 200 bytes, including the 160 bytes data
packet (codec G711, 64 kbps rate and 20 ms duration), 20 byte IP
header, 8 byte UDP header, and 12 byte RTP header. These parameters
can be changed by using other voice/audio codecs.
Wang, Xia and Harrison [Page 7]
Internet-Draft TMRG, Evaluation, Tool April 2007
2.3. Performance Metrics
A comprehensive list of the metrics for TCP performance evaluation is
described in [19]. In the first step, this tool tries to implement
some commonly used metrics described in [19]. Here we follow [19]
and classify the metrics into network metrics and application
metrics. They are listed as follows.
2.3.1. Throughput, Delay, Jitter and Loss Rate
2.3.1.1. Throughput
For network metrics, we collect bottleneck link utilization as the
aggregate link throughput.
Throughput is sometimes different from goodput, because goodput
consists solely of useful transmitted traffic, where throughput may
also include retransmitted traffic [19]. But users care more about
the useful bits the network can provide. So the tool collects
application level end-to-end goodput no matter what the transport
protocol is employed.
For long-lived FTP traffic, it measures the transmitted traffic
during some intervals in bits per second.
For short-lived web traffic, the PackMime HTTP model collects
request/response goodput and response time to measure web traffic
performance.
Voice and video traffic are different from above. Their performance
is affected by packet delay, delay jitter and packet loss rate as
well as goodput. So their goodput is measured in transmitted packet
rate excluding lost packets and delayed packets in excess of a
predefined delay threshold.
2.3.1.2. Delay
We use bottleneck queue size as an indication of queuing delay in
bottlenecks. Besides mean and max/min queue size statistics, we also
use percentile queue size to indicate the queue length during most of
the time.
Wang, Xia and Harrison [Page 8]
Internet-Draft TMRG, Evaluation, Tool April 2007
FTP traffic is not affected much by packet transmission delay.
For web traffic, we report on the response time, defined as the
duration between the client's sending out requests and receiving the
response from the server.
For streaming and interactive traffic, packet delay is a one-way
measurement, as defined by the duration between sending and receiving
at the end nodes.
2.3.1.3. Jitter
Delay jitter is quite important for delay sensitive traffic, such as
voice and video. Large jitter requires much more buffer size at the
receiver side and may cause high loss rates in strict delay
requirements. We employ standard packet delay deviation to show
jitter for interactive and streaming traffic.
2.3.1.4. Loss Rate
To obtain network statistics, we measure the bottleneck queue loss
rate.
We do not collect loss rates for FTP and web traffic because they are
less affected by this metric.
For interactive and streaming traffic, high packet loss rates result
in the failure of the receiver to decode the packet. In this tool,
they are measured during specified intervals. The received packet is
considered lost if its delay is beyond a predefined threshold.
2.3.2. Response Times and Oscillations
One of the key concerns in the design of congestion control
mechanisms has been the response time to sudden network changes. On
the one hand, the mechanism should respond rapidly to changes in the
network environment. On the other hand, it has to make sure changes
are not too severe to ensure the stability of the network [19]. This
tool is designed so the response time and fluctuations can be easily
observed using a series of figures it generates, if the simulation
Wang, Xia and Harrison [Page 9]
Internet-Draft TMRG, Evaluation, Tool April 2007
scenarios we use include variable bandwidth, round trip delay,
various traffic start times and other parameters.
2.3.3. Fairness and Convergence
In this tool, the fairness measurement uses Jain's fairness index
[20] to measure the fair bandwidth share of end-to-end FTP flows
that traverse the same route.
Convergence times are the time elapsed between multiple flows from an
unfair share of link bandwidth to a fair state. They are quite
important for environments with high-bandwidth, long-delay flows.
This tool includes scenarios to test the convergence performance.
2.3.4. Robustness in Challenging Environments
A static link packet error model has been introduced in the tool to
investigate TCP performance in challenging environments. Link
failure, routing changes and other diagnostic markers can easily be
tested by changing the tool's parameters.
2.4. Simulation Results
The tool includes a package [21] to automatically generate the above-
discussed performance metrics. At the end of a simulation, it also
automatically generates a series of user-defined statistics (e.g.
bottleneck average utilization, bottleneck 90-percentile queue
length, average per-flow goodput, etc.) and graphs (like bottleneck
utilization and queue length variation over time, per-flow throughput
over time, etc). It can create latex and html files in order to
present the simulation results in a paper or webpage form. All the
simulation-generated data is stored in a temporary directory for
later use.
3. Usage Description
Here we present a brief description on how to use this software.
For the details the reader is referred to the manual available at
Wang, Xia and Harrison [Page 10]
Internet-Draft TMRG, Evaluation, Tool April 2007
[22]. The main body of this tool includes three files:
create_topology.tcl, create_traffic.tcl, and create_graph.tcl. As
their file names indicate, create_topology.tcl implements the three
common network topologies discussed in Section 2.1,
create_traffic.tcl defines the traffic model parameters in the
simulation (see Section 2.2), and crate_graph.tcl generates
simulation statistics (see Section 2.3) and plots graphs at the end
of simulations. For the details on the graphs that can be generated,
please refer to the manual available at [21].
Three example scripts are given in the package. Let's take the dumb-
bell topology as an example. The simulation script is
test_dumb_bell.tcl, and its default parameters are set in
def_dumb_bell.tcl. Those parameters can be changed as needed.
Totally, the parameter setting includes three parts: topology setting
traffic setting, and simulation statistics and graph setting. The
topology setting defines the specific topology parameters. For dumb-
bell, it sets the bottleneck bandwidth, round trip time, propagation
delay, and packet error rate in the bottleneck link. The traffic
setting defines the traffic parameters used in the simulation, such
as the number of FTP traffic, what high-speed TCP protocol employed
by FTP, using AQM or not, and how long the simulation runs. Finally,
you choose the performance statistics to be generated (like
bottleneck utilization, packet loss rate, etc.), and what kinds of
figures will be displayed (e.g., queue length variation over time).
After running "ns test_dumb_bell.tcl", simulation statistics in text
format and graphs in encapsulated postscript format are stored in the
directory /tmp/expX, where X stands for the sequence number of the
simulation. At the same time, all the simulation results can be
viewed through a portal file /tmp/expX/index.html.
The parking-lot and network simulations are similar to the dumb-bell
topology.
4. Security Considerations
There are no security considerations in this document.
Wang, Xia and Harrison [Page 11]
Internet-Draft TMRG, Evaluation, Tool April 2007
5. IANA Considerations
There are no IANA considerations in this document.
6. Acknowledgments
The authors would like to thank Dr. Sally Floyd of ICIR for her
encouragement and a lot of valuable advice. Part of David Harrison
and Yong Xia's work was conducted when they were PhD students at
Rensselaer Polytechnic Institute (RPI). They thank Prof. Shivkumar
Kalyanaraman of RPI for his support and guidance.
7. Informative References
[1] V. Jacobson, "Congestion Avoidance and Control", SIGCOMM'88,
August 1998.
[2] S. Floyd, S. Ratnasamy, and S. Shenker, "Modifying TCP's
Congestion Control for High Speeds",
http://www.icir.org/floyd/notes.html.
[3] S. Floyd, "HighSpeed TCP for Large Congestion Windows", RFC
3649, December 2003.
[4] T. Kelly, "Scalable TCP: Improving Performance in HighSpeed
Wide Area Networks", ACM Computer Commun. Rev., vol.33, no.2,
pp.83-91, April 2003.
[5] L. Xu, K. Harfoush, and I. Rhee, "Binary Increase Congestion
Control (BIC) for Fast Long-Distance Networks", IEEE
INFOCOM'04, pp.2514-2524, March 2004.
[6] I. Rhee and L. Xu, "Cubic: A New TCP-Friendly High-Speed TCP
Variant", PFLDnet 2005.
[7] D. Leigh and R. Shorten, "H-TCP Protocol for High-Speed Long
Distance Networks", PFLDnet 2004.
[8] D. Kataabi, M. Handley, and C. Rohrs, "Congestion Control for
High Bandwidth-Delay Product Networks", SIGCOMM'02, August
2002.
Wang, Xia and Harrison [Page 12]
Internet-Draft TMRG, Evaluation, Tool April 2007
[9] Y. Xia, L. Subramanian, Ion Stoica, and S. Kalyanaraman, "One
More Bit is Enough", SIGCOMM'05, August 2005.
[10] C. Jin, D. Wei, and S. Low, "FAST TCP: Motivation, Architecture,
Algorithms, Performance", INFOCOM'04, March 2004.
[11] M. Nabeshima and K. Yata, "Performance Evaluation and
Comparison of Transport Protocols for Fast Long-Distance
Networks." IEICE Trans. Commun., vol.E89-B, no.4, April 2006.
[12] S. Mascolo and F. Vacirca, "The Effect of Reverse Traffic on
the Performance of New TCP Congestion Control Algorithms",
PFLDnet 2006.
[13] NS2. http://www.isi.edu/nsnam/ns.
[14] S. Ha, Y. Kim, L. Le, I. Rhee, and L. Xu, "A Step toward
Realistic Performance Evaluation of High-Speed TCP Variants",
PFLDnet 2006.
[15] H. Bullot, R.Les Cottrell, and R. Hughes-Jones, "Evaluation of
advanced TCP stacks on fast long distance production networks",
Journal of Grid Computing. vol.1, pp.345-359, 2003.
[16] GT-ITM.
http://www.static.cc.gatech.edu/fac/Ellen.Zegura/graphs.html.
[17] J. Cao, W.S. Cleveland, Y. Gao, K. Jeffay, F.D. Smith, and M.C,
Weigle, "Stochastic Models for Generating Synthetic HTTP Source
Traffic", INFOCOM'04, March 2004.
[18] ITU-T Recommendation, "Artificial conversational speech", ITU-T,
March 1993.
[19] S. Floyd, "Metrics for the Evaluation of Congestion Control
Mechanisms", http://www.icir.org/tmrg/.
[20] R. Jain, "The Art of Computer Systems Performance Analysis",
Wiley-Interscience, New York, NY, 1991.
[21] D. Harrison, RPI NS2 Graphing and Statistics Package,
http://networks.ecse.rpi.edu/~harrisod/graph.html
[22] G. Wang and Y. Xia, An NS2 TCP Evaluation Tool,
http://labs.nec.com.cn/tcpeval.htm
Wang, Xia and Harrison [Page 13]
Internet-Draft TMRG, Evaluation, Tool April 2007
Author's Addresses
Gang Wang
NEC Laboratories China
14/F, Bldg.A, Innovation Plaza, Tsinghua Science Park
No.1, Zhong Guan Chun East Road, Beijing, China
Phone: +86-10-6270-5962 ext 511
Email: wanggang@research.nec.com.cn
Yong Xia
NEC Laboratories China
14/F, Bldg.A, Innovation Plaza, Tsinghua Science Park
No.1, Zhong Guan Chun East Road, Beijing, China
Phone: +86-10-6270-5962 ext 320
Email: xiayong@research.nec.com.cn
David Harrison
BitTorrent
201 Mission Blvd Suite 900
San Francisco, CA 94105
Phone: +1-415-568-9000 ext 9012
Email: dave@bittorrent.com
Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
Wang, Xia and Harrison [Page 14]
Internet-Draft TMRG, Evaluation, Tool April 2007
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Disclaimer of Validity
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Copyright Statement
Copyright (C) The IETF Trust (2007).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
Acknowledgment
Funding for the RFC Editor function is currently provided by the
Internet Society.
Wang, Xia and Harrison [Page 15]
| PAFTECH AB 2003-2026 | 2026-04-23 06:11:06 |