One document matched: draft-morton-ippm-testplan-rfc2679-00.xml
<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?>
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?>
<rfc category="info" docName="draft-morton-ippm-testplan-rfc2679-00"
ipr="pre5378Trust200902">
<front>
<title abbrev="Stds Track Tests RFC2679">Test Plan and Results for
Advancing RFC 2679 on the Standards Track</title>
<author fullname="Len Ciavattone" initials="L." surname="Ciavattone">
<organization>AT&T Labs</organization>
<address>
<postal>
<street>200 Laurel Avenue South</street>
<city>Middletown</city>
<region>NJ</region>
<code>07748</code>
<country>USA</country>
</postal>
<phone>+1 732 420 1239</phone>
<facsimile></facsimile>
<email>lencia@att.com</email>
<uri></uri>
</address>
</author>
<author fullname="Ruediger Geib" initials="R." surname="Geib">
<organization>Deutsche Telekom</organization>
<address>
<postal>
<street>Heinrich Hertz Str. 3-7</street>
<!-- Reorder these if your country does things differently -->
<code>64295</code>
<city>Darmstadt</city>
<region></region>
<country>Germany</country>
</postal>
<phone>+49 6151 628 2747</phone>
<email>Ruediger.Geib@telekom.de</email>
<!-- uri and facsimile elements may also be added -->
</address>
</author>
<author fullname="Al Morton" initials="A." surname="Morton">
<organization>AT&T Labs</organization>
<address>
<postal>
<street>200 Laurel Avenue South</street>
<city>Middletown</city>
<region>NJ</region>
<code>07748</code>
<country>USA</country>
</postal>
<phone>+1 732 420 1571</phone>
<facsimile>+1 732 368 1192</facsimile>
<email>acmorton@att.com</email>
<uri>http://home.comcast.net/~acmacm/</uri>
</address>
</author>
<author fullname="Matthias Wieser" initials="M." surname="Wieser">
<organization>University of Applied Sciences Darmstadt</organization>
<address>
<postal>
<street>Birkenweg 8 Department EIT</street>
<!-- Reorder these if your country does things differently -->
<code>64295</code>
<city>Darmstadt</city>
<region></region>
<country>Germany</country>
</postal>
<phone></phone>
<email>matthias.wieser@stud.h-da.de</email>
<!-- uri and facsimile elements may also be added -->
</address>
</author>
<date day="6" month="March" year="2011" />
<abstract>
<t>This memo proposes to advance a performance metric RFC along the
standards track, specifically RFC 2679 on One-way Delay Metrics.
Observing that the metric definitions themselves should be the primary
focus rather than the implementations of metrics, this memo describes
the test procedures to evaluate specific metric requirement clauses to
determine if the requirement has been interpreted and implemented as
intended. Two completely independent implementations have been tested
against the key specifications of RFC 2679.</t>
</abstract>
<note title="Requirements Language">
<t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in <xref
target="RFC2119">RFC 2119</xref>.</t>
</note>
</front>
<middle>
<section title="Introduction">
<t>The IETF (IP Performance Metrics working group, IPPM) has considered
how to advance their metrics along the standards track since 2001, with
the initial publication of Bradner/Paxson/Mankin's memo [ref to work in
progress, draft-bradner-metricstest-]. The original proposal was to
compare the results of implementations of the metrics, because the usual
procedures for advancing protocols did not appear to apply. It was found
to be difficult to achieve consensus on exactly how to compare
implementations, since there were many legitimate sources of variation
that would emerge in the results despite the best attempts to keep the
network paths equal, and because considerable variation was allowed in
the parameters (and therefore implementation) of each metric.
Flexibility in metric definitions, essential for customization and broad
appeal, made the comparison task quite difficult.</t>
<t>A renewed work effort sought to investigate ways in which the
measurement variability could be reduced and thereby simplify the
problem of comparison for equivalence.</t>
<t>There is *preliminary* consensus <xref
target="I-D.ietf-ippm-metrictest"></xref> that the metric definitions
should be the primary focus of evaluation rather than the
implementations of metrics, and equivalent results are deemed to be
evidence that the metric specifications are clear and unambiguous. This
is the metric specification equivalent of protocol interoperability. The
advancement process either produces confidence that the metric
definitions and supporting material are clearly worded and unambiguous,
OR, identifies ways in which the metric definitions should be revised to
achieve clarity.</t>
<t>The process should also permit identification of options that were
not implemented, so that they can be removed from the advancing
specification (this is an aspect more typical of protocol advancement
along the standards track).</t>
<t>This memo's purpose is to implement the current approach for <xref
target="RFC2679"></xref>. It was prepared to help progress discussions
on the topic of metric advancement, both through e-mail and at the
upcoming IPPM meeting at IETF.</t>
<t>In particular, consensus is sought on the extent of tolerable errors
when assessing equivalence in the results. In discussions, the IPPM
working group agreed that test plan and procedures should include the
threshold for determining equivalence, and this information should be
available in advance of cross-implementation comparisons. This memo
includes procedures for same-implementation comparisons to help set the
equivalence threshold.</t>
<t>Another aspect of the metric RFC advancement process is the
requirement to document the work and results. The procedures of <xref
target="RFC2026"></xref> are expanded in<xref target="RFC5657"></xref>,
including sample implementation and interoperability reports. This memo
follows the template in <xref
target="I-D.morton-ippm-advance-metrics"></xref> for the report that
accompanies the protocol action request submitted to the Area Director,
including description of the test set-up, procedures, results for each
implementation and conclusions.</t>
<section title="RFC 2679 Coverage">
<t>This plan, in it's first draft version, does not cover all critical
requirements and sections of <xref target="RFC2679"></xref>. Material
will be added as it is "discovered" (not all requirements use
requirements language).</t>
</section>
</section>
<section title="A Definition-centric metric advancement process">
<t>The process described in Section 3.5 of <xref
target="I-D.ietf-ippm-metrictest"></xref> takes as a first principle
that the metric definitions, embodied in the text of the RFCs, are the
objects that require evaluation and possible revision in order to
advance to the next step on the standards track.</t>
<t>IF two implementations do not measure an equivalent singleton or
sample, or produce the an equivalent statistic,</t>
<t>AND sources of measurement error do not adequately explain the lack
of agreement,</t>
<t>THEN the details of each implementation should be audited along with
the exact definition text, to determine if there is a lack of clarity
that has caused the implementations to vary in a way that affects the
correspondence of the results.</t>
<t>IF there was a lack of clarity or multiple legitimate interpretations
of the definition text,</t>
<t>THEN the text should be modified and the resulting memo proposed for
consensus and advancement along the standards track.</t>
<t>Finally, all the findings MUST be documented in a report that can
support advancement on the standards track, similar to those described
in <xref target="RFC5657"></xref>. The list of measurement devices used
in testing satisfies the implementation requirement, while the test
results provide information on the quality of each specification in the
metric RFC (the surrogate for feature interoperability).</t>
<t>The figure below illustrates this process:</t>
<t><figure>
<preamble></preamble>
<artwork><![CDATA[ ,---.
/ \
( Start )
\ / Implementations
`-+-' +-------+
| /| 1 `.
+---+----+ / +-------+ `.-----------+ ,-------.
| RFC | / |Check for | ,' was RFC `. YES
| | / |Equivalence..... clause x -------+
| |/ +-------+ |under | `. clear? ,' |
| Metric \.....| 2 ....relevant | `---+---' +----+---+
| Metric |\ +-------+ |identical | No | |Report |
| Metric | \ |network | +---+---. |results+|
| ... | \ |conditions | |Modify | |Advance |
| | \ +-------+ | | |Spec +----+ RFC |
+--------+ \| n |.'+-----------+ +-------+ |request?|
+-------+ +--------+
]]></artwork>
<postamble></postamble>
</figure></t>
</section>
<section title="Test configuration">
<t></t>
<t>>>>> This section needs to be updated
<<<<</t>
<t>One metric implementation used was NetProbe version 5.8.5, (an
earlier version is used in the WIPM system and deployed world-wide).
NetProbe uses UDP packets of variable size, and can produce test streams
with periodic or Poisson sample distributions.</t>
<t>>>> Add DT's Perfas Description</t>
<t>Figure 2 shows a view of the test path as each Implementation's test
flows pass through the Internet and the L2TPv3 tunnel IDs (1 and 2),
based on Figure 1 of <xref
target="I-D.ietf-ippm-metrictest"></xref>.</t>
<t><figure align="center" anchor="L2TPv3 tunnel">
<preamble />
<artwork align="left"><![CDATA[
Implementations ,---. +--------+
+~~~~~~~~~~~/ \~~~~~~| Remote |
+------->-----F2->-| / \ |->---+ |
| +---------+ | Tunnel ( ) | | |
| | transmit|-F1->-| ID 1 ( ) |->+ | |
| | Imp 1 | +~~~~~~~~~| |~~~~| | | |
| | receive |-<--+ ( ) | F1 F2 |
| +---------+ | |Internet | | | | |
*-------<-----+ F2 | | | | | |
+---------+ | | +~~~~~~~~~| |~~~~| | | |
| transmit|-* *-| | | |--+<-* |
| Imp 2 | | Tunnel ( ) | | |
| receive |-<-F1-| ID 2 \ / |<-* |
+---------+ +~~~~~~~~~~~\ /~~~~~~| Router |
`-+-' +--------+
]]></artwork>
<postamble>Illustration of a test setup with a bi-directional
tunnel. For simplicity, only two measurement implementations and two
flows (F#) between them are shown.</postamble>
</figure></t>
<t>The testing employs the Layer 2 Tunnel Protocol, version 3 (L2TPv3)
<xref target="RFC3931"></xref> tunnel between test sites on the
Internet. The tunnel IP and L2TPv3 headers are intended to conceal the
test equipment addresses and ports from hash functions that would tend
to spread different test streams across parallel network resources, with
likely variation in performance as a result.</t>
<t>At each end of the tunnel, VLANs encapsulated in the tunnel are
looped-back so that test traffic is returned to each test site. Thus,
test streams traverse the L2TP tunnel twice, but appear to be one-way
tests from the test equipment point of view.</t>
<t>The network emulator is a host running Fedora Core Linux
[http://fedoraproject.org/] with IP forwarding enabled and the NIST Net
emulator 2.0.12b [http://snad.ncsl.nist.gov/nistnet/] loaded and
operating.</t>
<t>The links between NetProbe hosts and the NIST Net emulator host were
100baseTx-FD (100Mbps full duplex) as reported by "mii-tool", except as
noted below.</t>
<t>>>>> We need to decide on common packet rates,
Poisson/Periodic, packet sizes, etc.</t>
<t>For these tests, a stream of at least 30 packets were sent from
Source to Destination in each implementation. Periodic streams (as per
<xref target="RFC3432"></xref>) with 1 second spacing were used, except
as noted.</t>
<t>Thus, the metric name for the testing configured here, with respect
to the IP header exposed to Internet processing, is:</t>
<t>Type-IP-protocol-115-One-way-Delay-<StreamType>-Stream</t>
<t>With (Section 4.2. <xref target="RFC2679"></xref>) Metric Parameters:
+ Src, the IP address of a host + Dst, the IP address of a host + T0, a
time + Tf, a time + lambda, a rate in reciprocal seconds</t>
<t>+ Thresh, a maximum waiting time in seconds (see Section 3.82 of
<xref target="RFC2679"></xref>) And (Section 4.3. <xref
target="RFC2679"></xref>)Metric Units: A sequence of pairs; the elements
of each pair are: + T, a time, and + dT, either a real number or an
undefined number of seconds. The values of T in the sequence are
monotonic increasing. Note that T would be a valid parameter to
Type-P-One-way-Delay, and that dT would be a valid value of
Type-P-One-way-Delay.</t>
<t>Also, Section 3.8.4 of <xref target="RFC2679"></xref> recommends that
the path SHOULD be reported. In this test set-up, most of the path
details will be concealed from the implementations by the L2TPv3
tunnels, thus a more informative path trace route can be conducted by
the routers at each location.</t>
<t>When NetProbe is used in production, a trace route is conducted in
parallel at the outset of measurements.</t>
<t>In Perfas, ???</t>
</section>
<section title="Error Calibration, RFC 2679">
<t>An implementation is required to report on its error calibration in
Section 3.8 of <xref target="RFC2679"></xref> (also required in Section
4.8 for sample metrics). Sections 3.6, 3.7, and 3.8 of <xref
target="RFC2679"></xref> give the detailed formulation of the errors and
uncertainties for calibration. In summary, Section 3.7.1 of <xref
target="RFC2679"></xref> describes the total time-varying uncertainty
as:</t>
<t>Esynch(t)+ Rsource + Rdest</t>
<t>where:</t>
<t>Esynch(t) denotes an upper bound on the magnitude of clock
synchronization uncertainty.</t>
<t>Rsource and Rdest denote the resolution of the source clock and the
destination clock, respectively.</t>
<t>Further, Section 3.7.2 of <xref target="RFC2679"></xref> describes
the total wire-time uncertainty as</t>
<t>Hsource + Hdest</t>
<t>referring to the upper bounds on host-time to wire-time for source
and destination, respectively.</t>
<t>Section 3.7.3 of <xref target="RFC2679"></xref> describes a test with
small packets over an isolated minimal network where the results can be
used to estimate systematic and random components of the sum of the
above errors or uncertainties. In a test with hundreds of singletons,
the median is the systematic error and when the median is subtracted
from all singletons, the remaining variability is the random error.</t>
<t>>>>>> The RFC text indicates that the clock-related
errors are not included in this analysis, but a sufficiently long test
(under full test load) should include all forms of error, IAO (in Al's
opinion).</t>
<t>The test context, or Type-P of the test packets, must also be
reported, as required in Section 3.8 of <xref target="RFC2679"></xref>
and all metrics defined there. Type-P is defined in Section 13 of <xref
target="RFC2330"></xref> (as are many terms used below).</t>
<section title="NetProbe Error and Type-P">
<t>Type-P for this test was IP-UDP with Best Effort DCSP. These
headers were encapsulated according to the L2TPv3 specifications <xref
target="RFC3931"></xref>, and thus may not influence the treatment
received as the packets traversed the Internet.</t>
<t>In general, NetProbe error is dependent on the specific version and
installation details.</t>
<t>NetProbe operates using host time above the UDP layer, which is
different from the wire-time preferred in <xref
target="RFC2330"></xref>, but can be identified as a source of error
according to Section 3.7.2 of <xref target="RFC2679"></xref>.</t>
<t>Accuracy of NetProbe measurements is usually limited by NTP
synchronization performance (which is typically taken as ~+/-1ms error
or greater), although the installation used in this testing often
exhibits errors much less than typical for NTP. The primary stratum 1
NTP server is closely located on a sparsely utilized network
management LAN, thus it avoids many concerns raised in Section 10
of<xref target="RFC2330"></xref> (in fact, smooth adjustment,
long-term drift analysis and compensation, and infrequent adjustment
all lead to stability during measurement intervals, the main
concern).</t>
<t>The resolution of the reported results is 1us (us = microsecond) in
the version of NetProbe tested here, which contributes to at least
+/-1us error.</t>
<t>NetProbe implements a time-keeping sanity check on sending and
receiving time-stamping processes. When the significant process
interruption takes place, individual test packets are flagged as
possibly containing unusual time errors, and are excluded from the
sample used for all "time" metrics.</t>
<t>We performed a NetProbe calibration of the type described in
Section 3.7.3 of <xref target="RFC2679"></xref>, using 64 Byte packets
over a cross-connect cable. The results estimate systematic and random
components of the sum of the Hsource + Hdest errors or uncertainties.
In a test with 300 singletons conducted over 30 seconds (periodic
sample with 100ms spacing), the median is the systematic error and the
remaining variability is the random error. One set of results is
tabulated below:</t>
<t><figure>
<preamble>(Results from the "R" software environment for
statistical computing and graphics - http://www.r-project.org/
)</preamble>
<artwork><![CDATA[> summary(XD4CAL)
CAL1 CAL2 CAL3
Min. : 89.0 Min. : 68.00 Min. : 54.00
1st Qu.: 99.0 1st Qu.: 77.00 1st Qu.: 63.00
Median :110.0 Median : 79.00 Median : 65.00
Mean :116.8 Mean : 83.74 Mean : 69.65
3rd Qu.:127.0 3rd Qu.: 88.00 3rd Qu.: 74.00
Max. :205.0 Max. :177.00 Max. :163.00
> boxplot(XD4CAL$CAL1,XD4CAL$CAL2,XD4CAL$CAL3) ]]></artwork>
<postamble>NetProbe Calibration with Cross-Connect Cable, one-way
delay values in microseconds (us)</postamble>
</figure></t>
<t>The median or systematic error can be as high as 110 us, and the
range of the random error is also on the order of 110 us for all
streams.</t>
<t>Also, anticipating the Anderson-Darling K-sample (ADK) comparisons
to follow, we corrected the CAL2 values for the difference between
means between CAL2 and CAL3 (as specified in <xref
target="I-D.ietf-ippm-metrictest"></xref>), and found strong support
for the (Null Hypothesis that) the samples are from the same
distribution (resolution of 1 us and alpha equal 0.05 and 0.01)<figure>
<preamble></preamble>
<artwork><![CDATA[> XD4CVCAL2 <- XD4CAL$CAL2 - (mean(XD4CAL$CAL2)-mean(XD4CAL$CAL3))
> boxplot(XD4CVCAL2,XD4CAL$CAL3)
> XD4CV2_ADK <- adk.test(XD4CVCAL2, XD4CAL$CAL3)
> XD4CV2_ADK
Anderson-Darling k-sample test.
Number of samples: 2
Sample sizes: 300 300
Total number of values: 600
Number of unique values: 97
Mean of Anderson Darling Criterion: 1
Standard deviation of Anderson Darling Criterion: 0.75896
T = (Anderson Darling Criterion - mean)/sigma
Null Hypothesis: All samples come from a common population.
t.obs P-value extrapolation
not adj. for ties 0.71734 0.17042 0
adj. for ties -0.39553 0.44589 1
> ]]></artwork>
<postamble></postamble>
</figure></t>
</section>
<section title="Perfas Error and Type-P">
<t></t>
</section>
</section>
<section title="Pre-determined Limits on Equivalence">
<t>>>>> This section contains many proposals
<<<<<</t>
<t>In this section, we provide the numerical limits on comparisons
between implementations, in order to declare that the results are
equivalent and therefore, the tested specification is clear.</t>
<t>A key point is that the allowable errors, corrections, and confidence
levels only need to be sufficient to detect mis-interpretation of the
tested specification resulting in diverging implementations.</t>
<t>Also, the allowable error must be sufficient to compensate for
measured path differences. It was simply not possible to measure fully
identical paths in the VLAN-loopback test configuration used, and this
practical compromise must be taken into account.</t>
<t>For Anderson-Darling K-sample (ADK) comparisons, the required
confidence factor for the cross-implementation comparisons SHALL be the
smallest of:</t>
<t><list style="symbols">
<t>0.95 confidence factor at 1ms resolution, or</t>
<t>the smallest confidence factor (in combination with resolution)
of the two same-implementation comparisons for the same test
conditions.</t>
</list>A constant time accuracy error of as much as +/-0.5ms MAY be
removed from one implementation's distributions (all singletons) before
the ADK comparison is conducted.</t>
<t>A constant propagation delay error (due to use of different sub-nets
between the switch and measurement devices at each location) of as much
as +2ms MAY be removed from one implementation's distributions (all
singletons) before the ADK comparison is conducted.</t>
<t>For comparisons involving the mean of a sample or other central
statistics, the limits on both the time accuracy error and the
propagation delay error constants given above also apply.</t>
</section>
<section title="Tests to evaluate RFC 2679 Specifications">
<t>This section describes some results from real-world (cross-Internet)
tests with measurement devices implementing IPPM metrics and a network
emulator to create relevant conditions, to determine whether the metric
definitions were interpreted consistently by implementors.</t>
<t>The procedures are slightly modified from the original procedures
contained in Appendix A.1 of <xref
target="I-D.ietf-ippm-metrictest"></xref>. The modifications include the
use of the mean statistic for comparisons.</t>
<t>Note that there are only five instances of the requirement term
"MUST" in <xref target="RFC2679"></xref> outside of the boilerplate and
<xref target="RFC2119"></xref> reference.</t>
<section title="One-way Delay, ADK Sample Comparison - Same Implementation">
<t>This test determines if implementations produce results that appear
to come from the same delay distribution, as an overall evaluation of
Section 4 of <xref target="RFC2679"></xref>, "A Definition for Samples
of One-way Delay". Same-implementation comparison results help to set
the threshold of equivalence that will be applied to
cross-implementation comparisons.</t>
<t>This test is intended to evaluate measurements in sections 3 and 4
of <xref target="RFC2679"></xref>.</t>
<t>By testing the extent to which the distributions of one-way delay
singletons from two implementations of <xref target="RFC2679"></xref>
appear to be from the same distribution, we economize on comparisons,
because comparing a set of individual summary statistics (as defined
in Section 5 of <xref target="RFC2679"></xref>) would require another
set of individual evaluations of equivalence. Instead, we can simply
check which statistics were implemented, and report on those
facts.</t>
<t><list style="numbers">
<t>Configure an L2TPv3 path between test sites, and each pair of
measurement devices to operate tests in their designated pair of
VLANs.</t>
<t>Measure a sample of one-way delay singletons with 2 or more
implementations, using identical options.</t>
<t>Measure a sample of one-way delay singletons with *five*
additional instances of the *same* implementations, using
identical options, noting that connectivity differences SHOULD be
the same as for the cross implementation testing.</t>
<t>Apply the ADK comparison procedures (see Appendix C of <xref
target="I-D.ietf-ippm-metrictest"></xref>) and determine the
resolution and confidence factor for distribution equivalence of
each same-implementation comparison and each cross-implementation
comparison.</t>
<t>Take the coarsest resolution and confidence factor for
distribution equivalence from the same-implementation pairs, or
the limit defined in Section 5 above, as a limit on the
equivalence threshold for these experimental conditions.</t>
<t>Apply constant correction factors to all singletons of the
sample distributions, as described and limited in Section 5
above.</t>
<t>Compare the cross-implementation ADK performance with the
equivalence threshold determined in step 5 to determine if
equivalence can be declared.</t>
</list></t>
<section title="NetProbe Same-implementation results">
<t>To be provided,</t>
<t><figure title="NetProbe ADK Results for same-implementation">
<preamble></preamble>
<artwork align="center"><![CDATA[ ]]></artwork>
<postamble></postamble>
</figure></t>
<t></t>
</section>
<section title="Perfas Same-implementation results">
<t>To be provided,</t>
<t><figure title="Perfas ADK Results for same-implementation">
<preamble></preamble>
<artwork align="center"><![CDATA[ ]]></artwork>
<postamble></postamble>
</figure></t>
<t></t>
</section>
<section title="One-way Delay, Cross-Implementation ADK Comparison">
<t></t>
</section>
<section title="Conclusions on the ADK Results for One-way Delay">
<t>>>> Comment: this section is a placeholder</t>
</section>
</section>
<section title="One-way Delay, Loss threshold, RFC 2679">
<t>This test determines if implementations use the same configured
maximum waiting time delay from one measurement to another under
different delay conditions, and correctly declare packets arriving in
excess of the waiting time threshold as lost.</t>
<t>See Section 3.5 of <xref target="RFC2679"></xref>, 3rd bullet point
and also Section 3.8.2 of <xref target="RFC2679"></xref>.</t>
<t><list style="numbers">
<t>configure an L2TPv3 path between test sites, and each pair of
measurement devices to operate tests in their designated pair of
VLANs.</t>
<t>configure the network emulator to add 0.5 sec one-way constant
delay to each direction of transmission (or 1 second one-way).</t>
<t>measure (average) one-way delay with 2 or more implementations,
using identical waiting time thresholds (Thresh) for loss set at 2
seconds</t>
<t>configure the network emulator to add 1 sec one-way constant
delay to each direction of transmission equivalent to 2 seconds of
additional one-way delay, or change the path delay while test is
in progress, when there are sufficient packets at the first delay
setting)</t>
<t>repeat/continue measurements</t>
<t>observe that the increase measured in step 5 caused all packets
with 2 sec additional delay to be declared lost, and that all
packets that arrive successfully in step 3 are assigned a valid
one-way delay.</t>
</list></t>
<section title="NetProbe results for Loss Threshold">
<t>In NetProbe, the Loss Threshold is implemented uniformly over all
packets as a post-processing routine. With the Loss Threshold set at
2 seconds, all packets with one-way delay >2 seconds are marked
"Lost" and included in the Lost Packet list with their transmission
time (as required in Section 3.3 of <xref target="RFC2680"></xref>).
22 of 38 packets were declared lost.</t>
</section>
<section title="Perfas Results for Loss Threshold">
<t>>>> Comment: this section is a placeholder</t>
</section>
<section title="Conclusions on Lab Results for Loss Threshold">
<t>>>> Comment: this section is a placeholder</t>
</section>
</section>
<section title="One-way Delay, First-bit to Last bit, RFC 2679">
<t>This test determines if implementations register the same relative
increase in delay from one measurement to another under different
delay conditions. This test tends to cancel the sources of error which
may be present in an implementation.</t>
<t>See Section 3.7.2 of <xref target="RFC2679"></xref>, and Section
10.2 of <xref target="RFC2330"></xref>.</t>
<t><list style="numbers">
<t>configure an L2TPv3 path between test sites, and each pair of
measurement devices to operate tests in their designated pair of
VLANs, and ideally including a low-speed link</t>
<t>measure (average) one-way delay with 2 or more implementations,
using identical options and equal size small packets (e.g., 100
octet IP payload)</t>
<t>maintain the same path with X ms one-way delay</t>
<t>measure (average) one-way delay with 2 or more implementations,
using identical options and equal size large packets (e.g., 1500
octet IP payload)</t>
<t>observe that the increase measured in steps 2 and 4 is
equivalent to the increase in ms expected due to the larger
serialization time for each implementation. Most of the
measurement errors in each system should cancel, if they are
stationary.</t>
</list></t>
<section title="NetProbe Lab results for Serialization">
<t>For this test only, the link between the NetProbe Source host and
the NIST Net emulator host was changed to 10baseT-FD (10Mbps full
duplex) as configured by "mii-tool".</t>
<t>When the UDP payload size was increased from 32 octets to 1400
octets, the NIST Net emulator exhibited a bi-modal delay
distribution. Investigation confirmed that the NetProbe
implementations tested did not exhibit bi-modal delay on an
alternate (network management) path.</t>
<t><figure
title="Average Delay over 60 packets for different payload sizes with Delay computations and comparison with expected delay difference for serialization.">
<preamble></preamble>
<artwork align="center"><![CDATA[1400 byte payload 32 byte payload
Delay for each mode (one mode) Delay Diff Expected Diff
microseconds microseconds microseconds microseconds
1001621 1000356 1265 1094.4
1002735 1000356 2379 1094.4]]></artwork>
<postamble></postamble>
</figure></t>
</section>
</section>
<section title="One-way Delay, Difference Sample Metric (Lab)">
<t>This test determines if implementations register the same relative
increase in delay from one measurement to another under different
delay conditions. This test tends to cancel the sources of error which
may be present in an implementation.</t>
<t>This test is intended to evaluate measurements in sections 3 and 4
of <xref target="RFC2679"></xref>.</t>
<t><list style="numbers">
<t>configure an L2TPv3 path between test sites, and each pair of
measurement devices to operate tests in their designated pair of
VLANs.</t>
<t>measure (average) one-way delay with 2 or more implementations,
using identical options</t>
<t>configure the path with X+Y ms one-way delay</t>
<t>repeat measurements</t>
<t>observe that the (average) increase measured in steps 2 and 4
is ~Y ms for each implementation. Most of the measurement errors
in each system should cancel, if they are stationary.</t>
</list></t>
<section title="NetProbe Lab results for Differential Delay">
<t>In this test, X=1000ms and Y=2000ms.</t>
<t><figure title="Average delays before/after 2 second increase">
<preamble></preamble>
<artwork align="center"><![CDATA[Average pre-increase delay, microseconds 1000276.6
Average post 2s additional, microseconds 3000282.6
Difference (should be ~= Y = 2s) 2000006]]></artwork>
<postamble></postamble>
</figure></t>
<t>The NetProbe implementation exhibited a 2 second increase with a
6 microsecond error (assuming that the NIST Net emulated delay
difference is exact).</t>
</section>
</section>
<section title="Implementation of Statistics for One-way Delay">
<t>The ADK tests the extent to which the sample distributions of
one-way delay singletons from two implementations of <xref
target="RFC2679"></xref> appear to be from the same overall
distribution. By testing this way, we economize on the number of
comparisons, because comparing a set of individual summary statistics
(as defined in Section 5 of <xref target="RFC2679"></xref>) would
require another set of individual evaluations of equivalence. Instead,
we can simply check which statistics were implemented, and report on
those facts, noting that Section 5 of <xref target="RFC2679"></xref>
does not specify the calculations exactly, and gives only some
illustrative examples.<figure>
<preamble></preamble>
<artwork><![CDATA[ NetProbe Perfas
5.1. Type-P-One-way-Delay-Percentile yes
5.2. Type-P-One-way-Delay-Median yes
5.3. Type-P-One-way-Delay-Minimum yes
5.4. Type-P-One-way-Delay-Inverse-Percentile no
]]></artwork>
<postamble>Implementation of Section 5 Statistics</postamble>
</figure></t>
<t>5.1. Type-P-One-way-Delay-Percentile 5.2.
Type-P-One-way-Delay-Median 5.3. Type-P-One-way-Delay-Minimum 5.4.
Type-P-One-way-Delay-Inverse-Percentile</t>
</section>
</section>
<section anchor="Security" title="Security Considerations">
<t>The security considerations that apply to any active measurement of
live networks are relevant here as well. See <xref
target="RFC4656"></xref> and <xref target="RFC5357"></xref>.</t>
</section>
<section anchor="IANA" title="IANA Considerations">
<t>This memo makes no requests of IANA, and hopes that IANA will be as
accepting of our new computer overlords as the authors intend to be.</t>
</section>
<section anchor="Acknowledgements" title="Acknowledgements">
<t>The authors thank Lars Eggert for his continued encouragement to
advance the IPPM metrics during his tenure as AD Advisor.</t>
</section>
</middle>
<back>
<references title="Normative References">
<?rfc include="reference.RFC.2119"?>
<?rfc include='reference.RFC.2026'?>
<?rfc include='reference.RFC.2330'?>
<?rfc include='reference.RFC.2679'?>
<?rfc include='reference.RFC.2680'?>
<?rfc include='reference.RFC.3432'?>
<?rfc include='reference.RFC.4656'?>
<?rfc include='reference.RFC.4814'?>
<?rfc include='reference.RFC.5226'?>
<?rfc include='reference.RFC.5357'?>
<?rfc include='reference.RFC.5657'?>
<?rfc include='reference.I-D.ietf-ippm-metrictest'?>
</references>
<references title="Informative References">
<?rfc include='reference.I-D.morton-ippm-advance-metrics'?>
<?rfc include='reference.RFC.3931'?>
</references>
</back>
</rfc>| PAFTECH AB 2003-2026 | 2026-04-24 05:56:29 |