One document matched: draft-morton-ippm-advance-metrics-00.xml

<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?>
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?>
<rfc category="info" docName="draft-morton-ippm-advance-metrics-00"
     ipr="pre5378Trust200902">
  <front>
    <title abbrev="Advancing Metrics">Problems and Possible Solutions for
    Advancing Metrics on the Standards Track</title>

    <author fullname="Al Morton" initials="A." surname="Morton">
      <organization>AT&T Labs</organization>

      <address>
        <postal>
          <street>200 Laurel Avenue South</street>

          <city>Middletown,</city>

          <region>NJ</region>

          <code>07748</code>

          <country>USA</country>
        </postal>

        <phone>+1 732 420 1571</phone>

        <facsimile>+1 732 368 1192</facsimile>

        <email>acmorton@att.com</email>

        <uri>http://home.comcast.net/~acmacm/</uri>
      </address>
    </author>

    <date day="4" month="July" year="2009" />

    <abstract>
      <t>This memo identifies some issues with the process of progressing
      performance metric RFCs along the standards track. This memo takes the
      position that the metric definitions themselves should be the primary
      focus, rather than the implementations of metrics. This appears to allow
      some simplification of the task at hand and subsequently leads to
      solutions for the issues raised.</t>
    </abstract>

    <note title="Requirements Language">
      <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
      "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
      document are to be interpreted as described in <xref
      target="RFC2119">RFC 2119</xref>.</t>
    </note>
  </front>

  <middle>
    <section title="Introduction">
      <t>IPPM has been considering how to advance their metrics along the
      standards track since 2001, with the initial publication of
      Bradner/Paxson/Mankin's memo [ref to work in progress]. The original
      proposal was to compare the results of implementations of the metrics,
      because the usual procedures for advancing protocols did not appear to
      apply. It was found to be difficult to achieve consensus on exactly how
      to compare implementations, since there were many legitimate sources of
      variation that would emerge in the results despite the best attempts to
      keep the network measured equal for both.</t>

      <t>This author takes the position that the metric definitions themselves
      should be the primary focus, rather than the implementations of metrics.
      The advancement process should produce confidence that the metric
      definitions and supporting material are clearly worded and unambiguous,
      OR, it should identify ways in which the metric definitions should be
      revised to achieve clarity.</t>

      <t>The process should also permit identification of options that were
      not implemented, so that they can be removed from the advancing
      specification (this is an aspect more typical of protocol advancement
      along the standards track).</t>

      <t>This memo first identifies some issues with the current approach of
      comparing implementations (primarily on live network paths), and then
      looks at the metric RFC advancement process in a different way to see
      how some of the issues might be resolved.</t>

      <t>This memo's purpose is to make some constructive observations on the
      approach as the author perceives it to be (at the moment). It was
      prepared to help progress discussions on the topic of metric
      advancement, both through e-mail and at the upcoming IPPM meeting at
      IETF-75 in Stockholm.</t>
    </section>

    <section title="Issues with comparing implementations">
      <t>A few topics have been, and continue to be issues for comparing
      implementations. This section lists some of these issues, and
      simplifying solutions when they seem possible under the current
      approach.</t>

      <section title="Implementation variability">
        <t>The metric RFCs generally allow quite a bit of flexibility for
        implementators to design metrics that meet their particular needs.
        However, if two implementations cannot use identical options, it adds
        complexity to the comparison because the statistical analysis must
        allow for the differences.</t>

        <t>A solution would be to mandate that implementations used in the
        advancement process use identical metric parameters and options.</t>
      </section>

      <section title="Deciding on the statistical methods">
        <t>As mentioned above, allowing for implementation variability made
        this a complex process.</t>

        <t>Another complexity is the variability in metric statistics allowed
        in the RFCs: min, max, percentiles, ratios, etc. Comparisons vary by
        the statistic.</t>

        <t>This problem is being worked, but perhaps the problem statement
        will need simplification before it can be solved.</t>
      </section>

      <section title="Assumption of non-interoperable implementations">
        <t>The original stdmetrics memo was written before OWAMP <xref
        target="RFC4656"></xref> and TWAMP <xref target="RFC5357"></xref> were
        proposed, and correctly assumed that measurement implementations could
        not talk to each other, as this was the common case at the time. Only
        the results of measurements could be compared.</t>

        <t>With the approval of OWAMP and TWAMP, and the emergence of many
        implementations, this facility could be exploited to produce:</t>

        <t><list style="symbols">
            <t>testing opportunities with less coordination required to
            transport and install different implementations side-by-side.</t>

            <t>development of test vectors in OWAMP-Test or TWAMP-Test packet
            formats.</t>
          </list></t>
      </section>

      <section title="Determining whether Lab testing can serve the process">
        <t>Since the measurement systems developed for metrics in the RFC 2330
        framework are intended for use on live networks, it was taken that the
        comparison would also involve live network measurements. Lab
        measurements were not considered the primary basis for comparison, if
        they were to be used at all.</t>

        <t>A solution to this question may stem from deciding exactly what the
        RFC advancement process will entail.</t>

        <t>This memo posits that several key aspects of the metric definition
        implementations cannot be conveniently examined in field measurement
        scenarios, and that lab measurements could be a reasonable basis for a
        purely metric-definition-centric advancement process (as described in
        section 3).</t>
      </section>

      <section title="Achieving "identical" network conditions ">
        <t>The basis for this concern is the amount of parallelism that is
        common in devices/networks today. Parallel resources are applied to
        increase capacity, and hashing functions (on addresses and ports)
        determine which set of resources all the packets in a flow will
        traverse. For one analysis of the situation, see the Benchmarking WG's
        Hash and Stuffing <xref target="RFC4814"></xref>.</t>

        <t>Two measurement devices on the same sub-net will have different
        addresses, and their packet streams are likely to be assigned to
        different network resources where delay, loss, and other impairments
        can differ for legitimate reasons. Testing in the lab may not remove
        the parallel resources, but it can provide some time stability that's
        never assured in live network testing.</t>

        <t>One possible solution proposed in Geib's work-in-progress is to
        encapsulate all measurement streams in an IP tunnel, specifically and
        IPsec tunnel. This needs further investigation for feasibility and
        potential new issues raised by encaulation/de-encapsulation
        processes.</t>
      </section>

      <section title="IETF is not in the Certification Business">
        <t>We're not.</t>

        <t>The solution seems to be carefully wording the process of metric
        advancement so that it is clear that the metric definitions are the
        focus throughout, and not the implementations.</t>
      </section>
    </section>

    <section title="A Definition-centric metric advancement process">
      <t>The process described in this section takes as a first principle that
      the metric definitions, embodied in the text of the RFCs, are the
      objects that require evaluation and possible revision in order to
      advance to the next step on the standards track.</t>

      <t>IF two implementations do not measure an equivalent singleton, or
      sample, or produce the an equivalent statistic,</t>

      <t>AND sources of measurement error do not adequately explain the lack
      of agreement,</t>

      <t>THEN the details of each implementation should be audited along with
      the exact definition text, to determine if there is a lack of clarity
      that has caused the implementations to vary in a way that affects the
      correspondence of the results.</t>

      <t>IF there was a lack of clarity or multiple legitimate interpretations
      of the definition text,</t>

      <t>THEN the text should be modified and the resulting memo proposed for
      consensus and advancement along the standards track.</t>

      <t>The figure below illustrates this process:</t>

      <t><figure>
          <preamble></preamble>

          <artwork><![CDATA[     ,---.
    /     \
   ( Start )
    \     /    Implementations
     `-+-'        +-------+
       |         /|   1   `.
   +---+----+   / +-------+ `.-----------+      ,-------.
   |  RFC   |  /             |Check for  |    ,' was RFC `.  YES
   |        | /              |Equivalence.....  clause x   -------+
   |        |/    +-------+  |under      |    `. clear?  ,'       |
   | Metric \.....|   2   ....relevant   |      `---+---'    +----+---+
   | Metric |\    +-------+  |identical  |       No |        |Advance |
   | Metric | \              |network    |      +---+---.----+spec    |
   |  ...   |  \             |conditions |      |Modify |    +----+---+
   |        |   \ +-------+  |           |      |Spec   |         |
   +--------+    \|   n   |.'+-----------+      +-------+      +--+-+
                  +-------+                                    |DONE|
                                                               +----+
]]></artwork>

          <postamble></postamble>
        </figure></t>
    </section>

    <section title="Examples of checking metric definitions in the Lab">
      <t>This section describes some quick examples of lab tests with devices
      and/or simulators to create relevant conditions to test whether the
      metric definitions were interpreted consistently by implementors.</t>

      <t>For these tests, a stream of 100 (?) packets SHALL be sent from
      Source to Destination in each implementation.</t>

      <t>These examples do not entirely avoid the problem of declaring
      equivalence with a statistical test, but the lab conditions should
      simplify the problem by removing as much variability as possible.</t>

      <t>Note that there are only five instances of the requirement term
      "MUST" in <xref target="RFC2679"></xref> outside of the boilerplate and
      <xref target="RFC2119"></xref> reference.</t>

      <section title="One-way Delay, Loss threshold, RFC 2679">
        <t>This test determines if implementations use the same configured
        maximum waiting time delay from one measurement to another under
        different delay conditions, and correctly declare packets arriving in
        excess of the waiting time threshold as lost.</t>

        <t>See Section 3.5 of <xref target="RFC2679"></xref>, 3rd bullet point
        and also Section 3.8.2 of <xref target="RFC2679"></xref>.</t>

        <t><list style="numbers">
            <t>configure a path with 1 sec one-way constant delay</t>

            <t>measure one-way delay with 2 or more implementations, using
            identical waiting time thresholds for loss set at 2 seconds</t>

            <t>configure the path with 3 sec one-way delay</t>

            <t>repeat measurements</t>

            <t>observe that the increase measured in step 4 caused all packets
            to be declared lost, and that all packets that arrive successfully
            in step 2 are assigned a valid one-way delay.</t>
          </list></t>
      </section>

      <section title="One-way Delay, First-bit to Last bit, RFC 2679">
        <t>This test determines if implementations register the same relative
        increase in delay from one measurement to another under different
        delay conditions. This test tends to cancel the sources of error which
        may be present in an implementation.</t>

        <t>See Section 3.7.2 of <xref target="RFC2679"></xref>, and Section
        10.2 of <xref target="RFC2330"></xref>.</t>

        <t><list style="numbers">
            <t>configure a path with X ms one-way constant delay, and ideally
            including a low-speed link</t>

            <t>measure one-way delay with 2 or more implementations, using
            identical options and equal size small packets (e.g., 100 octet IP
            payload)</t>

            <t>maintain the same path with X ms one-way delay</t>

            <t>measure one-way delay with 2 or more implementations, using
            identical options and equal size large packets (e.g., 1500 octet
            IP payload)</t>

            <t>observe that the increase measured in steps 2 and 4 is
            equivalent to the increase in ms expected due to the larger
            serialization time for each implementation. Most of the
            measurement errors in each system should cancel, if they are
            stationary.</t>
          </list></t>
      </section>

      <section title="One-way Delay, RFC 2679">
        <t>This test determines if implementations register the same relative
        increase in delay from one measurement to another under different
        delay conditions. This test tends to cancel the sources of error which
        may be present in an implementation.</t>

        <t>This test is intended to evaluate measurments in sections 3 and 4
        of <xref target="RFC2679"></xref>.</t>

        <t><list style="numbers">
            <t>configure a path with X ms one-way constant delay</t>

            <t>measure one-way delay with 2 or more implementations, using
            identical options</t>

            <t>configure the path with X+Y ms one-way delay</t>

            <t>repeat measurements</t>

            <t>observe that the increase measured in steps 2 and 4 is ~Y ms
            for each implementation. Most of the measurement errors in each
            system should cancel, if they are stationary.</t>
          </list></t>
      </section>

      <section title="Error Calibration, RFC 2679">
        <t>This is a simple check to determine if an implementation reports
        the error calibration as required in Section 4.8 of <xref
        target="RFC2679"></xref>. Note that the context (Type-P) must also be
        reported.</t>
      </section>
    </section>

    <section anchor="Security" title="Security Considerations">
      <t>There are no security issues raised by discussing the topic of metric
      RFC advancement along the standards track.</t>

      <t>The security considerations that apply to any active measurement of
      live networks are relevant here as well. See <xref
      target="RFC4656"></xref> and <xref target="RFC5357"></xref>.</t>
    </section>

    <section anchor="IANA" title="IANA Considerations">
      <t>This memo makes no requests of IANA, and hopes that IANA will leave
      it alone, as well.</t>
    </section>

    <section anchor="Acknowledgements" title="Acknowledgements">
      <t>The author would like to thank anyone who reads this memo for helpful
      review and comments.</t>
    </section>
  </middle>

  <back>
    <references title="Normative References">
      <?rfc include="reference.RFC.2119"?>

      <?rfc include='reference.RFC.2330'?>

      <?rfc include='reference.RFC.2679'?>

      <?rfc include='reference.RFC.2680'?>

      <?rfc include='reference.RFC.4656'?>

      <?rfc include='reference.RFC.4814'?>

      <?rfc include='reference.RFC.5226'?>

      <?rfc include='reference.RFC.5357'?>
    </references>
  </back>
</rfc>