One document matched: draft-morton-perf-metrics-framework-00.xml

<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?>
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?>
<rfc category="info" docName="draft-morton-perf-metrics-framework-00"
     ipr="full3978">
  <front>
    <title abbrev="Perf. Metric Framework">Framework for Performance Metric
    Development</title>

    <author fullname="Al Morton" initials="A." surname="Morton">
      <organization>AT&T Labs</organization>

      <address>
        <postal>
          <street>200 Laurel Avenue South</street>

          <city>Middletown,</city>

          <region>NJ</region>

          <code>07748</code>

          <country>USA</country>
        </postal>

        <phone>+1 732 420 1571</phone>

        <facsimile>+1 732 368 1192</facsimile>

        <email>acmorton@att.com</email>

        <uri>http://home.comcast.net/~acmacm/</uri>
      </address>
    </author>

    <author fullname="Alan Clark" initials="A." surname="Clark">
      <organization>Telchemy</organization>

      <address>
        <postal>
          <street></street>

          <city></city>

          <region></region>

          <code></code>

          <country></country>
        </postal>

        <phone></phone>

        <facsimile></facsimile>

        <email></email>

        <uri></uri>
      </address>
    </author>

    <date day="15" month="October" year="2007" />

    <abstract>
      <t>This memo describes a framework and guidelines for the development of
      performance metrics that are beyond the scope of existing working group
      charters in the IETF. In this version, the memo refers to a Performance
      Metrics Entity, or PM Entity, which may in future be a working group or
      directorate or a combination of these two.</t>
    </abstract>

    <note title="Requirements Language">
      <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
      "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
      document are to be interpreted as described in <xref
      target="RFC2119">RFC 2119</xref>.</t>
    </note>
  </front>

  <middle>
    <section title="Introduction">
      <t>There have always been some uncertainties about the performance and
      suitability of new technologies and applications for their intended
      audience, and the Internet is no exception. Most uncertainties are
      effectively addressed through quantified assessment of key performance
      indicators. Standardized performance metrics add the desirable features
      of consistent implementation, interpretation, and comparison.</t>

      <t>There are at least three phases in the development of performance
      standards. They are:</t>

      <t><list style="numbers">
          <t>Definition of a Performance Metric and its units of measure</t>

          <t>Specification of a Method of Measurement</t>

          <t>Specification of the Reporting Format</t>
        </list>In some metric development activites, there are additional
      steps, such as setting numerical requirements or objectives. This memo
      will focus on metric definition, but it is worth noting that the
      products of this phase benefit when the challenges of measurement are
      considered.</t>

      <t>In this version, the memo refers to a Performance Metrics Entity, or
      PM Entity, which may in future be a working group or directorate or a
      combination of these two.</t>

      <section title="Background and Motivation">
        <t>Although the IETF has two Working Groups dedicated to the
        development of performance metrics, they each have strict limitations
        in their charters:</t>

        <t>- The Benchmarking Methodology WG has addressed a range of
        networking technologies and protocols in their long history (such as
        IEEE 802.3, ATM, Frame Relay, and Routing Protocols), but the charter
        strictly limits their performance characterizations to the laboratory
        environment.</t>

        <t>- The IP Performance Metrics WG has the mandate to develop metrics
        applicable to live IP networks, but it is specifically prohibited from
        developing metrics that characterize traffic (such as a VoIP
        stream).</t>

        <t>A BOF held at IETF-69 introduced the IETF community to the
        possibility of a generalized activity to define standardized
        performance metrics. The existence of a growing list of
        Internet-Drafts on performance metrics (with community interest in
        development, but in un-chartered areas) illustrates the need for
        additional performance work. The majority of people present at the BOF
        supported the proposition that IETF should be working in these areas,
        and no one objected to any of the proposals. </t>

        <t>The IETF does have current and completed activities related to the
        reporting of application performance metrics (e.g. RAQMON) and is also
        actively involved in the development of reliable transport protocols
        which would affect the relationship between IP performance and
        application performance.</t>

        <t>Thus there is a gap in the currently chartered coverage of IETF
        WGs: development of performance metrics for non-IP-layer protocols
        that can be used to characterize performance on live networks.</t>
      </section>

      <section title="Organization of this memo">
        <t>This memo is divided in two major sections beyond the Purpose and
        Scope section. The first is a definition and description of a
        performance metric and its key aspects. The second defines a process
        to develop these metrics that is applicable to the IETF
        environment.</t>
      </section>

      <t></t>
    </section>

    <section title="Purpose and Scope">
      <t>The purpose of this memo is to define a framework and a process for
      developing performance metrics in the IETF.</t>

      <t>The scope includes metric definition for any protocol developed by
      the IETF. However, other frameworks exist for IETF chartered work <xref
      target="RFC2330"></xref>, and this memo is not intended to superceed the
      existing working methods.</t>

      <t>Question: How do we write this scope so as not to conflict with IPPM
      and BMWG? Can we say that this framework may be useful to those
      activities as well, and leave it at that? Perhaps this will do:</t>

      <t>This process is not intended to govern performance metric development
      in existing IETF WG, such as IPPM and BMWG. However, the framework and
      guidelines may be useful in these activities, and MAY be applied where
      appropriate.</t>
    </section>

    <section title="Metrics Development">
      <t>This section provides key definitions and qualifications of
      performance metrics.</t>

      <section title="Definitions of a Metric">
        <t>A metric is a measure of an observable behavior of an application,
        protocol or other system. The definition of a metric often assumes
        some implicit or explicit underlying statistical process, and a metric
        is an estimate of a parameter of this process. If the assumed
        statistical process closely models the behavior of the system then the
        metric is "better" in the sense that it more accurately characterizes
        the state or behavior of the system.</t>

        <t>A metric should serve some defined purpose. This may include the
        measurement of capacity, quantifying how bad some problem is,
        measurement of service level, problem diagnosis or location and other
        such uses. A metric may also be an input to some other process, for
        example the computation of a composite metric or a model or simulation
        of a system. Tests of the "usefulness" of a metric include:</t>

        <t><list style="empty">
            <t>(i) the degree to which its absence would cause significant
            loss of information on the behavior or state of the application or
            system being measured</t>

            <t>(ii) the correlation between the metric and the quality of
            service / experience delivered to the user (person or other
            application)</t>

            <t>(iii) the degree to which the metric is able to support the
            identification and location of problems affecting service
            quality.</t>
          </list>For example, consider a distributed application operating
        over a network connection that is subject to packet loss. A Packet
        Loss Rate (PLR) metric is defined as the mean packet loss rate over
        some time period. If the application performs poorly over network
        connections with high packet loss rate and always performs well when
        the packet loss rate is zero then the PLR metric is useful to some
        degree. Some applications are sensitive to short periods of high loss
        (bursty loss). If both bursty and independent loss occur, it is
        possible that there may be periods during which the PLR metric would
        be the same however the application performance would be different -
        i.e. PLR is weakly correlated with application performance - which
        would suggest that the metric is weak.<!--This is a comment: suggesting a change to the above:  - which would suggest
the PLR metric dies not capture all the information necessary to infer the
performance of some applications under all conditions.--></t>
      </section>

      <section title="Composed Metrics">
        <t>Some metrics may not be measured directly, but may be composed from
        metrics that have been measured. Usually the contribution metrics have
        a limited scope in time or space, and they can be combined to estimate
        the performance of some larger entity.</t>

        <t><xref target="I-D.ietf-ippm-framework-compagg"></xref> gives the
        framework for three types of composed metrics: Temporal Aggregation,
        Spatial Aggregation, and Spatial Composition.</t>

        <t>Spatial Composition is described in <xref
        target="I-D.ietf-ippm-spatial-composition"></xref>. Other memos in
        this framework are TBD.</t>

        <t>>>>Not sure what's intended in this next one
        >>></t>

        <t>- Derivative / Composite metrics (e.g. combined effect of several
        different metrics)</t>
      </section>

      <section title="Metric Specification">
        <t>Process for specifying a metric</t>
      </section>

      <section title="Classes of Metrics">
        <t>Simplify process by identifying classes of metric</t>
      </section>

      <section title="Qualifying Metrics">
        <t>Each metric SHOULD be assessed according to the following list of
        qualifications:</t>

        <t><list style="symbols">
            <t>Unambiguously defined?</t>

            <t>Units of Measure Specified?</t>

            <t>Measurement Errors Identified?</t>

            <t>Repeatable?</t>

            <t>Implementable?</t>

            <t>Assumptions concerning underlying process?</t>

            <t>Use cases?</t>

            <t>Correlation with application performance/ user experience?</t>
          </list>(not sure that the last one is essential, it can be useful to
        characterize a network attribute without linking it to application
        performance/user experience)</t>
      </section>

      <section title="Reporting Models">
        <t>A metric, or some set of metrics, may be measured over some time
        period, measured continuously, sampled, aggregated and/or combined
        into composite metrics and then reported using a "push" or "pull"
        model. Reporting protocols typically introduce some limitations and
        assumptions with regard to the definition of a metric.</t>

        <t></t>
      </section>
    </section>

    <section title="Performance Metric Development Process">
      <t></t>

      <section title="New Proposals for Metrics">
        <t>The following entry criteria will be considered for each
        proposal.</t>

        <t>Proposals SHOULD be prepared as Internet Drafts, describing the
        metrics and conforming to the qualifications above as much as
        possible.</t>

        <t>Proposals SHOULD be vetted by the corresponding protocol
        development Working Group prior to discussion by the PM Entity. This
        aspect of the process includes an assessment of the need for the
        metrics proposed and assessment of the support for their development
        in IETF.</t>

        <t>Proposals SHOULD include an assessment of interaction and/or
        overlap with work in other Standards Development Organizations.</t>

        <t>Proposals SHOULD specify the intended audience and users of the
        metrics. The development process encourages participation by members
        of the intended audience.</t>
      </section>

      <section title="Proposal Approval">
        <t>Who does this???</t>

        <t>The IETF/IESG/Relevant ADs/Relevant WG/PM Entity ???</t>

        <t>This section depends on the direction of the solution, or form that
        the PM Entity takes.</t>
      </section>

      <section title="PM Entity Interaction with other WGs">
        <t>The PM Entity SHALL work in partnership with the related protocol
        development WG when considering an Internet Draft that specifies
        performance metrics for a protocol. A sufficient number of individuals
        with expertise must be willing to consult on the draft. If the related
        WG has concluded, comments on the proposal should still be sought from
        key RFC authors and former chairs, or from the WG mailing list if it
        was not closed.</t>

        <t>A dedicated mailing list MAY be initiated for each work area, so
        that protocol experts can subscribe to and receive the message traffic
        that is relevant to their work.</t>

        <t>In some cases, it will be appropriate to have the IETF session
        discussion during the related protocol WG session, to maximize
        visibility of the effort to that WG and expand the review.</t>
      </section>

      <section title="Standards Track Performance Metrics">
        <t>The PM Entity will manage the progression of PM RFCs along the
        Standards Track. See <xref target="I-D.bradner-metricstest"></xref>.
        This may include the preparation of test plans to examine different
        implementations of the metrics to ensure that the metric definitions
        are clear and unambiguous (depending on the final form of the draft
        above).</t>
      </section>
    </section>

    <section anchor="IANA" title="IANA Considerations">
      <t>This document makes no request of IANA.</t>

      <t>Note to RFC Editor: this section may be removed on publication as an
      RFC.</t>
    </section>

    <section anchor="Security" title="Security Considerations">
      <t>In general, the existence of framework for performance metric
      development does not constitute a security issue for the Internet.</t>

      <t>The security considerations that apply to any active measurement of
      live networks are relevant here as well. See <xref
      target="RFC4656"></xref>.</t>
    </section>

    <section anchor="Acknowledgements" title="Acknowledgements">
      <t>The authors would like to thank</t>
    </section>
  </middle>

  <back>
    <references title="Normative References">
      <?rfc include="reference.RFC.2119"?>

      <?rfc include='reference.RFC.2330'?>

      <?rfc include='reference.RFC.4656'?>

      <?rfc ?>
    </references>

    <references title="Informative References">
      <?rfc include='reference.I-D.ietf-ippm-framework-compagg'?>

      <?rfc include='reference.I-D.ietf-ippm-spatial-composition'?>

      <?rfc include='reference.I-D.bradner-metricstest'?>

      <reference anchor="Casner">
        <front>
          <title>A Fine-Grained View of High Performance Networking, NANOG 22
          Conf.; http://www.nanog.org/mtg-0105/agenda.html</title>

          <author fullname="S. Casner, C. Alaettinoglu, and C. Kuan,"
                  surname="">
            <organization></organization>
          </author>

          <date month="May 20-22" year="2001" />
        </front>
      </reference>
    </references>
  </back>
</rfc>