One document matched: draft-ietf-pmol-metrics-framework-08.xml
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?>
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?>
<rfc category="bcp" docName="draft-ietf-pmol-metrics-framework-08"
ipr="pre5378Trust200902">
<front>
<title abbrev="Guidelines Perf. Metric Devel.">Guidelines for Considering New Performance Metric Development</title>
<author fullname="Alan Clark" initials="A." surname="Clark">
<organization>Telchemy Incorporated</organization>
<address>
<postal>
<street>2905 Premiere Parkway, Suite 280</street>
<city>Duluth</city>
<region>Georgia</region>
<code>30097</code>
<country>USA</country>
</postal>
<phone></phone>
<facsimile></facsimile>
<email>alan.d.clark@telchemy.com</email>
<uri></uri>
</address>
</author>
<author fullname="Benoit Claise" initials="B." surname="Claise">
<organization>Cisco Systems, Inc.</organization>
<address>
<postal>
<street>De Kleetlaan 6a b1</street>
<city>Diegem</city>
<code>1831</code>
<country>Belgium</country>
</postal>
<phone>+32 2 704 5622</phone>
<facsimile></facsimile>
<email>bclaise@cisco.com</email>
<uri></uri>
</address>
</author>
<date day="28" month="January" year="2011" />
<abstract>
<t>
This document describes a framework and a process for developing
Performance Metrics of protocols and applications transported over
IETF-specified protocols, and that can be used to
characterize traffic on live networks and services.</t>
</abstract>
<note title="Requirements Language">
<t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in <xref
target="RFC2119">RFC 2119</xref>.</t>
</note>
</front>
<middle>
<section title="Introduction">
<t>Many networking technologies, applications, or services, are distributed
in nature, and their performance may be impacted by IP impairments, server
capacity, congestion and other factors. It is important to measure the
performance of applications and services to ensure that quality objectives
are being met and to support problem diagnosis. Standardized metrics help
to ensure that performance measurement is implemented consistently and facilitate
interpretation and comparison.</t>
<t>There are at least three phases in the development of performance
standards. They are:</t>
<t><list style="numbers">
<t>Definition of a Performance Metric and its units of measure</t>
<t>Specification of a method of measurement</t>
<t>Specification of the reporting format</t>
</list>
During the development of metrics, it is often useful to define
performance objectives and expected value ranges. However, this is not
defined as part of the metric specification. </t>
<t>
The intended audience for this document includes, but is not
limited to, IETF participants who write Performance Metrics documents
in the IETF, reviewers of such documents, and members of the Performance
Metrics Entity.
</t>
<section title="Background and Motivation">
<t>
Although the IETF has two active Working Groups (WGs) dedicated to the
development of Performance Metrics, they each have strict limitations
in their charters:
</t>
<t>
- The Benchmarking Methodology WG has addressed a range of
networking technologies and protocols in their long history (such as
IEEE 802.3, ATM, Frame Relay, and Routing Protocols), but the charter
strictly limits their performance characterizations to the laboratory
environment.
</t>
<t>
- The IP Performance Metrics (IPPM) WG has developed a set of
standard metrics that can be applied to the quality, performance, and reliability
of Internet data delivery services. The IPPM metrics development is applicable
to live IP networks, but it is specifically prohibited from developing metrics that
characterize traffic at upper layers, such as a VoIP stream.
</t>
<t>
A Birds Of a Feather (BOF) held at IETF-69 introduced the IETF community to the
possibility of a generalized activity to define standardized
Performance Metrics. The existence of a growing list of
Internet-Drafts on Performance Metrics (with community interest in
development, but in un-chartered areas) illustrates the need for
additional performance work. The majority of people present at the
BOF supported the proposition that IETF should be working in these areas,
and no one objected to any of the proposals.
</t>
<t>
Previous IETF work related to reporting of application Performance Metrics
includes the "Real-time Application Quality-of-Service Monitoring (RAQMON) Framework"
<xref target="RFC4710">RFC 4710</xref>, which extends the remote network monitoring
(RMON) family of specifications to allow real-time quality-of-service (QoS)
monitoring of various applications that run on devices such as IP phones,
pagers, Instant Messaging clients, mobile phones, and various other
handheld computing devices. Furthermore, the "RTP Control Protocol Extended
Reports (RTCP XR)" <xref target="RFC3611">RFC 3611</xref> and the "SIP RTCP Summary
Report Protocol" <xref target="RFC6035"></xref> are
protocols that support the real-time reporting of Voice over IP and other
applications running on devices such as IP phones and mobile handsets.
</t>
<t>
The IETF is also actively involved in the development of reliable
transport protocols, such as <xref target="RFC0793">TCP</xref> or
<xref target="RFC4960">SCTP</xref>, which would affect the relationship
between IP performance and application performance.
</t>
<t>
Thus there is a gap in the currently chartered coverage of IETF
WGs: development of Performance Metrics for protocols above and below
the IP-layer that can be used to characterize performance on live networks.
</t>
<t>
This document refers to the implementation of a Performance Metrics Entity,
whose goal is to advice and support the Performance Metric development at the IETF.
A recommendation about the Performance Metrics Entity is made
in Section 6.6.
</t>
<t>
Similarly to the "Guidelines for Considering Operations and Management of
New Protocols and Protocol Extensions" RFC 5706 [RFC5706], which is the
reference document for the IETF Operations Directorate, this document
should be consulted as part of the new Performance Metric review.
</t>
</section>
<section title="Organization of this document">
<t>This document is divided in two major sections beyond the "Purpose and
Scope" section. The first is a definition and description of a
Performance Metric and its key aspects. The second defines a process
to develop these metrics that is applicable to the IETF
environment.</t>
</section>
</section>
<section title="Terminology">
<section title="Performance Metrics Entity">
<t>
The Performance Metrics Entity is a directorate that coordinates
the Performance Metric development in the IETF.
</t>
<t>
The Performance Metrics Entity should be composed of experts in the performance community,
potentially selected from the IPPM, BMWG, and PMOL WGs.
</t>
</section>
<section title="Quality of Service">
<t>
Quality of Service (QoS) is defined in a similar way to the ITU
"QoS experienced/perceived by customer/user (QoE)"
<xref target="E.800">E.800</xref>, i.e.:
"Totality of characteristics of a telecommunications service that bear
on its ability to satisfy stated and implied needs of the user of the service."
</t>
</section>
<section title="Quality of Experience">
<t>
Quality of Experience (QoE) is defined in a similar way to the ITU
"QoS experienced/perceived by customer/user (QoE)"
<xref target="E.800">E.800</xref>, i.e.:
"a statement expressing the level of quality that customers/users believe
they have experienced."
</t>
<t>
NOTE 1 – The level of QoS experienced and/or perceived by the customer/user
may be expressed by an opinion rating.
</t>
<t>
NOTE 2 – QoE has two main components: quantitative and
qualitative. The quantitative component can be influenced by the
complete end-to-end system effects (including user devices and
network infrastructure).
</t>
<t>
NOTE 3 – The qualitative component can be influenced by user expectations,
ambient conditions, psychological factors, application context, etc.
</t>
<t>
NOTE 4 – QoE may also be considered as QoS delivered, received, and
interpreted by a user with the pertinent qualitative factors influencing
his/her perception of the service.
</t>
</section>
<section title="Performance Metric">
covers <t>
A quantitative measure of performance, specific to an IETF-specified protocol or
specific to an application transported over an IETF-specified protocol.
Examples of Performance Metrics are: the FTP response time for a complete file
download, the DNS response time to resolve the IP address, a database logging time,
etc.
</t>
</section>
</section>
<section title="Purpose and Scope">
<t>The purpose of this document is to define a framework and a process
for developing Performance Metrics for protocols above and below the IP-layer
(such as IP-based applications that operate over reliable or datagram transport
protocols), that can be used to characterize traffic on live networks and
services. As such, this document does not define any Performance Metrics.</t>
<t>The scope of this document covers guidelines for considering new Performance
Metric development. However this document is not intended to
supercede existing working methods within WGs that have existing
chartered work in this area. </t>
<t>This process is not intended to govern Performance Metric development
in existing IETF WG that are focused on metrics development, such as
IPPM and BMWG. However, this guidelines document may be useful in
these activities, and MAY be applied where appropriate. A typical example
is the development of Performance Metrics to be exported with the IPFIX
protocol <xref target="RFC5101">RFC 5101</xref>, with specific IPFIX information
elements <xref target="RFC5102">RFC 5102</xref>, which would benefit from
the framework in this document. </t>
<t>The framework in this document applies to Performance Metrics derived from
both active and passive measurements.</t>
</section>
<section title="Relationship between QoS, QoE and Application-specific Performance Metrics">
<t> Network QoS deals with the network and network protocol performance, while QoE
deals with the assessment of a user's experience in a context of a task or a
service. As a result, the topic of application-specific Performance Metrics includes the
opportunities to quantify performance at layers between IP and the user.
For example, network QoS metrics (packet loss, delay, and delay variation
<xref target="RFC5481"></xref>) can be used to estimate application-specific
Performance Metrics (de-jitter buffer size and RTP-layer packet loss), then
combined with other known aspects of a VoIP application (such as codec type)
to estimate a Mean Opinion Score (MOS) <xref target="P.800"></xref>.
However, the QoE for a particular VoIP user depends on the specific context,
such as a casual conversation, a business conference call, or an emergency call.
Finally, QoS and application-specific Performance Metrics are quantitative, while QoE is
qualitative. Also network QoS and application-specific Performance Metrics can be
directly or indirectly evident to the user, while the QoE is directly evident.</t>
</section>
<section title="Performance Metrics Development">
<t>This section provides key definitions and qualifications of
Performance Metrics.</t>
<section title="Identifying and Categorizing the Audience">
<t>Many of the aspects of metric definition and reporting, even the
selection or determination of the essential metrics, depend on who
will use the results, and for what purpose. Some examples of how the
reports may be used include the proper maintenance of service quality
or to identify and quantify problems. The question, "How will the
results be used?" usually yields important factors to consider when
developing Performance Metrics.</t>
<t>All documents defining Performance Metrics SHOULD identify the primary
audience and its associated requirements. The audience can influence
both the definition of metrics and the methods of measurement.</t>
<t>The key areas of variation between different metric users include:</t>
<t><list style="symbols">
<t>Suitability of passive measurements of live traffic, or active
measurements using dedicated traffic</t>
<t>Measurement in laboratory environment, or on a network of deployed
devices</t>
<t>Accuracy of the results</t>
<t>Access to measurement points and configuration information</t>
<t>Measurement topology (point-to-point, point-to-multipoint)</t>
<t>Scale of the measurement system</t>
<t>Measurements conducted on-demand, or continuously</t>
<t>Required reporting formats and periods</t>
</list></t>
</section>
<section title="Definitions of a Performance Metric">
<t>A metric is a measure of an observable behavior of a networking technology,
an application, or a service. Most of the time, the metric can be
directly measured. However, sometimes, the metric definition is computed:
it assumes some implicit or explicit underlying statistical process.
In such case, the metric is an estimate of a parameter of this process,
assuming that the statistical process closely models the behavior
of the system.</t>
<t>A metric should serve some defined purpose. This may include the
measurement of capacity, quantifying how bad some problem is,
measurement of service level, problem diagnosis or location and other
such uses. A metric may also be an input to some other process, for
example the computation of a composite metric or a model or simulation
of a system. Tests of the "usefulness" of a metric include:</t>
<t><list style="empty">
<t>(i) the degree to which its absence would cause significant
loss of information on the behavior or performance of the application or
system being measured</t>
<t>(ii) the correlation between the Performance Metric, the QoS
<xref target="G.1000"></xref> and QoE delivered to the
user (person or other application)</t>
<t>(iii) the degree to which the metric is able to support the
identification and location of problems affecting service
quality.</t>
<t>(iv) the requirement to develop policies (Service Level Agreement,
and potentially Service Level Contract) based on the metric.</t>
</list>For example, consider a distributed application operating
over a network connection that is subject to packet loss. A Packet
Loss Rate (PLR) metric is defined as the mean packet loss ratio over
some time period. If the application performs poorly over network
connections with high packet loss ratio and always performs well when
the packet loss ratio is zero then the PLR metric is useful to some
degree. Some applications are sensitive to short periods of high loss
(bursty loss) and are relatively insensitive to isolated packet loss
events; for this type of application there would be very weak
correlation between PLR and application performance. A "better" metric
would consider both the packet loss ratio and the distribution of loss
events. If application performance is degraded when the PLR exceeds
some rate then a useful metric may be a measure of the duration and
frequency of periods during which the PLR exceeds that rate.</t>
</section>
<section title="Computed Metrics">
<section title="Composed Metrics">
<t>Some metrics may not be measured directly, but can be composed from
base metrics that have been measured. A composed metric is derived from
other metrics by applying a deterministic process or function (e.g.,
a composition function). The process may use metrics that are identical
to the metric being composed, or metrics that are dissimilar, or some
combination of both types. Usually the base metrics have a limited scope
in time or space, and they can be combined to estimate the performance
of some larger entities.</t>
<t>Some examples of composed metrics and composed metric definitions
are:</t>
<t>Spatial composition is defined as the composition of metrics of the
same type with differing spatial domains
<xref target="RFC5835"></xref>
<xref target="RFC6049"></xref>. For spatially
composed metrics to be meaningful, the spatial domains should be non-overlapping
and contiguous, and the composition operation should be mathematically appropriate
for the type of metric.</t>
<t>Temporal composition is defined as the composition of sets of metrics
of the same type with differing time spans <xref target="RFC5835"></xref>. For temporally
composed metrics to be meaningful, the time spans should be
non-overlapping and contiguous, and the composition operation should
be mathematically appropriate for the type of metric.</t>
<t>Temporal aggregation is a summarization of metrics into a smaller
number of metrics that relate to the total time span covered by the
original metrics. An example would be to compute the minimum,
maximum and average values of a series of time sampled values of a
metric.</t>
<t>In the context of flow records in IP Flow Informatin eXport (IPFIX), the IPFIX Mediation:
Framework <xref target="I-D.ietf-ipfix-mediators-framework"></xref> also discusses some
aspects of the temporal and spatial composition. </t>
</section>
<section title="Index">
<t>An Index is a metric for which the output value range has been
selected for convenience or clarity, and the behavior of which is
selected to support ease of understanding; for example the R Factor
<xref target="G.107"></xref>. The deterministic function for an index
is often developed after the index range and behavior have been
determined.</t>
</section>
</section>
<section title="Performance Metric Specification">
<section title="Outline">
<t>A Performance Metric definition MUST have a normative part that defines what
the metric is and how it is measured or computed and SHOULD have an
informative part that describes the Performance Metric and its application. </t>
</section>
<section title="Normative parts of Performance Metric definition">
<t>The normative part of a Performance Metric definition MUST define at least the
following: </t>
<t>(i) Metric Name</t>
<t> Performance Metric names MUST be unique within the set of metrics being defined
and MAY be descriptive.</t>
<t> (ii) Metric Description</t>
<t> The Performance Metric description MUST explain what the metric is, what is being
measured and how this relates to the performance of the system being
measured.</t>
<t> (iii) Method of Measurement or Calculation</t>
<t>
This method of measurement or calculation MUST define what is being
measured or computed and the specific algorithm to be used. Does the
measurement involve active or only passive measurements? Terms such
as "average" should be qualified (e.g. running average or average over
some interval). Exception cases SHOULD also be defined with the
appropriate handling method. For example, there are a number of commonly
used metrics related to packet loss; these often don't define the criteria
by which a packet is determined to be lost (vs very delayed) or how
duplicate packets are handled. For example, if the average packet
loss rate during a time interval is reported, and a packet's arrival
is delayed from one interval to the next then was it "lost" during
the interval during which it should have arrived or should it be
counted as received?</t>
<t>Some parameters linked to the method MAY also be reported, in
order to fully interpret the Performance Metric. For example, the
time interval, the load, the minimum packet loss, the potential measurement
errors and their sources, the attainable accuracy of the metric (e.g. +/-0,1)
etc..
</t>
<t>(iv) Units of measurement </t>
<t>The units of measurement MUST be clearly stated.</t>
<t> (v) Measurement Point(s)</t>
<t>If the measurement is specific to a measurement point, this SHOULD be
defined. The measurement domain MAY also be defined. Specifically, if
measurement points are spread across domains, the measurement domain
(intra-, inter-) is another factor to consider.</t>
<t>In some cases, the measurement requires multiple measurement points: all measurement
points SHOULD be defined, including the measurement domain(s).</t>
<t> (vi) Measurement timing</t>
<t>The acceptable range of timing intervals or sampling intervals for a
measurement and the timing accuracy required for such intervals MUST
be specified. Short sampling intervals or frequent samples provide
a rich source of information that can help to assess application
performance but may lead to excessive measurement data. Long
measurement or sampling intervals reduce the amount of reported
and collected data such that it may be insufficient to understand
application performance or service quality insofar as the measured
quantity may vary significantly with time.</t>
<t>In case of multiple measurement points, the potential requirement
for synchronized clocks must be clearly specified. In the specific example
of the IP delay variation application metric, the different aspects of synchronized
clocks are discussed in <xref target="RFC5481"></xref>.</t>
</section>
<section title="Informative parts of Performance Metric definition"></section>
<t>The informative part of a Performance Metric specification is intended to support
the implementation and use of the metric. This part SHOULD provide
the following data:</t>
<t>(i) Implementation </t>
<t>The implementation description MAY be in the form of text, algorithm
or example software. The objective of this part of the metric
definition is to assist implementers to achieve consistents results.</t>
<t>(ii) Verification</t>
<t>The Performance Metric definition SHOULD provide guidance on verification
testing. This may be in the form of test vectors, a formal
verification test method or informal advice.</t>
<t>(iii) Use and Applications</t>
<t>The use and applications description is intended to assist the "user"
to understand how, when and where the metric can be applied, and what
significance the value range for the metric may have. This MAY
include a definition of the "typical" and "abnormal" range of the
Performance Metric, if this was not apparent from the nature of the metric.
The description MAY include information about the influence of extreme
measurement values, i.e. if the Performance Metric is sensitive to
outliers. The Use and Application section SHOULD also include the
security implications in the description.
</t>
<t>For example: </t>
<t>(a) it is fairly intuitive that a lower packet loss ratio
would equate to better performance. However the user may
not know the significance of some given packet loss ratio,</t>
<t>(b) the speech level of a telephone signal is commonly expressed
in dBm0. If the user is presented with:</t>
<t>Speech level = -7 dBm0 </t>
<t>this is not intuitively understandable, unless the user is a
telephony expert. If the metric definition explains that the
typical range is -18 to -28 dBm0, a value higher than -18
means the signal may be too high (loud) and less than -28
means that the signal may be too low (quiet), it is much
easier to interpret the metric. </t>
<t>(iv) Reporting Model</t>
<t>The reporting model definition is intended to make any relationship
between the metric and the reporting model clear. There are often
implied relationships between the method of reporting metrics and the
metric itself, however these are often not made apparent to the
implementor. For example, if the metric is a short term running average
packet delay variation (e.g. <xref target="RFC3550">
RFC 3550</xref>) and this value is reported at intervals of 6-10 seconds,
the resulting measurement may have limited accuracy when packet delay variation
is non-stationary.</t>
<section title="Performance Metric Definition Template">
<t></t>
</section>
<t>Normative</t>
<list style="symbols">
<t>Metric Name</t>
<t>Metric Description</t>
<t>Method of Measurement or Calculation</t>
<t>Units of Measurement</t>
<t>Measurement Point(s) with potential Measurement Domain</t>
<t>Measurement Timing</t>
</list>
<t>Informative</t>
<list style="symbols">
<t>Implementation</t>
<t>Verification</t>
<t>Use and Applications</t>
<t>Reporting Model</t>
</list>
<section title="Example: Burst Packet Loss Frequency">
<t>The burst packet loss frequency can be observed at different
layers. The following example is specific to RTP <xref target="RFC3550">
RFC 3550</xref>.</t>
<t>Metric Name: BurstPacketLossFrequency</t>
<t>Metric Description: A burst of packet loss is defined as a longest
period starting and ending with lost packets during which no more
than Gmin consecutive packets are received. The
BurstPacketLossFrequency is defined as the number of bursts of packet
loss occurring during a specified time interval (e.g. per minute, per
hour, per day). If Gmin is set to 0 then a burst of packet loss
would comprise only consecutive lost packets, whereas a Gmin of 16
would define bursts as periods of both lost and received packets
(sparse bursts) having a loss rate of greater than 5.9%.</t>
<t>Method: Bursts may be detected using the Markov Model
algorithm defined in <xref target="RFC3611">RFC 3611</xref>. The
BurstPacketLossFrequency is calculated by counting the number of burst
events within the defined measurement interval. A burst that spans the
boundary between two time intervals shall be counted within the later of
the two intervals.</t>
<t>Units of Measurement: Bursts per time interval (e.g. per second, per
hour, per day)</t>
<t>Measurement Timing: This metric can be used over a wide range of time
intervals. Using time intervals of longer than one hour may prevent
the detection of variations in the value of this metric due to time-
of-day changes in network load. Timing intervals should not vary
in duration by more than +/- 2%.</t>
<t>Implementation Guidelines: See <xref target="RFC3611">RFC 3611</xref>.</t>
<t>Verification Testing: See Appendix for C code to generate test
vectors.</t>
<t>Use and Applications: This metric is useful to detect IP network
transients that affect the performance of applications such as
Voice over IP or IP Video. The value of Gmin may be selected to
ensure that bursts correspond to a packet loss ratio that would
degrade the performance of the application of interest (e.g. 16 for
VoIP).</t>
<t>Reporting Model: This metric needs to be associated with a defined
time interval, which could be defined by fixed intervals or by a
sliding window.</t>
</section>
</section>
<section title="Dependencies">
<section title="Timing accuracy">
<t>The accuracy of the timing of a measurement may affect the accuracy
of the Performance Metric. This may not materially affect a sampled value metric
however would affect an interval based metric. Some metrics, for
example the number of events per time interval, would be directly
affected; for example a 10% variation in time interval would
lead directly to a 10% variation in the measured value. Other
metrics, such as the average packet loss ratio during some time
interval, would be affected to a lesser extent.
</t>
<t>If it is necessary to correlate sampled values or intervals then it
is essential that the accuracy of sampling time and interval start/
stop times is sufficient for the application (for example +/- 2%).
</t>
</section>
<section title="Dependencies of Performance Metric definitions on related events or metrics">
<t>Performance Metric definitions may explicitly or implicitly rely on factors that
may not be obvious. For example, the recognition of a packet as
being "lost" relies on having some method to know the packet was
actually lost (e.g. RTP sequence number), and some time threshold
after which a non-received packet is declared as lost. It is
important that any such dependencies are recognized and incorporated
into the metric definition.
</t>
</section>
<section title="Relationship between Performance Metric and lower layer
Performance Metrics">
<t>Lower layer Performance Metrics may be used to compute or infer the performance
of higher layer applications, potentially using an application
performance model. The accuracy of this will depend on many factors
including:
</t>
<t>
(i) The completeness of the set of metrics - i.e. are there metrics
for all the input values to the application performance model?
</t>
<t>
(ii) Correlation between input variables (being measured) and
application performance
</t>
<t>
(iii) Variability in the measured metrics and how this variability
affects application performance
</t>
</section>
<section title="Middlebox presence">
<t>Presence of a middlebox <xref target="RFC3303">RFC 3303</xref>, e.g.,
proxy, network address translation (NAT), redirect server, session border
controller (SBC), and application layer gateway (ALG) may add variability
to or restrict the
scope of measurements of a metric. For example, an SBC
that does not
process RTP loopback packets may block or locally terminate
this traffic
rather then pass it through to its target.</t>
</section>
</section>
<section title="Organization of Results">
<t>The IPPM Framework [RFC2330] organizes the results of metrics into
three related notions:</t>
<t><list style="symbols">
<t>singleton, an elementary instance, or "atomic" value.</t>
<t>sample, a set of singletons with some common properties and some
varying properties.</t>
<t>statistic, a value derived from a sample through deterministic
calculation, such as the mean.</t>
</list></t>
<t>Many Performance Metrics MAY use this organization for the results, with or
without the term names used by IPPM WG. Section 11 of
<xref target="RFC2330">RFC 2330</xref> should consulted for further details.</t>
</section>
<section title="Parameters, the variables of a Performance Metric">
<t>Metrics are completely defined when all options and input variables
have been identified and considered. These variables are sometimes
left unspecified in a metric definition, and their general name
indicates that the user must set them and report them with the
results. Such variables are called "parameters" in the IPPM metric
template. The scope of the metric, the time at which it was
conducted, the settings for timers and the thresholds for counters
are all examples of parameters.</t>
<t>All documents defining Performance Metric SHOULD identify all key
parameters for each Performance Metric.</t>
</section>
</section>
<section title="Performance Metric Development Process">
<t></t>
<section title="New Proposals for Metrics">
<t>This process is intended to add additional considerations to the
processes for adopting new work as described in RFC 2026 <xref
target="RFC2026"></xref> and RFC 2418 <xref
target="RFC2418"></xref>.
The following entry criteria will be considered for each
proposal.</t>
<t>Proposals SHOULD be prepared as Internet Drafts, describing the
Performance Metric and conforming to the qualifications above as much as
possible. Proposals SHOULD be deliverables of the corresponding protocol
development WG charters. As such, the Proposals SHOULD be vetted by that
WG prior to discussion by the Performance Metrics Entity. This
aspect of the process includes an assessment of the need for the
Performance Metric proposed and assessment of the support for their
development in IETF.
</t>
<t>Proposals SHOULD include an assessment of interaction and/or overlap
with work in other Standards Development Organizations. Proposals SHOULD
identify additional expertise that might be consulted.
</t>
<t>Proposals SHOULD specify the intended audience and users of the
Performance Metrics. The development process encourages participation by members
of the intended audience.</t>
<t>Proposals SHOULD identify any security and IANA requirements.
Security issues could potentially involve revealing of user
identifying data or the potential misuse of active test tools. IANA
considerations may involve the need for a Performance Metrics registry.</t>
</section>
<section title="Reviewing Metrics">
<t>Each Performance Metric SHOULD be assessed according to the following list of
qualifications:</t>
<list style="symbols">
<t>Unambiguously defined?</t>
<t>Units of Measure Specified?</t>
<t>Measurement Interval Specified?</t>
<t>Measurement Errors Identified?</t>
<t>Repeatable?</t>
<t>Implementable?</t>
<t>Assumptions concerning underlying process?</t>
<t>Use cases?</t>
<t>Correlation with application performance/ user experience?</t>
<t>security impact?</t>
</list>
</section>
<section title="Proposal Approval">
<t>New work item proposals SHALL be approved using the existing IETF
process.</t>
<t>In all cases, the proposal will need to achieve consensus, in the
corresponding protocol development WG (or alternatively,
an "Area" WG with broad charter), that there is interest
and a need for the work.</t>
<t>The approval SHOULD include the following steps</t>
<t><list style="symbols">
<t>consultation with the Performance Metrics Entity, using this document</t>
<t>consultation with Area Director(s)</t>
<t>and possibly IESG approval of a new or revised charter for the
WG</t>
</list></t>
</section>
<section title="Performance Metrics Entity Interaction with other WGs">
<t>The Performance Metrics Entity SHALL work in partnership with the related protocol
development WG when considering an Internet Draft that specifies
Performance Metrics for a protocol. A sufficient number of individuals
with expertise must be willing to consult on the draft. If the related
WG has concluded, comments on the proposal should still be sought from
key RFC authors and former chairs, or from the WG mailing list if it
was not closed.</t>
<t>A formal review is RECOMMENDED by the time the document is reviewed by the Area
Directors, or an IETF Last Call is being conducted - same as expert reviews are being
performed by other directorates.</t>
<t>Existing mailing lists SHOULD be used, however a dedicated mailing
list MAY be initiated if necessary to facilitate work on a draft.</t>
<t>In some cases, it will be appropriate to have the IETF session
discussion during the related protocol WG session, to maximize
visibility of the effort to that WG and expand the review.</t>
</section>
<section title="Standards Track Performance Metrics">
<t>
The Performance Metrics Entity will manage the progression of RFCs along the
Standards Track. See <xref target="I-D.bradner-metricstest"></xref>.
This may include the preparation of test plans to examine different
implementations of the metrics to ensure that the metric definitions
are clear and unambiguous (depending on the final form of the draft
above).
</t>
</section>
<section title="Recommendations">
<t>
This document recommends that the Performance Metrics Entity be
implemented (according to this memo) as a directorate in one of the IETF Areas,
providing advice and support as described in this document to all areas in the IETF.
</t>
</section>
</section>
<section anchor="IANA" title="IANA Considerations">
<t>This document makes no request of IANA.</t>
<t>Note to RFC EDITOR: this section may be removed on publication as an
RFC.</t>
</section>
<section anchor="Security" title="Security Considerations">
<t>In general, the existence of framework for Performance Metric
development does not constitute a security issue for the Internet.
Performance Metric definitions may introduce security issues and this framework
recommends that those defining Performance Metrics should identify any such risk
factors.</t>
<t>The security considerations that apply to any active measurement of
live networks are relevant here. See <xref target="RFC4656"></xref>.</t>
<t>The security considerations that apply to any passive measurement of
specific packets in live networks are relevant here as well. See the security
considerations in <xref target="RFC5475"></xref>.</t>
</section>
<section anchor="Acknowledgements" title="Acknowledgements">
<t>The authors would like to thank Al Morton, Dan Romascanu, Daryl
Malas and Loki Jorgenson for their comments and contributions.
The authors would like to thank Aamer Akhter, Yaakov Stein, Carsten Schmoll,
and Jan Novak for their reviews.</t>
</section>
</middle>
<back>
<references title="Normative References">
<?rfc include="reference.RFC.2026"?>
<?rfc include="reference.RFC.2119"?>
<?rfc include="reference.RFC.2418"?>
<?rfc include='reference.RFC.4656'?>
</references>
<references title="Informative References">
<?rfc include='reference.RFC.0793'?>
<?rfc include='reference.RFC.2330'?>
<?rfc include='reference.RFC.3303'?>
<?rfc include='reference.RFC.3550'?>
<?rfc include='reference.RFC.3611'?>
<?rfc include='reference.RFC.4710'?>
<?rfc include='reference.RFC.4960'?>
<?rfc include='reference.RFC.5101'?>
<?rfc include='reference.RFC.5102'?>
<?rfc include='reference.RFC.5481'?>
<?rfc include='reference.RFC.5835'?>
<?rfc include='reference.RFC.5475'?>
<?rfc include='reference.RFC.5706'?>
<?rfc include='reference.RFC.6035'?>
<?rfc include='reference.RFC.6049'?>
<?rfc include='reference.I-D.ietf-ipfix-mediators-framework'?>
<?rfc include='reference.I-D.bradner-metricstest'?>
<reference anchor="E.800">
<front>
<title>ITU-T Recommendation E.800. SERIES E: OVERALL NETWORK
OPERATION, TELEPHONE SERVICE, SERVICE OPERATION AND HUMAN FACTORS
</title>
</front>
</reference>
<reference anchor="G.1000">
<front>
<title>ITU-T Recommendation G.1000. Communications Quality of
Service: A framework and definitions</title>
</front>
</reference>
<reference anchor="P.800">
<front>
<title>ITU-T Recommendation P.800. : Methods for subjective
determination of transmission quality
</title>
</front>
</reference>
<reference anchor="G.107">
<front>
<title>ITU-T Recommendation G.107. : The E-model, a computational model for use in transmission planning.
</title>
</front>
</reference>
</references>
</back>
</rfc>| PAFTECH AB 2003-2026 | 2026-04-23 13:25:04 |