One document matched: draft-morton-perf-metrics-framework-02.txt
Differences from draft-morton-perf-metrics-framework-01.txt
Network Working Group A. Clark
Internet-Draft Telchemy Incorporated
Intended status: Best Current February 25, 2008
Practice
Expires: August 28, 2008
Framework for Performance Metric Development
draft-morton-perf-metrics-framework-02
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on August 28, 2008.
Copyright Notice
Copyright (C) The IETF Trust (2008).
Abstract
This memo describes a framework and guidelines for the development of
performance metrics that are beyond the scope of existing working
group charters in the IETF. In this version, the memo refers to a
Performance Metrics Entity, or PM Entity, which may in future be a
working group or directorate or a combination of these two.
Clark Expires August 28, 2008 [Page 1]
Internet-Draft Perf. Metric Framework February 2008
Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Background and Motivation . . . . . . . . . . . . . . . . 3
1.2. Organization of this memo . . . . . . . . . . . . . . . . 4
2. Purpose and Scope . . . . . . . . . . . . . . . . . . . . . . 4
3. Metrics Development . . . . . . . . . . . . . . . . . . . . . 4
3.1. Audience for Metrics . . . . . . . . . . . . . . . . . . . 5
3.2. Definitions of a Metric . . . . . . . . . . . . . . . . . 5
3.3. Composed Metrics . . . . . . . . . . . . . . . . . . . . . 6
3.4. Metric Specification . . . . . . . . . . . . . . . . . . . 6
3.4.1. Outline . . . . . . . . . . . . . . . . . . . . . . . 6
3.4.2. Normative parts of metric definition . . . . . . . . . 6
3.4.3. Informative parts of metric definition . . . . . . . . 7
3.4.4. Metric Definition Template . . . . . . . . . . . . . . 7
3.4.5. Examples . . . . . . . . . . . . . . . . . . . . . . . 7
3.5. Classes of Metrics . . . . . . . . . . . . . . . . . . . . 8
3.6. Qualifying Metrics . . . . . . . . . . . . . . . . . . . . 8
3.7. Reporting Models . . . . . . . . . . . . . . . . . . . . . 8
3.8. Dependencies . . . . . . . . . . . . . . . . . . . . . . . 9
3.9. Organization of Results . . . . . . . . . . . . . . . . . 9
3.10. Parameters, the variables of a metric . . . . . . . . . . 9
3.11. Packet Measurement Details . . . . . . . . . . . . . . . . 9
3.12. Identifying and Categorizing the Audience . . . . . . . . 10
4. Performance Metric Development Process . . . . . . . . . . . . 10
4.1. New Proposals for Metrics . . . . . . . . . . . . . . . . 10
4.2. Proposal Approval . . . . . . . . . . . . . . . . . . . . 11
4.3. PM Entity Interaction with other WGs . . . . . . . . . . . 12
4.4. Standards Track Performance Metrics . . . . . . . . . . . 12
5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12
6. Security Considerations . . . . . . . . . . . . . . . . . . . 13
7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 13
8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 13
8.1. Normative References . . . . . . . . . . . . . . . . . . . 13
8.2. Informative References . . . . . . . . . . . . . . . . . . 13
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 14
Intellectual Property and Copyright Statements . . . . . . . . . . 15
Clark Expires August 28, 2008 [Page 2]
Internet-Draft Perf. Metric Framework February 2008
1. Introduction
Many applications are distributed in nature, and their performance
may be impacted by a IP impairments, server capacity, congestion and
other factors. It is important to measure the performance of
applications and services to ensure that quality objectives are being
met and to support problem diagnosis. Standardized metrics help to
ensure that performance measurement is implemented consistently and
to facilitate interpretation and comparison.
There are at least three phases in the development of performance
standards. They are:
1. Definition of a Performance Metric and its units of measure
2. Specification of a Method of Measurement
3. Specification of the Reporting Format
During the development of metrics it is often useful to define
performance objectives and expected value ranges however this is not
defined as part of the metric specification.
This memo refers to a Performance Metrics Entity, or PM Entity, which
may in future be a working group or directorate or a combination of
these two.
1.1. Background and Motivation
Although the IETF has two Working Groups dedicated to the development
of performance metrics, they each have strict limitations in their
charters:
- The Benchmarking Methodology WG has addressed a range of networking
technologies and protocols in their long history (such as IEEE 802.3,
ATM, Frame Relay, and Routing Protocols), but the charter strictly
limits their performance characterizations to the laboratory
environment.
- The IP Performance Metrics WG has the mandate to develop metrics
applicable to live IP networks, but it is specifically prohibited
from developing metrics that characterize traffic (such as a VoIP
stream).
A BOF held at IETF-69 introduced the IETF community to the
possibility of a generalized activity to define standardized
performance metrics. The existence of a growing list of Internet-
Drafts on performance metrics (with community interest in
Clark Expires August 28, 2008 [Page 3]
Internet-Draft Perf. Metric Framework February 2008
development, but in un-chartered areas) illustrates the need for
additional performance work. The majority of people present at the
BOF supported the proposition that IETF should be working in these
areas, and no one objected to any of the proposals.
The IETF does have current and completed activities related to the
reporting of application performance metrics (e.g. RAQMON) and is
also actively involved in the development of reliable transport
protocols which would affect the relationship between IP performance
and application performance.
Thus there is a gap in the currently chartered coverage of IETF WGs:
development of performance metrics for non-IP-layer protocols that
can be used to characterize performance on live networks.
1.2. Organization of this memo
This memo is divided in two major sections beyond the Purpose and
Scope section. The first is a definition and description of a
performance metric and its key aspects. The second defines a process
to develop these metrics that is applicable to the IETF environment.
2. Purpose and Scope
The purpose of this memo is to define a framework and a process for
developing performance metrics for IP-based applications that operate
over reliable or datagram transport protocols, and that can be used
to characterize traffic on live networks and services.
The scope of this memo includes the support of metric definition for
any protocol developed by the IETF, however this memo is not intended
to supercede existing working methods within WGs that have existing
chartered work in this area.
This process is not intended to govern performance metric development
in existing IETF WG that are focused on metrics development, such as
IPPM and BMWG. However, the framework and guidelines may be useful
in these activities, and MAY be applied where appropriate.
3. Metrics Development
This section provides key definitions and qualifications of
performance metrics.
Clark Expires August 28, 2008 [Page 4]
Internet-Draft Perf. Metric Framework February 2008
3.1. Audience for Metrics
Metrics are intended for use in measuring the performance of an
application, network or service. A key first step in metric
definition is to identify what metrics are needed by the "user" in
order to properly maintain service quality and to identify and
quantify problems, i.e. to consider the audience for the metrics.
3.2. Definitions of a Metric
A metric is a measure of an observable behavior of an application,
protocol or other system. The definition of a metric often assumes
some implicit or explicit underlying statistical process, and a
metric is an estimate of a parameter of this process. If the assumed
statistical process closely models the behavior of the system then
the metric is "better" in the sense that it more accurately
characterizes the state or behavior of the system.
A metric should serve some defined purpose. This may include the
measurement of capacity, quantifying how bad some problem is,
measurement of service level, problem diagnosis or location and other
such uses. A metric may also be an input to some other process, for
example the computation of a composite metric or a model or
simulation of a system. Tests of the "usefulness" of a metric
include:
(i) the degree to which its absence would cause significant loss
of information on the behavior or state of the application or
system being measured
(ii) the correlation between the metric and the quality of service
/ experience delivered to the user (person or other application)
(iii) the degree to which the metric is able to support the
identification and location of problems affecting service quality.
For example, consider a distributed application operating over a
network connection that is subject to packet loss. A Packet Loss
Rate (PLR) metric is defined as the mean packet loss rate over some
time period. If the application performs poorly over network
connections with high packet loss rate and always performs well when
the packet loss rate is zero then the PLR metric is useful to some
degree. Some applications are sensitive to short periods of high
loss (bursty loss) and are relatively insensitive to isolated packet
loss events; for this type of application there would be very weak
correlation between PLR and application performance. A "better"
metric would consider both the packet loss rate and the distribution
of loss events. If application performance is degraded when the PLR
Clark Expires August 28, 2008 [Page 5]
Internet-Draft Perf. Metric Framework February 2008
exceeds some rate then a useful metric may be a measure of the
duration and frequency of periods during which the PLR exceeds that
rate.
3.3. Composed Metrics
Some metrics may not be measured directly, but may be composed from
metrics that have been measured. Usually the contribution metrics
have a limited scope in time or space, and they can be combined to
estimate the performance of some larger entity. Some examples of
composed metrics and composed metric definitions are:
Spatial Composition
Temporal Composition
Temporal Aggregation
Spatial Aggregation
Examples.....
3.4. Metric Specification
3.4.1. Outline
A metric definition MUST have a normative part that defines what the
metric is and how it is measured or computed and SHOULD have an
informative part that describes the metric and its application.
3.4.2. Normative parts of metric definition
The normative part of a metric definition MUST define at least the
following:
(i) Metric Name
(ii) Measurement Method
This defines what is being measured, estimated or computed and the
specific algorithm to be used. Terms such as "average" should be
qualified (e.g. running average or average over some interval). It
is important to also define exception cases and how these are
handled. For example, there are a number of commonly used metrics
related to packet loss; these often don't define the criteria by
which a packet is determined to be lost (vs very delayed) or how
duplicate are handled. If the average packet loss rate during a time
interval is reported, and a packet's arrival is delayed from one
Clark Expires August 28, 2008 [Page 6]
Internet-Draft Perf. Metric Framework February 2008
interval to the next then was it "lost" during the interval during
which it should have arrived?
(iii) Units of measurement
(iv) Timing of measurement
The timing interval or sampling interval for a measurement must be
specified. Short intervals or frequent sampling provides a richer
source of information that can be helpful in assessing application
performance however can lead to excessive measurement data. Long
measurement or sampling intervals reduce that amount of reported and
collected data however may be insufficient to truly understand
application performance or service quality.
3.4.3. Informative parts of metric definition
The informative part of a metric specification is intended to support
the implementation and use of the metric. This part SHOULD provide
the following data:
(i) Description of metric The description should explain (ii) Metric
use and applications (iii) Implementation (iv) Reporting Model
3.4.4. Metric Definition Template
Name
Description
Measurement
Method
Units of measurement
Measurement or Sampling Interval
Use and Applications
Implementation
Reporting Model
3.4.5. Examples
Example definitions
Clark Expires August 28, 2008 [Page 7]
Internet-Draft Perf. Metric Framework February 2008
1 burst loss
2 protocol with a request that can be retransmitted - e.g. count how
many requests, include retransmissions?
3.5. Classes of Metrics
Simplify process by identifying classes of metric
3.6. Qualifying Metrics
Each metric SHOULD be assessed according to the following list of
qualifications:
o Unambiguously defined?
o Units of Measure Specified?
o Measurement Interval Specified?
o Measurement Errors Identified?
o Repeatable?
o Implementable?
o Assumptions concerning underlying process?
o Use cases?
o Correlation with application performance/ user experience?
(not sure that the last one is essential, it can be useful to
characterize a network attribute without linking it to application
performance/user experience)
3.7. Reporting Models
A metric, or some set of metrics, may be measured over some time
period, measured continuously, sampled, aggregated and/or combined
into composite metrics and then reported using a "push" or "pull"
model. Reporting protocols typically introduce some limitations and
assumptions with regard to the definition of a metric.
Clark Expires August 28, 2008 [Page 8]
Internet-Draft Perf. Metric Framework February 2008
3.8. Dependencies
Brian Carpenter's follow-up comment: My thought is that the variance
in the "upper" metric will be very hard to understand if there is a
large variance in the "lower" metrics. There seem likely to be non-
linear effects at work. If the upper metric is being used only as an
observation (e.g. to check SLA conformance) that is one thing, but if
it's being used to make any kind of prediction or to analyze the
behaviour of the upper layer protocol, I think describing it as
unreliable is justified.
-----------------Al inserted the sections below ---------
3.9. Organization of Results
The IPPM Framework [RFC2330] organizes the results of metrics into
three related notions:
1. singleton, an elementary instance, or "atomic" value
2. sample, a set of singletons with some common properties and some
varying properties.
3. statistic, a value derived from a sample through deterministic
calculation, such as the mean.
Many metrics can use this organization for the results, with or
without the term names used by IPPM working group. Section 11 of
[RFC2330] should consulted for further details.
3.10. Parameters, the variables of a metric
Metrics are completely defined when all options and input variables
have been identified and considered. These variables are sometimes
left unspecified in a metric definition, and their general name
indicates that the user must set them and report them with the
results. Such variables are called "parameters" in the IPPM metric
template. The scope of the metric, the time at which it was
conducted, the settings for timers and the thresholds for counters
are all examples of parameters.
All memos defining performance metrics SHOULD identify ALL key
parameters for each metric.
3.11. Packet Measurement Details
Another key aspect of the IPPM Framework [RFC2330] is the discussion
of "wire time". Internet hosts require physical resources to observe
Clark Expires August 28, 2008 [Page 9]
Internet-Draft Perf. Metric Framework February 2008
packet traffic, and they can contribute to the performance observed.
Further, the observations SHOULD be related to some instant of time
(such as a specific bit of a packet), because a packet contains many
bits and exists on physical media long enough to introduce some
ambiguity. Section 10.2 of [RFC2330] should be consulted.
3.12. Identifying and Categorizing the Audience
Many of the aspects of metric definition and reporting, even the
selection or determination of the essential metrics, depend on who
will use the results, and for what purpose. The question, "How will
the results be used?" usually yields important factors to consider
when developing performance metrics.
All memos defining performance metrics SHOULD identify the primary
audience and its associated requirements. The audience can influence
both the definition of metrics and the methods of measurement.
The key areas of variation between different metric users include:
o Suitability of passive measurements of live traffic, or active
measurements using dedicated traffic
o Measurement in laboratory environment, or on a network of deployed
devices
o Accuracy of the results
o Access to measurement points and configuration information
o Measurement topology (point-to-point, point-to-multipoint)
o Scale of the measurement system
o Measurements conducted on-demand, or continuously
o Required reporting formats
------------------- end of Al's sections, but more in 4.2 -----
4. Performance Metric Development Process
4.1. New Proposals for Metrics
The following entry criteria will be considered for each proposal.
Proposals SHOULD be prepared as Internet Drafts, describing the
Clark Expires August 28, 2008 [Page 10]
Internet-Draft Perf. Metric Framework February 2008
metrics and conforming to the qualifications above as much as
possible.
Proposals SHOULD be vetted by the corresponding protocol development
Working Group prior to discussion by the PM Entity. This aspect of
the process includes an assessment of the need for the metrics
proposed and assessment of the support for their development in IETF.
Proposals SHOULD include an assessment of interaction and/or overlap
with work in other Standards Development Organizations.
Proposals SHOULD specify the intended audience and users of the
metrics. The development process encourages participation by members
of the intended audience.
Proposals SHOULD survey the existing standards work in the area and
identify additional expertise that might be consulted, or possible
overlap with other standards development orgs.
4.2. Proposal Approval
The process depends on the form that the PM Entity ultimately takes,
Directorate or working group.
In all cases, the proposal will need to achieve consensus, in the
corresponding protocol development working group (or alternatively,
an "Area" working group with broad charter), that there is interest
and a need for the work.
IF the PM Entity is a Directorate,
THEN Approval SHOULD include the following steps
o consultation with the PM Directorate, using this framework memo
o consultation with Area Director(s)
o and possibly IESG approval of a new or revised charter for the
working group
IF the PM Entity is a Working Group, and the protocol development
working group decides to take up the work under its charter,
THEN the approval is the same as the PM Directorate steps above, with
the possible additional assignment of a PM Advisor for the work item.
IF the PM Entity is a Working Group, and the protocol development
working group decides it does not have sufficient expertise to
Clark Expires August 28, 2008 [Page 11]
Internet-Draft Perf. Metric Framework February 2008
take-up the work, or the proposal falls outside the current charter,
THEN
Approval SHOULD include the following steps
o identification of protocol expertise to support metric development
o consensus in the PM working group that there is interest and a
need for the work, and that a memo conforming to this framework
can be successfully developed
o consultation with Area Director(s)
o IESG approval of a revised charter for the PM working group
4.3. PM Entity Interaction with other WGs
The PM Entity SHALL work in partnership with the related protocol
development WG when considering an Internet Draft that specifies
performance metrics for a protocol. A sufficient number of
individuals with expertise must be willing to consult on the draft.
If the related WG has concluded, comments on the proposal should
still be sought from key RFC authors and former chairs, or from the
WG mailing list if it was not closed.
A dedicated mailing list MAY be initiated for each work area, so that
protocol experts can subscribe to and receive the message traffic
that is relevant to their work.
In some cases, it will be appropriate to have the IETF session
discussion during the related protocol WG session, to maximize
visibility of the effort to that WG and expand the review.
4.4. Standards Track Performance Metrics
The PM Entity will manage the progression of PM RFCs along the
Standards Track. See [I-D.bradner-metricstest]. This may include
the preparation of test plans to examine different implementations of
the metrics to ensure that the metric definitions are clear and
unambiguous (depending on the final form of the draft above).
5. IANA Considerations
This document makes no request of IANA.
Note to RFC Editor: this section may be removed on publication as an
Clark Expires August 28, 2008 [Page 12]
Internet-Draft Perf. Metric Framework February 2008
RFC.
6. Security Considerations
In general, the existence of framework for performance metric
development does not constitute a security issue for the Internet.
The security considerations that apply to any active measurement of
live networks are relevant here as well. See [RFC4656].
7. Acknowledgements
The authors would like to thank Al Morton
8. References
8.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
8.2. Informative References
[RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
"Framework for IP Performance Metrics", RFC 2330,
May 1998.
[RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M.
Zekauskas, "A One-way Active Measurement Protocol
(OWAMP)", RFC 4656, September 2006.
[Casner] "A Fine-Grained View of High Performance Networking, NANOG
22 Conf.; http://www.nanog.org/mtg-0105/agenda.html", May
20-22 2001.
[I-D.bradner-metricstest]
Bradner, S. and V. Paxson, "Advancement of metrics
specifications on the IETF Standards Track",
draft-bradner-metricstest-03 (work in progress),
August 2007.
[I-D.ietf-ippm-framework-compagg]
Morton, A., "Framework for Metric Composition",
draft-ietf-ippm-framework-compagg-06 (work in progress),
February 2008.
Clark Expires August 28, 2008 [Page 13]
Internet-Draft Perf. Metric Framework February 2008
[I-D.ietf-ippm-spatial-composition]
Morton, A. and E. Stephan, "Spatial Composition of
Metrics", draft-ietf-ippm-spatial-composition-05 (work in
progress), November 2007.
Author's Address
Alan Clark
Telchemy Incorporated
2905 Premiere Parkway, Suite 280
Duluth, Georgia 30097
USA
Phone:
Fax:
Email: alan.d.clark@telchemy.com
URI:
Clark Expires August 28, 2008 [Page 14]
Internet-Draft Perf. Metric Framework February 2008
Full Copyright Statement
Copyright (C) The IETF Trust (2008).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Acknowledgment
Funding for the RFC Editor function is provided by the IETF
Administrative Support Activity (IASA).
Clark Expires August 28, 2008 [Page 15]
| PAFTECH AB 2003-2026 | 2026-04-24 05:56:26 |