One document matched: draft-trammell-ipfix-a8n-00.txt
IPFIX Working Group B. Trammell
Internet-Draft E. Boschi
Intended status: Experimental ETH Zurich
Expires: January 6, 2011 A. Wagner
Consecom AG
July 5, 2010
Exporting Aggregated Flow Data using the IP Flow Information Export
(IPFIX) Protocol
draft-trammell-ipfix-a8n-00.txt
Abstract
This document describes the export of aggregated Flow information
using IPFIX. An aggregated Flow is essentially an IPFIX Flow with an
externally imposed time interval, generally representing packets from
one or more original Flows. The document describes aggregated Flow
export within the framework of IPFIX Mediators, defines an
interoperable, implementation-independent method for aggregated Flow
export, covering time interval export, counter distribution and
export, and counting of original Flows.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 6, 2011.
Copyright Notice
Copyright (c) 2010 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
Trammell, et al. Expires January 6, 2011 [Page 1]
Internet-Draft IPFIX Aggregation July 2010
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Requirements for Aggregation Support in IPFIX . . . . . . . . 4
4. Aggregation of IP Flows . . . . . . . . . . . . . . . . . . . 5
4.1. A general model for IP Flow Aggregation . . . . . . . . . 5
4.2. Aggregating and Distributing Counters . . . . . . . . . . 5
4.3. Counting Original Flows . . . . . . . . . . . . . . . . . 6
5. Aggregation in the IPFIX Architecture . . . . . . . . . . . . 7
6. Export of Aggregated IP Flows using IPFIX . . . . . . . . . . 8
6.1. Flow Count Export . . . . . . . . . . . . . . . . . . . . 8
6.1.1. originalFlowsPresent Information Element . . . . . . . 9
6.1.2. originalFlowsInitiated InformationElement . . . . . . 9
6.1.3. originalFlowsCompleted InformationElement . . . . . . 9
6.1.4. originalFlows InformationElement . . . . . . . . . . . 9
6.2. Aggregate Counter Distibution Export . . . . . . . . . . . 10
6.3. Time Interval Export . . . . . . . . . . . . . . . . . . . 10
7. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
8. Security Considerations . . . . . . . . . . . . . . . . . . . 10
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 11
10.1. Normative References . . . . . . . . . . . . . . . . . . . 11
10.2. Informative References . . . . . . . . . . . . . . . . . . 11
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 12
Trammell, et al. Expires January 6, 2011 [Page 2]
Internet-Draft IPFIX Aggregation July 2010
1. Introduction
The aggregation of packet data into flows serves a variety of
different purposes, as noted in [RFC3917] and [RFC5472]. Aggregation
beyond the flow level, into records representing multiple flows, is a
common analysis and data reduction technique as well. Often these
aggregation operations result in a time series of keys and counters,
which is useful for gaining insight into trends and anomalies.
Aggregation may also has added benefits for privacy: when data about
specific transactions and hosts are aggregated together, it has an
anonymising effect on the data.
Aggregation can be applied in parallel with storage of original
packet and flow data. In certain situations, aggregation can be
performed at mediator, with aggregated data can be used for initial
analysis (e.g., to provide a reduced data stream for analysis at a
central point, with original flow data stored in a higher-access-
latency manner (e.g. tape storage at the mediator at a remote site).
Other applications may benefit from a reverse application: online
analysis of original flows, with long-term storage of aggregated
data. Aggregation can also be used to provide a lower-volume, lower-
privacy-risk source of data for reporting or data exchange across
organizational boundaries.
While IPFIX can be used to export [RFC5101] and store [RFC5655]
aggregated data, there exists as yet no common terminology or
aggregation metadata representation for this purpose, no set of
requirements to define what is meant by aggregation, and no facility
for counting original Flows from which Aggregated Flows are derived.
This document seeks to remedy this situation, defining a common basis
for the application of IPFIX to the handling of aggregated data, with
applicability to large-scale network data processing, archiving, and
inter-organization exchange.
2. Terminology
Terms used in this document that are defined in the Terminology
section of the IPFIX Protocol [RFC5101] document are to be
interpreted as defined there.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
In addition, this document defines the following terms
Trammell, et al. Expires January 6, 2011 [Page 3]
Internet-Draft IPFIX Aggregation July 2010
Aggregated Flow: A Flow, as defined by [RFC5101], derived from a
set of zero or more original Flows within a defined time interval.
The two primary differences between a Flow and an Aggregated Flow
are (1) that the time interval of a Flow is generally derived from
information about the timing of the packets comprising the Flow,
while the time interval of an Aggregated Flow are generally
externally imposed; and (2) that an Aggregated Flow may represent
zero packets (i.e., an assertion that no packets were seen for a
given Flow Key in a given time interval).
(Intermediate) Aggregation Function: A mapping from a set of zero
or more original Flows into a set of aggregated Flow, that
separates the original Flows into a set of one or more given time
intervals.
(Intermediate) Aggregation Process: An Intermediate Process, as in
[I-D.ietf-ipfix-mediators-framework], hosting an Intermediate
Aggregation Function.
[TODO: may want to define a more precise terminology for aggregated
flow time interval, "contribution"/"presence" of an original Flow
within an aggregated Flow...]
3. Requirements for Aggregation Support in IPFIX
In defining a terminology, additional information elements, and
metadata export methods for Aggregated Flow export using IPFIX, we
have sought to meet the following requirements.
First, a specification of Aggregated Flow export must seek to be as
interoperable as possible. Export of Aggregated Flows using the
techniques described in this document will result in Flow data which
can be collected by Collecting Processes and read by File Readers
which do not provide any special support for Aggregated Flow export.
Second, a specification of Aggregated Flow export must seek to be as
implementation-independent as the IPFIX protocol itself. In
Section 5, we specify the flow aggregation process as an intermediate
process within the IPFIX Mediator framework
[I-D.ietf-ipfix-mediators-framework], and specify a variety of
different architectural arrangements for flow aggregation; these are
meant to be descriptive as opposed to proscriptive. In metadata
export, we seek to define properties of the set of exported
Aggregated Flows, as opposed to the properties of the specific
algorithms used to aggregate these Flows. Specifically out of scope
for this effort are any definition of a language for proscribing
aggregation records, or the configuration parameters of Aggregation
Trammell, et al. Expires January 6, 2011 [Page 4]
Internet-Draft IPFIX Aggregation July 2010
Processes.
From the definition of presented in Section 2, an Aggregated Flow is
a Flow as in [RFC5101], with a restricted defintion as to the packets
making up the Flow. Practically speaking, Aggregated Flows are
derived from original Flows, as opposed to a raw packet stream. Key
to this definition of Aggregated Flow is how timing affects the
process of aggregation, as for the most part flow aggregation takes
place within some set of (usually regular) time intervals. Any
specification for Aggregated Flow export must account for the special
role time intervals play in aggregation, and the many-to-many
relationship between Aggregated Flows and original Flows which this
implies.
4. Aggregation of IP Flows
[TODO: frontmatter: restate definition: a flow is just a flow. Here
we talk about Aggregation Process actions.]
4.1. A general model for IP Flow Aggregation
[TODO: introduce as little of an Aggregation Process as necessary to
be able to talk about things practically. Define in simple terms the
role of externally imposed time intervals on aggregation. ]
4.2. Aggregating and Distributing Counters
In general, counters in Aggregated Flows are treated the same as in
any Flow: on a per-Information Element basis, the counters are
calculated as if they were derived from the set of packets in the
original flow. For the most part, when aggregating original Flows
into Aggregated Flows, this is simply done by summation.
However, this raises a complication when aggregating original Flows
for which original packet data is not available: often, when imposing
an external time interval on an original Flow, the original Flow will
incompletely cover one or more time intervals, and apply to one or
more Aggregated Flows; in this case, the Aggregation Process must
distribute the counters in the original Flows across the multiple
Aggregated Flows. There are several methods for doing this, listed
here in increasing order of complexity and accuracy:
End Interval: The counters for an original Flow are added to the
counters of the appropriate Aggregated Flow containing the end
time of the original Flow.
Trammell, et al. Expires January 6, 2011 [Page 5]
Internet-Draft IPFIX Aggregation July 2010
Start Interval: The counters for an original Flow are added to the
counters of the appropriate Aggregated Flow containing the start
time of the original Flow.
Mid Interval: The counters for an original Flow are added to the
counters of a single appropriate Aggregated Flow containing some
timestamp between start and end time of the original Flow.
Simple Uniform Distribution: Each counter for an original Flow is
divided by the number of time intervals the original Flow covers
(i.e., of appropriate Aggregated Flows sharing the same Flow Key),
and this number is added to each corresponding counter in each
Aggregated Flow.
Proportional Uniform Distribution: Each counter for an original
Flow is divided by the number of time _units_ the original Flow
covers, to derive a mean count rate. This mean count rate is then
multiplied by the number of time units in the intersection of the
duration of the original Flow and the time interval of each
Aggregated Flow. This is like simple uniform distribtion, but
accounts for the fractional portions of a time interval covered by
an original Flow in the first and last time interval.
Simulated Process: Each counter of the original Flow is distributed
among the intervals of the Aggregated Flows according to some
function the Aggregation Process uses based upon properties of
Flows presumed to be like the original Flow. For example, bulk
transfer flows might follow a more or less proportional uniform
distribtion, while interactive processes are far more bursty.
Direct: The Aggregating Process has access to the original packet
timings from the packets making up the original Flow, and uses
these to distribute or recalculate the counters.
A method for exporting the distribution of counters across multiple
Aggregated Flows is detailed in Section 6.2. In any case, counters
MUST be distributed across the multiple Aggregated Flows in such a
way that the total count is preserved; this property allows data to
be aggregated and re-aggregated without any loss of original count
information. To avoid confusion in interpretation of the aggregated
data, all the counters for a set of given original Flows SHOULD be
distributed via the same method.
4.3. Counting Original Flows
When aggregating multiple original Flows into an Aggregated Flow, it
is often useful to know how many original Flows are present in the
Aggregated Flow. This document introduces four new information
Trammell, et al. Expires January 6, 2011 [Page 6]
Internet-Draft IPFIX Aggregation July 2010
elements in Section 6.1 to export these counters.
There are two possible ways to count original Flows, which we call
here conservative and non-conservative. Conservative flow counting
has the property that each original Flow contributes exactly one to
the total flow count within a set of aggregated Flows. In other
words, conservative flow counters are distributed just as any other
counter, except each original Flow is assumed to have a flow count of
one. When a count for an original Flow must be distributed across a
set of Aggregated Flows, and a distribution method is used which does
not account for that original Flow completely within a single
Aggregated Flow, conservative flow counting requires a fractional
representation.
By contrast, non-conservative flow counting is used to count how many
flows are represented in an Aggregated Flow. Flow counters are not
distributed in this case. An original Flow which is present within N
Aggregated Flows would add N to the total non-conservative flow
count, one to each Aggregated Flow.
5. Aggregation in the IPFIX Architecture
[TODO: describe these diagrams]
+==========================================+
| Exporting Process |
+==========================================+
| |
| (Aggregated Flow Export) |
V |
+=============================+ |
| Mediator | |
+=============================+ |
| |
| (Aggregating Mediator) |
V V
+==========================================+
| Collecting Process |
+==========================================+
|
| (Aggregation for Storage)
V
+--------------------+
| IPFIX File Storage |
+--------------------+
Figure 1: Potential Aggregation Locations
Trammell, et al. Expires January 6, 2011 [Page 7]
Internet-Draft IPFIX Aggregation July 2010
packets --+ +- IPFIX Messages -+
| | |
V V V
+==================+ +====================+ +=============+
| Metering Process | | Collecting Process | | File Reader |
+==================+ +====================+ +=============+
| original | Flows |
V V V
+=========================================================+
| Intermediate Aggregation Process (IAP) |
+=========================================================+
| Aggregated Aggregated |
| Flows Flows |
V V
+===================+ +=============+
| Exporting Process | | File Writer |
+===================+ +=============+
| |
+------------> IPFIX Messages <----------+
Figure 2: Data flows through the aggregation process
+----+ +-----+ +----+
pkts -> | MP |->| IAP |->| EP |-> Export of Aggregated Flows
+----+ +-----+ +----+
+----+ +-----+ +----+
IPFIX -> | CP |->| IAP |->| EP |-> Aggregating Mediator
+----+ +-----+ +----+
+----+ +-----+ +----+
IPFIX -> | CP |->| IAP |->| FW |-> Aggregation for storage in IPFIX Files
+----+ +-----+ +----+
+----+ +-----+ +----+
IPFIX -> | FR |->| IAP |->| FW |-> Aggregation for analysis
File +----+ +-----+ +----+ (aggregating File manipulator)
Figure 3: Possible aggregation arrangements in the IPFIX architecture
6. Export of Aggregated IP Flows using IPFIX
[TODO: frontmatter: here we talk about export specifics.]
6.1. Flow Count Export
[TODO: describe these four IEs. for now, see Section 4.3]
Trammell, et al. Expires January 6, 2011 [Page 8]
Internet-Draft IPFIX Aggregation July 2010
6.1.1. originalFlowsPresent Information Element
Description: The non-conservative count of original Flows
containing packets from which a this Aggregated Flow was
aggregated.
Abstract Data Type: unsigned64
ElementId: TBD1
Status: Proposed
6.1.2. originalFlowsInitiated InformationElement
Description: The conservative count of original Flows whose first
packet is represented within this Aggregated Flow.
Abstract Data Type: unsigned64
ElementId: TBD2
Status: Proposed
6.1.3. originalFlowsCompleted InformationElement
Description: The conservative count of original Flows whose last
packet is represented within this Aggregated Flow.
Abstract Data Type: unsigned64
ElementId: TBD3
Status: Proposed
6.1.4. originalFlows InformationElement
Description: The conservative count of original Flows represented
within this Aggregated Flow; may be distributed via any of the
methods described in Section 4.2
Abstract Data Type: float64
ElementId: TBD4
Status: Proposed
Trammell, et al. Expires January 6, 2011 [Page 9]
Internet-Draft IPFIX Aggregation July 2010
6.2. Aggregate Counter Distibution Export
[TODO: options template and IE for representing the distribution
methods in Section 4.2]
6.3. Time Interval Export
Since an Aggregated Flow is simply a Flow, the existing timestamp
Information Elements in the IPFIX Information Model (e.g.,
flowStartMilliseconds, flowEndNanoseconds) are sufficient to specify
the time interval for aggregation. Therefore, this document
specifies no new aggregation-specific Information Elements for
exporting time interval information.
Each Aggregated Flow SHOULD contain both an interval start and
interval end timestamp. If an exporter of Aggregated Flows omits the
interval end timestamp from each Aggregated Flow, the time interval
for Aggregated Flows within an Observation Domain and Transport
Session MUST be regular and constant, and SHOULD be exported using
the Options Record described in this section. However, note that
this approach might lead to interoperability problems when exporting
Aggregated Flows to non-aggregation-aware Collecting Processes and
downstream analysis tasks; therefore, an Exporting Process capable of
exporting only interval start timestamps MUST provide a configuration
option to export interval end timestamps as well.
[TODO: options template and IE for binding a time interval to a
session.]
7. Examples
[TODO]
8. Security Considerations
[TODO]
9. IANA Considerations
[TODO: add all IEs defined in Section 6.]
10. References
Trammell, et al. Expires January 6, 2011 [Page 10]
Internet-Draft IPFIX Aggregation July 2010
10.1. Normative References
[RFC5101] Claise, B., "Specification of the IP Flow Information
Export (IPFIX) Protocol for the Exchange of IP Traffic
Flow Information", RFC 5101, January 2008.
[RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J.
Meyer, "Information Model for IP Flow Information Export",
RFC 5102, January 2008.
[RFC5610] Boschi, E., Trammell, B., Mark, L., and T. Zseby,
"Exporting Type Information for IP Flow Information Export
(IPFIX) Information Elements", RFC 5610, July 2009.
[RFC5655] Trammell, B., Boschi, E., Mark, L., Zseby, T., and A.
Wagner, "Specification of the IP Flow Information Export
(IPFIX) File Format", RFC 5655, October 2009.
[RFC3330] IANA, "Special-Use IPv4 Addresses", RFC 3330,
September 2002.
10.2. Informative References
[RFC5103] Trammell, B. and E. Boschi, "Bidirectional Flow Export
Using IP Flow Information Export (IPFIX)", RFC 5103,
January 2008.
[RFC5472] Zseby, T., Boschi, E., Brownlee, N., and B. Claise, "IP
Flow Information Export (IPFIX) Applicability", RFC 5472,
March 2009.
[RFC5470] Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek,
"Architecture for IP Flow Information Export", RFC 5470,
March 2009.
[I-D.ietf-ipfix-mediators-framework]
Kobayashi, A., Claise, B., Muenz, G., and K. Ishibashi,
"IPFIX Mediation: Framework",
draft-ietf-ipfix-mediators-framework-06 (work in
progress), April 2010.
[I-D.ietf-ipfix-mediators-problem-statement]
Kobayashi, A., Claise, B., Nishida, H., Sommer, C.,
Dressler, F., and E. Stephan, "IPFIX Mediation: Problem
Statement",
draft-ietf-ipfix-mediators-problem-statement-09 (work in
progress), March 2010.
Trammell, et al. Expires January 6, 2011 [Page 11]
Internet-Draft IPFIX Aggregation July 2010
[RFC5153] Boschi, E., Mark, L., Quittek, J., Stiemerling, M., and P.
Aitken, "IP Flow Information Export (IPFIX) Implementation
Guidelines", RFC 5153, April 2008.
[RFC3917] Quittek, J., Zseby, T., Claise, B., and S. Zander,
"Requirements for IP Flow Information Export (IPFIX)",
RFC 3917, October 2004.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
Authors' Addresses
Brian Trammell
Swiss Federal Institute of Technology Zurich
Gloriastrasse 35
8092 Zurich
Switzerland
Phone: +41 44 632 70 13
Email: trammell@tik.ee.ethz.ch
Elisa Boschi
Swiss Federal Institute of Technology Zurich
Gloriastrasse 35
8092 Zurich
Switzerland
Email: boschie@tik.ee.ethz.ch
Arno Wagner
Consecom AG
Bellariastrasse 11
8002 Zurich
Switzerland
Email: arno@wagner.name
Trammell, et al. Expires January 6, 2011 [Page 12]
| PAFTECH AB 2003-2026 | 2026-04-24 04:32:09 |