One document matched: draft-morton-bmwg-virtual-net-02.xml
<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?>
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?>
<rfc category="info" docName="draft-morton-bmwg-virtual-net-02"
ipr="trust200902">
<front>
<title abbrev="Benchmarking VNFs and Related Inf.">Considerations for
Benchmarking Virtual Network Functions and Their Infrastructure</title>
<author fullname="Al Morton" initials="A." surname="Morton">
<organization>AT&T Labs</organization>
<address>
<postal>
<street>200 Laurel Avenue South</street>
<city>Middletown,</city>
<region>NJ</region>
<code>07748</code>
<country>USA</country>
</postal>
<phone>+1 732 420 1571</phone>
<facsimile>+1 732 368 1192</facsimile>
<email>acmorton@att.com</email>
<uri>http://home.comcast.net/~acmacm/</uri>
</address>
</author>
<date day="26" month="October" year="2014"/>
<abstract>
<t>Benchmarking Methodology Working Group has traditionally conducted
laboratory characterization of dedicated physical implementations of
internetworking functions. This memo investigates additional
considerations when network functions are virtualized and performed in
commodity off-the-shelf hardware.</t>
</abstract>
<note title="Requirements Language">
<t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in <xref
target="RFC2119">RFC 2119</xref>.</t>
<t/>
</note>
</front>
<middle>
<section title="Introduction">
<t>Benchmarking Methodology Working Group (BMWG) has traditionally
conducted laboratory characterization of dedicated physical
implementations of internetworking functions. The Black-box Benchmarks
of Throughput, Latency, Forwarding Rates and others have served our
industry for many years. <xref target="RFC1242"/> and <xref
target="RFC2544"/> are the cornerstones of the work.</t>
<t>An emerging set of service provider and vendor development goals is
to reduce costs while increasing flexibility of network devices, and
drastically accelerate their deployment. Network Function Virtualization
(NFV) has the promise to achieve these goals, and therefore has garnered
much attention. It now seems certain that some network functions will be
virtualized following the success of cloud computing and virtual
desktops supported by sufficient network path capacity, performance, and
widespread deployment; many of the same techniques will help achieve
NFV.</t>
<t>See http://www.etsi.org/technologies-clusters/technologies/nfv for
more background, for example, the white papers there may be a useful
starting place. The Performance and Portability Best Practices <xref
target="NFV.PER001"/> are particularly relevant to BMWG. There are
currently work-in-progress documents available in the Open Area
http://docbox.etsi.org/ISG/NFV/Open/Latest_Drafts/ including drafts
describing Infrastructure aspects and service quality.</t>
</section>
<section title="Scope">
<t>BMWG will consider the new topic of Virtual Network Functions and
related Infrastructure to ensure that common issues are recognized from
the start, using background materials from industry and SDOs (e.g.,
IETF, ETSI NFV).</t>
<t>This memo investigates additional methodological considerations
necessary when benchmarking VNF instantiated and hosted in commodity
off-the-shelf (COTS) hardware. An essential consideration is
benchmarking both physical and virtual network functions, thereby
allowing direct comparison.</t>
<t>A clearly related goal: the benchmarks for the capacity of COTS to
host a plurality of VNF instances should be investigated. Existing
networking technology benchmarks will also be considered for adaptation
to NFV and closely associated technologies.</t>
<t>A non-goal is any overlap with traditional computer benchmark
development and their specific metrics (SPECmark suites such as
SPECCPU).</t>
<t>A colossal non-goal is any form of architecture development related
to NFV and associated technologies in BMWG, as has been the case since
BMWG began work in 1989.</t>
</section>
<section title="Considerations for Hardware and Testing">
<t>This section lists the new considerations which must be addressed to
benchmark VNF(s) and their supporting infrastructure.</t>
<section title="Hardware Components">
<t>New Hardware devices will become part of the test set-up.<list
style="numbers">
<t>High volume server platforms (COTS, possibly with virtual
technology enhancements).</t>
<t>Storage systems with large capacity, high speed, and high
reliability.</t>
<t>Network Interface ports specially designed for efficient
service of many virtual NICs.</t>
<t>High capacity Ethernet Switches.</t>
</list>Labs conducting comparisons of different VNFs may be able to
use the same hardware platform over many studies, until the steady
march of innovations overtakes their capabilities (as happens with the
lab's traffic generation and testing devices today).</t>
</section>
<section title="Configuration Parameters">
<t>It will be necessary to configure and document the settings for the
entire COTS platform, including:</t>
<t><list style="symbols">
<t>number of server blades (shelf occupation)</t>
<t>CPUs</t>
<t>caches</t>
<t>storage system</t>
<t>I/O</t>
</list>as well as configurations that support the devices which host
the VNF itself: <list style="symbols">
<t>Hypervisor</t>
<t>Virtual Machine</t>
<t>Infrastructure Virtual Network</t>
</list>and finally, the VNF itself, with items such as:<list
style="symbols">
<t>specific function being implemented in VNF</t>
<t>number of VNF components in the service function chain</t>
<t>number of physical interfaces and links transited in the
service function chain</t>
</list></t>
<t/>
</section>
<section title="Testing Strategies">
<t>The concept of characterizing performance at capacity limits may
change. For example:</t>
<t><list style="numbers">
<t>It may be more representative of system capacity to
characterize the case where Virtual Machines (VM, hosting the VNF)
are operating at 50% Utilization, and therefore sharing the "real"
processing power across many VMs.</t>
<t>Another important case stems from the need for partitioning
functions. A noisy neighbor (VM hosting a VNF in an infinite loop)
would ideally be isolated and the performance of other VMs would
continue according to their specifications.</t>
<t>System errors will likely occur as transients, implying a
distribution of performance characteristics with a long tail (like
latency), leading to the need for longer-term tests of each set of
configuration and test parameters.</t>
<t>The desire for Elasticity and flexibility among network
functions will include tests where there is constant flux in the
VM instances. Requests for new VMs and Releases for VMs hosting
VNFs no longer needed would be an normal operational
condition.</t>
<t>All physical things can fail, and benchmarking efforts can also
examine recovery aided by the virtual architecture with different
approaches to resiliency.</t>
</list></t>
</section>
</section>
<section title="Benchmarking Considerations">
<t>This section discusses considerations related to Benchmarks
applicable to VNFs and their associated technologies.</t>
<section title="Comparison with Physical Network Functions">
<t>In order to compare the performance of virtual designs and
implementations with their physical counterparts, identical benchmarks
must be used. Since BMWG has developed specifications for many network
functions already, there will be re-use of existing benchmarks through
references, while allowing for the possibility of benchmark curation
during development of new methodologies. Consideration should be given
to quantifying the number of parallel VNFs required to achieve
comparable performance with a given physical device, or whether some
limit of scale was reached before the VNFs could achieve the
comparable level.</t>
</section>
<section title="Continued Emphasis on Black-Box Benchmarks">
<t>When the network functions under test are based on Open Source
code, there may be a tendency to rely on internal measurements to some
extent, especially when the externally-observable phenomena only
support an inference of internal events (such as routing protocol
convergence). However, external observations remain essential as the
basis for Benchmarks. Internal observations with fixed specification
and interpretation may be provided in parallel, to assist the
development of operations procedures when the technology is deployed,
for example. Internal metrics and measurements from Open Source
implementations may be the only direct source of performance results
in a desired dimension, but corroborating external observations are
still required to assure the integrity of measurement discipline was
maintained for all reported results.</t>
<t>A related aspect of benchmark development is where the scope
includes multiple approaches to a common function under the same
benchmark. For example, there are many ways to arrange for activation
of a network path between interface points and the activation times
can be compared if the start-to-stop activation interval has a generic
and unambiguous definition. Thus, generic benchmark definitions are
preferred over technology/protocol specific definitions where
possible.</t>
</section>
<section title="New Benchmarks">
<t>There will be new classes of benchmarks needed for network design
and assistance when developing operational practices (possibly
automated management and orchestration of deployment scale). Examples
follow in the paragraphs below, many of which are prompted by the
goals of increased elasticity and flexibility of the network
functions, along with accelerated deployment times.</t>
<t>Time to deploy VNFs: In cases where the COTS hardware is already
deployed and ready for service, it is valuable to know the response
time when a management system is tasked with "standing-up" 100's of
virtual machines and the VNFs they will host.</t>
<t>Time to migrate VNFs: In cases where a rack or shelf of hardware
must be removed from active service, it is valuable to know the
response time when a management system is tasked with "migrating" some
number of virtual machines and the VNFs they currently host to
alternate hardware that will remain in-service.</t>
<t>Time to create a virtual network in the COTS infrastructure: This
is a somewhat simplified version of existing benchmarks for
convergence time, in that the process is initiated by a request from
(centralized or distributed) control, rather than inferred from
network events (link failure). The successful response time would
remain dependent on dataplane observations to confirm that the network
is ready to perform.</t>
</section>
<section title="Assessment of Benchmark Coverage">
<t>It can be useful to organize benchmarks according to their
applicable lifecycle stage and the performance criteria they intend to
assess. The table below provides a way to organize benchmarks such
that there is a clear indication of coverage for the intersection of
lifecycle stages and performance criteria.</t>
<t><figure align="center">
<artwork><![CDATA[|----------------------------------------------------------|
| | | | |
| | SPEED | ACCURACY | RELIABILITY |
| | | | |
|----------------------------------------------------------|
| | | | |
| Activation | | | |
| | | | |
|----------------------------------------------------------|
| | | | |
| Operation | | | |
| | | | |
|----------------------------------------------------------|
| | | | |
| De-activation | | | |
| | | | |
|----------------------------------------------------------|
]]></artwork>
</figure></t>
<t>For example, the "Time to deploy VNFs" benchmark described above
would be placed in the intersection of Activation and Speed, making it
clear that there are other potential performance criteria to
benchmark, such as the "percentage of unsuccessful VM/VNF stand-ups"
in a set of 100 attempts. This example emphasizes that the Activation
and De-activation lifecycle stages are key areas for NFV and related
infrastructure, and encourage expansion beyond traditional benchmarks
for normal operation. Thus, reviewing the benchmark coverage using
this table (sometimes called the 3x3 matrix) can be a worthwhile
exercise in BMWG.</t>
<t>Comment/Discussion:</t>
<t>In one of the first applications of the 3x3 matrix on BMWG, we
discovered that metrics on measured size, capacity, or scale do not
easily match one of the three columns above. There are three
alternatives to resolve this:</t>
<t><list style="numbers">
<t>Add a column, Scaleability, but then it would be expected to
have metrics in most of the Activation, Operation, and
De-activation functions (which may not be the case).</t>
<t>Include Scalability under Reliability: This fits the user
perspective of the 3x3 matrix because the size or capacity of a
device contributes to the likelihood that a request will be
blocked, or that operation will be un-reliable when operating in
an overload state.</t>
<t>Keep size, capacity, and scale metrics separate from the 3x3
matrix, and present the results for key benchmarks in different
versions of the matrix, and the titles of each matrix provide the
details of configuration and scale.</t>
</list>Alternative 3 would address a discussion comment from
IETF-90, so it seems to cover a range of wanted features.</t>
</section>
</section>
<section title="Security Considerations">
<t>Benchmarking activities as described in this memo are limited to
technology characterization of a Device Under Test/System Under Test
(DUT/SUT) using controlled stimuli in a laboratory environment, with
dedicated address space and the constraints specified in the sections
above.</t>
<t>The benchmarking network topology will be an independent test setup
and MUST NOT be connected to devices that may forward the test traffic
into a production network, or misroute traffic to the test management
network.</t>
<t>Further, benchmarking is performed on a "black-box" basis, relying
solely on measurements observable external to the DUT/SUT.</t>
<t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
benchmarking purposes. Any implications for network security arising
from the DUT/SUT SHOULD be identical in the lab and in production
networks.</t>
</section>
<section anchor="IANA" title="IANA Considerations">
<t>No IANA Action is requested at this time.</t>
</section>
<section title="Acknowledgements">
<t>The author acknowledges an encouraging conversation on this topic
with Mukhtiar Shaikh and Ramki Krishnan in November 2013. Bhuvaneswaran
Vengainathan, Bhavani Parise, and Ilya Varlashkin have provided useful
suggestions to expand these considerations.</t>
</section>
</middle>
<back>
<references title="Normative References">
<?rfc ?>
<?rfc include="reference.RFC.2119"?>
<?rfc include="reference.RFC.2330"?>
<?rfc include='reference.RFC.2544'?>
<?rfc include="reference.RFC.2679"?>
<?rfc include='reference.RFC.2680'?>
<?rfc include='reference.RFC.3393'?>
<?rfc include='reference.RFC.3432'?>
<?rfc include='reference.RFC.2681'?>
<?rfc include='reference.RFC.5905'?>
<?rfc include='reference.RFC.4737'?>
<?rfc include='reference.RFC.5357'?>
<reference anchor="NFV.PER001">
<front>
<title>Network Function Virtualization: Performance and Portability
Best Practices</title>
<author fullname="ETSI NFV" initials="" surname="">
<organization/>
</author>
<date month="June" year="2014"/>
</front>
<seriesInfo name="Group Specification"
value="ETSI GS NFV-PER 001 V1.1.1 (2014-06)"/>
<format type="PDF"/>
</reference>
</references>
<references title="Informative References">
<?rfc include='reference.RFC.1242'?>
<?rfc include='reference.RFC.5481'?>
<?rfc include='reference.RFC.6248'?>
<?rfc include='reference.RFC.6390'?>
</references>
</back>
</rfc>
| PAFTECH AB 2003-2026 | 2026-04-24 05:57:49 |