One document matched: draft-iesg-odell-mibtest-00.txt

Network Working Group                                        Mike O'Dell
Internet-Draft                                        UUNET Technologies
                                                            October 1997

                      MIB Interoperability Testing

                   <draft-iesg-odell-mibtest-00.txt>

1. Status of this Memo

   This document is an Internet Draft.  Internet Drafts are working
   documents of the Internet Engineering Task Force (IETF), its Areas,
   and its Working Groups.  Note that other groups may also distribute
   working documents as Internet Drafts.

   Internet Drafts are draft documents valid for a maximum of six
   months.  Internet Drafts may be updated, replaced, or obsoleted by
   other documents at any time.  It is not appropriate to use Internet
   Drafts as reference material or to cite them other than as a "working
   draft" or "work in progress".

   To learn the current status of any Internet-Draft, please check the
   1id-abstracts.txt listing contained in the Internet-Drafts Shadow
   Directories on ds.internic.net, nic.nordu.net, ftp.isi.edu, or
   munnari.oz.au.

   A revised version of this draft document will be submitted to the RFC
   editor as a Proposed Standard for the Internet Community. Discussion
   and suggestions for improvement are requested.  This document will
   expire before March, 1997. Distribution of this draft is unlimited.


2. Abstract

   This document specifies the IESG's interpretation of the Internet
   Standards Process for the progression of SNMP MIB specifications to
   the Draft Standard level of maturity.  It discusses the rationale for
   this interpretation and details the testing which is used to satisfy
   the advancement criteria.

2. Conventions used in this document

   In this document the words "MUST", "MUST NOT", "REQUIRED", "SHALL",
   "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and
   "OPTIONAL" are used in strict accordance with governing Canon Law -
   see [RFC-2119].





O'Dell 1.6                                                      [Page 1]

Internet-Draft        MIB Interoperability Testing         November 1995


3. The Nature of the Problem

   The Internet Standards Process [BCP-XYZZY] requires that for a
   protocol to advance to the Draft Standard level, two genetically
   unrelated implementations must be shown to interoperate correctly
   with all features and options. There are two distinct reasons for
   this requirement.

   The first reason is to verify that the Technical Specification is
   adequately clear and accurate.  This is demonstrated by showing
   that(at least) two different implementation efforts have worked from
   the specification and achieved interoperable implementations.

   The second reason is to discourage excessive options and "feature
   creep". This is accomplished by requiring interoperable
   implementation of all features, including options.  If an option is
   not included in at least two different interoperable implementations,
   it must be remove before the specification can advance.

   In the case of a true protocol which specifies the "bits on the wire"
   exchanged by executing state machines, the notion of
   "interoperability" is reasonably intuitive - the implementations must
   successfully "talk to each other", exchanging "bits on the wire",
   while exercising all features and options.

   In the case of an SNMP Management Information Base (MIB)
   specification, exactly what constitutes "interoperation" is less
   obvious.  It is precisely this notion of "MIB interoperability" that
   this document specifies.

   The difficulty is that there are a number of plausible
   interpretations of interoperability, all of which have merit but
   which have very different costs and difficulties in realization.

   The goal is to achieve an interpretation of interoperability and a
   testing model that strikes a balance between testing complexity and
   the competing desire to maximize the information generated by
   interoperability testing.

4. On The Nature of MIBs

   Compared to "state machine" protocols which focus on procedural
   specifications, a MIB is much more data oriented. To over-
   generalize, in a typical MIB the collection of data type and instance
   specifications outnumbers inter-object procedural or causal semantics
   by a significant amount.

   A central issue is that a MIB does not stand alone; it forms the



O'Dell 1.6                                                      [Page 2]

Internet-Draft        MIB Interoperability Testing         November 1995


   access interface to the instrumentation underneath it. Without the
   instrumentation, a MIB has form but no values. Coupled with the large
   number of objects even in a simple MIB, a MIB tends to have more of
   the look and feel of an API than a state machine protocol.

5. Some Alternatives

   The IESG debated a number of interoperability and testing models in
   formulating this specification.  The following list is incomplete but
   captures the major plausible models which developed in the course of
   that discussion.

5.1 Basic Object Comparison

   Assume the requisite two genetically unrelated implementations of the
   MIB in an SNMP agent and an SNMP management station which can do a
   "MIB Dump" (extract the complete set of MIB object types and values
   from the agent implementation).  Extract a MIB Dump from each
   implementation and compare the two to verify that both provide the
   complete set of mandatory and optional objects and that they are of
   the correct type.

5.2 Stimulus/Response Testing

   Proceed as in 5.1, but in addition, comprehensively exercise the two
   network elements containing the agent implementations to verify that
   all the MIB objects reflect plausible values in operational
   conditions.  An even stricter interpretation would require that the
   MIB objects in the two network elements track identically given the
   identical stimulus. This does good testing on "read-only" or
   "monitoring" information obtained from the underlying
   instrumentation.

5.3 Full Coverage Testing

   This extends the notion of Stimulus/Response Testing to its logical
   extreme. Given that a MIB can be viewed as an API, the software
   engineering notion of full coverage testing could be applied to a
   MIB.  This involves exercising all paths through the causal semantics
   and verifying that all objects change state correctly in all cases.

6. Discussion and Recommended Interpretation

   In a perfect world, Full Coverage or Stimulus-Response testing would
   materially increase the level of confidence in a MIB specification;
   this much we grant.  However, experience in the software engineering
   real world makes it very clear that such confidence comes at an
   extremely high price, and that even with the most exhaustive testing,



O'Dell 1.6                                                      [Page 3]

Internet-Draft        MIB Interoperability Testing         November 1995


   it is often not clear what precisely has been demonstrated. We
   believe that both of those standards of evidence are beyond what can
   be reasonably accomplished in an operational sense, and is probably
   well beyond the ability to specify in any formal semantic sense.

   Therefore, the Operations and Management Area and the IESG adopt
   Basic Object Comparison as specified in section 5.1 above as the
   minimum demonstration of interoperability for advancing a MIB to
   Draft Standard status.  This certainly does not preclude more
   aggressive testing, and the IESG reserves the right to take the
   advice of a Working Group who in strong consensus suggests that a
   particular MIB might require a stronger interoperability
   demonstration, but such advice should be quite rare.

8. Security Considerations

   Some will view this policy as possibly leading to a reduction in the
   level of confidence people can have in MIBs.  Others will view this
   policy as a countermeasure to a significant denial of service threat.
   The O&M Area and the IESG view this as a competent engineering
   trade-off of multiple competing factors.

9. References

   [RFC-2119] "Keywords for use in RFCs to Indicate Requirement Levels",
   Bradner, March 1997

   [BCP-XYZZY] "IETF Standards Process - The Standard Version", xxx,
   yyy, zzz

10. Author's Addresses

   Michael D. ODell
   UUNET Technologies, Inc.
   3060 Williams Drive
   Fairfax, VA 22031
   Email: mo@uu.net














O'Dell 1.6                                                      [Page 4]