One document matched: draft-allman-icar-wg-revcomm-00.txt

Internet Engineering Task Force                              Mark Allman
INTERNET DRAFT                                                      ICIR
File: draft-allman-icar-wg-revcomm-00.txt                    James Kempf
                                                         DoCoMo Labs USA
                                                             April, 2004
                                                  Expires: October, 2004
    
    
                 Using Working Group Review Committees
    


Status of this Memo


    This document is an Internet-Draft and is in full conformance with
    all provisions of Section 10 of [RFC2026].


    Internet-Drafts are working documents of the Internet Engineering
    Task Force (IETF), its areas, and its working groups.  Note that
    other groups may also distribute working documents as
    Internet-Drafts.


    Internet-Drafts are draft documents valid for a maximum of six
    months and may be updated, replaced, or obsoleted by other documents
    at any time.  It is inappropriate to use Internet-Drafts as
    reference material or to cite them other than as "work in progress."


    The list of current Internet-Drafts can be accessed at
    http://www.ietf.org/ietf/1id-abstracts.txt


    The list of Internet-Draft Shadow Directories can be accessed at
    http://www.ietf.org/shadow.html.


Abstract


    This document sketches a potential quality control mechanism for the
    IETF in the form of working group review committees.  The idea is to
    form a small committee per working group that will provide document
    review and advice throughout the working group's lifetime.  


1   Introduction


    In this document we sketch a possible quality control mechanism for
    the IETF that is based on forming a small review committee for each
    working group.  While the members of this committee would have no
    special privledges within the IETF process we believe that a small
    group may aid the WG in several ways:


      + Provide architectural advice from a broader vantage point than
        the WG's focused viewpoint to help the WG understand the
        interactions of their work with the entire system.


      + Provide a view point from a different area of the IETF (e.g., a
        security viewpoint in an applications area working group).


      + Provide a place for vetting the WGs output to a broader group


Expires: October 2004                                           [Page 1]
draft-allman-icar-wg-revcomm-00.txt                           April 2004


        than the WG itself before the output is sent to the IESG.


    The review committee notion is in some ways a refinement of the
    "stuckee" notion [Har03].  While the stuckee notion is focused on
    (slightly) formalizing roles within a WG, a review committee is
    mainly focused on obtaining "outside" input (important input that
    generally comes from people who would otherwise not be deeply
    involved with the WG).
    
    The ideas outlined in this document attempt to provide a similar
    kind of quality control provided by the SIRS proposal [CC03].
    Individual SIR members are logical candidates for the review
    committee, but the committee concept goes further in that the review
    committee may draw in people who are not traditionally active in
    IETF, like implementors, academics, or people active in other fora
    who may have an interest in the work and the experience to judge it.
    An additional difference between review committees and SIRS is in
    the collection of the reviews undertaken by given individuals.  With
    a review committee a group of related WG documents will be
    considered by the same group of reviewers.  There is no such
    grouping of documents for a given reviewer in the SIRS concept.


    The notion of review committees also is an extension of the
    "technical advisor" that has traditionally been appointed to various
    IETF WGs.


    In some WGs the notion of a "review committee" is already informally
    put to use.  For instance, when documents are at a critical stage a
    WG chair will ask several people from around the IETF to read and
    comment on the draft.  This document formalizes this notion so that
    the committee is setup apriori for the WG chair(s) to utilize.


2   Forming a Review Committee    
    
    Each WG forms a review committee whose size is dictated by the WG's
    workload and breadth.  There are no formal rules about who can sit
    on a review committee.  Likewise, there is no explicit rule on the
    size of the review committee, but the size should be somewhat
    dependent on the number of work items a particular WG has (an
    initial touchstone would be 2 members per document).  The committee
    is chosen by the WG chair(s) and the area directors.  While there
    are no hard and fast rules there are some guidelines in choosing
    committee members:


      + A security guru is a likely must for groups outside the security
        area.
      + If the WG is involved in writing MIBs, a MIB doctor would be
        useful. 
      + A transport guru is probably necessary for those groups where
        the choice of transport protocol is not obvious.


    Beyond obvious choices the WG chairs and ADs should pick several
    people from different areas of the IETF to ensure the documents
    within the group received a thorough review from a number of angles.


Expires: October 2004                                           [Page 2]
draft-allman-icar-wg-revcomm-00.txt                           April 2004


    Particular attention should be paid to choosing people who are
    otherwise not engaged with the working group -- to ensure that a
    truely different perspective is gained from employing the review
    committees.  In other words, review committees should not be
    constituted of the "usual suspects" who will review a given WGs
    documents anyway.  Finally, viewpoints from outside the IETF may
    also be useful to include (e.g., from operators that do not normally
    participate but have a particularly interesting vantage point on
    some work).
    
    A committee is formed by consultation between the WG chairs and
    shepherding AD.  The WG chairs generate a list of potential members,
    and discuss with the AD about who might be appropriate. After an
    agreed upon list of members has been generated, the WG chairs
    contact the committee members to assess their willingness to serve.
    
3   Role of the Review Committee


    The review committee looks at documents as requested by the WG
    chair(s).  All reviews are sent to the WG mailing list for public
    comment and archiving.  The comments from the review committee are
    to be treated no differently than reviews from WG members or any
    other member of the community.
    
    The review committee members are not expected to closely track the
    WG if they do not have the time or inclination.  Some members of the
    committee may want to engage the WG on the mailing list with regards
    to the work and their reviews, while other members may not.  Either
    of these is fine (although, the WG chair(s) and AD(s) may consider a
    potential member's stated participation plan when forming the
    committee).  Should a committee member not directly participate on
    the mailing list or in meetings they should be available for
    questions and clarifications about their reviews to the WG chair(s).


    Review committee members contributions are acknowledged in two ways:


      + By placing their names on a Working Group's Web page.


      + Within the Contributions section of the Working Group documents
        they have reviewed.


    In order for a review committee member to qualify for
    acknowledgment, they must generate at least one detailed review of
    a working group draft.  People who agree to particate but don't
    contribute will not be listed as contributors.
    
    WG chairs are encouraged to utilize review committees as early as
    possible to review WG documents (or, even potential WG work items).
    This early vetting is designed to illuminate large issues in the
    documents before much time and effort is spent polishing some facet
    of a protocol that will ultimately be fundamentally changed.


4   Experience with Review Committees



Expires: October 2004                                           [Page 3]
draft-allman-icar-wg-revcomm-00.txt                           April 2004


    Review committees have been used in the Seamoby WG and SEND WG. In
    both cases, high quality, detailed reviews of protocol documents
    were generated by most of the review committee members. Only a few
    of the reviews were perfunctory. In the Seamboy case, two reviewers
    were selected for their extensive expertise in implementing Mobile
    IP, though they did not typically participate in the IETF. Several
    of the reviewers on the Seamoby committee followed up with
    discussion on the mailing list, even though they did not typically
    participate in the WG. 


    One issue that arose with the Seamoby review committees is that
    because the WG is not obliged to accept the review committee's
    opinion, even if the review committee members are senior and
    respected people, there is no guarantee that problems identified by
    the review committee will be addressed by the WG. This is in
    contrast to review committees for academic conferences and journals,
    where authors are typically required to address points raised during
    peer review, or their paper will not be published.


    An attempt was made to limit the demands on the review committee
    members, so reviews were only solicited at the semi-final and final
    stages of document preparation. Since review committee members may
    not be full time WG members, gauging exactly how much to ask of them
    is critical to maintaining their commitment and interest.  All in
    all, the experience with review committees in these two cases has
    been extremely positive. The amount and quality of the technical
    input obtained from the review committees far and away exceeded the
    amount of input typically generated when reviews are not directly
    solicited, such as by issuance of a WG last call.


    To be effective, reviews need to be accompanied by effective WG
    issue management. Each review should be used to generate a list of
    issues recorded, along with other issues raised by the WG, on an
    issues Web page.  Document editors, or, in the absence of strong
    editorship, the WG chairs, then become responsible for posting
    issues on the mailing list and monitoring discussion until consensus
    is achieved on the issues. The results of this discussion then need
    to be edited back into the WG documents.  Editorial issues, which
    are also raised by reviewers, typically don't need as strict
    discussion, though editors and WG chairs need to assess carefully
    whether an editorial issue will affect interpretation of the
    document, and discuss it with the WG if so.


5   A Review Committee Experiment


    We believe that the review committee concept can be accomplished
    within the given IETF procedures.  We, however, encourage WG chairs
    and area directors to form WG review committees, utilize them and
    report the results back to the community (via the ICAR working
    group).


Non-Normative References
    
    [CC03] Brian Carpenter, Dave Crocker.  Careful Additional Review of


Expires: October 2004                                           [Page 4]
draft-allman-icar-wg-revcomm-00.txt                           April 2004


        Documents (CARD) by Senior IETF Reviewers (SIRS).
        Internet-Draft draft-carpenter-icar-sirs-00.txt, March 2004.
        Work in progress.


    [Har03] Ted Hardie.  Working Groups and Their Stuckees.
        Internet-Draft draft-hardie-wg-stuckees-00.txt, February 2003.
        Work in progress.


Authors' Addresses:


    Mark Allman
    ICSI Center for Internet Research
    1947 Center Street, Suite 600
    Berkeley, CA 94704-1198
    Phone: 216-243-7361
    mallman@icir.org
    http://www.icir.org/mallman/


    James Kempf
    DoCoMo Labs USA
    180 Metro Drive, Suite 300 
    San Jose, CA 95430 
    Phone: +1 408 451 4711 
    Email: kempf@docomolabs-usa.com 
































Expires: October 2004                                           [Page 5]