One document matched: draft-ietf-icar-experiment-early-review-00.txt
ICAR D. Partain
Internet-Draft Ericsson
Expires: November 30, 2004 June 2004
An Experiment in Early Cross-Area Review for the IETF
draft-ietf-icar-experiment-early-review-00.txt
Status of this Memo
By submitting this Internet-Draft, I certify that any applicable
patent or other IPR claims of which I am aware have been disclosed,
and any of which I become aware will be disclosed, in accordance with
RFC 3668.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that other
groups may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at http://
www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on November 30, 2004.
Copyright Notice
Copyright (C) The Internet Society (2004). All Rights Reserved.
Abstract
This memo captures current working group rough consensus on
procedures for improved cross-area early review in the IETF. It is a
product of the ICAR working group.
This memo describes an experiment to improve early cross-area review
in the IETF. Early cross-area review aims to improve quality of IETF
work by identifying problems early in document development. Improved
quality may, in turn, mean that documents can be processed faster in
the IESG.
Partain Expires November 30, 2004 [Page 1]
Internet-Draft ICAR Experiment June 2004
Table of Contents
1. Introduction to ICAR . . . . . . . . . . . . . . . . . . . . . 3
2. Overview of this memo . . . . . . . . . . . . . . . . . . . . 3
3. Prerequisites to the ICAR early review process . . . . . . . . 4
4. ICAR review pool . . . . . . . . . . . . . . . . . . . . . . . 4
4.1 Makeup of the review pool . . . . . . . . . . . . . . . . 4
4.1.1 Seeding the initial reviewer pool . . . . . . . . . . 5
4.1.2 Objective criteria for entering the review pool . . . 5
4.1.3 Subjective criteria for entering the review pool . . . 5
5. Requesting a review . . . . . . . . . . . . . . . . . . . . . 5
5.1 Who requests a review . . . . . . . . . . . . . . . . . . 5
5.2 Available information about reviewers . . . . . . . . . . 6
5.3 How the review is requested . . . . . . . . . . . . . . . 6
5.4 Review announcements . . . . . . . . . . . . . . . . . . . 7
6. Results of a review . . . . . . . . . . . . . . . . . . . . . 7
6.1 Openly published and archived . . . . . . . . . . . . . . 7
6.2 Review is not binding on the WG . . . . . . . . . . . . . 7
6.3 WG's response to the review . . . . . . . . . . . . . . . 7
7. An initial experiment . . . . . . . . . . . . . . . . . . . . 8
8. Open issues . . . . . . . . . . . . . . . . . . . . . . . . . 8
9. Security Considerations . . . . . . . . . . . . . . . . . . . 9
Author's Address . . . . . . . . . . . . . . . . . . . . . . . 9
Intellectual Property and Copyright Statements . . . . . . . . 10
Partain Expires November 30, 2004 [Page 2]
Internet-Draft ICAR Experiment June 2004
1. Introduction to ICAR
Improved cross-area review (ICAR) aims to improve the overall quality
of the work produced by the IETF's working groups and to increase the
speed of that work's progression through the standardization process.
This memo is an attempt to capture in a coherent form the rough
consensus from the working group as well as the state of the on-going
debate.
Cross-area review is not a new idea in the IETF. Most particularly,
the IESG has carried out this kind of review when documents have been
put on their table. The IETF as a whole could function as a large
review body and exactly this kind of review is theoretically
solicited during a document's IETF Last Call.
However, current review practices appear to have significant
problems. Little or no feedback is the norm during an IETF Last
Call. Waiting until the IESG review is deemed to be too late to
correct significant problems with a document.
ICAR proposes mechanisms by which early cross-area review is
solicited, the review is carried out within an IETF-wide pool of
reviewers, and the review itself as well as the working group's
response to the review are publicly documented.
The goal of the ICAR early review process is to ensure that reviews
actually happen and the results are adequately considered by the
working group.
2. Overview of this memo
This document begins by outlining the necessary prerequisites for a
successful ICAR early review process in Section 3.
Thereafter, Section 4 is a discussion of the makeup of the ICAR
review pool, how it is constituted, how reviewers are added, and how
reviewers are removed from the pool.
Section 5 then discusses who requests a review and how they do so.
Section 6 discusses how the results are handled and how the working
group requesting the review is expected to respond to the review.
In order to prove the value of ICAR reviews, the ICAR working group
aims to start a small-scale experiment, which is outlined in Section
7.
Partain Expires November 30, 2004 [Page 3]
Internet-Draft ICAR Experiment June 2004
Finally, Section 8 points out known open issues (to the editor...)
that have yet to be resolved.
3. Prerequisites to the ICAR early review process
In order for the ICAR early review process to be effective, the
following need to be in place.
1. An open mailing list will be created on which requests for ICAR
reviews will be posted. This will not be the mechanism by which
the reviews are requested but is instead the way that interested
parties and members of the IETF community who are not official
reviewers can find out what reviews are needed. Quite apart from
the ICAR process, any member of the IETF community may, of
course, review a WG document at any time s/he wishes and provide
that input to the WG.
2. A pool of volunteers needs to be available for reviews. This is
discussed in Section 4.
3. The processes for handling review requests and results need to be
ironed out (the goal of this memo).
4. There need to be tools available for managing the review flow.
This includes information about reviewers (rough availability,
review history, areas of expertise, etc.). The current tools at
publication time are available at http://www.machshav.com/~icar/
reviews/ but are rough and are expected to evolve with the
experiment.
5. The open issues listed in Section 8 should be resolved.
4. ICAR review pool
A pool of volunteers who can review documents is needed. The WG
considered the alternatives of one pool covering all IETF work and
multiple pools, each covering an area of the IETF. The WG consensus
is to have an IETF-wide review pool.
4.1 Makeup of the review pool
The initial pool of reviewers when starting the ICAR early review
process will be constructed as detailed in the following two
subsections.
The consensus of the ICAR working group is that there needs to be
some form of admission control to ensure that those in the review
pool have sufficient technical expertise to provide high-quality
reviews as well as an adequate understanding of the way that the IETF
works. The criteria listed below are tools to ensure that both of
these are true.
Partain Expires November 30, 2004 [Page 4]
Internet-Draft ICAR Experiment June 2004
4.1.1 Seeding the initial reviewer pool
In order to seed the review pool, ADs will suggest 5-6 people from
their area whom the ADs will "sponsor" and who will commit to
engaging in this activity. This would form the minimum pool that is
needed to get ICAR reviews underway.
In addition, there will be an open call for reviewers to join the
pool. Those who volunteer will need to meet one of the objective or
subjective criteria listed in the following subsections.
4.1.2 Objective criteria for entering the review pool
The following criteria will be used as one way to determine
eligibility for the review pool (and is based on the SIRS experiment
- editor: add reference). The criteria listed in the bullets below
will apply both when the initial review pool is constructed and on a
continuing basis for those who wish to be added later.
o people who have authored 2 IETF RFCs approved for publication by
the IESG, which includes non-WG documents submitted directly to
the IESG, but not documents submitted directly to the RFC editor
o current/former IAB members
o former IESG members
o former WG chairs
o current WG chairs whose WG has produced 2 RFCs
o MIB doctors
o directorate members
4.1.3 Subjective criteria for entering the review pool
There is some push within the ICAR working group to include a more
subjective method of getting into the review pool. However, there is
not yet consensus as to how this would happen in practice. This
section is simply a placeholder for the practice that the WG agrees
to.
5. Requesting a review
5.1 Who requests a review
In general, it is up to the working group in question to request an
ICAR review. However, an AD may also request a review.
ICAR reviews for working group documents are requested by a WG chair
/ secretary. If a document falls into two or more working group's
charters, the working group chairs are expected to coordinate their
calls for an ICAR review.
Partain Expires November 30, 2004 [Page 5]
Internet-Draft ICAR Experiment June 2004
A working group may request a review for any document it deems
relevant to its work. This may be official working group drafts, but
it may also include documents named as personal drafts but which are
functioning as working group drafts or are under consideration to be
WG items.
An AD may request an ICAR review, which would typically be when a
document does not clearly fall into a specific WG's domain.
Those requesting a review are expected to request reviews from people
with different areas of expertise.
5.2 Available information about reviewers
The following information is available about reviewers in the pool:
o Contact information
o Area(s) of expertise
o Some approximation of availability
o Archive of previous reviews
o Total number of pending requests to reviewer
5.3 How the review is requested
It is up to the WG chair(s) / secretary to choose the reviewers who
review the document. In order to do this, reviewers in the pool must
be clearly labeled with their area(s) of expertise they feel
qualified to provide .
The ICAR WG does not believe that a broadcast request for reviews
works. As such, when the requester decides to request an ICAR
review, s/he will look through the list of reviewers and the
information about them (see Section 5.2) and determine which
reviewers are available and have the appropriate technical expertise.
It is then the responsibility of the requester to make contact with
these individuals to request a review. If the reviewer cannot do the
review, the requester will have to find an alternative reviewer.
When a requester and a reviewer agree on doing a review then the
"ICAR infrastructure" should be notified (see Section 3 on tools for
the ICAR experiment) so that information about the review and the
review itself can be updated.
Editor: Some in the WG have suggested that a management function is
needed to shepherd the early review process. There is not yet any
consensus in that regard.
Partain Expires November 30, 2004 [Page 6]
Internet-Draft ICAR Experiment June 2004
5.4 Review announcements
In parallel to the closed ICAR reviewer pool, requests for reviews
will be sent to an announce list, allowing the community to track
which documents are actively being reviewed and encouraging people
outside the ICAR reviewer pool to submit their own reviews of those
documents.
In addition, the web infrastructure set up to aid the ICAR early
review process will include information about which documents are
currently under review.
6. Results of a review
6.1 Openly published and archived
All ICAR reviews should be openly published (on the WG list at a
minimum).
All ICAR reviews should be archived.
Reviews by individuals in response to ICAR review request
announcements are not archived in the ICAR tools infrastructure.
They are assumed to be archived in the normal WG mailing list
archives.
6.2 Review is not binding on the WG
The ICAR reviewers do not have any kind of veto power over the
working groups that request reviews. Reviewers can in no way force
working groups to follow their advice. That is, the reviews are not
binding on WGs.
6.3 WG's response to the review
Although reviews are not binding, WGs are nonetheless obligated to
consider and discuss each suggestion from the review. Not only must
it be discussed, but there must also be adequate discussion
documented to demonstrate why a working group chose not to implement
a particular change requested by the reviewer.
Like the review, the WG's response to the review should be archived.
This is important, because the reasons for a WG's decision to
disagree with a recommendation from a reviewer can be important input
to the IESG when the document is submitted to the IESG for
consideration.
Partain Expires November 30, 2004 [Page 7]
Internet-Draft ICAR Experiment June 2004
At least during the initial ICAR experiment, it would be useful for
the WG to note explicitly in documents that they have undergone an
ICAR review. For example, this could be noted in the "Status of this
Memo" section or in the abstract.
The ICAR process has no way of influencing how non-ICAR reviews are
handled by WGs and cannot force WGs to respond in a similar fashion
to these reviews. However, the IETF spirit is that all comments
should be addressed in an appropriate manner. Documentation of the
WG's response to such reviews is of course desirable.
7. An initial experiment
The ICAR WG intends to recommend that an "experiment" be carried out
with a group of willing participating reviewers and working groups.
This section outlines the steps in carrying out that initial
experiment.
o Ask the ADs to choose one or two WGs per area whose chairs are
amenable to using an ICAR review pool and to helping the ICAR
working group gauge how the experiment is working.
o Have the WGs in the WG pool solicit reviews on their current
documents, any new documents, etc. Of course, there will be a
mechanism by which we ensure that the initial burst will not be
overwhelming. Editor: two per week max?
o As we gain experience we can try to add more groups.
o WGs from outside the initial WG pool may request reviews if they
are so inclined. However, these WGs will not be expected to make
such requests, whereas the WGs participating in the experiment
will be expected to use the ICAR review-pool.
8. Open issues
Editor: This list is composed of issues that seem to still be open
in the working group. This list needs to be confirmed within the
working group, and, if necessary, addressed in this document.
1. What guidelines should we set, if any, for the form of the
review? Can we say that reviewers should provide something like
a numbered list of concise comments and not a rambling stream?
This facilitates solid responses from the WG and nicely separates
issues that will inevitably be dealt with differently.
2. Do we need addition information available about potential
reviewers as discussed in Section 5.2? For example, do we want
history of number of requests to this reviewer?
3. What are the subjective criteria to be used for admittance to the
pool (to be put into Section 4.1.3)?
Partain Expires November 30, 2004 [Page 8]
Internet-Draft ICAR Experiment June 2004
4. How are people removed from the pool or is such a process needed
at all?
5. How does one measure the success/failure of ICAR experiments?
6. What level of involvement in the ICAR early review process can we
expect / hope for from the IESG? In what way should the
reviewers, prospective reviewers, and working groups interact
with the IESG?
7. Do we want to stipulate the form in which a review and the WG's
response to it are documented?
9. Security Considerations
None.
Author's Address
David Partain
Ericsson AB
P.O. Box 1248
SE-581 12 Linkoping,
Sweden
EMail: david.partain@ericsson.com
Partain Expires November 30, 2004 [Page 9]
Internet-Draft ICAR Experiment June 2004
Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the IETF's procedures with respect to rights in IETF Documents can
be found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Disclaimer of Validity
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Copyright Statement
Copyright (C) The Internet Society (2004). This document is subject
to the rights, licenses and restrictions contained in BCP 78, and
except as set forth therein, the authors retain all their rights.
Acknowledgment
Funding for the RFC Editor function is currently provided by the
Internet Society.
Partain Expires November 30, 2004 [Page 10]
| PAFTECH AB 2003-2026 | 2026-04-24 03:10:41 |