One document matched: draft-carpenter-solution-sirs-01.txt
Differences from draft-carpenter-solution-sirs-00.txt
Network Working Group B. Carpenter
Internet Draft IBM
draft-carpenter-solution-sirs-01.txt D. Crocker
Expires: <12-03> Brandenburg
June 2003
Careful Additional Review of Documents (CARD)
by Senior IETF Reviewers (SIRS)
(draft-carpenter-solution-sirs-01.txt)
COPYRIGHT NOTICE
If published as an RFC this document will become
Copyright (C) The Internet Society (2003). All Rights
Reserved.
ABSTRACT
IETF specifications may not receive significant review,
especially beyond a single working group, until they
are submitted to the IESG. Hence, significant problems
with a specification often are not detected until a
problematic approach has been adopted and considerable
effort has been spent developing and documenting this
approach. Major changes to address problems identified
late in the process take considerable effort and
significantly delay a document that its authors
believed to be ready for publication. The procedure
described in this document is intended to solve, or
palliate, a number of related problems that have been
observed in the IETF process. The basic model is to
create a team of Senior IETF Reviewers (SIRS), and have
all documents receive a certain number of reviews by
SIRs, prior to being submitted for publication. Review
by SIRs at a very early stage is strongly encouraged.
STATUS OF THIS MEMO
This document is an Internet-Draft and is in full
conformance with all provisions of Section 10 of
RFC2026.
Internet-Drafts are working documents of the Internet
Engineering Task Force (IETF), its areas, and its
working groups. Note that other groups may also
distribute working documents as Internet Drafts.
Internet-Drafts are draft documents valid for a maximum
of six months and may be updated, replaced, or
obsoleted by other documents at any time. It is
inappropriate to use Internet- Drafts as reference
material or to cite them other than as "work in
progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/1id-abstracts.html
The list of Internet-Draft Shadow Directories can be
accessed at
http://www.ietf.org/shadow.html.
TABLE OF CONTENTS
1. Introduction
2. SIRs
2.1. The Body of Senior Internet Reviewers
2.2. Obtaining SIR Participation
3. Carding
3.1. Reviewing in Public
3.2. Spreading the load among SIRs
3.3. Form of a Review
3.4. Iterative Carding
4. Security considerations
5. Acknowledgements
6. Informative References
7. Authors' Addresses
8. Intellectual Property
9. Full Copyright Statement
1. INTRODUCTION
IETF specifications may not receive significant review,
especially beyond a single working group, until they
are submitted to the IESG or the RFC Editor. Hence,
significant problems with a specification often are not
detected until a problematic approach has been adopted
and considerable effort has been spent developing and
documenting this approach. Major changes to address
problems identified late in the process take
considerable effort and significantly delay a document
that its authors believed to be ready for publication.
We can speculate that this problem explains a large
part of the long tail in the distribution of IESG
document approval times, as well as a perception that
the IESG has too much authority.
The procedure described in this document is intended to
solve, or palliate, a number of related problems that
have been observed in the IETF process [PROBLEM]:
* perception that authority and influence in the IETF are
concentrated in too few hands
* lack of explicit quality auditing throughout the
standards development process.
* although there may be large attendances at many WG
meetings, in many cases 5% or less of the participants have
read the drafts which are under discussion or have a bearing
on the decisions to be made.
* commitments to review a document are not carried out in
a timely fashion
* little or no response is seen when a request for 'last-
call' review is issued either at WG or IETF level.
* lack of recognition of review work as a valuable
contribution
* submission of documents to the IESG that still have
significant problems (leading to overload and delay)
* failure to detect fundamental problems and Internet-
wide issues at an early stage; the IETF is not consistently
effective at resolving issues that cross WG or area
boundaries.
Particularly because of the last point, it is
impossible to resolve all these problems simply by
giving additional responsibility to working groups
themselves and relying on them for additional efforts.
An additional procedure is needed.
The procedure specified here calls for a team of 'SIRs'
to 'CARD'. The verb 'card' applies both to textiles
and public establishments. The former usage describes
the removal of detritus from textiles and their
preparation for weaving. The latter describes the
checking of participants' credentials at the entrance.
The term is also an acronym for 'Careful Additional
Review of Documents.'
The carding procedure initially makes no change to the
formal process of IETF document development, review,
approval, and publication. It is an additional
procedure intended to tackle the problems listed above.
If successful, it can be transformed into a formal
process in due course. For the moment, it is not
compulsory.
The basic model is that the IETF will create a team of
Senior IETF Reviewers (SIRs, who need be neither male
nor knighted) chosen in a way designed to create trust,
and that all documents should receive a certain number
of reviews by SIRs, prior to being submitted to the
IESG or the RFC Editor for publication. Furthermore,
review at a very early stage is strongly encouraged.
The model is not intended to replace existing
mechanisms such as the various Directorates within the
IETF and the MIB Doctor system. It will in fact build
on them.
The remainder of this document described how the team
of SIRs is created and refreshed, how the review
process works, and how it is used by document authors
and working groups to achieve their objectives.
2. SIRS
2.1. The Body of Senior Internet Reviewers
2.1.1. Initial Set of SIRs
With approximately 200 RFCs published per year, and
with most of them having existed in multiple Internet
Drafts, a large team of SIRs is needed.
An initial target of 100 people is suggested. It is
recognized that this may be insufficient, so the number
should be revised in the light of experience.
Q. Is this anything like the right number?
A. Assume we need 9 reviews per year (3 review cycles,
3 reviewers each time) for each of 200 RFCs. That gives each
SIR one review every 20 days, approximately. This seems
high, and suggests that the final target should be 200
people.
The goal is to identify a team of people with adequate
experience contributing to IETF technical work, who are
likely to be trusted as a group to be both technically
expert and unbiased.
The initial set of candidates to be SIRS is defined by
objective criteria, to avoid any bias in their
selection. It will consist of:
* all current IAB members
* all former IAB and IESG members, and former WG Chairs
* all current MIB Doctors
* all members of existing IETF Directorates
* all authors of at least three RFCs
* (other suggestions???)
Candidates must, of course, agree to participate and
agree to the terms of serving as a SIR.
Current IESG Area Directors are excluded from the pool.
Current WG Chairs are not explicitly included, but they
may qualify under other criteria.
Q: Why are all IAB members automatically included?
A: Because the IAB is in any case supposed to carry
out cross-area review. There should be no
significant extra workload for an IAB member to
act as a SIR.
Q: Why are MIB Doctors automatically included?
A: They already do the SIR job today for MIBs. There
is essentially no change for them.
Q: Why are current Directorates automatically
included?
A: Because they have been identified as experts by
Area Directors and their role normally includes
review duties already.
Q: Should all types of RFC count, or should it be
restricted to standards-track and BCP?
A: Some very valuable RFCs are Informational or
Experimental. It would seem unreasonable to
exclude their authors. On the other hand, not all
such RFCs have lasting value. Suggestions on how
to adjust this criterion are welcome.
Q: Why are ADs excluded?
A: Partly because they have to review all final
drafts anyway, partly to avoid potential conflict
of interest between reviewing and approving, and
partly because they are already very busy with
IESG work.
Q: Why are current WG Chairs not explicitly included?
A: Many of them will qualify
under other criteria. But the authors are open to
suggestions on this point - comments welcome.
2.1.2. Addition and Removal of SIRs
The team of SIRs is increased at least once each year
by a public nomination process and a voting procedure
among the existing SIRs.
Q. Why, since we have pre-qualified a team already?
A. Undoubtedly people will drop out, so new ones will
be needed. Renewal by open nomination seems the
most appropriate mechanism to ensure trust.
The nomination period will last one month, immediately
after the completion of the IESG and IAB nomination
cycle, i.e. it will start during the first IETF of each
calendar year. Anybody can nominate a SIR candidate,
and self-nominations are allowed. The list of nominees
will be public (on the IETF web site).
After the nomination period, a secret vote will be
conducted among the existing SIRs. Each SIR can vote
for none, some or all of the nominees. Nominees who
receive at least ten votes are elected.
Q: Why ten?
A: The basic criterion for being a SIR is community
agreement that one is, in fact, technically
"senior"; so someone who isn't viewed positively
by at least ten of his or her peers probably isn't
suitable. If the total number of SIRs becomes too
great, this rule would need to be revised.
SIRs who have been inactive for a full year will be
automatically retired from the team.
Q. Isn't this too tolerant? People may become SIRs
just to get it on their resume.
A. It allows for people who need to take a sabbatical
for whatever personal or professional reason. But
certainly the criterion for automatic retirement
should be reviewed when we have some experience.
In extremis, SIR status may be removed by a simple
majority vote of the team of SIRs. Such a vote will be
triggered by a recall petition from at least ten SIRs.
The nomination, voting, retirement and recall
procedures will be managed by the IETF Executive
Director. Any appeals under these procedures will be
handled by the IAB.
2.2. Obtaining SIR Participation
2.2.1. Working Group documents
For documents being processed by a Working Group (WG),
the WG solicits the assistance of SIRs. That is, the
general IETF community controls who is authorized as a
SIR, but WGs control which specific SIRs provides the
formal review that is needed for a given document.
However, SIRs may spontaneously and voluntarily review
a draft.
A primary goal of this proposal is to ensure that
Working Groups benefit from broad experience in the
design of Internet technology. Hence it is entirely
reasonable that some SIRs reviewing a given document
should be subject matter experts. However the full set
of input from SIRs is substantially more useful when it
includes SIRs from other areas. In particular, cross-
area review makes it more likely that architectural and
operational impacts outside of the subject matter will
be detected. It is therefore strongly recommended that
WGs seek a diverse set of SIRs to participate in
evaluations, able to cover most if not all IETF Areas
between them.
Each WG will make its own decision about how its SIRs
are selected (e.g. chosen by the WG Chairs, chosen by
the document authors concerned, etc.).
Q. If a SIR is active in the same WG, isn't there a
possible conflict of interest?
A. SIRs provide two benefits. One is expertise and
the other is independence. Expertise is always
helpful. However, yes, it will be important for
some of the SIR feedback to be independent of the
working group. In addition, reviewing will be a
public activity, so that any conflict will be
visible to the whole WG and can be dealt with
openly. But...
SIRs should not formally review documents in which they
have a direct interest as a contributor, or as a
contributor to a competing document.
There is no fixed number of SIR reviews required prior
to submission to the IESG. However, it is likely that
drafts with at least three positive reviews from SIRs
in different areas will experience much shorter IESG
review cycles than drafts with fewer positive reviews.
Other common sense rules will apply; for example a MIB
that has not been reviewed by a MIB Doctor is unlikely
to be published.
In all likelihood, Drafts without SIR reviews will get
worse IESG response time than today, whereas Drafts
with strongly positive SIR reviews will be processed
much more rapidly, especially as the IESG's confidence
in the SIR procedure increases. However, since the SIR
process does not change the formal document approval
process, the IESG will still be obliged to weigh Last
Call comments carefully, especially if they conflict
with the SIR reviews.
2.2.2. Individual submissions
For individual submissions, the document author(s)
should solicit SIR reviews, according to the same
principles applied to Working Group documents, and SIRs
may spontaneously and voluntarily review a draft.. When
a draft is submitted to the RFC Editor as an individual
submission, the SIR reviews are available to the RFC
Editor, and later to the IESG, to assist the
evaluation process.
Again, in all likelihood, Drafts without reviews will
get worse RFC Editor response time than today, whereas
Drafts with reviews will be processed much more
rapidly, especially as the RFC Editor's confidence in
the SIR procedure increases.
It should be clear that negative reviews do not
automatically lead to rejection and positive reviews do
not automatically lead to acceptance. The RFC Editor's
independence is intended to ensure that startling and
innovative ideas may be published if appropriate.
3. CARDING
3.1. Reviewing in Public
The current list of SIRs will be available on the IETF
web site, along with information about their areas of
expertise. All reviews will be published on the web
site, with adequate tooling (linked to
the ID Tracker) so that SIRs can publish reviews
without adminstrative overhead, every review can be
readily found, and all reviews will be automatically
archived. In fact, a WG or document author in need of
reviews should be able to request them through the web
site.
SIR reviews are not a formal part of the standards
process and do not have formal authority, so they are
not subject to formal appeal. However, since the
reviews are public, authors are able to include
rebuttals in subsequent drafts, or to issue rebuttals
on the appropriate mailing list.
SIR reviews are nevertheless contributions to the IETF
in the sense of [TECHNO] and therefore fall under the
relevant intellectual property rules of the IETF.
3.2. Spreading the load among SIRs
No doubt some SIRs will be asked for too many
reviews, and others will be asked for too few. SIRs are
expected to manage their own workload by declining to
commit to too many reviews. The mechanism for
requesting reviews should, if possible, display each
SIR's current review backlog.
3.3. Form of a Review
Each review must start with a summary statement chosen
from or adapted from the following list:
* This draft is ready for publication as a [type] RFC,
where [type] is Informational, Experimental, etc. (In some
cases, the review might recommend publication as a different
[type] than requested by the author.)
* This draft is on the right track but has open issues,
described in the review.
* This draft has serious issues, described in the review,
and needs to be rethought.
* This draft has very fundamental issues, described in
the review, and further work is not recommended.
* Unfortunately, I don't have the expertise to review
this draft.
The length of a review will vary greatly according to
circumstances, and it is acceptable for purely
editorial comments to be sent privately. All
substantive comments must be included in the public
review. Wherever possible, they should be written as
suggestions for improvement rather than as simple
criticism. Explicit references to prior work and prior
IETF discussion should be given.
SIRs should review for all kinds of problems, from
basic architectural or security issues, Internet-wide
impact, technical nits, problems of form and format
(such as IANA Considerations or incorrect references),
and editorial issues. As a draft progresses from its
initial, "-00" version towards one that is ready for
submission, successive SIR reviews should progress from
the general architectural level to the editorial level.
The intention is that before a draft is submitted by a
WG to the IESG, or by an individual to the RFC Editor,
it has already benefited from a level of review
equivalent to that traditionally applied by the IESG.
In fact, SIRs should apply generally agreed IETF
criteria, such as
[RFC1958] The Architectural Principles of the
Internet
[RFC3426] General Architectural and Policy
Considerations
[RFC3439] Some Internet Architectural Guidelines
and Philosophy
[NITS] The "ID Nits" document maintained by the
IESG
[RFC2223] Instructions to RFC Authors
[BCP26] Guidelines for Writing an IANA
Considerations Section in RFCs
[SECCONS] Guidelines for Writing RFC Text on
Security Considerations
as well as any other applicable architectural or
procedural documents. It is important that reviews give
precise references to such criteria when relevant.
Q: Should we have a complete set of guidelines for
reviewers?
A: Yes. Please start writing :-)
Q: Should a SIR engage actively in improving the
document?
A: It's not forbidden, but then s/he probably morphs
into a contributor and a new SIR will be needed.
3.4. Iterative Carding
The carding of textiles is an iterative process, and so
is the carding of documents by SIRs. It is not expected
that every version of an Internet Draft should be
submitted for SIR review. However, it is advisable to
request reviews at the very beginning (to check for
fundamental issues), as major technical issues are
resolved, and again just before the document is
submitted for IESG approval (or to the RFC Editor).
Thus three SIR review cycles per document may be
considered the minimum. The document author should
ensure that the SIRs reviewing a document understand
what stage in its life cycle it has reached, so that
they can review it appropriately.
Both Working Groups and individual submitters should
realise that carding should start early (to detect and
hopefully fix fundamental problems) and be repeated as
often as needed (to avoid submitting inadequate
documents to the IESG). By these means, it should be
possible to avoid most cases where a document spends a
long time in IESG review or, worse, is fundamentally
unacceptable to the IESG.
The feedback from carding is intended to be helpful and
beneficial to authors. Authors are encouraged to stay
with the same SIRs if possible. SIRs are encouraged to
stay with a given document as it evolves and to mentor
its author(s), especially if the latter are new to the
IETF process. Generally, SIRs are encouraged to
participate in the education of less experienced IETF
participants.
4. SECURITY CONSIDERATIONS
This document does not directly impact the operational
security of the Internet.
5. ACKNOWLEDGEMENTS
Valuable comments and ideas have come from many
sources, especially an earlier draft by Ted Hardie and
many members of the IETF 'problem' working group.
Specific thanks go to Mark Allman, Senthilkumar
Ayyasamy, Aaron Falk, Mike Heard, John Loughney, Jonne
Soininen.
Spencer Dawkins gets extra credit for having thoroughly
carded the first draft of this document and is hereby
appointed as the honorary First Sir.
6. INFORMATIVE REFERENCES
[PROBLEM] IETF Problem Statement, E. Davies (ed.),
draft-ietf-problem-issue-statement-
01.txt, work in progress.
[SECCONS] Guidelines for Writing RFC Text on
Security Considerations, E.Rescorla,
B.Korver, work in progress.
[TECHNO] Intellectual Property Rights in IETF
Technology, S.Bradner, work in progress.
[BCP26] Guidelines for Writing an IANA
Considerations Section in RFCs, T.
Narten, H. Alvestrand, RFC 2434, 1998.
[RFC1958] Architectural Principles of the
Internet, IAB, RFC 1958, 1996.
[RFC2223] Instructions to RFC Authors. J. Postel,
J. Reynolds, RFC 2223, 1997.
[RFC3426] General Architectural and Policy
Considerations, IAB, RFC 3426, 2002
[RFC3439] Some Internet Architectural Guidelines
and Philosophy, R.Bush, D.Meyer, RFC
3439, 2002
[NITS] AD Review of I-Ds, IESG,
<http://www.ietf.org/ID-nits.html>
7. AUTHORS' ADDRESSES
Brian E. Carpenter
IBM Zurich Research Laboratory
Saeumerstrasse 4
8803 Rueschlikon
Switzerland
Email: brian@hursley.ibm.com
Dave Crocker
Brandenburg InternetWorking
675 Spruce Drive
Sunnyvale, CA 94086 USA
Tel: +1.408.246.8253
dcrocker@brandenburg.com
8. INTELLECTUAL PROPERTY
PLACEHOLDER for full IETF IPR Statement if needed.
9. FULL COPYRIGHT STATEMENT
Copyright (C) The Internet Society (2003). All Rights
Reserved.
This document and translations of it may be copied and
furnished to others, and derivative works that comment
on or otherwise explain it or assist in its
implementation may be prepared, copied, published and
distributed, in whole or in part, without restriction
of any kind, provided that the above copyright notice
and this paragraph are included on all such copies and
derivative works. However, this document itself may
not be modified in any way, such as by removing the
copyright notice or references to the Internet Society
or other Internet organizations, except as needed for
the purpose of developing Internet standards in which
case the procedures for copyrights defined in the
Internet Standards process must be followed, or as
required to translate it into languages other than
English.
The limited permissions granted above are perpetual and
will not be revoked by the Internet Society or its
successors or assigns.
This document and the information contained herein is
provided on an "AS IS" basis and THE INTERNET SOCIETY
AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL
WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT
LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE.
| PAFTECH AB 2003-2026 | 2026-04-22 21:49:59 |