Sampling-Review Mini-Howto

David S. Lawyer

v0.0, November 2006

The Linux Documentation Project (LDP) requires that all new documents (docs) from volunteer authors be reviewed for quality. Normally, a reviewer will read over the entire document, but this may not be feasible if there is a shortage of reviewers. So to get new submissions rapidly published on the LDP websites, we sometimes use the fast sampling review method. The reviewer just takes some samples of the writing and if they pass, the document is provisionally accepted. But a full review at a later date might possibly result in rejection of the document.

For a sampling review, the reviewer scans over the document and selects some sections of it to read over. This selection could be at random, but it may be based on selecting those parts where the reviewer knows the subject best so that any technical errors will be easily caught. It's fast to just select the first sample of perhaps 1% of a long doc or say 4% of a short doc and read it over thoroughly. If needed, check the contents against reference material (printed books or stuff you trust on the Internet). Then repeat for the second sample, etc. The samples should be spread out over the doc since an author may have written one chapter well but neglected others.

After the reviewer has read over say about 10% of the entire doc this way it's time for a 3-way decision: accept, reject, or continue sampling some more. If everything looks good and there are almost no errors, then accept the doc for provisional publication. If there are too many errors (including lack of clarity or poor grammar) then the doc is rejected and returned to the author for improvement. But if the situation is somewhere in between these two extremes (accept, reject) then continue with some more sampling until it becomes clear whether or not to accept or reject the doc.

When a doc fails a sampling review, some authors have just fixed the errors found in the sample and resubmitted the doc again. This is not permitted. The author must carefully rework the whole doc before resubmitting since the sampling has indicated (with high probability) that the rest of the document needs a lot of work too.

The reviewer should read over the Author-Mini-Howto so as to know what we expect of authors. The reviewer should write up a brief report of the review and send it to the author, with a CC going to the review coordinator. If there are too many errors, it's OK just to say in the report that you found additional errors, etc. without giving all the details.