The quality practice II: quality control, assurance and the quality register

Lesson 1 dealt with quality planning: expectations, acceptance criteria, product descriptions and the quality management approach. This lesson deals with quality control: how products are actually checked, how the results are recorded, how products are approved and accepted, and what happens when a product fails. It also looks at how the quality register gives the project manager and the board a running picture of whether quality work is keeping pace with delivery.

Quality methods

A quality method is any way of checking that a product meets its quality criteria. Quality methods fall into two broad families. In-process methods build quality in while the product is being made: specialist tools and techniques, training, automated checks, peer working, and controlled production processes. Appraisal methods check a finished or partly finished product: testing, inspection, measurement, review and audit. Most products need both. Testing a billing interface at the end will find defects, but a team that writes automated checks as it builds will produce fewer defects for the test to find.

The quality method for each product is chosen during planning and written into its product description. The choice is a trade-off. Full testing of every meter in the factory would be the most thorough but would add cost and time; sampling is cheaper but accepts a small chance that a poor batch passes. Lesson 1's quality management approach records the general policy; each product description records the specific method.

The quality review technique

Many products, especially documents and designs, cannot be tested by a machine. They are checked by people who compare them with their quality criteria. PRINCE2 has long described a structured quality review technique for this. It defines four roles. The chair runs the review and makes sure it reaches a conclusion. The presenter introduces the product and represents the people who produced it. Reviewers check the product against its quality criteria and raise questions. The administrator records actions and results. One person may hold more than one role on a small project, but the chair and presenter should usually be different people so that the person defending the product is not also deciding the outcome.

A quality review can end in one of three ways: the product is complete and ready for approval; it is complete subject to agreed actions, which reviewers will confirm; or it is incomplete and needs another review. The technique also produces useful side effects: reviewers learn about products they will later use or support, and the presenter gets structured feedback rather than a collection of conflicting emails.

Approval and acceptance

Passing a quality check is not the same as being approved, and approval of individual products is not the same as acceptance of the project's final product. Quality checks produce quality records. The people named as approvers in each product description then give formal approval, producing approval records. At the end of the project, the people named in the project product description confirm that the final product meets its acceptance criteria, producing acceptance records. Some products are accepted in stages, for example as each district goes live.

The quality register

The quality register is a summary of all the quality activities that are planned or have taken place. It does not hold the detailed results, which sit in quality records; it gives a single view of what is due, what has been done and what passed. It is usually maintained by project support, and in PRINCE2 7 it is one of the records held in the project log, alongside the daily log, issue register, risk register, lessons log and product register. The project manager uses it to see whether quality work is on schedule; project assurance uses it to check that planned checks are actually happening.

The code below holds a small extract of Ridgeline Water's quality register in week 22 of the pilot stage and reports three things: which checks are overdue, the pass rate so far, and which products are ready for approval because every one of their planned checks has passed.

today = 22  # pilot stage week
register = [
    # id, product, method, planned week, actual week or None, result
    ("Q-010", "Pilot radio network", "coverage test", 12, 12, "pass"),
    ("Q-011", "Head-end configured", "acceptance test", 11, 13, "pass"),
    ("Q-012", "Meter batch 1", "factory sample of 50", 9, 9, "pass"),
    ("Q-013", "Meter batch 2", "factory sample of 50", 15, 15, "fail"),
    ("Q-014", "Billing interface", "end-to-end test", 20, 21, "conditional"),
    ("Q-015", "Alert design", "usability test", 19, None, None),
    ("Q-016", "Customer letters", "quality review", 17, 18, "pass"),
    ("Q-017", "Meter batch 2", "retest of replacement batch", 20, None, None),
]
done = [r for r in register if r[4] is not None]
overdue = [r for r in register if r[4] is None and r[3] < today]
passed = sum(1 for r in done if r[5] == "pass")
print(f"checks done {len(done)} of {len(register)}; passed first time {passed} of {len(done)}")
for r in overdue:
    print(f"OVERDUE {r[0]} {r[1]} ({r[2]}) planned week {r[3]}")
products = {}
for r in register:
    products.setdefault(r[1], []).append(r)
ready = [p for p, rs in products.items() if all(x[5] == "pass" for x in rs[-1:])]
print("ready for approval:", ", ".join(ready))

checks done 6 of 8; passed first time 4 of 6
OVERDUE Q-015 Alert design (usability test) planned week 19
OVERDUE Q-017 Meter batch 2 (retest of replacement batch) planned week 20
ready for approval: Pilot radio network, Head-end configured, Meter batch 1, Customer letters

Six of the eight planned checks have been done and four passed first time. Two checks are overdue: the alert usability test was due in week 19 and the retest of the replacement meter batch in week 20. Four products are ready for approval: the network, the head-end, the first meter batch and the customer letters. The billing interface has a conditional result and cannot yet be approved. For the project manager, the overdue usability test is the most worrying item, because PRIN-130's schedule showed alert design with little float; for the highlight report, the first-time pass rate is a useful indicator of supplier quality.

Sampling large batches

Where products arrive in large batches, checking every item may be impractical. Sampling accepts or rejects a batch based on a sample. Its weakness is that a poor batch can still pass if the sample happens to contain few defects. The code below estimates, for Ridgeline Water's rule of testing 50 meters per batch and rejecting the batch if more than one fails, the chance of accepting batches with different true defect rates.

from math import comb
n, max_fail = 50, 1
def p_accept(rate):
    return sum(comb(n, k) * rate**k * (1 - rate)**(n - k) for k in range(max_fail + 1))
for rate in [0.005, 0.01, 0.02, 0.05, 0.08]:
    print(f"true defect rate {rate:5.1%}: chance batch is accepted {p_accept(rate):5.1%}")

true defect rate  0.5%: chance batch is accepted 97.4%
true defect rate  1.0%: chance batch is accepted 91.1%
true defect rate  2.0%: chance batch is accepted 73.6%
true defect rate  5.0%: chance batch is accepted 27.9%
true defect rate  8.0%: chance batch is accepted  8.3%

A batch with a 0.5 % defect rate is accepted about 97 % of the time, which is what the project wants. A batch with a 5 % defect rate, ten times worse, is still accepted about 28 % of the time, and one at 8 % about 8 % of the time. If a 5 % defect rate would seriously harm the daily read rate, the sampling plan is too weak and should be strengthened, for example with a larger sample, a zero-failure rule or tightened inspection after a failed batch. The point for the quality practice is that the method must match the risk; the quality management approach should state the sampling rule and the reasoning behind it.

When a product fails

A product that fails a quality check is not automatically an issue. If it can be corrected within the work package's tolerances, the team manager simply fixes it and rechecks, and the quality register shows the retest. If it cannot be corrected within tolerance, or if the customer is asked to accept it as it is, it becomes an off-specification and is handled through the issues practice. An off-specification may lead to the product being fixed, to a concession in which the customer accepts it with a known shortfall, or to an exception if tolerances are threatened. Lessons 5 and 6 cover that route.