Delphi Consensus Protocol: When Is Consensus Reached?

"Consensus was reached" is the central claim of every Delphi study, and the first thing a reviewer tries to take apart. The defence is not a higher agreement percentage; it is a consensus protocol: a short, written set of rules, fixed before Round 1, that says exactly when a statement counts as agreed, when the study stops, and what happens to the statements that never get there.

This article covers what belongs in that protocol, the choices behind each rule, and a template you can paste into your study protocol or pre-registration.

What a Delphi Consensus Protocol Is

A consensus protocol is the part of your study protocol that operationalises "consensus". It usually fits on one page and answers five questions:

CREDES, the reporting guideline for Delphi studies, requires all five to be stated a priori and reported unchanged. Studies that define consensus after seeing the data are the single most common reason for a methods-section rejection.

Rule 1: The Metric

Percentage agreement is the share of panelists whose rating falls in the agreeing range, for example 7 to 9 on a 9-point scale or 4 to 5 on a 5-point scale. It is intuitive and easy to report, but it depends entirely on where you draw the agreeing range.

Interquartile range (IQR) is the spread of the middle 50 % of ratings. It does not depend on a category boundary and is treated as the more rigorous criterion in the methodological literature. An IQR of 1 or less on a 5-point scale, or 2 or less on a 9-point scale, is the common rule.

Most current protocols combine the two: a statement reaches consensus when at least X % of panelists rate within the agreeing range and the IQR is at or below Y. Using both protects you against the failure mode of each: a high percentage with a long tail of strong disagreement, or a tight IQR around a neutral midpoint.

Rule 2: The Threshold and the Scale

Across published Delphi studies the median percentage threshold is 75 %, and most sit between 70 % and 80 %. Choose by the stakes of the output:

Always state the scale with the threshold. "75 % agreement" means something different on a 4-point scale, where every rating is either agree or disagree, than on a 9-point scale with a three-point neutral zone. Write it as a sentence: "Consensus is defined as at least 75 % of panelists rating a statement 7 or higher on the 9-point scale, with an IQR of 2 or less."

Rule 3: Stability

A statement can cross the threshold in Round 2 by chance and fall back in Round 3. Stability is the check that agreement holds between consecutive rounds. The usual rule: a statement is stable when no more than 15 % of panelists (some protocols say 10 %, others 20 %) change their rating by more than one scale point between two consecutive rounds, or when the median and IQR are unchanged.

Stability is also what lets you stop a classic Delphi after two rounds with a clear conscience: if a statement is both above the threshold and stable, a third round adds fatigue, not information. In a real-time Delphi, where panelists revise continuously inside one survey, stability is observed directly as the distribution stops moving.

Rule 4: Stopping Criteria

State in advance when the study ends. The common options, often combined:

Pre-registering the maximum number of rounds is what prevents the appearance of running rounds until the numbers look right.

Rule 5: Statements Without Consensus

Not every statement will get there, and a good protocol says so. The options are to drop the statement after the maximum rounds, to revise it based on comments and re-rate it (document the change and the reason), or to report it as dissent with the distribution and the arguments on each side. The last option is increasingly expected: a documented disagreement among experts is a finding, and for guideline groups it is often the most useful one.

What is not acceptable is to lower the threshold, widen the agreeing range, or merge statements after the fact so that the final table reads as consensus. Reviewers compare the protocol to the results table, and the numbers must match.

A Protocol Template

Adapt the numbers; keep the structure.

Worked Example

A guideline panel of 24 experts rates 30 statements. After Round 1, 11 statements meet the 75 % / IQR ≤ 2 rule. They are not yet stable, because there is no previous round to compare with, so they are carried into Round 2 with the group feedback. After Round 2, 10 of the 11 are unchanged within the stability rule and are closed as consensus; one has drifted and goes to Round 3 together with 12 newly agreed statements. After Round 3, 26 statements are closed as consensus, 4 are reported as no consensus with their distributions and the panel's arguments. Nothing in the rule set changed after Round 1, and the results table says so.

Where Software Fits

The protocol is yours; the tool should apply it without you recomputing anything. Durvey stores the threshold per scale, reports every statement against it in every round, tracks convergence between rounds, and flags stable statements without consensus as dissent, so the results table and the protocol are produced from the same rules.