Delphi MethodResearch MethodsBest PracticesAcademic Writing

Delphi Questionnaire Design: How to Write Items That Work

•4 min read

Item quality is what separates a Delphi study that produces interpretable consensus from one that produces agreement on statements nobody actually disagreed with. Most Delphi questionnaire problems aren't visible during design — they only appear in Round 1 results, when you realise panelists were interpreting your items in three different ways. This guide covers how to avoid the most common design mistakes before they cost you a round.

Choosing a Rating Scale

The most common rating format in Delphi studies is a Likert-type scale. The two most widely used options are:

  • 5-point scale: 1 = strongly disagree to 5 = strongly agree. Simpler to complete, used more in education and social sciences research
  • 9-point scale: 1 = extremely inappropriate to 9 = extremely appropriate (or equivalent). Used widely in clinical research, particularly in RAND/UCLA-method studies, because the wider range makes IQR-based consensus calculation more meaningful
  • For most academic thesis work, a 5-point or 7-point scale is sufficient. Use a 9-point scale if your study follows RAND/UCLA methodology or if the reporting framework you've chosen recommends it.

    Open vs. Closed Round 1

    If your item pool isn't already well-defined by the literature, consider a semi-open Round 1: present a preliminary list of items derived from literature and ask panelists to rate them, but also include open-ended fields where panelists can suggest additional items or reword existing ones. This hybrid approach gives you a starting structure while surfacing important items you may have missed.

    A fully open Round 1 (no items, just open questions) generates richer initial content but requires significantly more between-round analysis to convert qualitative responses into rateable statements for Round 2.

    Writing Good Items

  • One construct per item — the most common wording mistake is double-barrelled items: "This approach is efficient and cost-effective" conflates two distinct judgments. If panelists disagree on one dimension and agree on the other, the combined item produces meaningless results
  • Avoid negative phrasing — "This should not be excluded from the curriculum" is harder to rate consistently than "This should be included in the curriculum"
  • Use consistent rating anchors — if some items use "agree/disagree" anchors and others use "important/unimportant," panelists shift their mental reference point between items. Use one anchor pair throughout
  • Avoid vague quantifiers — "often," "regularly," and "sometimes" mean different things to different panelists. If frequency matters, specify it numerically
  • Keep items at a consistent level of specificity — mixing very broad items ("Student wellbeing is important") with very specific ones ("Mental health check-ins should occur at least weekly") creates a questionnaire where agreement means different things on different items
  • Open-Ended Questions Between Rounds

    Include a brief open-ended field after each rated item, especially in Round 1, allowing panelists to explain unusual ratings or flag problems with item wording. This qualitative data is what makes between-round feedback meaningful — without it, you can only show panelists where they disagreed, not why.

    In Round 2 and later, display the anonymized reasoning from panelists who rated outside the consensus range. This is the mechanism that distinguishes a Delphi study from a repeated survey.

    Piloting

    Pilot the questionnaire with 2–3 people who have the relevant domain knowledge but are not part of your expert panel. Ask them to rate the items and then tell you: were any items ambiguous? Did any items seem to measure two things at once? Were the anchors clear? A pilot often surfaces wording problems that seem obvious only in retrospect — before they become a methodological limitation in your published paper.

    Item Length and Questionnaire Length

  • Individual items should be one sentence, two at most. Longer items are harder to rate consistently
  • Keep total questionnaire length to what can be completed in 15–20 minutes. Rounds longer than this produce measurably higher dropout rates
  • If your content requires more items than fit in 20 minutes, split them across more specific rounds rather than adding items to a single round
  • What to Report in the Methods Chapter

    Describe: how items were generated (literature review, expert interviews, open Round 1), the rating scale and anchors used with justification, the pilot process if conducted, and how open-ended responses were used between rounds. A methods chapter that describes item generation and piloting demonstrates methodological care — the absence of these elements is a standard reviewer complaint.

    Continue Learning in Delphi Academy

    Dive deeper into these topics with our comprehensive guides:

    Ready to Implement Your Delphi Study?

    Our platform streamlines every step of the Delphi process—from expert recruitment to automated statistical analysis and report generation.

    Related Articles

    Delphi MethodResearch Methods

    Delphi Consensus Protocol: When Is Consensus Reached?

    A Delphi consensus protocol fixes, before Round 1, exactly when a statement counts as agreed, when the study stops, and what happens to the items that never get there. Here is what goes into it and how to write it so that reviewers accept the consensus claim.

    6 min read•
    Delphi Questionnaire Design: How to Write Items That Work | Durvey