Delphi Questionnaire Design: How to Write Items That Work
Item quality is what separates a Delphi study that produces interpretable consensus from one that produces agreement on statements nobody actually disagreed with. Most Delphi questionnaire problems aren't visible during design — they only appear in Round 1 results, when you realise panelists were interpreting your items in three different ways. This guide covers how to avoid the most common design mistakes before they cost you a round.
Choosing a Rating Scale
The most common rating format in Delphi studies is a Likert-type scale. The two most widely used options are:
For most academic thesis work, a 5-point or 7-point scale is sufficient. Use a 9-point scale if your study follows RAND/UCLA methodology or if the reporting framework you've chosen recommends it.
Open vs. Closed Round 1
If your item pool isn't already well-defined by the literature, consider a semi-open Round 1: present a preliminary list of items derived from literature and ask panelists to rate them, but also include open-ended fields where panelists can suggest additional items or reword existing ones. This hybrid approach gives you a starting structure while surfacing important items you may have missed.
A fully open Round 1 (no items, just open questions) generates richer initial content but requires significantly more between-round analysis to convert qualitative responses into rateable statements for Round 2.
Writing Good Items
Open-Ended Questions Between Rounds
Include a brief open-ended field after each rated item, especially in Round 1, allowing panelists to explain unusual ratings or flag problems with item wording. This qualitative data is what makes between-round feedback meaningful — without it, you can only show panelists where they disagreed, not why.
In Round 2 and later, display the anonymized reasoning from panelists who rated outside the consensus range. This is the mechanism that distinguishes a Delphi study from a repeated survey.
Piloting
Pilot the questionnaire with 2–3 people who have the relevant domain knowledge but are not part of your expert panel. Ask them to rate the items and then tell you: were any items ambiguous? Did any items seem to measure two things at once? Were the anchors clear? A pilot often surfaces wording problems that seem obvious only in retrospect — before they become a methodological limitation in your published paper.
Item Length and Questionnaire Length
What to Report in the Methods Chapter
Describe: how items were generated (literature review, expert interviews, open Round 1), the rating scale and anchors used with justification, the pilot process if conducted, and how open-ended responses were used between rounds. A methods chapter that describes item generation and piloting demonstrates methodological care — the absence of these elements is a standard reviewer complaint.
Continue Learning in Delphi Academy
Dive deeper into these topics with our comprehensive guides:
Ready to Implement Your Delphi Study?
Our platform streamlines every step of the Delphi process—from expert recruitment to automated statistical analysis and report generation.
Related Articles
How to Recruit Experts for a Delphi Study: A Step-by-Step Guide
Recruiting the wrong experts is the most common way a Delphi study fails before data collection even starts. Here's how to do it right.
Delphi Consensus Protocol: When Is Consensus Reached?
A Delphi consensus protocol fixes, before Round 1, exactly when a statement counts as agreed, when the study stops, and what happens to the items that never get there. Here is what goes into it and how to write it so that reviewers accept the consensus claim.