5 Common Mistakes in Delphi Studies and How to Avoid Them

Running a Delphi study looks straightforward on paper: recruit experts, send questionnaires, gather consensus. But in practice, even experienced researchers make avoidable mistakes that compromise their results — or derail the study entirely.

Here are the five most common pitfalls, and what to do instead.

Mistake 1: Treating the Delphi Method as Purely Quantitative

This is the most widespread misunderstanding. Researchers focus exclusively on numerical ratings, percentage agreement, and convergence scores — and completely miss the most valuable part of the study.

The numbers tell you what the group thinks. The qualitative justifications tell you why.

When you reduce Delphi to rating scales and statistics, you lose the nuance, assumptions, and divergent reasoning that make expert input genuinely useful. A panelist who rates an item 2 out of 7 may have spotted a fundamental flaw that everyone else missed. If you never read their comment, you'll never know.

What to do instead: Treat qualitative feedback from each round as primary data. Summarize it carefully and feed it back to panelists in the next round. The quantitative scores are a navigation tool — not the destination.

Mistake 2: Not Defining Consensus Before You Start

Many researchers begin a Delphi study without a clear definition of what "consensus" means for their study. They run rounds until responses "look similar enough" — which is neither rigorous nor reproducible.

This creates two problems:

What to do instead: Define your consensus threshold before recruiting a single expert. Common approaches include:

Document your criteria in your methods section. Your committee and reviewers will thank you.

Mistake 3: Selecting the Wrong Expert Panel

Delphi studies are only as good as the experts they involve. Two common panel errors destroy validity before the study even begins:

Too small a panel. A panel of 5–7 people is too narrow to represent the breadth of a field. Most methodologists recommend 10–50 panelists depending on scope; some studies go higher for broad interdisciplinary topics.

Wrong expertise mix. Selecting only senior professors, or only practitioners, or only people from one country introduces systematic bias. Delphi works best when the panel reflects genuine diversity of perspective, background, and experience.

What to do instead: Define inclusion criteria for your experts before recruiting. Consider: seniority, institution type, geographic region, sub-discipline, and practitioner vs. academic balance. Aim for heterogeneity as a feature, not an afterthought.

Mistake 4: Poor Questionnaire Design in Round 1

Round 1 of a Delphi study is typically open-ended — its job is to generate ideas and surface important dimensions you may not have anticipated. But many researchers overstructure Round 1 by presenting a pre-defined list of items and asking experts to simply rate them.

This forces panelists into your framework from the start, eliminating one of the core advantages of the method: the ability to surface unknown unknowns.

What to do instead: Keep Round 1 genuinely open. Ask broad, generative questions like:

Then synthesize the responses thematically and use that synthesis to build the rating items for Round 2. This is more work — but it's what makes Delphi different from a regular survey.

Mistake 5: Marginalizing Minority Opinions

After two or three rounds, outlier responses often disappear from the analysis. Researchers summarize "the group reached consensus" and move on. But persistent minority opinions are frequently the most interesting finding in the study.

A panelist who consistently disagrees may be ahead of the curve. Or they may be identifying a genuine limitation in the consensus view. Either way, dismissing them as statistical noise is a mistake.

What to do instead: When a subset of experts consistently diverges from the majority, analyze that divergence explicitly. What distinguishes these panelists — background, region, sub-discipline? What reasoning do they offer? Document it. In many research fields, "the group reached strong consensus with one dissenting cluster arguing X" is a richer contribution than "the group agreed."

The Underlying Pattern

Most Delphi mistakes share a common root: treating the method as a faster survey rather than as a structured conversation between experts. When you focus on efficiency over rigor — skipping qualitative coding, rushing to consensus, using convenience sampling for your panel — you undermine the very thing that makes Delphi worth doing.

The good news: all five mistakes are avoidable with proper planning. Define your criteria upfront, recruit thoughtfully, and respect both the numbers and the reasoning behind them.