Survey Design

Once your expert panel is in place, it's time to design the survey environment. This step lays the groundwork for clean data and meaningful insights. Think beyond simple questions—plan for the analysis you want to conduct later.

If you skip demographic and sentiment questions now, you cannot conduct those analyses later. Think ahead about every comparison and insight you might want to extract from your data, and design your survey accordingly.

Define the Response Format

Choose how experts will evaluate your theses. The format should match your research objectives and enable the analysis you plan to conduct.

Common Response Formats

• Likert scales: "Rate your agreement from 1 (strongly disagree) to 5 (strongly agree)"
• Probability ratings: "What is the likelihood this will occur by 2035? (0-100%)"
• Impact ratings: "Rate the expected impact from 1 (negligible) to 5 (transformative)"
• Timeline estimates: "In which year do you expect this to become mainstream?"
• Open comments: "Please explain your reasoning" (optional but valuable)
• Ranking: "Rank these items from 1 (highest priority) to 5 (lowest priority)"

Example: "Please rate your level of agreement from 1 (strongly disagree) to 5 (strongly agree) and optionally explain your answer."

Include Stakeholder-Identifying Questions

Collect metadata to enable segmentation later. This demographic data allows you to compare responses across different expert groups—a powerful analytical capability.

Essential Demographic Variables

• Professional background: Academia, policy, clinical practice, tech industry
• Years of experience: <5, 5-10, 10-20, 20+ years
• Geographic region: North America, Europe, Asia, etc.
• Organizational type: University, hospital, government, corporation, NGO
• Career stage: Early-career, mid-career, senior

Example: "What is your professional background? ☐ Academia ☐ Policy ☐ Clinical Practice ☐ Tech Industry"

Plan for Capturing Sentiment

Make it possible to measure optimism, critical attitudes, or other emotional nuances. Sentiment analysis can reveal whether experts view developments positively or with concern.

Sentiment Capture Methods

• Overall optimism scale: "How optimistic are you about these developments? (1-5)"
• Desirability ratings: "How desirable is this outcome? (1-5)"
• Concern levels: "How concerned are you about potential risks? (1-5)"
• Pro/Con comment fields for each thesis

Example: "How optimistic are you about the future impact of these technologies? (1=Very pessimistic, 5=Very optimistic)"

Anticipate Subgroup Analysis

If you want to compare responses by role, region, or experience level, capture that data now. You cannot add these questions retroactively.

Analytical Possibilities with Proper Design

• Compare academics vs. practitioners
• Analyze regional differences
• Examine experience-based variations
• Identify optimist vs. skeptic patterns
• Track consensus differences across stakeholder groups

Additional Design Elements

Choose Your Delphi Variant

• Classical Delphi: Start with open-ended Round 1, then structured rounds
• Modified Delphi: Begin with structured questionnaire based on literature
• Real-Time Delphi: Continuous feedback, experts see live updates (Durvey specialty)

Set Study Parameters

• Determine number of rounds (typically 2-4)
• Set consensus criteria and stopping rules
• Establish timeline and deadlines for each round
• Define what constitutes "consensus" (e.g., IQR ≤1, ≥70% agreement)

Create Supporting Materials

• Invitation letter: Explain purpose, time commitment, why they were selected
• Informed consent: Ethical approval, anonymity assurance, data usage
• Instructions: Clear guidance on completing questionnaires
• Glossary: Define key terms for consistent understanding
• Timeline: When each round occurs and expected time commitment

Select Your Technology Platform

Choose the right tool for data collection and management.

Pilot Test Your Questionnaire

Test with 2-3 people (colleagues or non-panel experts) to identify issues before launch.

• Check for clarity and ambiguity in questions
• Assess completion time (aim for ≤20 minutes per round)
• Test technical platform functionality
• Verify instructions are clear and complete
• Ensure rating scales work as intended
• Make revisions based on pilot feedback

Practical Example: Survey Design

For our public health technologies Delphi, we designed the following structure:

Each thesis was rated on:

• Impact: 1-5 scale (How transformative will this be?)
• Desirability: 1-5 scale (How desirable is this outcome?)
• Expected year: When will this become mainstream? (2025-2040)

We included:

• Sentiment question: "How optimistic are you? (1-5)"
• Pro/con comment fields for qualitative insights
• Professional background, experience, and region demographics

This design ensured seamless consensus detection, sentiment analysis, and stakeholder group comparisons in the analysis phase—all handled within Durvey.org's integrated analytics.

Durvey's Analysis Advantage

One of Durvey.org's most powerful features is its integrated analysis toolkit. Unlike traditional Delphi platforms that require exporting data to external tools, Durvey provides everything you need:

• Real-time consensus tracking: Monitor agreement levels as responses come in
• Sentiment analysis: Analyze pro/contra comments and expert reasoning
• Subgroup comparisons: Compare stakeholder segments automatically
• Interactive visualizations: Charts and graphs update dynamically
• Paper-ready outputs: Export publication-quality tables and summaries
• All-in-one platform: No need for SPSS, Excel, or external analysis tools

Survey Design by Academic Level

Common Survey Design Pitfalls

Missing Demographic Questions

Forgetting to collect stakeholder metadata means you cannot conduct subgroup analysis later. Plan ahead for every comparison you might want to make.

Overly Long Questionnaires

Keep each round to 20 minutes max. Longer surveys lead to abandonment and poor-quality responses. Be ruthless about cutting unnecessary questions.

No Pilot Testing

Launching without testing leads to discovered errors mid-study. Always pilot test with 2-3 people to catch issues before your expert panel sees them.

Ambiguous Questions

Unclear or leading questions produce unreliable data. Each question should be precise, neutral, and easily understood by all experts.