"Our data is high quality" is not a metric. When a provider says it, the right response is, "Great, show me the numbers from my last project."

Here's the short list I'd want on every quality report.

Key takeaways

  • Ask for metrics at three stages: entry, in-survey and post-field.
  • Blocked-entrant counts by reason show how much fraud was stopped before it cost you.
  • A removal rate with no breakdown tells you very little.
  • Compare metrics across projects and sources, not in isolation.

Entry-stage metrics

These tell you what never reached your survey.

  • Entrants blocked, by reason: duplicate device, proxy or VPN, geo mismatch, known fraud profile.
  • Cross-source duplicate rate: how often the same person arrived through more than one supplier.

A provider that can't report these probably isn't checking at entry, which means you're paying for fraud to get in and then paying again to clean it out. QualityShield reports blocked entrants by layer on every project.

In-survey metrics

  • Speeder rate against a section-level threshold.
  • Straight-lining rate on grids.
  • Attention check pass rate. Useful, but remember AI agents pass these easily, so a high pass rate is not proof of quality on its own.
  • Paste events on open ends.

Post-field metrics

  • Removal rate, by reason. Speeding, patterning, open-end quality, inconsistency.
  • Replacement rate and whether replacements were free.
  • Open-end quality score distribution, not just the average.

Reading the report

Very low removal rates are not automatically good news. They can mean excellent sample, or weak checks. Very high rates signal a source problem. The useful view is trend: the same study type, same audience, compared over time and across sources. That's the idea behind the benchmarking work coming out of the Global Data Quality initiative, and it's worth following.

When you evaluate a sample provider, ESOMAR's "37 Questions to Help Buyers of Online Sample" is still a solid checklist to send along with your RFP.

FAQ

What is an acceptable survey removal rate?

It depends on audience and source. For verified panels with entry screening, post-field removals are typically modest. Much higher numbers point to a source or questionnaire issue worth investigating.

Which data quality metrics matter most?

Blocked entrants by reason and removals by reason. Together they show how much bad traffic was stopped before your survey and how much still needed cleaning afterward.