← Back to work

Synthetic Panel

Simulated survey respondents, built from census data and checked against real polls — an experiment in whether synthetic response data is trustworthy enough to decide on.

Type
Research prototype
Stack
Python · Claude API · HTML
Sample
50 personas
Role
Design & build

The question

Research is expensive and slow, so there's obvious appeal in asking a language model to stand in for respondents. The appeal is also the problem: a model will produce plausible survey answers for any prompt you give it, whether or not those answers correspond to anything real.

Published research suggests this can work — condition a model with enough specific demographic detail and its responses start tracking how that kind of person actually answers. I wanted to find out how much of that holds up when you build it yourself, on a real question, and check the output against a known answer.

How it works

  • The sample frame. Fifty personas built from census data — housing tenure, income bracket, language spoken at home, transit reliance, neighbourhood, age. Distributed evenly across every electoral ward so the sample isn't accidentally a downtown monoculture.
  • The conditioning. Each row becomes a first-person backstory rather than an instruction to role-play. Specificity is what produces fidelity; “simulate a voter” produces an average of everyone.
  • The runner. Loops the panel, asks each persona a fixed question set, parses responses into structured output, and exports to CSV alongside a topline.
  • The events log. A context block appended to every backstory, so the same panel can be rerun after a real-world development and the movement between runs becomes the finding.

Calibration is the whole thing

Without a baseline this is a random number generator with extra steps. Every run is compared against published polls covering the same question and period. That comparison isn't a formality — it's the only mechanism that turns output into evidence.

The interesting result isn't when the synthetic topline matches. It's when it misses, and the miss is structured — because a systematic gap tells you which populations the model represents badly, which is information you can correct for.

What it can't do

  • Models over-represent some demographics badly. Baselines are the bias correction, not a nice-to-have.
  • An even split across geography isn't a probability sample. Comparing toplines honestly means weighting to real population distribution first.
  • Stated response and actual behaviour are different things. This is a pre-filter for killing weak ideas early — not a replacement for research that touches real people.

Why I built it

The same question runs through everything else here: not whether a system produces output, but whether the output can be trusted enough to act on. Synthetic respondents are a clean version of that problem — the data looks entirely convincing whether or not it means anything, and the only defence is checking it against something real.

kevin@kevrichard.com →