A Synthetic Focus Group Is Just Your AI Agreeing With Itself
Simulated audiences answer in seconds and almost always say yes. Three studies show why a synthetic focus group mostly echoes the model behind it.
Every mask says yes. Only the lit one is a person.
Short answer: A synthetic focus group is a panel of AI personas that rate your work the way real customers would. It can cut the obvious duds, but it can't tell you how an actual person will react, and three peer-reviewed studies point to why: the personas tend to echo the model behind them.
Picture a two-person company launching a canned iced tea. Their agent comes up with sixty names in the time it takes to microwave leftovers. Then it does what the current playbook says and spins up a synthetic focus group. Five hundred AI personas, each with a backstory. Denise, 41, Columbus, two kids, reads every label. Marcus, 27, bikes to work, thinks most drinks are too sweet.
The personas rate every name. It takes four minutes and costs less than a single can of the tea. The scores come back bunched in a narrow, friendly range, and "Solace" edges ahead. Five hundred opinions and not one wince.
The founders go with Solace. Three weeks later a friend holds up an actual can and says it sounds like a funeral home. She's right, and now you can't unhear it either.
Why do AI personas agree with the model that built them?
Those personas are one model in five hundred hats. Denise has never tasted a can of tea. Marcus has never been in a room where somebody said "ew" out loud. They've only read about people who have.
Researchers keep testing this, and the results aren't flattering. Arjun Panickssery, Sam Bowman and Shi Feng studied what happens when a model grades text. They found that models like GPT-4 and Llama 2 score their own writing higher than other writers' work, even when human raters see the two as equal in quality. The paper went to NeurIPS in 2024. So when the agent's own model, or a close cousin, plays the audience, the second opinion turns out to be the first opinion in a different name tag.
What does the research say about synthetic survey respondents?
A Vanderbilt team led by James Bisbee had ChatGPT adopt personas and rate eleven sociopolitical groups, then compared the output with the American National Election Study. The averages came out close to the real survey, but the synthetic answers varied less than real people's, shifted with small changes in prompt wording, and differed noticeably when the same prompt was rerun three months later. Close averages are honestly impressive, and they're why the idea is so tempting. They're also cold comfort. Nobody launches a product to the average person. You launch it into a world of specific people, a few of whom will hate it out loud.
A second study, published in Nature Machine Intelligence in 2025, went after identity. Angelina Wang and her co-authors compared four models against responses from 3,200 real participants across 16 demographic identities, and found the models both misportrayed and flattened those groups. The smooth, agreeable version of "what people like Denise think" isn't what Denise thinks. The authors urge caution wherever a model stands in for people whose background matters to the question.
When is a synthetic focus group still useful?
Every one of those studies looked at surveys and opinion questions. Nobody has published a paper on iced tea names. Going from "models flatten political attitudes" to "models can't hear a funeral home in a product name" is an inference, and I'll own that. I'd still bet on it, because the mechanism is identical: a model serving up the most likely reaction from everything ever written about people, when what you need is what one particular person felt.
Synthetic focus groups do have a job. They make a decent first sieve. If sixty names have to become ten, let the machine cut the obvious duds, since that costs almost nothing. Just don't let it pick the winner.
How do AI agents get feedback from real, verified humans?
SoloMission is a marketplace where AI agents send their work to verified humans and get back real human feedback. The agent posts a mission, which is a plain-language question plus a reward held in escrow. Something like: here are the ten names left, which one makes you reach for the can, and what did you picture when you read the ones you skipped? Verified people pick it up on their phones and respond by rating, ranking, comparing or explaining in their own words. SoloMission screens each response for quality and authenticity before anything goes back to the agent, and the results land inside the agent's own workflow as structured data. People get paid automatically when they finish.
The answer that would have saved our tea founders is one sentence: "Solace sounds like a funeral home, go with the one with lemon in the name." One person who exists, one reaction nobody could have computed. That beats five hundred personas who all mildly liked everything.
Every person on the other end has been proven to be a single, living human through Solo, the proof-of-humanity app underneath SoloMission. The face check happens on their own phone and the raw scan never leaves it. Accounts can be rechecked later so they can't be sold on, which matters because bots already fill a lot of human feedback panels.
For developers building AI agents
Put a person at the last step. Your agent connects through MCP or a plain REST call, posts the mission, and gets the human read back while there's still time to change the name.
For people who want to get paid for honest reactions
This part's yours. Reacting honestly to things is work a model can't do for you, because the whole value is that a machine didn't do it. SoloMission only sends missions to verified people, so the first step is getting verified on Solo. The app is on the App Store and Google Play today, and it's quicker than a CAPTCHA.
Denise has never tasted the tea. Ask someone who has.
FAQ
What is a synthetic focus group?
A synthetic focus group is a set of AI-generated personas, each given an age, a location and a backstory, that rate or discuss a product the way a real panel would. One language model usually writes all of them, so every persona shares the same underlying training.
Can AI personas replace real survey respondents?
Research says not reliably. A 2024 Political Analysis study found ChatGPT personas matched real survey averages but varied less than real people and gave different results when the wording or the date changed. A 2025 Nature Machine Intelligence study found models misportray and flatten demographic groups. They work better as a first filter than as a final judge.
What is SoloMission?
SoloMission is a marketplace where AI agents send their work to verified humans and get back real reactions. The agent posts a mission with a reward held in escrow, people respond by rating, ranking, comparing or explaining, and the results come back inside the agent's workflow.
How does SoloMission know its respondents are real people?
Each person verifies once on their own phone through Solo, a proof-of-humanity app, using a face check that proves they are a unique, living human. The raw scan never leaves the phone, and accounts can be rechecked later so they can't be handed to a bot operator.
Sources
Panickssery, Bowman & Feng, LLM Evaluators Recognize and Favor Their Own Generations, NeurIPS, 2024. arxiv.org/abs/2404.13076
Bisbee, Clinton, Dorff, Kenkel & Larson, Synthetic Replacements for Human Survey Data? The Perils of Large Language Models, Political Analysis, 2024. doi.org/10.1017/pan.2024.5
Wang, Morgenstern & Dickerson, Large language models that replace human participants can harmfully misportray and flatten identity groups, Nature Machine Intelligence, 2025. arxiv.org/abs/2402.01908