Introducing Minds Mosaic: Pacing the frontier of audience simulation

Minds Mosaic is our frontier model for audience simulation. Give it a defined population and a survey question, and it predicts how that population will answer: the full distribution across every answer option, before the survey runs.

A forecast like that is only worth something if it holds up against real results. So we measure Mosaic in the open. Every forecast is locked and timestamped before the publisher releases its results, and every score goes into a public ledger, misses included. We have entered Mosaic in an independent arena that scores forecasts sealed in advance, and we have run it head to head on a competitor's own public benchmark.

Every synthetic-respondent vendor now publishes an accuracy number, but the numbers come from different data, different metrics and different conditions. This post starts with each vendor's overall figure, then sorts the published results into the use cases buyers actually run and compares numbers head to head where they were measured on the same data.

The short version

• We sort audience simulation into 15 use-case categories, from political polling to advertising and open-ended interviews, and compare numbers head to head only within a category and on the same data.
• On political and public opinion polls, Minds Mosaic's forecasts, locked before the results were released, show a distribution error (TVD) of 0.023 across 24 questions from 7 surveys. On economic sentiment surveys it is 0.039 across 53 questions from 6 surveys.
• On Artificial Societies' own public benchmark, Mosaic has the lower error on both social attitude sources, on two of three personality sources and on demographic subgroups (0.053 against 0.063). Overall the two are level on their convention, 0.083 against 0.082.
• None of the providers covered here publishes a running record of survey forecasts locked before the survey runs. Ours is public at forecasts.getminds.ai: plain Markdown and JSON files, timestamp proofs and a script that verifies them.

Overall accuracy across categories

Minds Mosaic and two other providers publish one overall distribution error (TVD) that pools all the categories tested. Lower is better; the next section explains the metric.

The table names the data and source behind each figure, with the average miss per answer option (MAE) and the pooled correlation where the provider publishes them.

Four companies publish their headline on a scale of their own, which cannot be converted to TVD or MAE:

• Simile: agents reproduce individuals' survey answers at about 85% of the consistency people show with themselves two weeks later (82% to 86% by data source in the latest version of the preprint).
• Evidenza: "88% accuracy", described as the average match between synthetic and traditional research in more than 100 validations.
• Synthetic Users: "Synthetic Organic Parity" of 85 to 92%, a weighted rubric for interview content.
• Electric Twin: in addition to the 96% above, 92% on its own full-distribution score (NDAM).

How to read the numbers

Two metrics appear most often, and they are not interchangeable.

Total variation distance (TVD) runs from 0 to 1. It is the share of the answer distribution that sits in the wrong place. A TVD of 0.05 means five points of the distribution would have to move to match the real result.

Mean absolute error (MAE) is the average miss per answer option, in percentage points. For the same forecast, MAE is usually much smaller than TVD expressed in points, because TVD adds up the misses across options and MAE averages them. A 3.5-point MAE and a 7.6-point TVD can describe the same result.

Correlations need a label too. A correlation pooled across every answer option of every question is high for almost any reasonable method, because some options are always large and others always small. A correlation within each question is much stricter. Where we quote a pooled correlation below, we say so, for ours as well as for others.

Fifteen categories of audience simulation

Buyers use audience simulation for very different jobs, and a system that forecasts vote intention well has not shown that it can predict how people react to a video ad. We group the work into 15 categories by what gets predicted and what the ground truth is.

The next two tables show who publishes a scored result in each category, as of 1 November 2026. "Yes" means at least one public, scored result exists in that category; the sections below name the data behind each one. Art. Soc. is Artificial Societies, E. Twin is Electric Twin, Syn. Users is Synthetic Users.

How each accuracy claim was produced

The category tells you what was measured. The conditions tell you how much the number can be trusted.

Based on each company's public materials as of 1 November 2026. An empty cell means the published claim does not show it.

A retrospective test runs a system on a survey that has already been published or is held by the vendor. It is useful for development, and every team in this field, ours included, uses it. Its limits are well known: the model may have seen the results, choices about which questions to include are made with the answers available, and the vendor scores itself.

A pre-registered forecast removes those problems. The forecast is fixed before the results exist, so it cannot be tuned to them, and it is scored against numbers the forecaster could not have seen. Locking before the first interview is stronger still, because nothing about the survey can have leaked. This is the standard used for election and economic forecasts, and it is the standard audience simulation should meet.

None of the providers above publishes a running record of survey forecasts locked before the survey runs and scored afterwards. Aaru published a forecast of the 2024 US presidential election the day before the vote. Electric Twin funded a pre-registered academic study led by its chief science adviser at LSE, comparing synthetic answers with seven human sampling services. Simile's partner whitepaper describes internal tests that anticipated new results ahead of fieldwork, without publishing the forecasts. Our ledger publishes every forecast and every score.

How the Minds Mosaic forecast ledger works

For each target survey we write down the exact question wording, answer options and population before the lock. Minds Mosaic then forecasts the full answer distribution of every question. Each forecast is hashed, added to an append-only ledger, batched and timestamped on the Bitcoin blockchain through OpenTimestamps. When the publisher releases its results, we score every locked forecast against the official release, including misses and forecasts that turn out not to be scoreable. Every survey carries an evidence class: locked before fieldwork began, or locked after fieldwork but before any result was public.

The ledger is deliberately plain. At forecasts.getminds.ai anyone can download it as ordinary files: a Markdown README with the current scorecard, one JSON file per locked forecast, the OpenTimestamps proofs, and a script that checks every hash and timestamp. The JSON files double as a machine-readable feed, so anyone can verify our record or rescore it without asking us.

Thirteen surveys have released results so far, covering 77 questions in Austria, France, Germany, Japan, Spain, the United Kingdom and the United States, and every one of those forecasts carries a verified Bitcoin timestamp from before the results were published. Across them, Mosaic's distribution error is 0.034, and 0.023 on the 4 surveys locked before fieldwork began.

Across all 77 questions, the average miss per answer option is 1.37 percentage points, and the pooled correlation between forecast and actual answer shares is 0.993. Minds Mosaic forecasts with all public information available before the lock, including earlier published results of the same survey. Released values from the publishers stay with the publishers: the ledger publishes our forecasts and our scores. The categories below split these results by use case.

Political and public opinion

Seven of the scored surveys are political and public opinion polls in the United States and the United Kingdom: The Economist/YouGov, YouGov's GB trackers, Emerson, NPR/PBS News/Marist, Reuters/Ipsos, Find Out Now and Opinium. Across their 24 questions, Mosaic's locked forecasts have a TVD of 0.023 and an average miss per answer option of 1.10 percentage points. The four surveys locked before fieldwork began are all in this category, with 19 questions and a TVD of 0.023.

We have also entered Minds Mosaic in the Social Simulation Arena, an independent academic leaderboard that scores forecasts sealed in advance, with forecasts for the 2026 US midterm rounds.

Other published results in this category rest on different data, so we list them rather than chart them. Aaru reports an average miss per answer option of 2.21 points on 573 politics, policy and public affairs questions from its 2026 replication study, after its sampling adjustment, and published one forecast of the 2024 US presidential election. Our own retrospective studies of the ANES 2024 and CES 2024 election surveys report 8.33 and 4.72 points of mean distribution error at the level of individual respondents.

Economic sentiment

Six of the scored surveys measure consumer sentiment: the European Commission's harmonised consumer surveys for Austria, France, Germany and Spain, Japan's ESRI consumer confidence survey and the Conference Board's US index. Across their 53 questions, Mosaic has a TVD of 0.039 and an average miss per answer option of 1.50 points.

Locked forecasts for the University of Michigan survey, the New York Fed's Survey of Consumer Expectations and the November waves of the consumer surveys are pending. No other provider covered here publishes a scored result in this category.

Social attitudes and values

Artificial Societies published a public benchmark with seven survey sources and a scoring contract, described in their paper arXiv 2609.30030. We ran Minds Mosaic on it under a protocol hashed before any model call, sealed the raw outputs before scoring, and used their data and scoring code. Two sources come from the 2024 General Social Survey.

Mosaic figures are the mean of three runs; Artificial Societies' are as published. Mosaic received one question per request, with earlier public waves of the General Social Survey as labelled context. Mosaic forecasts the whole sample's distribution, while their published numbers come from simulating each respondent.

Two other results in this category use their own data. Simile's agents of 1,052 interviewed Americans reproduce individuals' General Social Survey answers at about 85% of the consistency people show with themselves two weeks later. Our GSS 2024 study reports 7.53 points of mean distribution error across 17 questions and 360 matched respondents.

Personality and psychometrics

Three sources of the Artificial Societies benchmark measure personality rather than attitudes. This is not one of the 15 categories, so we report it separately.

Mosaic, mean of three runs, has the lower error on facets and on the twin retest and is level on traits.

Wording and survey design effects

Two sources of the same benchmark are split-ballot wording experiments: the same question asked in two versions, to measure how much the answer shifts.

Mosaic is level on the Jordan experiment, and Artificial Societies has the lower error on Albertson. Mosaic predicts the direction of both shifts and overstates their size. Wording effects are a different task from forecasting a distribution, which is why we score them as their own category.

Subgroups and segments

Subgroups are where synthetic research is most often used and where it most often fails. On the 145 demographic subgroup cells of the General Social Survey confidence items in Artificial Societies' benchmark, Mosaic has a TVD of 0.053 against their 0.063.

Our ledger also scores subgroups. The Economist/YouGov and YouGov GB crosstabs give 69 locked subgroup cells, such as answers by age group, party or 2024 vote, where Mosaic has a TVD of 0.046.

Advertising, video and creative

No two providers have published results on the same creative data, so this category has no head-to-head yet. Electric Twin reports correlations of 0.909 for images and 0.861 for video between predicted and actual mean Likert scores on its own benchmark. In our Super Bowl study, 100 grounded simulated respondents rated all 54 ads in the USA TODAY Ad Meter 2026 and ranked them in close agreement with the human panel, with a Spearman correlation of 0.85.

Concepts, pricing and brand

Published results here also rest on each provider's own data.

Open-ended answers and behaviour

Synthetic Users scores interview content with a weighted rubric it calls "Synthetic Organic Parity", at 85 to 92%. Our held-out interview study tested whether grounded simulated respondents reflect what 22 real participants later said in 66 interview answers, and found a clear gain from participant evidence over generic prompting.

For behaviour versus stated intent, Aaru compared its brand rankings with credit-card purchase data and reports agreement on 92.4% of brand pairs, against 77.8% for published survey consideration, in the same replication study.

Artificial Societies' benchmark overall

Across all seven sources, Mosaic and Artificial Societies are level. On Artificial Societies' convention, the first run with each source weighted equally, Mosaic scores 0.083 against their 0.082; averaged over all three runs, Mosaic scores 0.085. Mosaic has the lower error on five of the seven sources. Without the two wording experiments, Mosaic leads 0.084 against 0.091 on the three-run mean.

What a fair benchmark should look like

Our experiments over the past two months, the field's published numbers and our entry in a public arena all point to the same four rules.

1. Lock before fieldwork. A forecast fixed before the first interview cannot be tuned to the answer. Locking only before release is weaker, because results sometimes leak once fieldwork has closed. Record the actual fieldwork dates and the evidence class of each result.
2. Publish the noise floor. Even a perfect forecaster carries sampling error: about 0.024 TVD for a three-option question at a sample of around 1,200. Differences smaller than that are not meaningful without intervals.
3. Report by use-case category and stimulus. Polls, consumer sentiment, ads, concepts and websites are different tasks, and text questions, images, video, audio and interactive flows are different stimuli. A pooled headline across them says little about any one of them.
4. Split by response type. Single choice, Likert and ordinal scales, multi-select, numeric answers, rankings and forced-choice designs such as MaxDiff, and open-ended text each need their own metric. Multi-select options should be scored as separate yes/no shares, ordinal scales with an order-aware score, and open answers with theme shares or embedding distances rather than a single judge.

We would add two practical rules: report every run and the spread between runs, and state which correlation is reported, pooled or within question.

Other public benchmarks worth following

No single benchmark covers the field yet. These are the public ones we follow, grouped by the kind of evidence they produce.

Only the first one scores forecasts made before the outcome existed. The others are valuable for method development, and most are public, so any team can rerun them. Their items are also public, which means a model may have seen them during training. That is why prospective results carry the most weight.

What comes next

Several categories have no public scored result from anyone yet: messaging and copy, website and app behaviour, and media and entertainment demand. Others rest on a single study. These are open ground, and the pre-registered ledger is how we will fill them: lock the forecast, wait for the real result, publish the score.

Minds Mosaic is entered in the Social Simulation Arena as minds-mosaic, with forecasts sealed in advance for the 2026 US midterm rounds and for weekly survey rounds that resolve from November. We will keep adding surveys to our forecast ledger, publish every result as it resolves, and report it by category and against the noise floor. We are also running Aaru's published protocol on fresh 2026 studies.

Minds Mosaic is our frontier model for audience simulation: give it a population and a question, and it forecasts the full answer distribution before the real one exists. Published surveys are the first test. Next, Mosaic will be measured on behaviour: forecasts of purchases and viewing, scored against what people actually bought and watched. It will also be measured on the open categories above and in the arena rounds that resolve over the coming months, each one locked before the outcome and scored in public.

If you publish accuracy numbers for an audience simulation model, we would welcome a shared, forward-locked test in any of these categories. It is the fastest way for this category to earn the trust of the people who buy research.