Echo or estimate?

A simulated respondent is useful only if its answers come from something like an opinion. The usual evidence is agreement with a survey's published results, and that agreement has two possible sources. The model may be estimating how a population would answer, or it may be reproducing numbers it met in training. Agreement alone does not separate them, and one recent study reports that matching published marginals does not discriminate respondent simulation from direct population estimation^[1].

This inquiry asks whether the second source can be measured and sized.

Language models conditioned on demographic backstories have been reported to reproduce the response patterns of human subgroups^[2]. Later work qualifies how far that agreement can be trusted.

Two 2026 preprints suggest that agreement with published results is weak evidence of simulation. The 2026 sources here are preprints, two of them single-author, and have not been peer reviewed; the design tests their claims rather than assuming them. In one, across six multiselect batteries, the response format dominated measured fidelity, and a no-persona query asking for population prevalence had lower error than simulated respondents that each commit to a set of options in every one of nine aligned comparisons^[1]. In the other, much of the aggregate accuracy of language models on survey items came from predicting each item's mean: once item means were removed, predictions explained about 3% of the remaining variation between respondents^[3].

Memorisation has been examined directly in at least one study. A verbatim-recall probe across 37 models on one Lithuanian organisational-psychology dataset did not track how the models ranked on a psychometric similarity score^[4]. That compares models on a single dataset. A different design uses time: personas built from respondents' answers collected before 2023 were tested against the same people's later answers, and improved alignment with population-level response distributions, most in domains tied to stable attributes and values, while remaining limited for individual prediction^[5].

The exposure measure carries its own doubt. Membership inference attacks, which try to tell whether a given text was in a model's training data, barely outperform chance on large language models in most settings, and apparent success often reflects a distribution shift such as members and non-members coming from different periods^[6]. Blind attacks that never examine a trained model beat published attacks on eight evaluation datasets^[7].

The public stage uses published instruments only. Forty survey instruments will be matched by construct, half published before the documented training cut-off of every model tested and half after the latest of those cut-offs. The split spreads exposure across the set; it is not the exposure measure. Each model is asked to answer as respondents would, with scale points, wording and prompt frame held identical across every instrument, because the response contract otherwise dominates the result^[1].

Exposure is measured, not inferred from publication date. A verbatim-recall or membership probe estimates, for each instrument and each model, how much of the instrument the model reproduces. Three pinned models are tested. A fourth, open-weight model is reported as corroboration and cannot decide the outcome alone.

The outcome is the distance between the model's simulated answer distribution and the instrument's published item-level results, expressed in floor units. The floor comes from independent replications of the same instrument, so a gap smaller than the difference between two real replications does not count. Every gap is scored against an item-mean baseline built from the instrument's own repeated results, so that a model returning generic average answers cannot look anchored. The baseline matters because item means alone explain much of the accuracy models show on survey items^[3].

The question is whether the distance shrinks as measured exposure rises. The exposure gap is that change: for each model, how much closer its answers sit to the published results on instruments the probe scores as more exposed than on those it scores as less exposed, in floor units and net of the item-mean baseline. The pre-registration will fix its estimator, whether a split or a slope, and the confidence level of its interval. A 20-item pilot sets the equivalence margin and the smallest gap worth detecting, and both are registered before the full forty instruments are run. The design allows three weeks from start to an initial signal.

The rule is fixed in advance. The numbers it depends on will be set in the pre-registration before the full run: the equivalence margin and the smallest detectable gap from the pilot, the estimator and confidence level of the exposure gap, the number of pinned models a bounded null requires, the probe's validation threshold and its margin over a blind baseline.

1. Continue. For at least two of the three pinned models, the pre-registered interval on the exposure gap, in floor units, excludes zero and the gap clears the item-mean baseline. The fourth, open-weight model does not count towards the two.
2. Stop and publish a bounded null. The gap sits inside the pre-registered equivalence margin on the pinned models; the pre-registration will state how many of the three must show this, alongside the margin. The null is reported with its bounds.
3. Stop before measuring. Fewer than 40 construct-matched instruments with full item text and published item-level results can be sourced with exposure checked by the probe rather than by publication date, or the probe fails to separate documents known to be in a model's training data from documents known to be out, on its own validation set, at a threshold the pre-registration will fix before the probe is applied to any instrument; or the probe does not beat a blind baseline, one that never queries the model, on the same validation set by a margin the pre-registration fixes. The design does not fall back to publication date.

A gap that neither excludes zero nor sits inside the margin answers nothing and is reported as unresolved. The pre-registration will set the equivalence margin and the smallest detectable gap from the 20-item pilot, before the forty instruments are run.

A gradient between exposure and closeness has explanations other than recall. Popular, older instruments may be both easier to recall and more often replicated, and they may measure stable constructs that a model can estimate without recall; construct matching and floor units from independent replications are meant to reduce this, and the limits state that it cannot be removed. A model returning generic average answers can look close, because item means alone explain much of the accuracy models show on survey items^[3], so every gap will be scored net of the item-mean baseline. In one study of multiselect batteries the response contract dominated measured fidelity^[1], so it will be held identical across instruments and models. The probe may track how memorable an instrument's wording is rather than whether the model trained on it; it will be validated on documents known to be in and out of training data, and its error reported. Validation sets can be built from text created before and after model training, as in the WikiMIA benchmark^[8], and apparent detection success on such splits can come from the difference in period rather than from membership^[6]. Opinion also moves, so instruments fielded after a cut-off may sit further from a model's answers because the population changed. Because exposure will be measured per instrument rather than read from the date, the gradient can be checked within the pre-cut-off half, where period cannot explain it. One model's quirk cannot decide the outcome: two of the three pinned models must agree, and the fourth, open-weight model will be reported alongside and cannot supply either of the two. In the other direction, probe noise can flatten a true gradient, so a bounded null will be stated relative to the probe's measured error.

A pass would show that closeness to published results and measured exposure move together in these models on these instruments. Exposure is measured, not assigned, so topic, age, wording and popularity of an instrument can travel with it. Construct matching reduces this and cannot remove it.

The probe measures recall, not training membership. Failing to recall an item does not show the item was absent from training, so the probe's error is reported, not removed^[7][6].

The stage uses published instruments. It says nothing about whether a model estimates a fresh population's opinion; that needs fresh answers from real respondents and is a separate stage. The result is dated to the pinned model versions. A bounded null would show no exposure gradient of the size the design can detect, not that no recall exists.