A hand-written world as a test for world models

Far Akin Research Labs

Why this question

A world model is trusted to stand in for a world: to say what follows from an action, what is still there when it is out of sight, what cannot be undone. Common video metrics reward how the output looks^[1]. Whether the stand-in keeps the facts of the world it imitates is a separate question, and answering it needs a reference that holds those facts.

A fictional world written by hand can serve as that reference. Its canon is a finite record of events and of the rules that governed them. It may be incomplete in places, but it is fixed, so a check built from it can be repeated, and it needs no human annotators.

What is already known

Grading a simulator against a declared use is established practice. In verification, validation and accreditation (VV&A), accreditation is the official certification that a model is acceptable for a specific purpose^[2]. A model is valid for a set of conditions when its accuracy lies within the range its intended purpose requires, and that range should be fixed early^[3].

The practice has begun to reach learned world models. A 2026 preprint argues that a world model used as a test oracle must be accredited first, defines a five-level admissibility ladder, and applies it to two driving models; the one with better visual quality ranked lower on following actions^[1]. A benchmark of video generators finds physical understanding severely limited and unrelated to visual realism^[4].

Probes of the kind proposed here have precedents. Violation-of-expectation tests from developmental psychology, including a probe of object persistence behind an occluder, have been used on a learned model^[5]. A 2015 task suite generated questions and answers together from a simulated world of characters and objects, so the true labels were known^[6]. The failure the revisit class targets is documented: a short temporal context lets a world simulator lose consistency over long spans, and a memory bank is one proposed remedy^[7].

None of these sources uses a hand-written canon as the reference.

Definitions

World ledger: A typed record of one arc of the canon: entities, the relations between them, what each can do, what each owes, and events with a time and a cause. It is designed to be the only input the question generator reads.
Entailed answer: The answer that the ledger plus the written rule set fixes for a question. A question whose answer would need new canon will be dropped, not guessed.
World consistency: Agreement between what a model's rollout shows and what the ledger entails at the same point: a place as it was left, an object still present out of view, an irreversible act still done, a handoff still in effect.
Class score: For one model and one of the five classes, the share of that class's questions on which the model's output matches the entailed answer, read through the adapter that turns typed events into model inputs and outputs back into answers. The adapter will be frozen in the pre-registration before any score is read.
Separation: The competent model's class score exceeds the weak model's by more than the margin the pre-registration will fix. Each class that separates counts towards the two of five that a pass needs.
Added value: Neither the visual-fidelity score nor the average benchmark score on the same rollouts reproduces the questions' ranking on every class where the questions separate the models.

The approach

One arc of a fictional world written by the founder, about a tenth of its canon, is typed into a ledger: entities, relations, what each can do, what each owes, and events with a time and a cause. A second world written by someone else is needed to test transfer. An extraction pass builds the ledger, and a contradiction checker runs against the canon itself, so the reference is tested before it is used. A written rule set states what the canon entails, so that consequences the canon leaves unstated can still be adjudicated by machine.

Test questions are generated from the ledger alone, in five classes:

Revisit: whether a place is as it was left on return.
Occlusion: whether something out of view persists.
Irreversibility: whether an act that cannot be undone stays done.
Handoff: whether what passes between agents stays passed.
Counterfactual: what follows if one event is removed.

The same questions go to at least two world models chosen to differ sharply in quality: one competent, one deliberately weak, such as a random-action or memoryless model.

A world model that renders video or acts on commands does not answer questions, so each question passes through an adapter: the layer that turns a typed ledger event into an input the model accepts and reads the model's output back as an answer. Two adapter forms are candidates. In render-and-check, the ledger event is staged as an action sequence and a fixed vision checker reads the rollout for the entailed fact. In the second, text-world models are queried directly. The pilot uses render-and-check on one model, and the adapter will be frozen in the pre-registration. If neither form works on a queryable model, the inquiry stops.

The measure, per class, is whether the competent model scores above the weak one. A visual-fidelity score and an average benchmark score on the same rollouts serve as the comparison, because the interest lies in a ranking those scores do not already give. Results would be reported as a stated domain of use with error bounds, not a single number^[3]. The decision rules below, including the margin, will be fixed in a dated pre-registration before any score is read. The pilot window is five weeks. Nothing has been built or run.

What would count as an answer

Written in advance, for the pilot.

Pass. On the one ledgered arc, the generated questions rank the competent model above the weak one, by more than a margin to be fixed in the pre-registration before any score is read, on at least two of the five classes. A pass shows that the instrument discriminates. It does not show that a grade is trustworthy for any use.

Stop. Fewer than two classes separate the models. The instrument then fails to tell the models apart, and the full ledger build does not start. Also a stop: no world model can be queried inside the pilot, or neither adapter form works on one, logged as blocked on model access, neither skipped nor read as a result about the science.

Pass without added value. If the visual-fidelity or average benchmark score on the same rollouts reproduces the ranking on every class where the questions separate the models, the instrument discriminates but adds nothing, and the inquiry records it so.

Inconclusive. The interval on the class-score gap includes both no effect and the pre-registered margin on enough classes that neither a pass nor a stop is settled. Reported as such, not as a pass or a stop.

Two further conditions will be checked inside the same pilot, and neither depends on the scores. If the rule set can only be completed by inventing canon, the reference becomes opinion and the inquiry stops. If no second world, written by someone else, can be named, the full ledger build does not start, even after a pass, because the author-idiom objection would have no test.

What else could explain a success

A pass could have causes other than world modelling. The first is distance between the models. A random-action or memoryless model fails almost any check, so beating it shows that the questions work, not that they are sensitive; the page treats a pass as evidence of discrimination only. The same distance means a fidelity score may well rank the two models the same way, so the pilot may show discrimination without added value, and the page will record it so if it does. The second is rendering quality: a model that draws badly may fail consistency questions because its output cannot be read. The visual-fidelity and benchmark scores on the same rollouts will test this. The third is answer bias. If most correct answers to revisit and occlusion questions are “unchanged”, a model that changes nothing looks consistent. The generator is therefore designed to draw questions whose correct answer is a change as well, from events the ledger will record between visits, and to report the balance. The fourth is an unsound reference. The contradiction checker will count conflicts in the canon before any question is drawn, and counterfactual questions will be kept only where the canon and the written rule set entail the answer. The fifth is a shortcut, a rule that works on the test at hand and fails under harder conditions^[8]. Here the shortcut would be the author's habits. One arc of one world cannot rule that out. Only a second, independently written world can, which is why it must be named before the build continues.

What it would not show

A pass would show that questions from one arc of a world written by the founder separate a strong model from a weak one. It would not show that the grade transfers to a world written by someone else. A model could pass by echoing the founder's habits, and only a second, independent world can test that.

It would not show that the canon is a complete account of its world. A canon is a story plus rules, so counterfactual questions are adjudicable only where the canon and the written rule set entail the answer, and that rule set is itself an unaccredited simulator.

Two models far apart in quality separate extremes, not near neighbours. The pilot reads one model through one adapter, so a pass would not show that the adapter suits other models. A clean pass would show consistency with a written world on these questions, not that a model understands the world.

Next step

Check that at least one competent world model and one weak baseline can be queried through the render-and-check adapter, without which the pilot cannot start. In parallel, draft the ledger schema for one arc and name the second, independently written world before any further event is typed.

What it would build

World ledger and question generator

References

Oefinger, C., Schäfer, F. R., Moller, K., Piccinini, M., Betz, J. (2026). Validate the Dream Before You Trust Its Verdict: Admissibility for World-Model Simulators. arXiv preprint 2607.07196.
Balci, O. (1998). Verification, validation, and accreditation. Proceedings of the 1998 Winter Simulation Conference, 41–48.
Sargent, R. G. (2008). Verification and validation of simulation models. Proceedings of the 2008 Winter Simulation Conference, 157–169.
Motamed, S., Culp, L., Swersky, K., Jaini, P., Geirhos, R. (2025). Do generative video models understand physical principles? arXiv preprint 2501.09038.
Piloto, L. S., Weinstein, A., Battaglia, P., Botvinick, M. (2022). Intuitive physics learning in a deep-learning model inspired by developmental psychology. Nature Human Behaviour 6(9), 1257–1267.
Weston, J., Bordes, A., Chopra, S. et al. (2015). Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. arXiv preprint 1502.05698.
Xiao, Z., Lan, Y., Zhou, Y., Ouyang, W., Yang, S., Zeng, Y., et al. (2025). WorldMem: Long-term Consistent World Simulation with Memory. Advances in Neural Information Processing Systems 38 (NeurIPS 2025).
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M. et al. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence 2(11), 665–673.

Cite as

Far Akin Research Labs (2026). A hand-written world as a test for world models. Version 1.0. Far Akin Research Labs. https://farakin.com/research/a-written-world-as-a-test

BibTeX

@techreport{farakin2026-a-written-world-as-a-test,
  title = {{A hand-written world as a test for world models}},
  author = {Far Akin Research Labs},
  institution = {Far Akin Research Labs},
  type = {Inquiry design},
  year = {2026},
  month = oct,
  note = {Version 1.0},
  url = {https://farakin.com/research/a-written-world-as-a-test}
}

Noticed an error or a missing source? Send a correction