Skip to content
Menu
Essay

Far apart, possibly akin

Why one lab studies learning machines, human minds and simulated worlds together, what it would take for them to be akin, and how that claim could fail.

Written by
Far Akin Research Labs
Published
Generated cyanotype print of two stones on wet sand, one smooth and rounded, the other angular and faceted like a crystal.Generated photograph of a white plaster spiral shell and a white card spiral staircase far apart on a table, joined at matching points by a vermilion thread.Generated render of two unlike islands far apart above the waterline, joined under water by one shared ridge.
Plate ITwo unlike stones on one shore.A shell and a stair. Same spiral, different reasons, and the reason is what the essay asks about.Apart at the surface. Joined below it, perhaps.Generated image in a cyanotype treatment. Far Akin art direction.Generated image. Far Akin art direction, Datum look.Generated image. Far Akin art direction, Swath look.

One problem in three places

Learning machines, human minds and simulated worlds are mostly studied in separate fields, by people with different training and different journals. They share an awkward property. In each, the part that does the work cannot be opened and read. The weights and activations of a trained network can all be recorded, but what they encode cannot be read off by inspection. A person offers reports, and a report is not the process behind it. The rules of a simulated world reach an agent inside it only as consequences.

The pairing is not new. Wiener's Cybernetics (1948) carried the subtitle Control and Communication in the Animal and the Machine[1], which put the animal and the machine under one theory of control and communication. Five years earlier, Kenneth Craik had argued that an organism carries a “small-scale model” of external reality and of its own possible actions, and uses it to try options before acting[2]. Tolman inferred a map in rats from where they ran when a familiar route was blocked[3]. Ha and Schmidhuber (2018) trained an agent inside a generative model of its environment and moved the resulting policy back into the real one[4]. The vocabulary changes across the decades and the structure recurs: a system holds a model of its world and acts on what the model predicts.

That a description recurs suggests that the description is natural. It does not show that the things described are the same.

What “akin” would have to mean

The lab's name states a relation and leaves it unproved. To be a claim and not a mood, the relation needs content. Three kinds of content are available, and each can fail.

Shared mechanism. The systems compute the same thing in the same way. Conant and Ashby showed that every good regulator of a system must be a model of that system[5], a result about regulators whatever they are made of. Whether a given brain or a given trained network meets the conditions of the proof is an empirical matter. Marr's three levels mark where such claims go wrong: the problem a system solves, the representation and algorithm that solve it, and the hardware that runs them[6]. Two systems can be akin in what they compute and unlike in how. A kinship claim has to name its level, and a claim made at the first level fails if it is read at the second.

Shared method. The same test, applied to both, returns answers that mean the same thing. This fails when a test is valid for one system and empty for the other, for instance when it presumes a quantity the second system does not defend.

Shared failure. The systems break under the same conditions for the same reason. This fails when the breaks look alike and have different causes. One candidate is already in the literature. A probe trained to read a variable out of a network's activations may have learned the task itself, which is why probes need control tasks[7]. Decoding a variable from neural activity invites a related step, from “a reader can extract it” to “the system represents it”, and a philosophical analysis argues that decodability is a poor guide to what is represented[8]. Whether these are one problem or two resemblances with different causes is the kind of question the name asks.

At a fixed level, the claim fails if the resemblance disappears under a stricter control, if the test that works on one system has no counterpart in the other, or if the failures share a name and not a cause.

Two interiors, no assumed equivalence

Human inner life and the hidden state of a machine are both treated as serious territories. Neither is a metaphor for the other. A person has first-person access, a developmental history and more than a century of experimental constraint. A network has state that can in principle be read exactly and altered at will, and no first-person report whose standing is settled. These differences inform the comparison as much as any similarity does.

The stance is symmetric in seriousness and silent on equivalence. It does not assume that a model has an inner life, that a mind is a computation of the kind a network performs, or that either is the other under another name. It assumes that each interior has structure that outside evidence bears on, and that likeness is measured case by case, with the cases where it fails kept on record.

Knowing an interior from outside

Neither interior can be inspected directly, so the evidence comes from outside. Behaviour, probes and disturbances recur across the fields above.

Behaviour is the oldest. In the experiment Tolman reports, rats trained to run down one alley found it blocked and faced a fan of new paths; the preponderant choice was the path pointing towards where the food had been[3]. That separates a stored route from a stored layout without opening anything. Probes read a variable from hidden state and need controls, as above.

Disturbance comes from control theory. Powers argued that behaviour is the control of input: a system produces whatever output holds a sensed quantity steady against disturbances[9]. The corollary is a test. If a quantity is defended, pushing on it should draw an opposing output, and a quantity the system ignores should not. A system that holds a quantity steady shows nothing varying in that quantity while undisturbed, so watching it at rest says little about what it holds. A disturbance is the behavioural twin of an intervention on hidden state: both push on the inside and record what holds.

Generated cyanotype print of two hands on a wooden spirit level, one pushing it off true and the other bringing it back.Generated photograph of a level white plaster beam on a wooden pivot with matching white weights hanging at mirrored positions; vermilion beads mark the pivot and both hanging points.Generated render of a pontoon on a swell whose inner ballast shifts against the wave so its deck stays level.
Plate IIDisturbance and compensation: the correction mirrors the push, and the pair is the evidence.Disturbance and compensation: the correction mirrors the push, and the beam stays level.Disturbance and compensation: the correction mirrors the push, and the level holds.Generated image in a cyanotype treatment. Far Akin art direction.Generated image. Far Akin art direction, Datum look.Generated image. Far Akin art direction, Swath look.

Two inquiries build on these methods. What a learned policy holds still takes the disturbance test to robot policies and asks whether the variables a policy defends predict where it fails better than a probe does. Instruments for a workspace asks whether the measures of a model's supposed internal workspace can be trusted before anything is concluded from them. Both are designed, with written stop rules. Neither has run.

Old questions, new instruments

Some precise questions stalled for want of an instrument. Fechner's Elemente der Psychophysik (1860) gave psychophysics its name and its methods for locating thresholds[10]. Powers's test needs a system that can be pushed continuously and a record fine enough to show both the push and the answer. Both demand many controlled trials. A human observer tires and learns the experiment. A frozen model can be asked the same question thousands of times with one stimulus varied in exact steps, and a policy in a simulator can be disturbed on every step of a trajectory.

Generated worlds supply the other half. A world model can be graded exactly only against a world whose rules are known, and a world written down in full is such a world. A written world as a test designs that grading and asks whether questions drawn from one written world can tell a strong world model from a weak one; whether a grade carries over to a world someone else wrote needs a second world and is left open. A policy trained inside a model's own rollouts, as in Ha and Schmidhuber's agent[4], is only as good as what the generated world preserves, and checking what it preserves needs a reference whose rules are known.

None of this is a result. The premise is that unlike systems may share a mechanism, a method or a failure. The test is to state the claim at a level and put it to behaviour, probe and disturbance against a world whose rules are known. A kinship that survives that is worth the name. One that breaks leaves a record of where, which is also an answer.

References

  1. Wiener, N. (1948). Cybernetics: or Control and Communication in the Animal and the Machine. Hermann, Paris; The Technology Press, Cambridge, MA; John Wiley & Sons, New York.
  2. Craik, K. J. W. (1943). The Nature of Explanation. Cambridge University Press.
  3. Tolman, E. C. (1948). Cognitive maps in rats and men. Psychological Review 55(4), 189–208.
  4. Ha, D., Schmidhuber, J. (2018). World Models. arXiv preprint 1803.10122.
  5. Conant, R. C. and Ashby, W. R. (1970). Every good regulator of a system must be a model of that system. International Journal of Systems Science 1(2), 89–97.
  6. Marr, D. (1982). Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W. H. Freeman (reissued by MIT Press, 2010).
  7. Hewitt, J., Liang, P. (2019). Designing and Interpreting Probes with Control Tasks. Proceedings of EMNLP-IJCNLP 2019, 2733–2743.
  8. Ritchie, J. B., Kaplan, D. M. and Klein, C. (2019). Decoding the Brain: Neural Representation and the Limits of Multivariate Pattern Analysis in Cognitive Neuroscience. The British Journal for the Philosophy of Science 70(2), 581–607.
  9. Powers, W. T. (1973). Feedback: Beyond Behaviorism. Science 179(4071), 351–356.
  10. Fechner, G. T. (1860). Elemente der Psychophysik. Breitkopf und Härtel, Leipzig.

Cite as

Far Akin Research Labs (2026). Far apart, possibly akin. Far Akin Research Labs. https://farakin.com/notes/far-apart-possibly-akin

BibTeX
@misc{farakin2026-far-apart-possibly-akin,
  title = {{Far apart, possibly akin}},
  author = {Far Akin Research Labs},
  howpublished = {\url{https://farakin.com/notes/far-apart-possibly-akin}},
  year = {2026},
  month = oct,
  url = {https://farakin.com/notes/far-apart-possibly-akin}
}

Noticed an error or a missing source? Send a correction

Related inquiries

  • Generated cyanotype print of a heron standing in rippling shallows, its head held level above the moving water.Generated photograph of a white gimballed compass model on a steeply tilted plaster deck; the inner ring stays level, its centre marked by a vermilion bead above a vermilion pin.Generated render of a heaving sea surface made of points, with one slender buoy standing upright at the centre.
    Designed

    What a learned policy holds still

    Does a frozen robot policy resist disturbance along a few specific scene directions more than a decoding probe on its internal state would predict, and does that resistance predict which held-out disturbance conditions the policy survives?

    Cybernetics and adaptive systems · Robotics and embodied intelligence

  • Generated cyanotype print of three weighted lines lowered into one dark pool, their rings of ripples overlapping but not quite matching.Generated photograph of three different white probes lowered into one plaster basin, each tip marked by a vermilion bead; the three beads land near each other but not together.Generated night-survey render of three side-by-side strips of mapped sea floor that show the same ridge, nearly aligned.
    Designed

    Can the instruments for a model's workspace be trusted?

    On open-weight language models that already exist, do three tests of an internal “workspace” give repeatable readings (test-retest r of at least 0.8), and do they rank the same models alike or differently?

    Latent space and internal cognition · Cognitive science and psychology · Machine-learning foundations

  • Generated cyanotype print of a folded paper model of an imaginary island with paper trees, small paper houses and a paper lighthouse.Generated photograph of a small all-white hand-made world on a wooden board, with plaster hills, a paper river, card houses and a few vermilion pins at particular places.Generated render of a small hand-made landscape model of islands and a delta, fully mapped and floating in darkness.
    Designed

    A hand-written world as a test for world models

    Can test questions generated automatically from one arc of a hand-written fictional world tell a competent world model from a deliberately weak one, on at least two of five question classes, in a way a fidelity score does not?

    Simulation and world models