Learning systems and mechanisms
4 designs · 3 questions
What does a network hold, which of it does it use, and what could it report?
A probe can decode information a network never uses, and a network can act on information it cannot report. This area keeps present, used and reportable apart and tests each instrument before trusting its readings. It covers workspace measures, fact revision, agent memory and whether a learned state beats careful inference.
Representation and use
What a network holds against what it uses
Can the instruments for a model's workspace be trusted?
Recent preprints propose ways to measure a reportable workspace inside language models. Before any model is trained to have one, this study asks whether three such measures give repeatable readings on eight existing open-weight models, and whether they rank the models alike.
Can a model be trained so that its workspace is easier to inspect, at a bounded cost to what it can do?
If a workspace can be measured reliably, the next question is whether one can be built. A model whose stated reasoning is easier to check against what drives its behaviour matters to anyone who oversees a system from outside.
Memory and revision
How a stored fact changes, and what changes with it
When one fact changes, do the facts that depend on it follow?
Correcting one fact in a model should also correct the facts that rest on it. This study measures how updating of dependent facts, and wrong changes to unrelated facts, vary as the number of dependents grows, on a public record of superseded facts.
What should a memory keep apart?
An agent that works with someone over months is told things that later change, conflict or prove to be one person's view. This study tests whether a memory that records those differences as separate states earns its cost against simpler memories at matched budgets.
Can a memory foresee what will break it?
Later events can overwrite or blur a stored memory, and the failure usually shows only when the memory is needed. This study tests whether a trained model can forecast which of two later disruptions will break a memory, better than an equal-information rival.
Learning against inference
When a learned state earns its keep
Beyond settings small enough to enumerate, does a learned internal state ever beat careful Bayesian inference?
When a rival holds the true law and the same data, a learner can at best tie it. A learned state can earn its place only where exact inference is out of reach, structure is withheld and acquiring information has a cost.
Does a network acquire an abstraction, or only receive its structure from how it was trained?
A network trained with supplied relations may take its structure from the training signal without acquiring an abstraction it can use on new cases.
Open problems in this area
- Do proposed measures of an internal workspace in language models give repeatable readings, and do different measures rank models alike?
- When one fact in a model's weights is edited, how does correct updating of the facts that depend on it change as the number of dependents grows?
- Can a model forecast, before new learning happens, which of its stored items that learning will disrupt?
- Which structure beyond plain retrieval improves an agent's long-term memory enough to justify its cost?
- For which kinds of domain question does retrieval over a graph outperform plain text retrieval, and does a graph built from a curated ontology change the answer?