In the weights or beside them

Models built for specific problems are one of the main ways machine learning meets a particular field. A team that maintains a standard, a taxonomy or a body of rules has two broad options. It can train a small model on that material. Or it can keep the material outside any model and hand it to a large general model at the moment of use, through retrieval, deterministic checks and a loop that repairs answers the checks reject. The choice decides what gets built, what it costs to run and keep current, and who can inspect the knowledge. It may be made on preference rather than evidence.

Work on factual knowledge favours keeping it outside. When the task is to recall information, retrieval has tended to beat fine-tuning^[1]⁠^[2]. Judgement is a different task. Deciding which rule applies to an untidy situation, rejecting a tempting option that does not fit, and asking for missing information when a confident answer would be unsafe are skills of application, not recall. A small model trained on many worked applications of a domain's structure might acquire them in a way that retrieval does not supply. Equally, a frontier model given the same structure and a checker might already do better. This inquiry is designed to tell those outcomes apart, and it treats the second as the default expectation.

Software architecture is the first test bed. Its pattern literature is public and long established. Each pattern comes with the forces that call for it and the costs it brings, and the common mistakes have recognisable shapes: a fashionable pattern applied where its forces are absent, a large pattern where a small one would do, a firm answer where the situation is underdetermined. Experienced practitioners can grade answers, and every test case can be published.

The same knowledge will be arranged two ways, once held inside a model and once consulted from outside, and the two arrangements will be judged on identical tasks.

Three lines of work bear on the question.

The first compares fine-tuning with retrieval as ways of adding knowledge. Ovadia and colleagues found that retrieval-augmented generation consistently outperformed unsupervised fine-tuning on knowledge-intensive tasks, for knowledge seen in training and for new knowledge, and that models struggled to learn new facts through unsupervised fine-tuning^[1]. Soudani and colleagues found that fine-tuning helped across entities of varying popularity but that retrieval surpassed it by a large margin for the least popular facts^[2]. Gekhman and colleagues report that fine-tuning examples carrying new knowledge are learned more slowly and, once learned, increase the tendency to hallucinate^[3]. All three study facts, mostly through question answering, not judgement.

The second trains structure into weights. TaxoLLaMA adapts a 7-billion-parameter model on WordNet with 4-bit quantisation and low-rank adapters and reports state-of-the-art results on 11 of 16 lexical-semantic tasks^[4]. OntoTune uses the SNOMED CT medical ontology to select training data for self-training and reports state-of-the-art results on hypernym discovery and medical question answering, with better preservation of the base model's knowledge than TaxoLLaMA^[5]. Zhao and colleagues report that models fine-tuned with 4-bit low-rank adapters, one per task, outperformed GPT-4 by 10 points on average across 31 tasks^[6]. Belcak and colleagues argue that small models suit the repetitive, specialised calls made inside agent systems^[7]. These tasks are taxonomic, conceptual or narrowly defined; none asks for judgement among competing designs.

The third keeps structure outside. Pan and colleagues set out three ways of combining language models with knowledge graphs, including graphs used at inference time^[8]. Han and colleagues compare ordinary and graph-based retrieval under one protocol and report distinct strengths for each^[9]. Kamoi and colleagues' survey concludes that self-correction works well where reliable external feedback is available^[10], which is what a deterministic validator offers. In software architecture, Dhar and colleagues report that retrieval and fine-tuning each improved the generation of architecture decisions, and that combining them with few-shot examples did best on a dataset of 4,911 decision records^[11]. De Luca and colleagues found language models competitive with a static analyser on several microservice anti-patterns but less reliable where explicit structural or cross-service evidence is needed^[12].

None of the sources read for this page compares a small model trained on a domain's structure with a frontier model given that same structure, validators and repair, on adjudicated judgement. A bounded search cannot show that no such study exists, so the search will be repeated first.

The inquiry will hold the knowledge fixed and vary where it lives.

Domain structure. An openly licensed catalogue of software-architecture patterns, to be built from published pattern literature: each pattern with its forces, consequences and typed relations to others (refines, conflicts with, commonly paired with, commonly misapplied in place of). Applicability rules and deterministic validators will be written over the catalogue. A validator will check, for example, that every named pattern exists in the catalogue, that two recommended patterns are not marked as conflicting, and that each recommendation states which of the pattern's forces it relies on. The catalogue will be released before any arm runs. Only public sources will be used.

Task family and fixtures. Pattern judgement: given a short, user-level design problem, recommend patterns, reject unsuitable ones, explain trade-offs, ask a clarifying question only when one is needed, and give an order of next steps. Fixtures will be written for the inquiry and published with a scoring rubric, not only a gold label. They will cover pattern choice, migration strategy, distributed reliability, domain modelling, anti-pattern repair, false friends and cases where clarification is the right answer. The set must include harder cases: underdetermined prompts, problems no single pattern solves, and sequencing traps. Three adversarial kinds will test for vocabulary without judgement: jargon decoys, specificity traps and suggestive-syntax cases. False friends are a regular fixture kind and are not counted among the adversarial kinds. Each fixture will have paraphrased variants. The held-out set will be frozen before any training data is generated.

Arms. Seven, each to receive the same catalogue content and return the same output schema:

1. Frontier model, no catalogue. A floor, not decisive.
2. Frontier model with the catalogue retrieved as plain text.
3. Frontier model with structured retrieval of pattern cards and relations.
4. Arm 3 plus the validators and a repair loop. The decisive comparator.
5. Small open-weight model fine-tuned on the catalogue's plain text.
6. The same small model trained on structure-derived supervision.
7. Frontier model consulting the arm 6 model as a critic.

Training. Arm 6 will be trained with low-rank adapters^[13] on traces generated from the catalogue's rules and relations: a situation, the forces present, the patterns that apply and those that do not, and a critique and repair of a wrong answer. Traces must pass the validators, a sample will be checked by a rater, and none may reuse a fixture scenario. Arm 5 is designed to isolate whether structure matters in training beyond domain text.

Scoring. Two raters, one from outside the lab, will score each answer blind to arm on five criteria of 0 to 2 points: pattern fit, trade-off reasoning, rejection of unsuitable options, clarification judgement and usefulness of the next steps. Deductions will apply for naming a fashionable pattern without its force, a firm recommendation on an underdetermined case, and distributed-systems advice that ignores idempotency, consistency, rollback or duplicate effects. To be recorded separately: false acceptance of jargon decoys, over-broad answers to specificity traps, invented doctrine, and the cost to build and run each arm.

Order. Arms 1 to 4 will run first. Arms 5 to 7 will be built only if the gates below pass.

Margins, floors and the agreement statistic will be written down before any model output is read. No values are set on this page.

1. Release gate. The catalogue, validators, fixtures, rubric and held-out split will be frozen and published before any arm runs. A repeated literature search that finds this comparison already made stops the inquiry, or narrows it to a stated difference.
2. Agreement gate. The two raters must reach the agreement floor on a calibration sample. If they do not after one round of rubric revision, the inquiry stops, because the scores cannot carry a result.
3. Headroom gate. If arm 4 scores within the pre-set margin of the rubric maximum on the hard fixtures, the set cannot separate a specialist from it. The fixtures will be rebuilt once. If the rebuilt set also saturates, the inquiry stops and reports that retrieval with validators leaves no room on this task family. That report is a result.
4. Positive answer. Arm 6, alone or as arm 7, exceeds arm 4 on mean rubric score by the pre-set margin, does not do worse on clarification, and has an equal or lower rate of accepting decoys and of over-broad answers. A win bought at much higher build or run cost will be reported alongside that cost.
5. Negative answer. Arm 4 scores at least as high as arms 6 and 7 on mean rubric score, or arm 6 uses the catalogue's vocabulary correctly while failing the adversarial cases. The negative answer will be published.
6. Inconclusive. A lead for arm 6 or 7 smaller than the margin, a lead on mean score with a loss on adversarial cases, or a ranking that reverses across fixture kinds or paraphrases. Reported as such and not turned into a claim.

Several designs were considered and set aside. Tasks that only test selecting a rule set were set aside because they test selection, not judgement. Closed rule sets were set aside because their fixtures could not be published. UX and product judgement was considered as the first family, but its labels are less settled; it is the candidate for a second family that tests transfer. Exact-match scoring was rejected for open-ended advice. A comparison against a frontier model with no catalogue alone would flatter any specialist, so that arm is kept only as a floor.

A positive answer would show that, for one task family, one catalogue, named models and stated budgets, structure in the weights added something that structure beside a frontier model did not. It would not show that small specialised models beat frontier models in general, nor that the same holds in a field whose rules are less settled than software design.

Public pattern literature is almost certainly in the training data of every model involved. That helps the frontier arms and the small model's base alike, and it means the inquiry will measure the arrangement of known material, not the learning of new material.

Rubric grading carries the raters' views of good design. Blind scoring, an outside rater and an agreement floor are meant to reduce that; they cannot remove it. Generated training traces may teach the surface of the fixtures rather than judgement, which the paraphrases and adversarial cases are designed to expose.

The comparison is only as fair as the comparators. A weak retrieval or validator arm would flatter the specialist, so arm 4 will receive at least the engineering effort that arm 6 does. Frontier models change, so any result will be dated to the model versions used.