Judgement, conversation and evidence
1 design · 7 questions
What is an instrument actually measuring, and does it support the decision made from it?
Raters, judges, interviews, surveys and diagnostics all turn behaviour into evidence, and any of them can mislead. A score can sit inside ordinary expert disagreement. An interview can change the person it studies. This area checks the ruler before it is used: what an instrument measures, how reliable its readings are and which decision they support.
Checking the instruments
The floor a rater is judged against
How much do qualified experts disagree, and is that the floor any simulated rater should be judged against?
Simulated raters are usually scored against an average of human ratings. Qualified experts also disagree with one another, and that sets a floor: no rater should be expected to agree with one expert more closely than a second expert does.
Does a judge's role predict how others in the same role rate quality?
Quality ratings are often pooled across judges, or split by judge group, without a test of whether the grouping carries signal. If a judge's occupational role does, ratings from one judge predict others in the same role, and a simulated judge would need that role.
How should errors shared between automated judges be corrected when the reference labels are themselves noisy?
When automated judges make the same mistakes their agreement is not independent evidence, and noisy reference labels leave no answer key to correct against.
Interviews and decisions
What a question does to the person who answers it
Does an interviewer change the person it studies?
A planner that rehearses interview questions on a simulated person can be rewarded for changing the person instead of learning about them. The design compares four ways of holding that person, scored by hidden attributes recovered within a fixed question budget.
When budget, authority or evidence changes, what does a decision-maker try to keep constant?
Models of people usually predict a choice from past choices. A control account predicts it from what a person tries to hold steady, such as spending exposure, approval risk or enough evidence, and expects that to carry over when circumstances change.
When a new option appears, which standard do people start judging by?
An unfamiliar option can lead people to adopt a new criterion for a whole category instead of weighing it on the old ones.
Evidence on synthetic respondents
What has actually been validated
What have published studies of simulated respondents validated?
Simulated respondents are often called valid, but the word covers different checks: agreement on averages, on spread, on individual answers or on a decision. A model may also have read the published results it is said to reproduce.
Does a diagnostic built on planted defects find why a model fails better than ordinary tools, in two fields?
A diagnostic is usually trusted because its account of a failure sounds right. A defect planted at a known place gives a ground truth the diagnostic must find.