Notes · 01 · Harness engineering

The harness raises the floor.

Why the same AI model is chaos for one team and a reliable colleague for another — and why actuaries are well placed for the discipline that closes the gap.

Picture two teams licensing the same model, the same version, the same settings. Six months later, one team says agents are a toy that can't be trusted with real work. The other has an agent producing work they would put in front of a client. Same model. Opposite verdicts. I have seen versions of both.

The difference is not the model, and it is not prompt-writing skill. It is everything around the model — and in early 2026 the software community finally gave that everything a name: harness engineering. The lineage is short. Around 2022 we learned to phrase requests well (prompt engineering). Around 2025 we learned to feed the model the right information at the right time (context engineering). The harness is the third step: the role definition, working rules, templates, self-checks, and memory that surround the model — so that the system behaves reliably even though the model, run to run, does not.

Most explanations of harness engineering stop at that definition. I want to make a narrower, more useful claim — one you can test with your fingers in about thirty seconds:

A harness barely raises the ceiling. It raises the floor. And professionals don't sign ceilings — they sign floors.

Stop looking at the average

A language model is probabilistic. Ask it to do the same study twice and you get two slightly different performances — most days good, some days inexplicably not. So the honest unit of analysis is never "the run I saw in the demo." It is the distribution of runs. This is the single most important mental shift, and it happens to be a shift actuaries do not need to make, because distributions are our native tongue.

Below is a toy model of that idea. Each bar chart shows 200 simulated runs of the same agent on the same task, scored 0–100. The four switches are the four working parts of a harness. Turn them on and watch what moves and what doesn't.

The reliability simulator — same model, different harness
200 simulated runs, scored 0–100. Gray bars: no harness at all. Indigo bars: your current configuration.
30405060708090100
No harness (baseline)Current configuration
Mean score (the ceiling story)
5th percentile — the floor
Worst run out of 200
Runs you'd sign off (≥ 80)

An illustrative toy model, built to make one idea tactile — not data from a real experiment. For measured numbers from six real experience-study runs, see the AAC 2026 experiment.

If you flipped all four switches, you saw the pattern this whole essay hangs on. The mean crept up a little — pleasant, unremarkable. But the 5th percentile climbed roughly three times as far, the worst run stopped being a catastrophe, and the share of runs you would actually sign went from a coin flip to nearly all of them.

That is the harness premium, and notice what it is not: it is not a smarter model. The ceiling was already acceptable — the best no-harness run scored well, which is exactly what a demo shows you. The problem was never capability. For work like this, capability is rarely the bottleneck. Consistency is. And consistency is a property of the distribution, which means it is a property of the environment — the only part you control.

Why this is actuarial home ground

Read that last section again and replace "agent" with "junior actuary," and you'll notice the profession has been doing something very like harness engineering for a long time — we just never needed a name for it. An onboarding pack is a charter. Work manuals are playbooks. Peer review before release is a self-check gate. The experience-monitoring cycle — observe, measure against expected, update the basis, document why — is a retrospective loop with actuarial standards of practice wrapped around it.

But the deeper affinity is conceptual, and it's the reason I think actuaries can be unusually good at this. Harness engineering is, at its core, variance management under a sign-off constraint: you are paid to make the tail of a distribution acceptable and to document your reasoning so someone else can audit it. That is close to a description of actuarial work — we usually just point it at mortality instead of at a model's behavior. Credibility theory even gives us the honest way to think about a new agent: little history, wide uncertainty, low credibility; every documented cycle earns it more. Few professions walk into this discipline with the mental furniture already installed.

The engineers named the harness. Actuaries have been working inside one for years.

There is one caution, and it comes from the same analogy. A harness is an asset that compounds — every run should leave it richer — but it compounds in whatever direction you let it. Rules that encode answers ("the factor is 1.45") rot instantly and overfit forever. Rules that encode method ("when count-based and amount-based A/E disagree, check for split claims before counting") appreciate. The gate you keep as the human on the loop is exactly the gate a good reviewer keeps: method may enter the manual; answers may not.

The expertise doesn't leave the profession. It moves into the environment. I've made that argument with a real experiment, real defects, and a real audit trail; if this essay was the theory, the experiment is the evidence.