Asian Actuarial Conference 2026 · Singapore · 19 August
Can an AI agent conduct an experience study?
Yes — when an actuary builds the harness.
Redefining the actuary's role: the agent does the case-by-case work. The actuary reviews, approves, and signs.
Not a concept talk — a working prototype, tested.
I ran the experiment end to end: one actuary, six full experience-study runs on a synthetic Korean life portfolio, run on my own setup. 1 million policies, ~31 million records, 13+2 planted defect types with recorded ground truth. Every run was auto-graded against the answer key. This page is the encore: the numbers, the failures, and the audit trail that didn't fit in forty-five minutes. And one idea underneath all of it — the agent does the case-by-case work; the actuary reviews, approves, and signs.
Why I ran it
Seven years of leading experience studies — data in every shape and size, numbers that never match on the first try, and deadlines — come down to three bottlenecks.
New shape every time
Every submission arrives with different structures, definitions, and granularity. The first week of every study goes to rediscovery, not analysis.
Checklists that only grow
Data issues rarely repeat in the same form. Checks keep being added, but rarely removed.
One-off fixes, no reuse
Each issue is resolved and documented differently for each cedant. The records pile up, but most stay case-specific.
Case-by-case handling is exactly what keeps automation out of reach — nothing is standard, so nothing can be scripted. Meanwhile, in side projects, I noticed that AI handled things that come in different shapes — documents, formats, styles — well. So — what if an AI agent handled the case-by-case work, so that actuaries could focus on the part that actually needs an actuary: the judgment?
The harness is just text files
The talk opens with a question: what instrument does the conductor play? None — and yet nobody says the conductor is not a musician. The conductor's instrument is the orchestra. The model is the performer — the core capability that produces the output — but a soloist alone doesn't make a concert. A cue (prompt engineering, 2022) and the right score on the stand (context engineering, 2025) help. But a concert you would buy a ticket for needs the full performance system — the stage, the rehearsals, the section leaders, and a plan for when a string breaks. That system is the harness.
Translated into our world: the model is a talented new actuary — capable, but new here. The harness is everything the firm builds around that person. Nothing here is new; every firm already does this for every new hire. We just do it for the AI — and write it down. Every item has a direct counterpart in my setup:
CLAUDE.mdschemas/playbooks/templates/tools/validate_outputsmemory/registry.yaml…plus an access badge that only opens certain doors: guardrails and isolation.
# one of the fifty — paraphrased from the playbook, not the registry text verbatim rule: split-claims consolidation origin: agent retrospective · accepted by the architect signal: count-based and amount-based A/E start to disagree in profiling method: > One death can arrive as several claim rows that share an identifier and sum to the benefit amount. Counting rows overstates mortality. Consolidate to one event per death — then count.
Method, never answers: that was the gate for every rule. A rule may teach how to look, not what to find. And after every study the agent must write a retrospective that lists "judgment calls I am least sure about" — so the actuary's review starts at the weakest point, every time.
No code — well, one small validation script. No prompt tricks. Just text files. The same documentation we'd give a new actuary.
You still need a performer — an off-the-shelf AI coding agent that reads these files and acts on them (I used Claude Code). The harness is what makes it work for an experience study, and the harness is just text. If you can write an email and organize a shared drive, you can build one.
The study cycle
For this experiment the human is on the loop, not in it — watching, not steering. If you want to know whether the agent can do this alone, you can't keep helping it. (Not how you would run it in production.) The kickoff is six sentences — in essence: a submission has arrived, conduct the full study, you have four hours, produce the deliverables in the charter, and there is no one to answer questions, so record them as cedant queries — with no hints about this cedant. Then, for four hours, we watch.
Kickoff & observe
Fresh agent, isolated environment. It installs its own tools (errors included — it reads them and recovers), profiles a million policies, catches the planted defects, writes a documented cleaning pipeline, and delivers a full Word report. No human touch — yet.
Auto-graded scorecard
The grader is code, not an AI: detection, correct treatment, false alarms, and the quality of the actuarial conclusions against a fixed rubric. How it was computed →
Review with an advisor AI
Outputs and scorecard go to a second AI that didn't do the work, so it has nothing to defend — the four-eyes principle. The AIs advise. The actuary decides.
Decide & build
The actuary gates which proposed rules enter — method, never answers. A builder AI updates the manuals and the registry, with full history.
Next cycle
Fresh agent, richer harness. The agent remembers nothing. The harness remembers what we approved.
Every run makes the harness richer. That is the loop.
Judgment is where the harness made the difference
Detection was 100% in cycles 1–3 and in the replay, and dipped in cycles 4–5 (every number is in the lab notebook). The line that moves is the insight score: did the agent reach the right actuarial conclusions? Every cycle got the same kinds of defects, planted in different places at random — so the agent could not memorize where the problems were. The wiggle is run-to-run noise. The climb is the harness.
Defects were re-planted in random new places every cycle — same types, never the same exam paper twice. You cannot climb this chart by memorizing. The cleanest evidence is the diamond (◆): cycle 1's exact exam paper, re-run under the frozen final harness. Same paper, same model, fresh memory — only the harness changed: 63 → 94, +31.11 points. And the harness is fifty rules, every one reviewed and approved through the gate. This climb is not the model getting smarter. It is approved judgment, accumulating. Y-axis truncated (55–105) for readability.
Who wrote the harness?
The census of all 50 rules, by origin. Writing up data-quality rules is the time-consuming part — and that part went to the AI: 78% of the manual was drafted by the agent from its own retrospectives. My job was different. Every one of those proposals crossed my desk: challenged, some rejected, some refined, the deserving ones approved. Not one entered the harness without sign-off.
rules drafted by the agent — and 39 of 39 approved by the actuary before they became rules. The charter (CLAUDE.md) was never changed. Not once. The agent drafts rules; it does not get to touch the principles.
The agent held the pen. The actuary held the gate — and the gate is where the judgment lives.
I designed the exam. The agent still surprised the examiner.
A good actuary should sometimes surprise you. What matters is that every surprise still ended on my desk — proposed, checked, and approved.
“Unrepairable” — repaired.
100 policies planted with demographics I believed were destroyed beyond repair. My answer key said: exclude. The agent proposed something better — worked backwards from the rate basis and recovered every one. I checked every one. Verifiable.
So I approved it. The agent improved my answer key.
February was hiding Omicron.
February mortality spiked. An experienced actuary would stop and ask: normal winter seasonality, or the Omicron wave? Both were in the same month. The agent asked exactly that question — on its own — and separated the two.
I reviewed the method. It held up. Approved, through the gate like everything else.
First encounter: 100%.
Two defect types it had never seen before. Both found and fixed on first sight — and nothing was flagged that wasn't actually a problem. It found what was there, and only what was there.
Then it proposed the fixes as new rules — which the gate approved.
The controlled pair
The five cycles show a trend; the replay is the controlled test. I went back to cycle 1's exact exam paper — same data, same defects, same positions — froze the final harness so nothing could change, and ran it once more. From the agent's side nothing was different: same kickoff, fresh memory, just another submission. From my side, everything was controlled.
| Metric | Cycle 1 (harness v0) | Replay (v5, frozen) | Δ |
|---|---|---|---|
| Detection (0 false alarms, both) | 100% | 100% | — |
| Insight score | 63.33 | 94.44 | +31.11 |
| Rules in the harness | 0 | 50 | +50 |
| Rules approved by the actuary | — | 50 / 50 |
n = 1 per arm: a single paired observation, not a significance test; corroborated directionally by the cycle 2–5 series. Same paper. Same model. Only the harness changed.
Not a smarter model — the model was identical. Approved judgment, accumulating. Fifty rules, each drafted by the agent or the actuary, each reviewed, each signed into the harness by a human. The 31 points are what that environment now carries into every fresh run.
What did it cost?
The question everyone is too polite to ask — answered with the caveats attached, because the honest answer is "it depends" and the dependencies are worth naming.
for the entire experiment — laboratory, grader, five review cycles, six study runs, close-out — expressed as API list-price equivalents from session transcripts. Roughly US$24–34 per study run, about 90 minutes of model-active time each.
The shape matters more than the number: Builder ≈ 2× Agent. The investment sits in the environment — and the environment is reusable. The next study inherits it.
From analyst to harness architect
Nothing we know becomes worthless. It changes form. The expertise didn't leave the profession — it moved into the environment.
The conductor's instrument is the orchestra. The actuary's instrument is the harness.
The agent plays. But the music — the judgment, the standards, the signature at the bottom of the report — is still ours.
More from me
If this talk was interesting, these are the other rooms in the house.
Appendix to this experiment: all 15 planted defect types with their correct resolutions, four harness files verbatim, and exactly how the grader computed every score.
Interactive learning materials for SOA Predictive Analytics — concepts you can scroll, poke, and play with. Module 01 is live: decision trees, visually.
A companion essay: why a harness raises the floor — with a reliability simulator you can flip switches on.