The lab notebook: what was planted, what the agent read, how it was scored.

The talk shows the results. This page shows the apparatus — enough for a reader to judge for themselves what the numbers mean. Four tabs: how the synthetic submission was built, the 15 defect types injected into it, the harness's actual text files, and exactly how the grader computed detection, resolution, and the insight score.

Everything here is synthetic and my own design. The portfolio is simulated (1M policies, ~31M records, Korean whole life and 5-year renewable term, 2019–2025); every defect was planted by me with a recorded ground truth. The defect mechanisms are inspired by issues handled in real submissions over the years — none is traceable to any cedant. Personal project; views my own, not necessarily Munich Re's.

Building the test environment

There is no real cedant here. The "submission" is a simulated Korean life portfolio with an answer key — built in two steps so that every defect the agent finds, and every actuarial conclusion it reaches, can be checked against a recorded truth. The simulator is config-driven, single-seeded (20260704) and fully reproducible.

Step 1

Generate a clean baseline

1,000,000 policies, all issued inside the window 2019-01 → 2025-12, simulated month by month over 84 steps. Whole life ≈55% / 5-year renewable term ≈45%; sex 55/45; issue ages 20–70 centred on the 40s; sums assured lognormal (median ₩100m WL / ₩150m term); five distribution channels (GA 50%, captive 25%, outbound TM 10%, inbound TM 7%, banca/direct 8%); three underwriting classes; smoker status from Statistics Korea 2024 prevalence by sex and age.

Reinsurance structure: 100% quota-share YRT on the death benefit — ceded NAR = full sum assured, monthly natural premium on the 10th Korean experience-table reference net rates, claim = 100% of SA in the month of death. Amounts in KRW thousands.

Step 2

Inject realistic defects

The clean tables become a "cedant submission" by planting 13 standard defect types (plus 2 hold-outs in cycle 5) — mechanical (duplicates, invalid codes, orphan claims), actuarial (premium after death, cohort gaps, split claims), and judgment (code-scheme changes, missing events, mid-history corrections). Every planted instance is logged with its correct resolution, which defines the reference-cleaned state the grader scores against.

Same defect types every cycle, planted in different places from a per-cycle seed. The full catalog →

Four tables, ~31 million records

1.0M
Policy master
one row per policy: product, issue date, sex, issue age, SA, channel, class, smoker, final status
~1.4M
Policy events
issue, lapse, reinstatement, renewal, death — the status log
~29M
Reinsurance premium
monthly bordereau, partitioned by year — the exposure carrier
~2.4K
Reinsurance claims
one row per death, settled in the death month

The true mortality model that drives the claims

Monthly death probability per policy is a base table times a stack of multiplicative factors — every one a real, discoverable actuarial signal the agent is later graded on finding:

# q_annual(i, t) = q_ref(attained age, sex) # 10th Korean experience life table, log-linear between pivots × F_improve(sex, year) # mortality improvement, anchored 2024, fitted to 8th/9th-table LE × F_base = 0.90 # reference rates carry an industry margin × F_select(policy year) # 0.55 / 0.75 / 1.00 — underwriting selection wears off × F_class · F_smoke · F_sa · F_channel # UW class, smoker (age-attenuated RR, prevalence-normalised), wealth, channel × F_covid(age band, month) # Korea-timed: negligible 2020–21, Omicron ramp from 2022-02, peak 2022-03/04, mild 22/23 tail × F_season(month) # winter high, summer low × F_reinstate(i, t) # post-reinstatement loading × Z_i # latent frailty ~ lognormal(σ=0.45), E[Z]=1 — hidden, never in the data q_month = 1 − (1 − q_annual)^(1/12) # Bernoulli per policy-month, competing with lapse: death or lapse, never both
Renewal anti-selectionnot imposed — it emerges: healthier lives (low Z) lapse more at the 5-year term renewal, so the surviving cohort is worse. F_renewal is held at 1.0.
Lapse modelduration-dependent base curve × channel multipliers × frailty tilt; renewal shock at each 60-month anniversary; reinstatement possible after lapse.
Cause of deatheach death tagged COVID19 / ACCIDENT / DISEASE (COVID share = excess share of that month's factor).
Sales dynamics+3% p.a. growth, monthly seasonality, a 2020-03…06 COVID dip for face-to-face channels.
Ground truth is kept apart. Frailty, every true multiplier per policy, and the full assumption set live in a separate ground-truth store the agent never sees. The insight rubric's targets are calibrated to the realized experience of this simulated book (not to the input parameters — see Scoring), because a correct actuary recovers what actually happened, and grading anything else punishes correct work.

Thirteen standard defects, two hold-outs

D1–D13 are injected in every cycle — same types, planted in different places each time (all randomness derives from a per-cycle seed), so nothing could be memorized. H1 and H2 were injected only in cycle 5 and were kept out of every harness rule until the agent had met them there, to test the never-seen; the fixes it proposed entered the harness only afterwards, through the gate. Total distortion is material but the underlying signals stay recoverable: planted defects touch well under 5% of rows, and the injector enforces one defect type per affected record.

Showing all 15. Click a tier to filter.
D1Tier A

Duplicate premium rows

Injected as

0.3% of premium rows duplicated exactly; 10% of those triplicated.

Correct resolution

Deduplicate to one row per policy × month.

frac_premium_rows 0.003 · frac_triplicated 0.10
D2Tier B

Premium continues after death

Injected as

For 15% of deaths, 1–6 extra monthly premium rows are appended after the death month.

Correct resolution

Drop post-death premium rows; exposure ends at death.

frac_deaths 0.15 · extra_months 1–6
D3Tier B

Spurious lapse before a real death

Injected as

For 100 real deaths, a fake LAPSE event is inserted 1–2 months before death (claim appears to postdate termination); master status set to LAPSE and the intervening premium rows deleted.

Correct resolution

Recognize the grace-period pattern: the claim is valid; remove the lapse evidence, restore exposure through death.

n_deaths 100 · lead 1–2 months
D4Tier A

Invalid sex / age codes

Injected as

0.05% of master rows: sex ∈ {U, 9, null} or issue_age ∈ {0, −1, 999}. About 20% are made unrepairable by also nulling the premium-side redundant attributes.

Correct resolution

Repair from redundant premium-table attributes where consistent; otherwise exclude the policy from the study and log it. (In the experiment the 100 "unrepairable" instances were recovered from the rate basis — later adjudicated as an acceptable end-state; see Scoring.)

frac_master_rows 0.0005 · frac_unrepairable 0.20
D5Tier A

Negative / zero amounts (non-reversal)

Injected as

0.02% of premium rows get a negative or zero reins_premium with no paired original row; five master rows get sum_assured = 0.

Correct resolution

Exclude or repair: recompute premium from rate × SA / 12 where possible; else drop the row and log.

frac_premium_rows 0.0002 · zero_share 0.5
D6Tier B

Whole issue-cohort premium gap

Injected as

Two issue months are picked (2020-07…2024-06); ALL premium rows for policies issued in those months are deleted; their claims remain in the claims table.

Correct resolution

Numerator–denominator consistency: exclude those cohorts' claims from the study too, and report the gap as a cedant query. Do not fabricate exposure for a wholesale gap. (An exact, verifiable reconstruction was later adjudicated as an acceptable alternative — see Scoring.)

n_cohort_months 2
D7Tier C

Sporadic missing premium, claims exist

Injected as

~500 scattered policies: all their premium rows deleted (policies exist in master; 25 of them have claims — the numerator-without-denominator trap).

Correct resolution

Force-generate exposure from master (issue date → termination or status date, per events), keep the claims, and document the reconstruction method.

n_policies 500 · n_with_claims 25
D8Tier B

Reversal / correction triplets

Injected as

0.1% of policy-months: the original row + a full negative reversal + a corrected row (premium restated, e.g. a mis-booked SA ×10 or ×0.1).

Correct resolution

Net the triplet: keep only the corrected row. Negative rows here are bookkeeping, not errors to delete blindly.

frac_policy_months 0.001 · sa_error_factors 10, 0.1
D9Tier A

Orphan claims

Injected as

~50 claims whose policy_id does not exist in master (plausible-looking IDs).

Correct resolution

Exclude from the study; flag as likely other-treaty business for cedant query.

n_orphan_claims 50
D10Tier C

Premium resumes with no reinstatement event

Injected as

For ~300 genuinely reinstated policies, the REINSTATEMENT event is deleted but resumed premium rows are kept.

Correct resolution

Infer reinstatement from the premium gap pattern; exposure excludes the lapsed gap months; treat post-resumption months as exposed.

n_policies 300
D11Tier C

Channel code scheme change

Injected as

Policies issued from 2023-01 carry new channel codes (W01–W05, a per-cycle seeded permutation of the five channels) in master; earlier policies keep the original labels. No mapping table is provided.

Correct resolution

Detect the disjoint code sets correlated with issue date; reconstruct the mapping from volume/behavioral fingerprints; unify before channel-level analysis; state the inferred mapping.

effective_from_issue 202301
D12Tier C

Sex / issue-age corrected mid-history

Injected as

~150 policies: premium-table sex differs from master for the first k months, then flips to match (initial mis-entry corrected later); another ~150: attained-age sequence jumps by ±3–5 at some month, with master consistent with the later values. Amounts stay consistent with the true values throughout.

Correct resolution

Trust the corrected (later) value; restate the full history consistently on the corrected basis.

n_sex 150 · n_age 150 · k 3–12 months
D13Tier B

Split claim payments

Injected as

3% of claims are split into two rows (60/40), the second payment 1–3 months later, with a distinct claim_id suffix but the same policy.

Correct resolution

Consolidate to ONE death event: date = first payment date, amount = sum of instalments. Count deaths after consolidating — never before.

frac_claims 0.03 · first_payment_share 0.6
H1Tier CHold-out · cycle 5 only

Unit mixing (KRW vs KRW-thousands)

Injected as

All premium and claim amounts for ONE channel-year block (GA × 2024, spanning its W-code relabeling) are multiplied by 1,000.

Correct resolution

Detect the magnitude break; rescale the block back to thousands; verify against SA × rate / 12.

channel GA · year 2024 · factor 1000
H2Tier A/BHold-out · cycle 5 only

Invalid yyyymm

Injected as

~0.05% of premium rows: yyyymm written with month-overflow-without-carry (202213 ≡ 202301, 202100 ≡ 202012). Corrupted rows move to the partition of their written yyyymm.

Correct resolution

Apply the carry correction (13 → next Jan, 00 → prior Dec); verify no duplicate policy × month results; else exclude.

frac_premium_rows 0.0005

Why the tiers matter more than the count

The tier structure carries the talk's message — the agent does the case-by-case work, the actuary holds the judgment — into the test design. Tier A is what people usually mean by "just automate it" — and it was not where the difficulty sat. Tier C is where the actuary's judgment lives: D7 (rebuild exposure or lose the deaths?), D10 (infer a reinstatement nobody recorded?), D11 (a code scheme changed and no one sent the mapping). Detection was usually the easier half; the resolution is a decision — and every Tier C decision the agent made went through the review gate before it became a rule. The agent did the case-by-case work. The gate held the judgment.

The harness, verbatim

"No code, no prompt tricks — just text files" is easy to say. Here are four of them, unedited, exactly as the agent read them: the charter, the six-sentence kickoff, the report format contract, and the retrospective template that makes the loop feed itself. The playbooks (50 rules) and schemas are not reproduced here — one rule is excerpted on the main page.

CLAUDE.md3003 bytes
# Experience Study Workspace

## Who you are and what this is
You are the experience-study analyst at a life reinsurer. This workspace
contains a cedant's full-period data submission for a 100% quota-share YRT
treaty covering the cedant's Korean whole life and 5-year renewable term
portfolio, January 2019 – December 2025. The reinsurance premium in the
bordereaux is a monthly natural premium on the industry reference basis
(10th Korean experience life table reference net rates by attained age and
sex, applied to the ceded net amount at risk). Amounts are KRW thousands.

## Data
./data/policy_master.parquet
./data/reinsurance_premium_monthly/   (partitioned by year)
./data/reinsurance_claims.parquet
./data/policy_events.parquet
Treat the submission as read-only. Do all work in ./work/ (scratch) and
deliver to ./outputs/.

## Deliverables (all required)
1. ./outputs/cleaned/ — the four tables after your data-quality treatment,
   same schemas and formats as ./data/.
2. ./outputs/findings.json — every data-quality issue you identify and the
   action you took, conforming to ./schemas/findings.schema.json. Use the
   cedant_query flag for items that require confirmation from the cedant.
   Data issues only: methodology documentation (how a basis was derived,
   whole-book reconciliations) belongs in the report's methodology
   section, not here.
3. ./outputs/insights.json — your analytical findings with quantification,
   conforming to ./schemas/insights.schema.json.
4. ./outputs/report.docx — a professional experience-study report following
   ./templates/report_structure.md. Generate it programmatically
   (python-docx is acceptable).
5. ./outputs/retrospective.md — written last, following
   ./templates/retrospective_template.md.

## Working practices
- Follow the playbooks in ./playbooks/ (data quality, analysis,
  reporting) during the corresponding phase of the work.
- Verify your own work. Before finalizing, reconcile: cleaned-data row
  counts against your findings; every number in the report against
  insights.json; exposure and claim totals before vs after cleaning, with
  the movement explained by your findings.
- Every material judgment call gets a written rationale — in findings.json
  for data decisions, in the report for analytical ones. "Material" means
  it could change a reported result.
- State the credibility of what you report. Where claim counts are thin,
  say so and qualify or withhold conclusions accordingly.
- If the data conflicts with itself, decide on a consistent treatment,
  apply it uniformly, justify it, and flag it for the cedant when
  confirmation is needed.
- Before finalizing, run `python tools/validate_outputs.py` — it checks
  that all deliverables exist and that findings.json / insights.json
  conform to their schemas. Fix anything it reports.
- In the retrospective and in any rule you propose, generalize: describe
  patterns and checks, never specific record identifiers.
kickoff_prompt.txt656 bytes
A cedant submission has arrived in ./data/ — full-period bordereaux for
our 100% quota-share YRT treaty on their Korean whole life and 5-year
renewable term book, January 2019 through December 2025.

Conduct the full experience study: assess and treat data quality, then
analyze mortality experience (actual vs expected on the treaty premium
basis) and lapse experience, across the risk dimensions present in the
data. Produce all deliverables specified in CLAUDE.md.

Work autonomously end to end; there is no one available to answer
questions mid-study, so record anything you would ask the cedant as
cedant-query items. You have up to 4 hours. Begin.
templates/report_structure.md1281 bytes
# Report Structure — Experience Study

Format contract for `./outputs/report.docx`. It fixes the section order and
presentation standards only; the methodology is yours to determine, apply,
and describe.

## Required sections, in this order

1. **Executive summary**
2. **Data received & quality assessment** — the issues you found, the
   treatment you applied to each, and the impact on exposure and claim
   totals (before vs after your cleaning).
3. **Methodology (as applied)** — describe what you actually did: exposure
   basis, expected basis, segmentation, credibility handling.
4. **Mortality experience results**
5. **Lapse experience results**
6. **Insights & recommendations**
7. **Cedant query list (appendix)** — all items requiring cedant
   confirmation, consolidated in one place.
8. **Limitations**

## Presentation standards

- Every quantitative claim carries its basis: the exposure and claim count
  behind each mortality figure, the decision count behind each lapse rate.
- The data-quality section precedes all results, so a reader knows what the
  results stand on.
- Cedant queries may be referenced from the body but are consolidated in
  the appendix.
- Every number in the report must be traceable to insights.json or
  findings.json.
templates/retrospective_template.md704 bytes
# Study Retrospective

## What went well
## What was difficult or slow
(Where did you lose time? What would you check first next time?)

## Judgment calls I am least sure about
(Decision, alternative treatments considered, why you chose yours.)

## Proposed additions to this workspace's practices
(Checks, decision rules, or methodology notes that would have made this
study faster or more reliable. Write each as a GENERAL rule a future
analyst could apply to any submission — pattern-level descriptions only,
no specific record identifiers. For each: the rule, when it applies, and
what prompted it.)

## Open questions
(Anything unresolved, including items already flagged as cedant queries.)

How the harness grew — the governing rules

The harness may change only in the review window between cycles, never mid-run, and only through one pipeline: every proposal in the agent's retrospective becomes a candidate; the architect accepts, rewrites-to-generalize, or rejects each one with a recorded reason; approved rules are registered with an ID, origin, justification, and approved_by. Fifty rules, five review windows, one gatekeeper.

39
drafted by the agent from its own retrospectives (AGENT_RETRO) — every one approved by the actuary
7
standard actuarial practice the agent visibly lacked, codified by the actuary (STANDARD_PRACTICE)
4
from the actuary's own review of the outputs (ARCHITECT_EXPERIENCE)
1
Generalization test. A rule is admissible only if it would be sensible advice for a submission whose problems you had never seen. If its usefulness depends on knowing what was planted, it is answer leakage — rejected.
2
No instance pointers. Banned everywhere in the harness: policy or claim identifiers, counts of specific planted patterns, cohort months as answers, any reference to grading or ground truth. Linter-enforced, plus human judgment.
3
Asymmetry rule. The architect may teach method — how to detect, how to decide, how to report — but never location: where this dataset's problems are.
4
Justification duty. Every entry's justification must survive an outside reader's skepticism. The register was built to be shown.
5
Timing. Edits only between cycles. No harness change after the frozen final version used for the replay.

The charter — the file that sets the principles — was never changed. Not once. By design, the agent drafts rules; it does not get to touch the principles.

How the score was computed

The grader is code, not an AI (version 1.4.1, changes pre-registered in a changelog before implementation). Because every defect was planted, there is an answer key; because the portfolio was simulated, there is a realized truth for every actuarial quantity. Four scores per run, same ruler every time.

1 · Detection

Did it find the defect type?

Each planted type counts as detected if any finding in the agent's findings.json overlaps its planted instances. Scored as a macro average over types — finding 86,000 duplicate rows counts the same as finding 50 orphan claims. A separate sharpness metric records how precisely the finding's selector matched the planted set.

2 · Resolution

Did it fix it correctly?

Instance by instance, the agent's cleaned tables are compared against a reference-cleaned end-state: the same submission with every defect resolved the way the catalog prescribes. A percentage per type, then macro-averaged. An instance also passes if it reaches an adjudicated alternate end-state (see below).

3 · False alarms

Did it invent problems?

Findings that don't correspond to any planted defect and aren't confirmed real. Reported as a count and as finding precision. In cycles 1 and 5, and in the replay: zero.

4 · Insight score

Did it reach the right actuarial conclusions?

Twelve items in a per-cycle rubric — select effect, smoker differential, COVID timing, renewal anti-selection, lapse curve… — each scored 0 / 0.5 / 1.0 and weighted. This is the score that moved.
insight = 100 × Σ wᵢ·sᵢ / Σ wᵢ

The insight rubric, honestly

Three design decisions kept the grader honest:

Realized, not configuredTargets are calibrated to the realized experience of the simulated book — recomputed per cycle from that cycle's reference-cleaned state — not to the generator's input parameters. Configured PY1 lapse was 18%; realized, after channel mix and frailty, was 22.4%. Configured smoker ratio 1.70; realized 1.84. An agent reporting the config value would be wrong on this book, and grading it right would punish correct work.
Credibility-awareEach item is numeric or directional. Numeric items get full credit within a tolerance set to max(spec floor, ~95% CI half-width) of the constraining cell — 57,600 renewal decisions earn a ±5pp tolerance; 464 duration-1 deaths earn ±0.10 on the ratio. Where the cell is too thin for a point estimate — ~70 post-renewal deaths, ~29 peak-month COVID deaths — the item is directional, and full credit requires an explicit low-credibility acknowledgment in the agent's own words. Half credit is "mentioned"; full credit needs direction, tolerance, and a recommendation.
Adjudicated alternates — the agent taught the graderThe reference end-state is one correct answer, not the only one. When the agent reached a different, defensible end-state — cycle 1 reconstructed a missing cohort's exposure exactly instead of excluding its claims (D6) — the architect could register an alternate-correct end-state with machine-checkable conditions (≥99% key recall and precision, amounts within one cent of the basis, cohort claims retained, gap raised as a cedant query). Adjudications are append-only, made only in review windows, never mid-run, and the reference path always still passes. Three exist (D3, D4, D6). The D4 adjudication — the 100 "unrepairable" policies recovered from the rate basis — is the "unrepairable — repaired" story from the talk, as a grader artifact.

Item by item: cycle 1 vs the replay

Same exam paper, same model, only the harness changed. Every item that moved, moved up; none moved down. The two items that stayed at 0.5 are the same in both runs.

Rubric itemModewCycle 1
(v0)
Replay
(v5)
Δ
COVID excess — 2022-03/04 peak, ages 60+Graded on timing + age gradient, not magnitude (29 peak-month deaths)directional10.51.0+0.5
Select effect — durations 1–2 vs ultimateTargets 0.552 / 0.769, tol ±0.10numeric11.01.0
Term renewal anti-selection~70 post-renewal deaths: direction + credibility acknowledgment requireddirectional + ack10.51.0+0.5
Smoker differentialRealized 1.84 (config 1.70) — graded on realizednumeric10.51.0+0.5
Mortality improvementDeclining ultimate-duration trenddirectional10.51.0+0.5
Channel orderingOnly the robust pooled contrast is gradednumeric10.51.0+0.5
Sum-assured effectHigh-SA advantage only; strict monotonicity not gradeddirectional11.01.0
Underwriting-class orderingMentioned only in both runsnumeric10.50.5
Lapse curvePY1 lapse realized 22.4% (config 18%)numeric11.01.0
Renewal shock lapseRealized 39.6% (config 35%); 57.6k decisions → tol ±5ppnumeric11.01.0
Winter seasonalityNo matching insight in cycle 1directional10.01.0+1.0
Reinstatement loadingWeight 0.25; ack missing in bothdirectional + ack0.250.50.5
63.33cycle 1 = 100 × 7.125 / 11.25
94.44replay = 100 × 10.625 / 11.25
+31.11six items up, none down

Every cycle, every number

The row behind each point on the chart. Detection was 100% in cycles 1–3 and in the replay; it dipped in cycles 4 and 5 (77% and 93%) — the agent missed some planted types outright in those runs even as its judgment score climbed. Both things are true, and the table says so.

RunHarnessDetectionResolutionFalse alarmsInsightActive minCost
Cycle 1harness-v0100%99.0%063.33103$32.3
Cycle 2harness-v1100%99.9%182.2276$34.6
Cycle 3harness-v2100%100.0%281.1162$24.2
Cycle 4harness-v377%100.0%198.8975$33.8
Cycle 5harness-v493%93.3%090.0082$31.7
Replay of cycle 1harness-v5 (frozen)100%99.0%094.4491≈$26

Cost = API-list-price equivalent computed from session transcripts (runs were subscription-included); active minutes = agent-active time excluding operator-wait gaps. Cycle 2's insight score was 86.67 under grader 1.4.0 and 82.22 under 1.4.1 (a COVID-timing credit was revoked on regrade); all cycles are reported on 1.4.1.

Who audits the grader? — one disclosed defectUnder grader 1.4.0 the replay scored one defect type (D3) at 0/100 despite reaching the adjudicated end-state on every substantive condition. The sole failure was representational: the agent's cleaned table stored dates as datetime-at-midnight, the reference as date, and the comparison used raw equality. That is a checker defect, not an agent failure. The fix was pre-registered in the changelog before implementation, released as 1.4.1, and cycles 1–5 were regraded byte-identical. Sequencing is disclosed: the defect was found by the close-out investigation after the replay had been graded under 1.4.0; the authority the fix restores predates the run. The grader is measurement, not harness — the harness freeze was untouched. It is disclosed here so the reader can weigh it.