Duplicate premium rows
0.3% of premium rows duplicated exactly; 10% of those triplicated.
Deduplicate to one row per policy × month.
The talk shows the results. This page shows the apparatus — enough for a reader to judge for themselves what the numbers mean. Four tabs: how the synthetic submission was built, the 15 defect types injected into it, the harness's actual text files, and exactly how the grader computed detection, resolution, and the insight score.
There is no real cedant here. The "submission" is a simulated Korean life portfolio with an answer key — built in two steps so that every defect the agent finds, and every actuarial conclusion it reaches, can be checked against a recorded truth. The simulator is config-driven, single-seeded (20260704) and fully reproducible.
1,000,000 policies, all issued inside the window 2019-01 → 2025-12, simulated month by month over 84 steps. Whole life ≈55% / 5-year renewable term ≈45%; sex 55/45; issue ages 20–70 centred on the 40s; sums assured lognormal (median ₩100m WL / ₩150m term); five distribution channels (GA 50%, captive 25%, outbound TM 10%, inbound TM 7%, banca/direct 8%); three underwriting classes; smoker status from Statistics Korea 2024 prevalence by sex and age.
Reinsurance structure: 100% quota-share YRT on the death benefit — ceded NAR = full sum assured, monthly natural premium on the 10th Korean experience-table reference net rates, claim = 100% of SA in the month of death. Amounts in KRW thousands.
The clean tables become a "cedant submission" by planting 13 standard defect types (plus 2 hold-outs in cycle 5) — mechanical (duplicates, invalid codes, orphan claims), actuarial (premium after death, cohort gaps, split claims), and judgment (code-scheme changes, missing events, mid-history corrections). Every planted instance is logged with its correct resolution, which defines the reference-cleaned state the grader scores against.
Same defect types every cycle, planted in different places from a per-cycle seed. The full catalog →
Monthly death probability per policy is a base table times a stack of multiplicative factors — every one a real, discoverable actuarial signal the agent is later graded on finding:
D1–D13 are injected in every cycle — same types, planted in different places each time (all randomness derives from a per-cycle seed), so nothing could be memorized. H1 and H2 were injected only in cycle 5 and were kept out of every harness rule until the agent had met them there, to test the never-seen; the fixes it proposed entered the harness only afterwards, through the gate. Total distortion is material but the underlying signals stay recoverable: planted defects touch well under 5% of rows, and the injector enforces one defect type per affected record.
0.3% of premium rows duplicated exactly; 10% of those triplicated.
Deduplicate to one row per policy × month.
For 15% of deaths, 1–6 extra monthly premium rows are appended after the death month.
Drop post-death premium rows; exposure ends at death.
For 100 real deaths, a fake LAPSE event is inserted 1–2 months before death (claim appears to postdate termination); master status set to LAPSE and the intervening premium rows deleted.
Recognize the grace-period pattern: the claim is valid; remove the lapse evidence, restore exposure through death.
0.05% of master rows: sex ∈ {U, 9, null} or issue_age ∈ {0, −1, 999}. About 20% are made unrepairable by also nulling the premium-side redundant attributes.
Repair from redundant premium-table attributes where consistent; otherwise exclude the policy from the study and log it. (In the experiment the 100 "unrepairable" instances were recovered from the rate basis — later adjudicated as an acceptable end-state; see Scoring.)
0.02% of premium rows get a negative or zero reins_premium with no paired original row; five master rows get sum_assured = 0.
Exclude or repair: recompute premium from rate × SA / 12 where possible; else drop the row and log.
Two issue months are picked (2020-07…2024-06); ALL premium rows for policies issued in those months are deleted; their claims remain in the claims table.
Numerator–denominator consistency: exclude those cohorts' claims from the study too, and report the gap as a cedant query. Do not fabricate exposure for a wholesale gap. (An exact, verifiable reconstruction was later adjudicated as an acceptable alternative — see Scoring.)
~500 scattered policies: all their premium rows deleted (policies exist in master; 25 of them have claims — the numerator-without-denominator trap).
Force-generate exposure from master (issue date → termination or status date, per events), keep the claims, and document the reconstruction method.
0.1% of policy-months: the original row + a full negative reversal + a corrected row (premium restated, e.g. a mis-booked SA ×10 or ×0.1).
Net the triplet: keep only the corrected row. Negative rows here are bookkeeping, not errors to delete blindly.
~50 claims whose policy_id does not exist in master (plausible-looking IDs).
Exclude from the study; flag as likely other-treaty business for cedant query.
For ~300 genuinely reinstated policies, the REINSTATEMENT event is deleted but resumed premium rows are kept.
Infer reinstatement from the premium gap pattern; exposure excludes the lapsed gap months; treat post-resumption months as exposed.
Policies issued from 2023-01 carry new channel codes (W01–W05, a per-cycle seeded permutation of the five channels) in master; earlier policies keep the original labels. No mapping table is provided.
Detect the disjoint code sets correlated with issue date; reconstruct the mapping from volume/behavioral fingerprints; unify before channel-level analysis; state the inferred mapping.
~150 policies: premium-table sex differs from master for the first k months, then flips to match (initial mis-entry corrected later); another ~150: attained-age sequence jumps by ±3–5 at some month, with master consistent with the later values. Amounts stay consistent with the true values throughout.
Trust the corrected (later) value; restate the full history consistently on the corrected basis.
3% of claims are split into two rows (60/40), the second payment 1–3 months later, with a distinct claim_id suffix but the same policy.
Consolidate to ONE death event: date = first payment date, amount = sum of instalments. Count deaths after consolidating — never before.
All premium and claim amounts for ONE channel-year block (GA × 2024, spanning its W-code relabeling) are multiplied by 1,000.
Detect the magnitude break; rescale the block back to thousands; verify against SA × rate / 12.
~0.05% of premium rows: yyyymm written with month-overflow-without-carry (202213 ≡ 202301, 202100 ≡ 202012). Corrupted rows move to the partition of their written yyyymm.
Apply the carry correction (13 → next Jan, 00 → prior Dec); verify no duplicate policy × month results; else exclude.
The tier structure carries the talk's message — the agent does the case-by-case work, the actuary holds the judgment — into the test design. Tier A is what people usually mean by "just automate it" — and it was not where the difficulty sat. Tier C is where the actuary's judgment lives: D7 (rebuild exposure or lose the deaths?), D10 (infer a reinstatement nobody recorded?), D11 (a code scheme changed and no one sent the mapping). Detection was usually the easier half; the resolution is a decision — and every Tier C decision the agent made went through the review gate before it became a rule. The agent did the case-by-case work. The gate held the judgment.
"No code, no prompt tricks — just text files" is easy to say. Here are four of them, unedited, exactly as the agent read them: the charter, the six-sentence kickoff, the report format contract, and the retrospective template that makes the loop feed itself. The playbooks (50 rules) and schemas are not reproduced here — one rule is excerpted on the main page.
# Experience Study Workspace ## Who you are and what this is You are the experience-study analyst at a life reinsurer. This workspace contains a cedant's full-period data submission for a 100% quota-share YRT treaty covering the cedant's Korean whole life and 5-year renewable term portfolio, January 2019 – December 2025. The reinsurance premium in the bordereaux is a monthly natural premium on the industry reference basis (10th Korean experience life table reference net rates by attained age and sex, applied to the ceded net amount at risk). Amounts are KRW thousands. ## Data ./data/policy_master.parquet ./data/reinsurance_premium_monthly/ (partitioned by year) ./data/reinsurance_claims.parquet ./data/policy_events.parquet Treat the submission as read-only. Do all work in ./work/ (scratch) and deliver to ./outputs/. ## Deliverables (all required) 1. ./outputs/cleaned/ — the four tables after your data-quality treatment, same schemas and formats as ./data/. 2. ./outputs/findings.json — every data-quality issue you identify and the action you took, conforming to ./schemas/findings.schema.json. Use the cedant_query flag for items that require confirmation from the cedant. Data issues only: methodology documentation (how a basis was derived, whole-book reconciliations) belongs in the report's methodology section, not here. 3. ./outputs/insights.json — your analytical findings with quantification, conforming to ./schemas/insights.schema.json. 4. ./outputs/report.docx — a professional experience-study report following ./templates/report_structure.md. Generate it programmatically (python-docx is acceptable). 5. ./outputs/retrospective.md — written last, following ./templates/retrospective_template.md. ## Working practices - Follow the playbooks in ./playbooks/ (data quality, analysis, reporting) during the corresponding phase of the work. - Verify your own work. Before finalizing, reconcile: cleaned-data row counts against your findings; every number in the report against insights.json; exposure and claim totals before vs after cleaning, with the movement explained by your findings. - Every material judgment call gets a written rationale — in findings.json for data decisions, in the report for analytical ones. "Material" means it could change a reported result. - State the credibility of what you report. Where claim counts are thin, say so and qualify or withhold conclusions accordingly. - If the data conflicts with itself, decide on a consistent treatment, apply it uniformly, justify it, and flag it for the cedant when confirmation is needed. - Before finalizing, run `python tools/validate_outputs.py` — it checks that all deliverables exist and that findings.json / insights.json conform to their schemas. Fix anything it reports. - In the retrospective and in any rule you propose, generalize: describe patterns and checks, never specific record identifiers.
A cedant submission has arrived in ./data/ — full-period bordereaux for our 100% quota-share YRT treaty on their Korean whole life and 5-year renewable term book, January 2019 through December 2025. Conduct the full experience study: assess and treat data quality, then analyze mortality experience (actual vs expected on the treaty premium basis) and lapse experience, across the risk dimensions present in the data. Produce all deliverables specified in CLAUDE.md. Work autonomously end to end; there is no one available to answer questions mid-study, so record anything you would ask the cedant as cedant-query items. You have up to 4 hours. Begin.
# Report Structure — Experience Study Format contract for `./outputs/report.docx`. It fixes the section order and presentation standards only; the methodology is yours to determine, apply, and describe. ## Required sections, in this order 1. **Executive summary** 2. **Data received & quality assessment** — the issues you found, the treatment you applied to each, and the impact on exposure and claim totals (before vs after your cleaning). 3. **Methodology (as applied)** — describe what you actually did: exposure basis, expected basis, segmentation, credibility handling. 4. **Mortality experience results** 5. **Lapse experience results** 6. **Insights & recommendations** 7. **Cedant query list (appendix)** — all items requiring cedant confirmation, consolidated in one place. 8. **Limitations** ## Presentation standards - Every quantitative claim carries its basis: the exposure and claim count behind each mortality figure, the decision count behind each lapse rate. - The data-quality section precedes all results, so a reader knows what the results stand on. - Cedant queries may be referenced from the body but are consolidated in the appendix. - Every number in the report must be traceable to insights.json or findings.json.
# Study Retrospective ## What went well ## What was difficult or slow (Where did you lose time? What would you check first next time?) ## Judgment calls I am least sure about (Decision, alternative treatments considered, why you chose yours.) ## Proposed additions to this workspace's practices (Checks, decision rules, or methodology notes that would have made this study faster or more reliable. Write each as a GENERAL rule a future analyst could apply to any submission — pattern-level descriptions only, no specific record identifiers. For each: the rule, when it applies, and what prompted it.) ## Open questions (Anything unresolved, including items already flagged as cedant queries.)
The harness may change only in the review window between cycles, never mid-run, and only through one pipeline: every proposal in the agent's retrospective becomes a candidate; the architect accepts, rewrites-to-generalize, or rejects each one with a recorded reason; approved rules are registered with an ID, origin, justification, and approved_by. Fifty rules, five review windows, one gatekeeper.
The charter — the file that sets the principles — was never changed. Not once. By design, the agent drafts rules; it does not get to touch the principles.
The grader is code, not an AI (version 1.4.1, changes pre-registered in a changelog before implementation). Because every defect was planted, there is an answer key; because the portfolio was simulated, there is a realized truth for every actuarial quantity. Four scores per run, same ruler every time.
Each planted type counts as detected if any finding in the agent's findings.json overlaps its planted instances. Scored as a macro average over types — finding 86,000 duplicate rows counts the same as finding 50 orphan claims. A separate sharpness metric records how precisely the finding's selector matched the planted set.
Instance by instance, the agent's cleaned tables are compared against a reference-cleaned end-state: the same submission with every defect resolved the way the catalog prescribes. A percentage per type, then macro-averaged. An instance also passes if it reaches an adjudicated alternate end-state (see below).
Findings that don't correspond to any planted defect and aren't confirmed real. Reported as a count and as finding precision. In cycles 1 and 5, and in the replay: zero.
Twelve items in a per-cycle rubric — select effect, smoker differential, COVID timing, renewal anti-selection, lapse curve… — each scored 0 / 0.5 / 1.0 and weighted. This is the score that moved.
insight = 100 × Σ wᵢ·sᵢ / Σ wᵢ
Three design decisions kept the grader honest:
Same exam paper, same model, only the harness changed. Every item that moved, moved up; none moved down. The two items that stayed at 0.5 are the same in both runs.
| Rubric item | Mode | w | Cycle 1 (v0) | Replay (v5) | Δ |
|---|---|---|---|---|---|
| COVID excess — 2022-03/04 peak, ages 60+Graded on timing + age gradient, not magnitude (29 peak-month deaths) | directional | 1 | 0.5 | 1.0 | +0.5 |
| Select effect — durations 1–2 vs ultimateTargets 0.552 / 0.769, tol ±0.10 | numeric | 1 | 1.0 | 1.0 | — |
| Term renewal anti-selection~70 post-renewal deaths: direction + credibility acknowledgment required | directional + ack | 1 | 0.5 | 1.0 | +0.5 |
| Smoker differentialRealized 1.84 (config 1.70) — graded on realized | numeric | 1 | 0.5 | 1.0 | +0.5 |
| Mortality improvementDeclining ultimate-duration trend | directional | 1 | 0.5 | 1.0 | +0.5 |
| Channel orderingOnly the robust pooled contrast is graded | numeric | 1 | 0.5 | 1.0 | +0.5 |
| Sum-assured effectHigh-SA advantage only; strict monotonicity not graded | directional | 1 | 1.0 | 1.0 | — |
| Underwriting-class orderingMentioned only in both runs | numeric | 1 | 0.5 | 0.5 | — |
| Lapse curvePY1 lapse realized 22.4% (config 18%) | numeric | 1 | 1.0 | 1.0 | — |
| Renewal shock lapseRealized 39.6% (config 35%); 57.6k decisions → tol ±5pp | numeric | 1 | 1.0 | 1.0 | — |
| Winter seasonalityNo matching insight in cycle 1 | directional | 1 | 0.0 | 1.0 | +1.0 |
| Reinstatement loadingWeight 0.25; ack missing in both | directional + ack | 0.25 | 0.5 | 0.5 | — |
The row behind each point on the chart. Detection was 100% in cycles 1–3 and in the replay; it dipped in cycles 4 and 5 (77% and 93%) — the agent missed some planted types outright in those runs even as its judgment score climbed. Both things are true, and the table says so.
| Run | Harness | Detection | Resolution | False alarms | Insight | Active min | Cost |
|---|---|---|---|---|---|---|---|
| Cycle 1 | harness-v0 | 100% | 99.0% | 0 | 63.33 | 103 | $32.3 |
| Cycle 2 | harness-v1 | 100% | 99.9% | 1 | 82.22 | 76 | $34.6 |
| Cycle 3 | harness-v2 | 100% | 100.0% | 2 | 81.11 | 62 | $24.2 |
| Cycle 4 | harness-v3 | 77% | 100.0% | 1 | 98.89 | 75 | $33.8 |
| Cycle 5 | harness-v4 | 93% | 93.3% | 0 | 90.00 | 82 | $31.7 |
| Replay of cycle 1 | harness-v5 (frozen) | 100% | 99.0% | 0 | 94.44 | 91 | ≈$26 |
Cost = API-list-price equivalent computed from session transcripts (runs were subscription-included); active minutes = agent-active time excluding operator-wait gaps. Cycle 2's insight score was 86.67 under grader 1.4.0 and 82.22 under 1.4.1 (a COVID-timing credit was revoked on regrade); all cycles are reported on 1.4.1.