Workflow evaluation

examples/compliance-reviewer/runs/exec-59.harness.json scored against examples/compliance-reviewer/key.json · 40 records · strict mode · generated 2026-09-23T00:00:00.000Z
A record needed a person if the answer key says it should not have gone out without review.

Silent error rate

6.7%

1 of the 15 records that went out without review should have had one
The true rate on records like these is likely between 1.2% and 29.8% (95% range).
With 15 records, each one moves it by 6.7 points.

Straight-through

37.5%

went out without review
15 of 40 · likely 24.2–53.0%

Silent omissions

0.0%

blocked or suppressed, but should have gone out or to a person
0 of 40 · likely 0.0–8.8%

Review precision

100.0%

sent to a person, and needed one
25 of 25 · likely 86.7–100.0%

Safeguard failures

went out past the workflow's own confidence gate or floor

Silent errors

Records that went out without review, but needed a person.

Record	Severity	Types	Went	Answer key	What differed	Trap
`E38`	medium	SP-SHOULD-REVIEW SP-MISSING-REJECTED	PASS_WITH_NOTES	VIOLATION	violations found, but set aside: `AWARD_CLAIM_REVIEW`	llm

What the types mean

SP-SHOULD-REVIEW: went out, but should have gone to a person
SP-MISSING-REJECTED: the model found a correct value, but it was set aside for low confidence

Silent omissions

None.

Safeguard failures

None.

Unnecessary reviews (sent to a person, but could have gone out): none

Output quality

40 records. Precision: of the values the workflow applied, the share the answer key lists. Recall: of the values the answer key lists, the share the workflow applied. F1 balances the two (1 is perfect). A value the workflow found but set aside does not count as applied.

Output	Precision	Recall	F1	Exact match	TP / FP / FN
violations	85.4%	87.5%	0.864	75.0%	35 / 6 / 5

Per trap

A trap is a record built to provoke one kind of mistake. Each row shows how the records of one trap were handled.

Trap	Records	Routed right	Silent errors	Omissions	Unnecessary reviews
both	1	1 of 1	none	none	none
deterministic	14	14 of 14	none	none	none
llm	11	10 of 11	E38	none	none
none	14	14 of 14	none	none	none

Calibration

Does a stated confidence mean what it says? 40 claims, one group per stated value. Average gap 0.330 (0 is perfectly calibrated). The grey diagonal is where stated equals actual; a dot above it was right more often than stated, below it less often. Each thin line is the likely range (95%) for that dot. With 40 claims, read the ranges, not the dots.

Stated	Claims	Right	Accuracy	Gap	Likely range
0.60	1	0	0.0%	-0.60	0.0–79.3%
0.65	2	0	0.0%	-0.65	0.0–65.8%
0.72	1	0	0.0%	-0.72	0.0–79.3%
0.75	7	2	28.6%	-0.46	8.2–64.1%
0.80	4	3	75.0%	-0.05	30.1–95.4%
0.85	12	6	50.0%	-0.35	25.4–74.6%
0.88	2	2	100.0%	+0.12	34.2–100.0%
0.90	4	3	75.0%	-0.15	30.1–95.4%
0.92	2	2	100.0%	+0.08	34.2–100.0%
0.95	1	1	100.0%	+0.05	20.7–100.0%
0.97	4	2	50.0%	-0.47	15.0–85.0%

Every record

Record	Went	Answer key	Needed a person	Outcome	Trap
`E01`	BLOCK	VIOLATION	yes	ok	deterministic
`E02`	BLOCK	VIOLATION	yes	ok	deterministic
`E03`	BLOCK	VIOLATION	yes	ok	deterministic
`E04`	BLOCK	VIOLATION	yes	ok	deterministic
`E05`	BLOCK	VIOLATION	yes	ok	deterministic
`E06`	BLOCK	VIOLATION	yes	ok	deterministic
`E07`	BLOCK	VIOLATION	yes	ok	deterministic
`E08`	BLOCK	VIOLATION	yes	ok	both
`E09`	BLOCK	VIOLATION	yes	ok	deterministic
`E10`	BLOCK	VIOLATION	yes	ok	deterministic
`E11`	BLOCK	VIOLATION	yes	ok	deterministic
`E12`	REVIEW_REQUIRED	VIOLATION	yes	ok	deterministic
`E13`	REVIEW_REQUIRED	VIOLATION	yes	ok	deterministic
`E14`	PASS	CLEAN	no	ok	none
`E15`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E16`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E17`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E18`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E19`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E20`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E21`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E22`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E23`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E24`	PASS	CLEAN	no	ok	none
`E25`	BLOCK	VIOLATION	yes	ok	deterministic
`E26`	BLOCK	VIOLATION	yes	ok	deterministic
`E27`	PASS	CLEAN	no	ok	none
`E28`	PASS	CLEAN	no	ok	none
`E29`	PASS	CLEAN	no	ok	none
`E30`	PASS	CLEAN	no	ok	none
`E31`	PASS	CLEAN	no	ok	none
`E32`	PASS	CLEAN	no	ok	none
`E33`	PASS	CLEAN	no	ok	none
`E34`	PASS	CLEAN	no	ok	none
`E35`	PASS	CLEAN	no	ok	none
`E36`	PASS	CLEAN	no	ok	none
`E37`	PASS	CLEAN	no	ok	none
`E38`	PASS_WITH_NOTES	VIOLATION	yes	medium silent error	llm
`E39`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E40`	PASS	CLEAN	no	ok	none

Not measured

block precision: no decision maps to block
SP-INVALID: no allowed_values list
SG-GATE: no thresholds.auto_publish
SG-FLOOR: no thresholds.floor