Workflow evaluation

examples/compliance-reviewer/runs/exec-50.harness.json scored against examples/compliance-reviewer/key.json · 40 records · strict mode · generated 2026-09-23T00:00:00.000Z
A record needed a person if the answer key says it should not have gone out without review.

Silent error rate

30.0%

6 of the 20 records that went out without review should have had one
The true rate on records like these is likely between 14.5% and 51.9% (95% range).
With 20 records, each one moves it by 5.0 points.

Straight-through

50.0%

went out without review
20 of 40 · likely 35.2–64.8%

Silent omissions

0.0%

blocked or suppressed, but should have gone out or to a person
0 of 40 · likely 0.0–8.8%

Review precision

100.0%

sent to a person, and needed one
20 of 20 · likely 83.9–100.0%

Safeguard failures

went out past the workflow's own confidence gate or floor

Silent errors

Records that went out without review, but needed a person.

Record	Severity	Types	Went	Answer key	What differed	Trap
`E15`	medium	SP-SHOULD-REVIEW SP-MISSING-REJECTED	PASS_WITH_NOTES	VIOLATION	violations found, but set aside: `IMPLIED_GUARANTEE, RESULTS_NO_DISCLAIMER`	llm
`E16`	medium	SP-SHOULD-REVIEW SP-MISSING-REJECTED SP-MISSING	PASS_WITH_NOTES	VIOLATION	violations found, but set aside: `IMPLIED_GUARANTEE` violations never found: `RESULTS_NO_DISCLAIMER`	llm
`E20`	medium	SP-SHOULD-REVIEW SP-MISSING-REJECTED	PASS_WITH_NOTES	VIOLATION	violations found, but set aside: `IMPLIED_SUPERIORITY, UNVERIFIABLE_COMPARISON`	llm
`E22`	medium	SP-SHOULD-REVIEW SP-MISSING-REJECTED	PASS_WITH_NOTES	VIOLATION	violations found, but set aside: `UNVERIFIABLE_COMPARISON, IMPLIED_SUPERIORITY, IMPLIED_GUARANTEE`	llm
`E38`	medium	SP-SHOULD-REVIEW SP-MISSING-REJECTED	PASS_WITH_NOTES	VIOLATION	violations found, but set aside: `AWARD_CLAIM_REVIEW`	llm
`E39`	medium	SP-SHOULD-REVIEW SP-MISSING-REJECTED	PASS_WITH_NOTES	VIOLATION	violations found, but set aside: `STATISTIC_NO_CONTEXT`	llm

What the types mean

SP-SHOULD-REVIEW: went out, but should have gone to a person
SP-MISSING-REJECTED: the model found a correct value, but it was set aside for low confidence
SP-MISSING: a correct value was never found

Silent omissions

None.

Safeguard failures

None.

Unnecessary reviews (sent to a person, but could have gone out): none

Output quality

40 records. Precision: of the values the workflow applied, the share the answer key lists. Recall: of the values the answer key lists, the share the workflow applied. F1 balances the two (1 is perfect). A value the workflow found but set aside does not count as applied.

Output	Precision	Recall	F1	Exact match	TP / FP / FN
violations	90.0%	67.5%	0.771	72.5%	27 / 3 / 13

Per trap

A trap is a record built to provoke one kind of mistake. Each row shows how the records of one trap were handled.

Trap	Records	Routed right	Silent errors	Omissions	Unnecessary reviews
both	1	1 of 1	none	none	none
deterministic	14	14 of 14	none	none	none
llm	11	5 of 11	E15, E16, E20, E22, E38, E39	none	none
none	14	14 of 14	none	none	none

Calibration

Does a stated confidence mean what it says? 37 claims, one group per stated value. Average gap 0.228 (0 is perfectly calibrated). The grey diagonal is where stated equals actual; a dot above it was right more often than stated, below it less often. Each thin line is the likely range (95%) for that dot. With 37 claims, read the ranges, not the dots.

Stated	Claims	Right	Accuracy	Gap	Likely range
0.50	2	0	0.0%	-0.50	0.0–65.8%
0.55	3	1	33.3%	-0.22	6.1–79.2%
0.60	3	1	33.3%	-0.27	6.1–79.2%
0.65	1	1	100.0%	+0.35	20.7–100.0%
0.70	3	1	33.3%	-0.37	6.1–79.2%
0.75	11	6	54.5%	-0.20	28.0–78.7%
0.80	3	2	66.7%	-0.13	20.8–93.9%
0.85	5	4	80.0%	-0.05	37.6–96.4%
0.90	3	2	66.7%	-0.23	20.8–93.9%
0.95	2	1	50.0%	-0.45	9.5–90.5%
0.97	1	1	100.0%	+0.03	20.7–100.0%

Every record

Record	Went	Answer key	Needed a person	Outcome	Trap
`E01`	BLOCK	VIOLATION	yes	ok	deterministic
`E02`	BLOCK	VIOLATION	yes	ok	deterministic
`E03`	BLOCK	VIOLATION	yes	ok	deterministic
`E04`	BLOCK	VIOLATION	yes	ok	deterministic
`E05`	BLOCK	VIOLATION	yes	ok	deterministic
`E06`	BLOCK	VIOLATION	yes	ok	deterministic
`E07`	BLOCK	VIOLATION	yes	ok	deterministic
`E08`	BLOCK	VIOLATION	yes	ok	both
`E09`	BLOCK	VIOLATION	yes	ok	deterministic
`E10`	BLOCK	VIOLATION	yes	ok	deterministic
`E11`	BLOCK	VIOLATION	yes	ok	deterministic
`E12`	REVIEW_REQUIRED	VIOLATION	yes	ok	deterministic
`E13`	REVIEW_REQUIRED	VIOLATION	yes	ok	deterministic
`E14`	PASS	CLEAN	no	ok	none
`E15`	PASS_WITH_NOTES	VIOLATION	yes	medium silent error	llm
`E16`	PASS_WITH_NOTES	VIOLATION	yes	medium silent error	llm
`E17`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E18`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E19`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E20`	PASS_WITH_NOTES	VIOLATION	yes	medium silent error	llm
`E21`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E22`	PASS_WITH_NOTES	VIOLATION	yes	medium silent error	llm
`E23`	REVIEW_REQUIRED	VIOLATION	yes	ok	llm
`E24`	PASS	CLEAN	no	ok	none
`E25`	BLOCK	VIOLATION	yes	ok	deterministic
`E26`	BLOCK	VIOLATION	yes	ok	deterministic
`E27`	PASS	CLEAN	no	ok	none
`E28`	PASS	CLEAN	no	ok	none
`E29`	PASS	CLEAN	no	ok	none
`E30`	PASS	CLEAN	no	ok	none
`E31`	PASS	CLEAN	no	ok	none
`E32`	PASS	CLEAN	no	ok	none
`E33`	PASS	CLEAN	no	ok	none
`E34`	PASS	CLEAN	no	ok	none
`E35`	PASS	CLEAN	no	ok	none
`E36`	PASS	CLEAN	no	ok	none
`E37`	PASS	CLEAN	no	ok	none
`E38`	PASS_WITH_NOTES	VIOLATION	yes	medium silent error	llm
`E39`	PASS_WITH_NOTES	VIOLATION	yes	medium silent error	llm
`E40`	PASS	CLEAN	no	ok	none

Not measured

block precision: no decision maps to block
SP-INVALID: no allowed_values list
SG-GATE: no thresholds.auto_publish
SG-FLOOR: no thresholds.floor