Workflow evaluation
Silent error rate
6.5%
13 of the 199 records that went out without review should have had one
The true rate on records like these is likely between 3.9% and 10.9% (95% range).
With 199 records, each one moves it by 0.5 points.
The true rate on records like these is likely between 3.9% and 10.9% (95% range).
With 199 records, each one moves it by 0.5 points.
Straight-through
39.8%
went out without review
199 of 500 · likely 35.6–44.2%
199 of 500 · likely 35.6–44.2%
Review precision
44.0%
sent to a person, and needed one
118 of 268 · likely 38.2–50.0%
118 of 268 · likely 38.2–50.0%
Safeguard failures
0
went out past the workflow's own confidence gate or floor
Silent errors
Records that went out without review, but needed a person.
| Record | Severity | Types | Went | Answer key | What differed | Trap |
|---|---|---|---|---|---|---|
B050 | high | SP-WRONG | AUTO_PUBLISH | GEOGRAPHY applied, not in the answer key: GEO-US | ||
B144 | medium | SP-MISSING-REJECTED | AUTO_PUBLISH | INDUSTRY found, but set aside: IND-REALEST | ||
B172 | medium | SP-MISSING | AUTO_PUBLISH | SUBJECT never found: SUBJ-LIT | ||
B186 | medium | SP-MISSING | AUTO_PUBLISH | SUBJECT never found: SUBJ-FRAUD | ||
B292 | high | SP-WRONG | AUTO_PUBLISH | GEOGRAPHY applied, not in the answer key: GEO-UK | ||
B299 | medium | SP-MISSING | AUTO_PUBLISH | SUBJECT never found: SUBJ-LIT | ||
B340 | medium | SP-MISSING | AUTO_PUBLISH | SUBJECT never found: SUBJ-DATA, SUBJ-REG | ||
B353 | high | SP-WRONG | AUTO_PUBLISH | GEOGRAPHY applied, not in the answer key: GEO-UK | ||
B413 | medium | SP-MISSING | AUTO_PUBLISH | SUBJECT never found: SUBJ-REG | ||
B424 | medium | SP-MISSING | AUTO_PUBLISH | INDUSTRY never found: IND-TECH | ||
B427 | high | SP-WRONG | AUTO_PUBLISH | INDUSTRY applied, not in the answer key: IND-AERO | ||
B460 | high | SP-WRONG | AUTO_PUBLISH | INDUSTRY applied, not in the answer key: IND-AERO | ||
B471 | medium | SP-MISSING | AUTO_PUBLISH | INDUSTRY never found: IND-REALEST |
What the types mean
- SP-WRONG
- a wrong value went out
- SP-MISSING-REJECTED
- the model found a correct value, but it was set aside for low confidence
- SP-MISSING
- a correct value was never found
Safeguard failures
None.
Unnecessary reviews (sent to a person, but could have gone out): A04, A09, A17, A24, B001, B012, B015, B016, B021, B023, B033, B041, B042, B043, B044, B046, B047, B049, B051, B054, B056, B059, B060, B063, B065, B080, B096, B098, B102, B103, B106, B107, B110, B111, B114, B115, B128, B131, B133, B134, B150, B151, B152, B153, B157, B161, B162, B163, B165, B166, B171, B173, B175, B180, B188, B189, B190, B194, B198, B203, B204, B206, B208, B209, B211, B212, B217, B218, B219, B222, B225, B228, B229, B230, B234, B236, B237, B240, B244, B245, B248, B253, B254, B257, B260, B262, B265, B271, B274, B279, B281, B282, B283, B284, B285, B286, B296, B297, B303, B304, B311, B314, B316, B325, B327, B335, B337, B339, B343, B345, B346, B352, B356, B367, B368, B369, B370, B374, B379, B380, B382, B383, B385, B386, B387, B391, B396, B403, B405, B409, B417, B420, B421, B422, B425, B432, B436, B437, B440, B443, B445, B446, B462, B463, B464, B466, B473, B474, B475, B476
Output quality
466 records; left out: B224 (model failed), A02 (suppressed duplicate), A19 (suppressed duplicate), B025 (suppressed duplicate), B026 (suppressed duplicate), B027 (suppressed duplicate), B028 (suppressed duplicate), B062 (suppressed duplicate), B064 (suppressed duplicate), B066 (suppressed duplicate), B068 (suppressed duplicate), B122 (suppressed duplicate), B130 (suppressed duplicate), B137 (suppressed duplicate), B141 (suppressed duplicate), B192 (suppressed duplicate), B201 (suppressed duplicate), B214 (suppressed duplicate), B242 (suppressed duplicate), B250 (suppressed duplicate), B255 (suppressed duplicate), B261 (suppressed duplicate), B302 (suppressed duplicate), B306 (suppressed duplicate), B320 (suppressed duplicate), B324 (suppressed duplicate), B362 (suppressed duplicate), B378 (suppressed duplicate), B400 (suppressed duplicate), B402 (suppressed duplicate), B434 (suppressed duplicate), B441 (suppressed duplicate), B447 (suppressed duplicate), B452 (suppressed duplicate). Precision: of the values the workflow applied, the share the answer key lists. Recall: of the values the answer key lists, the share the workflow applied. F1 balances the two (1 is perfect). A value the workflow found but set aside does not count as applied.
| Output | Precision | Recall | F1 | Exact match | TP / FP / FN |
|---|---|---|---|---|---|
| SUBJECT | 99.3% | 95.2% | 0.972 | 90.8% | 864 / 6 / 44 |
| INDUSTRY | 99.0% | 87.6% | 0.930 | 86.7% | 495 / 5 / 70 |
| GEOGRAPHY | 99.1% | 94.6% | 0.968 | 92.3% | 646 / 6 / 37 |
| COMPANY | 100.0% | 100.0% | 1.000 | 100.0% | 184 / 0 / 0 |
| overall | 99.2% | 93.5% | 0.963 | 2189 / 17 / 151 |
Calibration
Does a stated confidence mean what it says? 1994 claims, groups of 0.1. Average gap 0.140 (0 is perfectly calibrated). The grey diagonal is where stated equals actual; a dot above it was right more often than stated, below it less often. Each thin line is the likely range (95%) for that dot.
| Stated | Claims | Right | Accuracy | Gap | Likely range |
|---|---|---|---|---|---|
| 0.20–0.30 | 1 | 0 | 0.0% | -0.25 | 0.0–79.3% |
| 0.30–0.40 | 11 | 7 | 63.6% | +0.32 | 35.4–84.8% |
| 0.40–0.50 | 32 | 24 | 75.0% | +0.35 | 57.9–86.7% |
| 0.50–0.60 | 82 | 62 | 75.6% | +0.23 | 65.3–83.6% |
| 0.60–0.70 | 144 | 134 | 93.1% | +0.33 | 87.7–96.2% |
| 0.70–0.80 | 256 | 247 | 96.5% | +0.23 | 93.5–98.1% |
| 0.80–0.90 | 475 | 472 | 99.4% | +0.15 | 98.2–99.8% |
| 0.90–1.00 | 993 | 988 | 99.5% | +0.07 | 98.8–99.8% |
- floor at 0.6: claims at or above it were right 98.6% of the time (1841 of 1868); below it, 73.8% (93 of 126).
- provisional_below at 0.75: claims at or above it were right 99.1% of the time (1627 of 1641).
- Gate on SUBJECT at 0.85: claims at or above it were right 100.0% of the time (619 of 619).
- Gate on INDUSTRY at 0.7: claims at or above it were right 99.2% of the time (352 of 355).
Every record
| Record | Went | Answer key | Needed a person | Outcome | Trap |
|---|---|---|---|---|---|
A01 | AUTO_PUBLISH | no | ok | ||
A02 | DUPLICATE_SUPPRESSED | no | ok | ||
A03 | AUTO_PUBLISH | no | ok | ||
A04 | EDITOR_REVIEW | no | unnecessary review | ||
A05 | AUTO_PUBLISH | no | ok | ||
A06 | AUTO_PUBLISH | no | ok | ||
A07 | EDITOR_REVIEW | yes | ok | ||
A08 | EDITOR_REVIEW | yes | ok | ||
A09 | EDITOR_REVIEW | no | unnecessary review | ||
A10 | EDITOR_REVIEW | yes | ok | ||
A11 | AUTO_PUBLISH | no | ok | ||
A12 | AUTO_PUBLISH | no | ok | ||
A13 | EDITOR_REVIEW | yes | ok | ||
A14 | AUTO_PUBLISH | no | ok | ||
A15 | EDITOR_REVIEW | yes | ok | ||
A16 | AUTO_PUBLISH | no | ok | ||
A17 | EDITOR_REVIEW | no | unnecessary review | ||
A18 | EDITOR_REVIEW | yes | ok | ||
A19 | DUPLICATE_SUPPRESSED | no | ok | ||
A20 | EDITOR_REVIEW | yes | ok | ||
A21 | EDITOR_REVIEW | yes | ok | ||
A22 | AUTO_PUBLISH | no | ok | ||
A23 | AUTO_PUBLISH | no | ok | ||
A24 | EDITOR_REVIEW | no | unnecessary review | ||
B001 | EDITOR_REVIEW | no | unnecessary review | ||
B002 | AUTO_PUBLISH | no | ok | ||
B003 | AUTO_PUBLISH | no | ok | ||
B004 | EDITOR_REVIEW | yes | ok | ||
B005 | AUTO_PUBLISH | no | ok | ||
B006 | AUTO_PUBLISH | no | ok | ||
B007 | AUTO_PUBLISH | no | ok | ||
B008 | AUTO_PUBLISH | no | ok | ||
B009 | AUTO_PUBLISH | no | ok | ||
B010 | AUTO_PUBLISH | no | ok | ||
B011 | EDITOR_REVIEW | yes | ok | ||
B012 | EDITOR_REVIEW | no | unnecessary review | ||
B013 | AUTO_PUBLISH | no | ok | ||
B014 | AUTO_PUBLISH | no | ok | ||
B015 | EDITOR_REVIEW | no | unnecessary review | ||
B016 | EDITOR_REVIEW | no | unnecessary review | ||
B017 | AUTO_PUBLISH | no | ok | ||
B018 | AUTO_PUBLISH | no | ok | ||
B019 | AUTO_PUBLISH | no | ok | ||
B020 | AUTO_PUBLISH | no | ok | ||
B021 | EDITOR_REVIEW | no | unnecessary review | ||
B022 | AUTO_PUBLISH | no | ok | ||
B023 | EDITOR_REVIEW | no | unnecessary review | ||
B024 | EDITOR_REVIEW | yes | ok | ||
B025 | DUPLICATE_SUPPRESSED | no | ok | ||
B026 | DUPLICATE_SUPPRESSED | no | ok | ||
B027 | DUPLICATE_SUPPRESSED | yes | ok | ||
B028 | DUPLICATE_SUPPRESSED | no | ok | ||
B029 | EDITOR_REVIEW | yes | ok | ||
B030 | EDITOR_REVIEW | yes | ok | ||
B031 | AUTO_PUBLISH | no | ok | ||
B032 | AUTO_PUBLISH | no | ok | ||
B033 | EDITOR_REVIEW | no | unnecessary review | ||
B034 | AUTO_PUBLISH | no | ok | ||
B035 | AUTO_PUBLISH | no | ok | ||
B036 | EDITOR_REVIEW | yes | ok | ||
B037 | AUTO_PUBLISH | no | ok | ||
B038 | AUTO_PUBLISH | no | ok | ||
B039 | AUTO_PUBLISH | no | ok | ||
B040 | AUTO_PUBLISH | no | ok | ||
B041 | EDITOR_REVIEW | no | unnecessary review | ||
B042 | EDITOR_REVIEW | no | unnecessary review | ||
B043 | EDITOR_REVIEW | no | unnecessary review | ||
B044 | EDITOR_REVIEW | no | unnecessary review | ||
B045 | EDITOR_REVIEW | yes | ok | ||
B046 | EDITOR_REVIEW | no | unnecessary review | ||
B047 | EDITOR_REVIEW | no | unnecessary review | ||
B048 | AUTO_PUBLISH | no | ok | ||
B049 | EDITOR_REVIEW | no | unnecessary review | ||
B050 | AUTO_PUBLISH | yes | high silent error | ||
B051 | EDITOR_REVIEW | no | unnecessary review | ||
B052 | AUTO_PUBLISH | no | ok | ||
B053 | AUTO_PUBLISH | no | ok | ||
B054 | EDITOR_REVIEW | no | unnecessary review | ||
B055 | EDITOR_REVIEW | yes | ok | ||
B056 | EDITOR_REVIEW | no | unnecessary review | ||
B057 | AUTO_PUBLISH | no | ok | ||
B058 | EDITOR_REVIEW | yes | ok | ||
B059 | EDITOR_REVIEW | no | unnecessary review | ||
B060 | EDITOR_REVIEW | no | unnecessary review | ||
B061 | AUTO_PUBLISH | no | ok | ||
B062 | DUPLICATE_SUPPRESSED | no | ok | ||
B063 | EDITOR_REVIEW | no | unnecessary review | ||
B064 | DUPLICATE_SUPPRESSED | no | ok | ||
B065 | EDITOR_REVIEW | no | unnecessary review | ||
B066 | DUPLICATE_SUPPRESSED | no | ok | ||
B067 | EDITOR_REVIEW | yes | ok | ||
B068 | DUPLICATE_SUPPRESSED | yes | ok | ||
B069 | AUTO_PUBLISH | no | ok | ||
B070 | AUTO_PUBLISH | no | ok | ||
B071 | AUTO_PUBLISH | no | ok | ||
B072 | AUTO_PUBLISH | no | ok | ||
B073 | EDITOR_REVIEW | yes | ok | ||
B074 | AUTO_PUBLISH | no | ok | ||
B075 | AUTO_PUBLISH | no | ok | ||
B076 | EDITOR_REVIEW | yes | ok | ||
B077 | AUTO_PUBLISH | no | ok | ||
B078 | AUTO_PUBLISH | no | ok | ||
B079 | AUTO_PUBLISH | no | ok | ||
B080 | EDITOR_REVIEW | no | unnecessary review | ||
B081 | AUTO_PUBLISH | no | ok | ||
B082 | AUTO_PUBLISH | no | ok | ||
B083 | EDITOR_REVIEW | yes | ok | ||
B084 | EDITOR_REVIEW | yes | ok | ||
B085 | AUTO_PUBLISH | no | ok | ||
B086 | AUTO_PUBLISH | no | ok | ||
B087 | AUTO_PUBLISH | no | ok | ||
B088 | AUTO_PUBLISH | no | ok | ||
B089 | AUTO_PUBLISH | no | ok | ||
B090 | EDITOR_REVIEW | yes | ok | ||
B091 | EDITOR_REVIEW | yes | ok | ||
B092 | EDITOR_REVIEW | yes | ok | ||
B093 | EDITOR_REVIEW | yes | ok | ||
B094 | EDITOR_REVIEW | yes | ok | ||
B095 | EDITOR_REVIEW | yes | ok | ||
B096 | EDITOR_REVIEW | no | unnecessary review | ||
B097 | AUTO_PUBLISH | no | ok | ||
B098 | EDITOR_REVIEW | no | unnecessary review | ||
B099 | AUTO_PUBLISH | no | ok | ||
B100 | AUTO_PUBLISH | no | ok | ||
B101 | EDITOR_REVIEW | yes | ok | ||
B102 | EDITOR_REVIEW | no | unnecessary review | ||
B103 | EDITOR_REVIEW | no | unnecessary review | ||
B104 | EDITOR_REVIEW | yes | ok | ||
B105 | EDITOR_REVIEW | yes | ok | ||
B106 | EDITOR_REVIEW | no | unnecessary review | ||
B107 | EDITOR_REVIEW | no | unnecessary review | ||
B108 | EDITOR_REVIEW | yes | ok | ||
B109 | AUTO_PUBLISH | no | ok | ||
B110 | EDITOR_REVIEW | no | unnecessary review | ||
B111 | EDITOR_REVIEW | no | unnecessary review | ||
B112 | AUTO_PUBLISH | no | ok | ||
B113 | EDITOR_REVIEW | yes | ok | ||
B114 | EDITOR_REVIEW | no | unnecessary review | ||
B115 | EDITOR_REVIEW | no | unnecessary review | ||
B116 | EDITOR_REVIEW | yes | ok | ||
B117 | AUTO_PUBLISH | no | ok | ||
B118 | AUTO_PUBLISH | no | ok | ||
B119 | AUTO_PUBLISH | no | ok | ||
B120 | EDITOR_REVIEW | yes | ok | ||
B121 | AUTO_PUBLISH | no | ok | ||
B122 | DUPLICATE_SUPPRESSED | no | ok | ||
B123 | AUTO_PUBLISH | no | ok | ||
B124 | AUTO_PUBLISH | no | ok | ||
B125 | AUTO_PUBLISH | no | ok | ||
B126 | EDITOR_REVIEW | yes | ok | ||
B127 | AUTO_PUBLISH | no | ok | ||
B128 | EDITOR_REVIEW | no | unnecessary review | ||
B129 | AUTO_PUBLISH | no | ok | ||
B130 | DUPLICATE_SUPPRESSED | no | ok | ||
B131 | EDITOR_REVIEW | no | unnecessary review | ||
B132 | AUTO_PUBLISH | no | ok | ||
B133 | EDITOR_REVIEW | no | unnecessary review | ||
B134 | EDITOR_REVIEW | no | unnecessary review | ||
B135 | EDITOR_REVIEW | yes | ok | ||
B136 | AUTO_PUBLISH | no | ok | ||
B137 | DUPLICATE_SUPPRESSED | no | ok | ||
B138 | AUTO_PUBLISH | no | ok | ||
B139 | AUTO_PUBLISH | no | ok | ||
B140 | AUTO_PUBLISH | no | ok | ||
B141 | DUPLICATE_SUPPRESSED | no | ok | ||
B142 | AUTO_PUBLISH | no | ok | ||
B143 | AUTO_PUBLISH | no | ok | ||
B144 | AUTO_PUBLISH | yes | medium silent error | ||
B145 | AUTO_PUBLISH | no | ok | ||
B146 | EDITOR_REVIEW | yes | ok | ||
B147 | AUTO_PUBLISH | no | ok | ||
B148 | AUTO_PUBLISH | no | ok | ||
B149 | EDITOR_REVIEW | yes | ok | ||
B150 | EDITOR_REVIEW | no | unnecessary review | ||
B151 | EDITOR_REVIEW | no | unnecessary review | ||
B152 | EDITOR_REVIEW | no | unnecessary review | ||
B153 | EDITOR_REVIEW | no | unnecessary review | ||
B154 | AUTO_PUBLISH | no | ok | ||
B155 | AUTO_PUBLISH | no | ok | ||
B156 | AUTO_PUBLISH | no | ok | ||
B157 | EDITOR_REVIEW | no | unnecessary review | ||
B158 | EDITOR_REVIEW | yes | ok | ||
B159 | AUTO_PUBLISH | no | ok | ||
B160 | EDITOR_REVIEW | yes | ok | ||
B161 | EDITOR_REVIEW | no | unnecessary review | ||
B162 | EDITOR_REVIEW | no | unnecessary review | ||
B163 | EDITOR_REVIEW | no | unnecessary review | ||
B164 | EDITOR_REVIEW | yes | ok | ||
B165 | EDITOR_REVIEW | no | unnecessary review | ||
B166 | EDITOR_REVIEW | no | unnecessary review | ||
B167 | EDITOR_REVIEW | yes | ok | ||
B168 | EDITOR_REVIEW | yes | ok | ||
B169 | EDITOR_REVIEW | yes | ok | ||
B170 | AUTO_PUBLISH | no | ok | ||
B171 | EDITOR_REVIEW | no | unnecessary review | ||
B172 | AUTO_PUBLISH | yes | medium silent error | ||
B173 | EDITOR_REVIEW | no | unnecessary review | ||
B174 | AUTO_PUBLISH | no | ok | ||
B175 | EDITOR_REVIEW | no | unnecessary review | ||
B176 | AUTO_PUBLISH | no | ok | ||
B177 | EDITOR_REVIEW | yes | ok | ||
B178 | AUTO_PUBLISH | no | ok | ||
B179 | AUTO_PUBLISH | no | ok | ||
B180 | EDITOR_REVIEW | no | unnecessary review | ||
B181 | AUTO_PUBLISH | no | ok | ||
B182 | AUTO_PUBLISH | no | ok | ||
B183 | AUTO_PUBLISH | no | ok | ||
B184 | EDITOR_REVIEW | yes | ok | ||
B185 | EDITOR_REVIEW | yes | ok | ||
B186 | AUTO_PUBLISH | yes | medium silent error | ||
B187 | EDITOR_REVIEW | yes | ok | ||
B188 | EDITOR_REVIEW | no | unnecessary review | ||
B189 | EDITOR_REVIEW | no | unnecessary review | ||
B190 | EDITOR_REVIEW | no | unnecessary review | ||
B191 | AUTO_PUBLISH | no | ok | ||
B192 | DUPLICATE_SUPPRESSED | no | ok | ||
B193 | AUTO_PUBLISH | no | ok | ||
B194 | EDITOR_REVIEW | no | unnecessary review | ||
B195 | EDITOR_REVIEW | yes | ok | ||
B196 | AUTO_PUBLISH | no | ok | ||
B197 | AUTO_PUBLISH | no | ok | ||
B198 | EDITOR_REVIEW | no | unnecessary review | ||
B199 | AUTO_PUBLISH | no | ok | ||
B200 | AUTO_PUBLISH | no | ok | ||
B201 | DUPLICATE_SUPPRESSED | yes | ok | ||
B202 | AUTO_PUBLISH | no | ok | ||
B203 | EDITOR_REVIEW | no | unnecessary review | ||
B204 | EDITOR_REVIEW | no | unnecessary review | ||
B205 | AUTO_PUBLISH | no | ok | ||
B206 | EDITOR_REVIEW | no | unnecessary review | ||
B207 | EDITOR_REVIEW | yes | ok | ||
B208 | EDITOR_REVIEW | no | unnecessary review | ||
B209 | EDITOR_REVIEW | no | unnecessary review | ||
B210 | AUTO_PUBLISH | no | ok | ||
B211 | EDITOR_REVIEW | no | unnecessary review | ||
B212 | EDITOR_REVIEW | no | unnecessary review | ||
B213 | EDITOR_REVIEW | yes | ok | ||
B214 | DUPLICATE_SUPPRESSED | no | ok | ||
B215 | EDITOR_REVIEW | yes | ok | ||
B216 | EDITOR_REVIEW | yes | ok | ||
B217 | EDITOR_REVIEW | no | unnecessary review | ||
B218 | EDITOR_REVIEW | no | unnecessary review | ||
B219 | EDITOR_REVIEW | no | unnecessary review | ||
B220 | AUTO_PUBLISH | no | ok | ||
B221 | AUTO_PUBLISH | no | ok | ||
B222 | EDITOR_REVIEW | no | unnecessary review | ||
B223 | AUTO_PUBLISH | no | ok | ||
B224 | SPECIALIST_REVIEW | yes | ok | ||
B225 | EDITOR_REVIEW | no | unnecessary review | ||
B226 | AUTO_PUBLISH | no | ok | ||
B227 | EDITOR_REVIEW | yes | ok | ||
B228 | EDITOR_REVIEW | no | unnecessary review | ||
B229 | EDITOR_REVIEW | no | unnecessary review | ||
B230 | EDITOR_REVIEW | no | unnecessary review | ||
B231 | EDITOR_REVIEW | yes | ok | ||
B232 | AUTO_PUBLISH | no | ok | ||
B233 | AUTO_PUBLISH | no | ok | ||
B234 | EDITOR_REVIEW | no | unnecessary review | ||
B235 | AUTO_PUBLISH | no | ok | ||
B236 | EDITOR_REVIEW | no | unnecessary review | ||
B237 | EDITOR_REVIEW | no | unnecessary review | ||
B238 | EDITOR_REVIEW | yes | ok | ||
B239 | AUTO_PUBLISH | no | ok | ||
B240 | EDITOR_REVIEW | no | unnecessary review | ||
B241 | AUTO_PUBLISH | no | ok | ||
B242 | DUPLICATE_SUPPRESSED | no | ok | ||
B243 | AUTO_PUBLISH | no | ok | ||
B244 | EDITOR_REVIEW | no | unnecessary review | ||
B245 | EDITOR_REVIEW | no | unnecessary review | ||
B246 | EDITOR_REVIEW | yes | ok | ||
B247 | AUTO_PUBLISH | no | ok | ||
B248 | EDITOR_REVIEW | no | unnecessary review | ||
B249 | AUTO_PUBLISH | no | ok | ||
B250 | DUPLICATE_SUPPRESSED | no | ok | ||
B251 | AUTO_PUBLISH | no | ok | ||
B252 | AUTO_PUBLISH | no | ok | ||
B253 | EDITOR_REVIEW | no | unnecessary review | ||
B254 | EDITOR_REVIEW | no | unnecessary review | ||
B255 | DUPLICATE_SUPPRESSED | no | ok | ||
B256 | EDITOR_REVIEW | yes | ok | ||
B257 | EDITOR_REVIEW | no | unnecessary review | ||
B258 | EDITOR_REVIEW | yes | ok | ||
B259 | AUTO_PUBLISH | no | ok | ||
B260 | EDITOR_REVIEW | no | unnecessary review | ||
B261 | DUPLICATE_SUPPRESSED | no | ok | ||
B262 | EDITOR_REVIEW | no | unnecessary review | ||
B263 | AUTO_PUBLISH | no | ok | ||
B264 | EDITOR_REVIEW | yes | ok | ||
B265 | EDITOR_REVIEW | no | unnecessary review | ||
B266 | AUTO_PUBLISH | no | ok | ||
B267 | AUTO_PUBLISH | no | ok | ||
B268 | AUTO_PUBLISH | no | ok | ||
B269 | EDITOR_REVIEW | yes | ok | ||
B270 | AUTO_PUBLISH | no | ok | ||
B271 | EDITOR_REVIEW | no | unnecessary review | ||
B272 | AUTO_PUBLISH | no | ok | ||
B273 | EDITOR_REVIEW | yes | ok | ||
B274 | EDITOR_REVIEW | no | unnecessary review | ||
B275 | AUTO_PUBLISH | no | ok | ||
B276 | AUTO_PUBLISH | no | ok | ||
B277 | AUTO_PUBLISH | no | ok | ||
B278 | AUTO_PUBLISH | no | ok | ||
B279 | EDITOR_REVIEW | no | unnecessary review | ||
B280 | EDITOR_REVIEW | yes | ok | ||
B281 | EDITOR_REVIEW | no | unnecessary review | ||
B282 | EDITOR_REVIEW | no | unnecessary review | ||
B283 | EDITOR_REVIEW | no | unnecessary review | ||
B284 | EDITOR_REVIEW | no | unnecessary review | ||
B285 | EDITOR_REVIEW | no | unnecessary review | ||
B286 | EDITOR_REVIEW | no | unnecessary review | ||
B287 | EDITOR_REVIEW | yes | ok | ||
B288 | EDITOR_REVIEW | yes | ok | ||
B289 | EDITOR_REVIEW | yes | ok | ||
B290 | AUTO_PUBLISH | no | ok | ||
B291 | EDITOR_REVIEW | yes | ok | ||
B292 | AUTO_PUBLISH | yes | high silent error | ||
B293 | EDITOR_REVIEW | yes | ok | ||
B294 | EDITOR_REVIEW | yes | ok | ||
B295 | EDITOR_REVIEW | yes | ok | ||
B296 | EDITOR_REVIEW | no | unnecessary review | ||
B297 | EDITOR_REVIEW | no | unnecessary review | ||
B298 | AUTO_PUBLISH | no | ok | ||
B299 | AUTO_PUBLISH | yes | medium silent error | ||
B300 | EDITOR_REVIEW | yes | ok | ||
B301 | EDITOR_REVIEW | yes | ok | ||
B302 | DUPLICATE_SUPPRESSED | no | ok | ||
B303 | EDITOR_REVIEW | no | unnecessary review | ||
B304 | EDITOR_REVIEW | no | unnecessary review | ||
B305 | AUTO_PUBLISH | no | ok | ||
B306 | DUPLICATE_SUPPRESSED | no | ok | ||
B307 | AUTO_PUBLISH | no | ok | ||
B308 | AUTO_PUBLISH | no | ok | ||
B309 | AUTO_PUBLISH | no | ok | ||
B310 | EDITOR_REVIEW | yes | ok | ||
B311 | EDITOR_REVIEW | no | unnecessary review | ||
B312 | AUTO_PUBLISH | no | ok | ||
B313 | AUTO_PUBLISH | no | ok | ||
B314 | EDITOR_REVIEW | no | unnecessary review | ||
B315 | AUTO_PUBLISH | no | ok | ||
B316 | EDITOR_REVIEW | no | unnecessary review | ||
B317 | AUTO_PUBLISH | no | ok | ||
B318 | EDITOR_REVIEW | yes | ok | ||
B319 | AUTO_PUBLISH | no | ok | ||
B320 | DUPLICATE_SUPPRESSED | no | ok | ||
B321 | EDITOR_REVIEW | yes | ok | ||
B322 | EDITOR_REVIEW | yes | ok | ||
B323 | AUTO_PUBLISH | no | ok | ||
B324 | DUPLICATE_SUPPRESSED | no | ok | ||
B325 | EDITOR_REVIEW | no | unnecessary review | ||
B326 | AUTO_PUBLISH | no | ok | ||
B327 | EDITOR_REVIEW | no | unnecessary review | ||
B328 | AUTO_PUBLISH | no | ok | ||
B329 | AUTO_PUBLISH | no | ok | ||
B330 | EDITOR_REVIEW | yes | ok | ||
B331 | AUTO_PUBLISH | no | ok | ||
B332 | EDITOR_REVIEW | yes | ok | ||
B333 | EDITOR_REVIEW | yes | ok | ||
B334 | EDITOR_REVIEW | yes | ok | ||
B335 | EDITOR_REVIEW | no | unnecessary review | ||
B336 | AUTO_PUBLISH | no | ok | ||
B337 | EDITOR_REVIEW | no | unnecessary review | ||
B338 | AUTO_PUBLISH | no | ok | ||
B339 | EDITOR_REVIEW | no | unnecessary review | ||
B340 | AUTO_PUBLISH | yes | medium silent error | ||
B341 | AUTO_PUBLISH | no | ok | ||
B342 | EDITOR_REVIEW | yes | ok | ||
B343 | EDITOR_REVIEW | no | unnecessary review | ||
B344 | EDITOR_REVIEW | yes | ok | ||
B345 | EDITOR_REVIEW | no | unnecessary review | ||
B346 | EDITOR_REVIEW | no | unnecessary review | ||
B347 | EDITOR_REVIEW | yes | ok | ||
B348 | EDITOR_REVIEW | yes | ok | ||
B349 | AUTO_PUBLISH | no | ok | ||
B350 | EDITOR_REVIEW | yes | ok | ||
B351 | AUTO_PUBLISH | no | ok | ||
B352 | EDITOR_REVIEW | no | unnecessary review | ||
B353 | AUTO_PUBLISH | yes | high silent error | ||
B354 | EDITOR_REVIEW | yes | ok | ||
B355 | EDITOR_REVIEW | yes | ok | ||
B356 | EDITOR_REVIEW | no | unnecessary review | ||
B357 | EDITOR_REVIEW | yes | ok | ||
B358 | AUTO_PUBLISH | no | ok | ||
B359 | AUTO_PUBLISH | no | ok | ||
B360 | AUTO_PUBLISH | no | ok | ||
B361 | AUTO_PUBLISH | no | ok | ||
B362 | DUPLICATE_SUPPRESSED | no | ok | ||
B363 | AUTO_PUBLISH | no | ok | ||
B364 | EDITOR_REVIEW | yes | ok | ||
B365 | AUTO_PUBLISH | no | ok | ||
B366 | EDITOR_REVIEW | yes | ok | ||
B367 | EDITOR_REVIEW | no | unnecessary review | ||
B368 | EDITOR_REVIEW | no | unnecessary review | ||
B369 | EDITOR_REVIEW | no | unnecessary review | ||
B370 | EDITOR_REVIEW | no | unnecessary review | ||
B371 | AUTO_PUBLISH | no | ok | ||
B372 | EDITOR_REVIEW | yes | ok | ||
B373 | EDITOR_REVIEW | yes | ok | ||
B374 | EDITOR_REVIEW | no | unnecessary review | ||
B375 | AUTO_PUBLISH | no | ok | ||
B376 | AUTO_PUBLISH | no | ok | ||
B377 | AUTO_PUBLISH | no | ok | ||
B378 | DUPLICATE_SUPPRESSED | no | ok | ||
B379 | EDITOR_REVIEW | no | unnecessary review | ||
B380 | EDITOR_REVIEW | no | unnecessary review | ||
B381 | AUTO_PUBLISH | no | ok | ||
B382 | EDITOR_REVIEW | no | unnecessary review | ||
B383 | EDITOR_REVIEW | no | unnecessary review | ||
B384 | AUTO_PUBLISH | no | ok | ||
B385 | EDITOR_REVIEW | no | unnecessary review | ||
B386 | EDITOR_REVIEW | no | unnecessary review | ||
B387 | EDITOR_REVIEW | no | unnecessary review | ||
B388 | EDITOR_REVIEW | yes | ok | ||
B389 | EDITOR_REVIEW | yes | ok | ||
B390 | EDITOR_REVIEW | yes | ok | ||
B391 | EDITOR_REVIEW | no | unnecessary review | ||
B392 | EDITOR_REVIEW | yes | ok | ||
B393 | EDITOR_REVIEW | yes | ok | ||
B394 | EDITOR_REVIEW | yes | ok | ||
B395 | AUTO_PUBLISH | no | ok | ||
B396 | EDITOR_REVIEW | no | unnecessary review | ||
B397 | EDITOR_REVIEW | yes | ok | ||
B398 | EDITOR_REVIEW | yes | ok | ||
B399 | AUTO_PUBLISH | no | ok | ||
B400 | DUPLICATE_SUPPRESSED | no | ok | ||
B401 | EDITOR_REVIEW | yes | ok | ||
B402 | DUPLICATE_SUPPRESSED | no | ok | ||
B403 | EDITOR_REVIEW | no | unnecessary review | ||
B404 | AUTO_PUBLISH | no | ok | ||
B405 | EDITOR_REVIEW | no | unnecessary review | ||
B406 | EDITOR_REVIEW | yes | ok | ||
B407 | EDITOR_REVIEW | yes | ok | ||
B408 | AUTO_PUBLISH | no | ok | ||
B409 | EDITOR_REVIEW | no | unnecessary review | ||
B410 | AUTO_PUBLISH | no | ok | ||
B411 | EDITOR_REVIEW | yes | ok | ||
B412 | AUTO_PUBLISH | no | ok | ||
B413 | AUTO_PUBLISH | yes | medium silent error | ||
B414 | AUTO_PUBLISH | no | ok | ||
B415 | AUTO_PUBLISH | no | ok | ||
B416 | EDITOR_REVIEW | yes | ok | ||
B417 | EDITOR_REVIEW | no | unnecessary review | ||
B418 | AUTO_PUBLISH | no | ok | ||
B419 | EDITOR_REVIEW | yes | ok | ||
B420 | EDITOR_REVIEW | no | unnecessary review | ||
B421 | EDITOR_REVIEW | no | unnecessary review | ||
B422 | EDITOR_REVIEW | no | unnecessary review | ||
B423 | AUTO_PUBLISH | no | ok | ||
B424 | AUTO_PUBLISH | yes | medium silent error | ||
B425 | EDITOR_REVIEW | no | unnecessary review | ||
B426 | AUTO_PUBLISH | no | ok | ||
B427 | AUTO_PUBLISH | yes | high silent error | ||
B428 | EDITOR_REVIEW | yes | ok | ||
B429 | EDITOR_REVIEW | yes | ok | ||
B430 | AUTO_PUBLISH | no | ok | ||
B431 | AUTO_PUBLISH | no | ok | ||
B432 | EDITOR_REVIEW | no | unnecessary review | ||
B433 | AUTO_PUBLISH | no | ok | ||
B434 | DUPLICATE_SUPPRESSED | no | ok | ||
B435 | AUTO_PUBLISH | no | ok | ||
B436 | EDITOR_REVIEW | no | unnecessary review | ||
B437 | EDITOR_REVIEW | no | unnecessary review | ||
B438 | AUTO_PUBLISH | no | ok | ||
B439 | EDITOR_REVIEW | yes | ok | ||
B440 | EDITOR_REVIEW | no | unnecessary review | ||
B441 | DUPLICATE_SUPPRESSED | no | ok | ||
B442 | AUTO_PUBLISH | no | ok | ||
B443 | EDITOR_REVIEW | no | unnecessary review | ||
B444 | AUTO_PUBLISH | no | ok | ||
B445 | EDITOR_REVIEW | no | unnecessary review | ||
B446 | EDITOR_REVIEW | no | unnecessary review | ||
B447 | DUPLICATE_SUPPRESSED | no | ok | ||
B448 | AUTO_PUBLISH | no | ok | ||
B449 | AUTO_PUBLISH | no | ok | ||
B450 | EDITOR_REVIEW | yes | ok | ||
B451 | AUTO_PUBLISH | no | ok | ||
B452 | DUPLICATE_SUPPRESSED | yes | ok | ||
B453 | EDITOR_REVIEW | yes | ok | ||
B454 | AUTO_PUBLISH | no | ok | ||
B455 | EDITOR_REVIEW | yes | ok | ||
B456 | AUTO_PUBLISH | no | ok | ||
B457 | AUTO_PUBLISH | no | ok | ||
B458 | AUTO_PUBLISH | no | ok | ||
B459 | AUTO_PUBLISH | no | ok | ||
B460 | AUTO_PUBLISH | yes | high silent error | ||
B461 | AUTO_PUBLISH | no | ok | ||
B462 | EDITOR_REVIEW | no | unnecessary review | ||
B463 | EDITOR_REVIEW | no | unnecessary review | ||
B464 | EDITOR_REVIEW | no | unnecessary review | ||
B465 | EDITOR_REVIEW | yes | ok | ||
B466 | EDITOR_REVIEW | no | unnecessary review | ||
B467 | AUTO_PUBLISH | no | ok | ||
B468 | EDITOR_REVIEW | yes | ok | ||
B469 | EDITOR_REVIEW | yes | ok | ||
B470 | EDITOR_REVIEW | yes | ok | ||
B471 | AUTO_PUBLISH | yes | medium silent error | ||
B472 | AUTO_PUBLISH | no | ok | ||
B473 | EDITOR_REVIEW | no | unnecessary review | ||
B474 | EDITOR_REVIEW | no | unnecessary review | ||
B475 | EDITOR_REVIEW | no | unnecessary review | ||
B476 | EDITOR_REVIEW | no | unnecessary review |
Not measured
- SP-FORBIDDEN, SP-SHOULD-REVIEW, SP-DUPLICATE, silent omissions, block precision: the answer key has no correct route (routing.gold_field)