Devin Huber

Free tool · runs in your browser

How wrong is your AI, really?

You tested your AI on some examples where you knew the right answer, and counted its mistakes. But a different batch of examples would give a different count, so that count isn't the whole story. This free calculator shows how often your AI is really likely to be wrong, how many examples you need to test to be sure, and whether a change you made actually helped.

Try an example:

Step 1 · Your real error rate

How often is it really wrong?

Enter your test results. An example is one test case you checked by hand: one question, one document, one task.

Step 2 · Test size

How many examples should I test?

Pick the error rate you need to stay under. This uses your results from step 1.

Step 3 · Before and after

Did my change actually help?

You changed something, like the prompt, the model or a rule, and tested before and after. Did it really get better, or could the difference be luck?

Did you test both versions on the same examples?

Before the change

After the change

Measured, not assumed

How often is this calculator wrong?

A tool about error rates should know its own. I checked it three ways: against a standard statistics library, against error rates I set myself (so the right answer was known exactly), and by grading it with my own testing tool.

65,808 of 65,808 answers match a standard statistics library I ran 65,808 inputs through this calculator and through statsmodels, a widely used statistics library. Every answer matched. The same-examples answers in step 3 also matched its standard test on all 6,955 inputs tried.
95.0–95.4% of ranges held the real error rate A "probably between" range should hold the real rate 95 times in 100, and on tests of 10 to 1,000 examples it does. The risky kind of miss, a range that makes your AI look better than it really is, happened at most 4.3% of the time (for error rates up to 50%).
87 of 87 test-size answers kept their promise Step 2 says "test this many and you have an 80% (or 90%, or 95%) chance of proving it." I checked 87 of those answers, at every size up to four times the suggestion, and all 87 came true. My first version's answers came true only 42–92% of the time, so I rebuilt it.
2.5% false alarms when nothing changed If a change did nothing at all, step 3 still says "Real improvement" 2.5% of the time, as designed. That rises to 3.3% when one test is much bigger than the other, and stays at or under 2.65% for same-examples tests. My testing tool also graded 2,600 scenarios.
37% vs 8% real improvements spotted: same vs. different examples Take a typical prompt fix that cuts mistakes from 20% to 12%, tested on 50 examples. Comparing the same examples before and after spots it 37% of the time. Treating the two runs as separate tests spots it only 8% of the time.

Read the full audit: every table, what it found, and what changed → Run it yourself: npm test, then node audit/accuracy.js.

How it works

The math, in plain words

What "probably between" means

Your count came from one batch of examples, and a different batch would give a different count. The range covers every real error rate that could easily have produced your count. "Probably" means 95% confidence: build ranges like this again and again, and about 95 in 100 hold the real rate. With small tests and very low error rates it misses more often, but those misses are on the safe side: they overstate the mistakes.

Why not just "6%, give or take"?

The textbook formula says 0 mistakes in 7 examples means an error rate of exactly 0%, which is the kind of overconfidence this tool exists to catch. This calculator uses a better formula (Wilson's score interval) that stays honest with small numbers and with zero mistakes: a perfect 7 out of 7 really means "anywhere up to about 35%."

How many examples

A new test's count bounces around the real rate. Testing only enough for your current rate to just squeak under your goal works about half the time. So the calculator works out the exact odds at each test size, and suggests the smallest one that reaches the odds you picked, at that size and every size above it.

Comparing two versions

A common rule of thumb says that if the two ranges overlap, nothing really changed. That rule is too cautious and misses real improvements. This calculator builds a range for the change itself (Newcombe's method), and calls it real only when that whole range is on one side of zero.

The same examples both times

When both versions ran on the same examples, the ones that came out the same tell you nothing about the change. Only two counts matter: how many it fixed and how many it broke. The calculator weighs just those (Tango's score interval, which agrees with McNemar's test), so it can spot changes that two separate tests would blur.

Same math as my testing tool

Every rate in workflow-eval-harness, my open-source tool for grading AI workflows, uses this same range, so the two tools agree to the decimal. Nothing you type here is sent anywhere.