Your AI Scored 100%. Half the Evaluation Is Missing.

A model evaluation dashboard says candidate A passed 100% of its cases. Candidate B passed 75%. Before changing your business workflow, ask one more question: how many planned cases actually received a grade?

This guide builds a small CSV audit that keeps the evaluation plan separate from the result rows. It reports missing grades, graded-case coverage and two explicitly different denominators. You can use that audit before comparing models or prompt revisions. It does not call a model, judge answer quality or select a winner for you.

The missing half of a perfect score

Orbit Desk is a fictional business evaluating a support assistant. Its authored sample plan contains four case IDs: quote, refund, privacy and handoff. Candidate A has two passing grades and no rows for the last two cases. Candidate B has three passing grades and one failing grade. These numbers are invented fixtures, not measurements of real models, customers or Jason’s experience.

Missing does not mean incorrect. It may mean a timeout, an exporter bug, an unfinished human review or a case that never ran. This CSV cannot distinguish those causes. The planned-case rate counts only observed passes in its numerator and retains every planned case in its denominator. Label it that way; do not relabel it as answer accuracy or silently turn absent rows into graded failures.

Freeze the plan before reading the results

The plan is the two explicit tuples at the top of the script, not a list inferred from successful output rows. Each candidate is expected to have one binary grade per case. Exact candidate/case pairs must be unique. Unknown IDs, nonbinary or empty grades, duplicate pairs, wrong headers, malformed quoting and wrong field counts stop the audit. A header-only export remains valid but yields zero coverage and a null graded-case pass rate.

The Python CSV reader handles quoted fields and returns strings by default. Schema and grade validation are this example’s application logic. Inspection of the CPython v3.14.3 source confirms that the module imports its reader from the underlying _csv implementation; it does not provide this evaluation policy. Avoid splitting CSV with a bare comma split.

Run the reusable audit

Prerequisite: Python 3.14.3 was used on Windows for these checks. There are no third-party dependencies, API keys, downloads or model charges. Save the following as eval_coverage.py in a new disposable directory. It reads only its embedded authored CSV and writes the report to standard output. It does not open a business file, change a model configuration or contact a service.

"""Audit authored binary evaluation grades against a fixed case plan."""
import csv
import io
import json
import unittest

CASES = ('quote', 'refund', 'privacy', 'handoff')
CANDIDATES = ('A', 'B')
SAMPLE = '''candidate,case_id,passed
A,quote,1
A,refund,1
B,quote,1
B,refund,1
B,privacy,1
B,handoff,0
'''


def identifiers(values):
    values = tuple(values)
    if not values or any(not isinstance(x, str) or not x or x != x.strip()
                         for x in values) or len(set(values)) != len(values):
        raise ValueError('Plan identifiers must be unique nonblank strings')
    return values


def audit(text, cases=CASES, candidates=CANDIDATES):
    cases, candidates = identifiers(cases), identifiers(candidates)
    reader = csv.reader(io.StringIO(text, newline=''), strict=True)
    try:
        if next(reader, None) != ['candidate', 'case_id', 'passed']:
            raise ValueError('Expected exact candidate,case_id,passed header')
        grades = {}
        for row in reader:
            if len(row) != 3:
                raise ValueError('Every data row must contain three fields')
            candidate, case_id, passed = row
            if candidate not in candidates or case_id not in cases:
                raise ValueError('Unknown candidate or case ID')
            if passed not in ('0', '1'):
                raise ValueError('Grade must be exactly 0 or 1')
            key = (candidate, case_id)
            if key in grades:
                raise ValueError('Duplicate candidate/case grade')
            grades[key] = int(passed)
    except csv.Error as exc:
        raise ValueError('Malformed CSV') from exc
    report = {}
    for candidate in candidates:
        present = [c for c in cases if (candidate, c) in grades]
        passes = sum(grades[candidate, c] for c in present)
        report[candidate] = {
            'planned': len(cases), 'graded': len(present), 'passes': passes,
            'missing': [c for c in cases if (candidate, c) not in grades],
            'graded_coverage': len(present) / len(cases),
            'pass_rate_graded': passes / len(present) if present else None,
            'pass_rate_planned': passes / len(cases),
        }
    return report


class CoverageTests(unittest.TestCase):
    def test_denominator_changes_apparent_winner(self):
        a, b = audit(SAMPLE).values()
        self.assertEqual((a['pass_rate_graded'], a['pass_rate_planned']), (1, .5))
        self.assertEqual((b['pass_rate_graded'], b['pass_rate_planned']), (.75, .75))
        self.assertEqual(a['missing'], ['privacy', 'handoff'])

    def test_no_grades_is_not_perfect(self):
        a = audit('candidate,case_id,passed\n')['A']
        self.assertIsNone(a['pass_rate_graded'])
        self.assertEqual(a['pass_rate_planned'], 0)
        self.assertEqual(a['graded_coverage'], 0)

    def test_duplicate_pair_rejected(self):
        with self.assertRaisesRegex(ValueError, 'Duplicate'):
            audit(SAMPLE + 'A,quote,1\n')

    def test_unknown_case_rejected(self):
        with self.assertRaisesRegex(ValueError, 'Unknown'):
            audit(SAMPLE + 'A,invented,1\n')

    def test_unknown_candidate_rejected(self):
        with self.assertRaisesRegex(ValueError, 'Unknown'):
            audit(SAMPLE + 'C,quote,1\n')

    def test_nonbinary_grade_rejected(self):
        for grade in ('', 'true', '2', 'NaN', ' 1'):
            with self.subTest(grade=grade), self.assertRaises(ValueError):
                audit('candidate,case_id,passed\nA,quote,' + grade + '\n')

    def test_wrong_width_rejected(self):
        for row in ('A,quote', 'A,quote,1,extra', ''):
            with self.subTest(row=row), self.assertRaises(ValueError):
                audit('candidate,case_id,passed\n' + row + '\n')

    def test_header_rejected(self):
        with self.assertRaisesRegex(ValueError, 'header'):
            audit('candidate,case_id,case_id\n')

    def test_bad_plan_rejected(self):
        for cases in ((), ('x', 'x'), (' x',), ('',)):
            with self.subTest(cases=cases), self.assertRaises(ValueError):
                audit(SAMPLE, cases=cases)

    def test_malformed_quoting_rejected(self):
        with self.assertRaisesRegex(ValueError, 'Malformed CSV'):
            audit('candidate,case_id,passed\n"A,quote,1\n')

    def test_quoted_comma_identifier(self):
        result = audit('candidate,case_id,passed\nA,"quote, revised",1\n',
                       cases=('quote, revised',), candidates=('A',))
        self.assertEqual(result['A']['pass_rate_planned'], 1)


if __name__ == '__main__':
    print(json.dumps(audit(SAMPLE), indent=2))

Run both commands from that directory. The first uses the standard library’s unittest command-line runner. It must report 11 passing test methods. The second prints the report shown below. Expected-exception tests actually exercise invalid input; they are not syntax-only checks.

python -X utf8 -m unittest -v eval_coverage.py
python -X utf8 eval_coverage.py

{
  "A": {
    "planned": 4,
    "graded": 2,
    "passes": 2,
    "missing": [
      "privacy",
      "handoff"
    ],
    "graded_coverage": 0.5,
    "pass_rate_graded": 1.0,
    "pass_rate_planned": 0.5
  },
  "B": {
    "planned": 4,
    "graded": 4,
    "passes": 3,
    "missing": [],
    "graded_coverage": 1.0,
    "pass_rate_graded": 0.75,
    "pass_rate_planned": 0.75
  }
}

Use the result to investigate, not to crown a winner

When adapting the function, keep your own frozen case list and candidate identifiers, then pass the CSV text to audit. If reading a local export, open that file with explicit UTF-8 encoding and newline set to an empty string as the CSV documentation recommends. Preserve the raw export separately. This example deliberately requires exact identifiers and exact 0/1 grades; it does not trim whitespace, infer synonyms or convert arbitrary truthy values.

Investigate A’s privacy and handoff records before treating its 100% graded-case score as a complete comparison. Retain run IDs, model/version, prompt version, generation settings, grader/rubric version and missing-case reasons in your actual evaluation system. If you repeat cases, extend the key with a trial ID and specify the expected number of trials; do not remove the duplicate check to make repeated rows disappear.

Use coverage as one release-review input. Complete grades can still be wrong, and a candidate with complete coverage can still fail a critical privacy case. Choose critical-case requirements before looking at the scores. For a broader business decision, combine this report with human review cost and inspect the types of errors, as in the routing regression guide.

What the checks establish

Eleven test methods executed on Python 3.14.3. They cover denominator differences, no grades, duplicate pairs, unknown IDs, invalid grades, field counts, headers, invalid plans, malformed quoting and a quoted comma in an identifier. The final article’s code was extracted and run again in a separate temporary directory. The published code is checked against independently preserved hashes and reproduced after publication.

This is a small in-memory binary-grade audit. It does not verify grader honesty, answer provenance, statistical significance, equal difficulty, latency, token cost, model capabilities or production integrations. It loads the CSV text into memory, assumes one grade per planned candidate/case pair and has no streaming or resource-limit design for large untrusted uploads. No real model, assistant session, vendor API, CI service or customer dataset was tested. Four invented cases cannot support a purchasing recommendation.

Jason Ead publishes AlgoThoughts AI News and connects practical engineering with My Biz Heroes: websites, automation, workflows, AI tools and training. Explore Jason’s work. The futuristic racing cover is generated conceptual artwork; the interior table shows the executed fixture’s arithmetic.