The coding assistant reports that tests passed. The app still contains a misspelled name in a function path that the tests never call. Both statements can be true. A passing test suite only proves what those tests actually checked.
This guide reproduces that gap, catches it with Ruff’s F821 rule, executes the broken path to confirm the runtime failure, then repairs the code and expands the tests. The reusable drill works entirely in a new temporary directory. No existing business repository or assistant configuration is changed.
The fictional intake refactor
Orbit Desk is a fictional business with a small request-routing function. Standard requests go to review; priority requests should go to priority-review. The authored broken fixture has a constant named PRIORITY_QUEUE, but its priority branch returns PRIORITY_QUEU. Its only initial test calls the standard branch.
This is deliberately constructed example code, not a measured failure from Codex, Claude or Antigravity. It does not show that one assistant produces more bugs than another. It demonstrates why the evidence attached to a coding task should include both appropriate static checks and behavior tests.
What Ruff adds to the review
Ruff’s F821 documentation describes checks for undefined names and the resulting risk of a runtime NameError. The 0.16.0 rule definition provides a versioned diagnostic reference. Here the installed executable was Ruff 0.16.0; Python execution used 3.14.3 on Windows.
The linter documentation covers rule selection and command exit codes. The drill selects only F821, avoids fixes and checks the returned diagnostic. It also distinguishes the expected lint exit of 1 from a tool/configuration failure. A syntax-only or formatting pass is not the evidence used here.
| Executed stage | Observed result | Meaning |
|---|---|---|
| Broken code, standard-only test | 1 test passes | Priority behavior was not exercised |
| F821 lint on the same code | Exit 1; PRIORITY_QUEU flagged | Undefined name found without taking the branch |
| Call the priority branch | Exit 1; NameError | The authored defect fails at runtime |
| Repair name and expand tests | 3 tests pass | Standard, priority and unsupported kind checked |
| F821 lint after repair | Exit 0; empty diagnostic list | This selected lint check is clean |
Reproduce the complete failure and repair
Prerequisites: Python 3.14 and Ruff on PATH. This run used an existing Ruff installation; the drill does not install or upgrade anything. Ruff’s 0.16.0 license file starts with the MIT license and includes notices for incorporated work. Use an official distribution and retain the relevant notices if redistributing it. No repository install scripts or paid services are used.
Save the following as review_drill.py. The embedded module and tests are authored fixtures. The runner writes them only inside its new temporary directory, captures every subprocess result and removes that directory on normal exit. It disables Python bytecode writes for the rapid before/after imports and uses isolated, noncached lint settings so a parent project’s Ruff configuration does not change the demonstration.
"""Run an authored lint/runtime review drill in a new temporary directory."""
from pathlib import Path
import json
import shutil
import subprocess
import sys
import tempfile
BROKEN = '''STANDARD_QUEUE = "review"
PRIORITY_QUEUE = "priority-review"
def route(kind):
if kind == "standard":
return STANDARD_QUEUE
if kind == "priority":
return PRIORITY_QUEU
raise ValueError("Unsupported request kind")
'''
THIN_TESTS = '''import unittest
from intake import route
class RouteTests(unittest.TestCase):
def test_standard(self):
self.assertEqual(route("standard"), "review")
'''
FULL_TESTS = THIN_TESTS + '''
def test_priority(self):
self.assertEqual(route("priority"), "priority-review")
def test_unknown(self):
with self.assertRaises(ValueError):
route("unknown")
'''
def drill():
if not __debug__:
raise RuntimeError('Run without Python optimization so drill assertions remain enabled')
ruff = shutil.which('ruff')
if not ruff:
raise RuntimeError('Install Ruff from its official source and place it on PATH')
logs = []
with tempfile.TemporaryDirectory(prefix='orbit-ruff-review-') as folder:
root = Path(folder)
def run(executable, args, expected=0):
r = subprocess.run([executable, *args], cwd=root, capture_output=True,
text=True, timeout=30)
logs.append({'args': args, 'returncode': r.returncode,
'stdout': r.stdout, 'stderr': r.stderr})
assert r.returncode == expected, logs[-1]
return r
version = run(ruff, ['--version']).stdout.strip()
(root / 'intake.py').write_text(BROKEN, encoding='utf-8')
(root / 'test_intake.py').write_text(THIN_TESTS, encoding='utf-8')
test_args = ['-B', '-X', 'utf8', '-m', 'unittest', '-v', 'test_intake.py']
lint_args = ['check', '--isolated', '--no-cache', '--select', 'F821',
'--target-version', 'py314', '--output-format', 'json', 'intake.py']
thin = run(sys.executable, test_args)
assert 'Ran 1 test' in thin.stderr
lint = run(ruff, lint_args, expected=1)
diagnostics = json.loads(lint.stdout)
assert len(diagnostics) == 1 and diagnostics[0]['code'] == 'F821'
assert 'PRIORITY_QUEU' in diagnostics[0]['message']
failed = run(sys.executable, ['-B', '-c',
'from intake import route; print(route("priority"))'], expected=1)
assert 'NameError' in failed.stderr and 'PRIORITY_QUEU' in failed.stderr
fixed = BROKEN.replace('return PRIORITY_QUEU\n', 'return PRIORITY_QUEUE\n')
assert fixed != BROKEN
(root / 'intake.py').write_text(fixed, encoding='utf-8')
(root / 'test_intake.py').write_text(FULL_TESTS, encoding='utf-8')
full = run(sys.executable, test_args)
assert 'Ran 3 tests' in full.stderr
assert json.loads(run(ruff, lint_args).stdout) == []
return {'passed': True, 'ruff': version, 'python': sys.version,
'stages': ['thin test passes', 'F821 identifies typo', 'priority path raises NameError',
'repaired three-test suite passes', 'repaired F821 check passes'],
'logs': logs}
if __name__ == '__main__':
print(json.dumps(drill(), indent=2))
Run this command with ordinary Python assertions enabled:
python -X utf8 review_drill.pyThe output contains the actual tool versions, five stage labels and full stdout/stderr/exit-code logs. The summary fields from the executed run were:
{
"passed": true,
"ruff": "ruff 0.16.0",
"stages": [
"thin test passes",
"F821 identifies typo",
"priority path raises NameError",
"repaired three-test suite passes",
"repaired F821 check passes"
]
}The final article’s script was extracted and this command reproduced in another disposable directory. It confirmed the initial test pass, the specific F821 finding, the actual NameError, the repaired three-test suite and clean selected lint output. A missing Ruff executable or unexpected exit stops the drill; it must not be reported as an application pass.
Ask a coding assistant for evidence, not a green sentence
For a real change, define the affected behavior before implementation. A useful task note is: “Run the repository’s existing checks. Report exact commands and results. Identify changed branches not exercised by tests. Add a regression case for the intended behavior, and resolve relevant undefined-name diagnostics without suppressing the rule to obtain a pass.”
That note is a proposed review contract, not a tested host integration. The drill’s isolated F821-only command is for this demonstration; it should not replace a repository’s broader lint, typing, build or test configuration. Review the diff and confirm why a name or business behavior is changing. Renaming the symbol is appropriate for this authored typo because the fixture’s required queue value is explicit, not because a tool guessed the intended workflow.
Keep static and runtime evidence together in the handoff. For a task in an existing app, combine this with an isolated worktree and dependency consistency checks. The earlier Ruff formatting-hook example addresses a different concern: formatting alone would not demonstrate this behavior.
The boundary of the result
F821 cannot prove that a defined name has the correct business value. Three toy tests cannot establish complete branch coverage, correct authorization, concurrency behavior, persistence or deployment safety. The drill intentionally leaves input typing, complex routing and integrations outside its contract.
No assistant session, hook, CI runner, customer record, routing action or production repository was exercised. Tool execution on authored local code is the scope of the evidence. In your application, select tests from the actual business change, preserve the real project configuration and review unexplored behavior before approval.
Jason Ead publishes AlgoThoughts AI News and connects practical engineering with My Biz Heroes: websites, automation, workflows, AI tools and training. Explore Jason’s work. The cover is generated conceptual artwork; the interior table reports actual drill results.