A policy hook that only judges the lines you touched

Project
qa-automation-framework
Component
bin/qa-policy, run from a PostToolUse hook
Stack
Python, stdlib only, runs on the critical path of every write

The problem

A hook that fails on any forbidden pattern anywhere in a file sounds like a guardrail until it meets a real codebase. Drop it into a repo that already has two hundred hard sleeps in it, and it fails every edit to any file that happens to contain one, including edits that have nothing to do with the sleep. That is not a guardrail, it is a wall, and a wall like that gets the hook deleted within a day.

The pre-existing sleeps are still real problems. The question is where a decision about two hundred of them belongs, and it is not inside a hook that blocks one unrelated line.

What I built

bin/qa-policy checks a file's text against policies.forbid from qa.config.yml: hard sleeps, absolute xpath, an assertion buried inside a conditional so the check can be skipped, a test with no assertion at all, shared mutable state between tests. Five rules, each a small regex or a short heuristic, textual on purpose, because the job is catching the obvious thing quickly in a hook, not parsing five languages correctly. Anything subtler is the suite auditor's job, and that one has a model to reason with.

Wired through hooks/hooks.json as a PostToolUse hook on Write and Edit, it runs after every file an agent touches, reads the tool payload from stdin, and checks that one file.

What it found

The check that makes it survivable is changed_lines(): it reads the file as committed at git show HEAD:<path>, diffs it against the working copy with difflib.SequenceMatcher, and keeps only the line numbers that were inserted or replaced. --hook mode implies --changed-only, so a violation is only ever reported if it sits on a line this edit actually touched.

SituationResult
File has 3 pre-existing sleeps, edit touches an unrelated line0 violations reported
Same file, edit adds one new sleep1 violation reported, that line only
File is new, or git cannot answer the diffevery line checked, the safe direction

When it does fire in hook mode it exits 2, not 1, which is the exit code that feeds the message back to the model that just wrote the file, so the fix happens before the file lands rather than after a human notices it later.

What it cost

A hook that is honest about what it cannot see. It reports two of the five rules as heuristics in its own output, because a conditional assertion or an assertion-free test is a pattern match, not a proof, and it can be wrong in either direction. The pre-existing violations a diff-scoped hook lets through do not disappear; they surface later, in /qa:audit, which is where a decision about two hundred old sleeps actually belongs, made once, on purpose, instead of one file at a time under a red hook.

Back to selected work