A policy hook that only judges the lines you touched
- Project
- qa-automation-framework
- Component
bin/qa-policy, run from aPostToolUsehook- Stack
- Python, stdlib only, runs on the critical path of every write
The problem
A hook that fails on any forbidden pattern anywhere in a file sounds like a guardrail until it meets a real codebase. Drop it into a repo that already has two hundred hard sleeps in it, and it fails every edit to any file that happens to contain one, including edits that have nothing to do with the sleep. That is not a guardrail, it is a wall, and a wall like that gets the hook deleted within a day.
The pre-existing sleeps are still real problems. The question is where a decision about two hundred of them belongs, and it is not inside a hook that blocks one unrelated line.
What I built
bin/qa-policy checks a file's text against policies.forbid from
qa.config.yml: hard sleeps, absolute xpath, an assertion buried inside a
conditional so the check can be skipped, a test with no assertion at all, shared mutable
state between tests. Five rules, each a small regex or a short heuristic, textual on
purpose, because the job is catching the obvious thing quickly in a hook, not parsing
five languages correctly. Anything subtler is the suite auditor's job, and that one has a
model to reason with.
Wired through hooks/hooks.json as a PostToolUse hook on
Write and Edit, it runs after every file an agent touches, reads
the tool payload from stdin, and checks that one file.
What it found
The check that makes it survivable is changed_lines(): it reads the file as
committed at git show HEAD:<path>, diffs it against the working copy
with difflib.SequenceMatcher, and keeps only the line numbers that were
inserted or replaced. --hook mode implies --changed-only, so a
violation is only ever reported if it sits on a line this edit actually touched.
| Situation | Result |
|---|---|
| File has 3 pre-existing sleeps, edit touches an unrelated line | 0 violations reported |
| Same file, edit adds one new sleep | 1 violation reported, that line only |
| File is new, or git cannot answer the diff | every line checked, the safe direction |
When it does fire in hook mode it exits 2, not 1, which is the exit code that feeds the message back to the model that just wrote the file, so the fix happens before the file lands rather than after a human notices it later.
What it cost
A hook that is honest about what it cannot see. It reports two of the five rules as
heuristics in its own output, because a conditional assertion or an assertion-free test
is a pattern match, not a proof, and it can be wrong in either direction. The pre-existing
violations a diff-scoped hook lets through do not disappear; they surface later, in
/qa:audit, which is where a decision about two hundred old sleeps actually
belongs, made once, on purpose, instead of one file at a time under a red hook.