# The verification loop: giving an agent something it cannot fake

> Source: https://learn-python.com/ai/feedback-loops/
> Part of Learn Python, free to read.

The quality of agent output is mostly a function of one thing: **how good is the signal it gets after each edit, and how fast does it arrive?**

A model asked to write code with no way to run it is doing creative writing. The same model with a four-second test suite is doing engineering, because it can be wrong five times in a row and you will never see the first four attempts.

So the interesting question is not *"which model is best at Python"*. It is *"what can I hand it that tells the truth quickly"*.

## Speed is a correctness feature

There is a threshold effect here that is easy to underestimate.

| Suite runtime | What the agent does |
|---|---|
| under 5s | Runs it after every edit. Converges. |
| 5–30s | Runs it after every few edits. Usually fine. |
| 30s–2min | Runs it once at the end. You review guesses. |
| over 2 min | Stops running it. Tells you it "should work". |

Nobody instructs an agent to stop running slow tests; it just happens, the same way it happens to humans. If your suite takes four minutes, the highest-leverage thing you can do for agent output quality is not a better prompt — it is splitting the suite.

```toml pyproject.toml
[tool.pytest.ini_options]
addopts = "-q --strict-markers -p no:cacheprovider"
markers = [
  "integration: needs a database or network. Excluded by default.",
  "slow: over one second. Excluded by default.",
]
# the default run is the fast one
addopts = "-q --strict-markers -m 'not integration and not slow'"
```

```makefile
test:            # the loop. must stay under ~5 seconds.
	uv run pytest -q -m "not integration and not slow"

test-all:        # pre-push and CI
	uv run pytest -q
```

:::tip Parallelise before you optimise
`uv add --dev pytest-xdist` then `pytest -q -n auto` is usually a 3–4x win for a few minutes of work. Do this before you spend an afternoon making individual tests faster.
:::

## Types are the cheapest signal you have

A type checker catches an entire class of generated-code error — wrong argument order, `None` where a value is required, a method that does not exist on that object — at a fraction of the cost of a test, with no test to write.

The problem is that most existing Python codebases produce thousands of errors on day one, so the check is turned off, so the signal is lost.

**Baseline it instead.** Turn strictness on for new code only, then ratchet.

```toml pyproject.toml
[tool.mypy]
python_version = "3.12"
strict = true
warn_unreachable = true
files = ["src", "tests"]

# Existing debt: opted out module by module, deleted as it is paid down.
[[tool.mypy.overrides]]
module = ["myapp.legacy.*", "myapp.reporting.old_exports"]
ignore_errors = true
```

Now `make typecheck` is green, so it can go in `make check`, so the agent gets the signal on everything it writes — and the overrides list is a visible, shrinking to-do list rather than an invisible surrender.

:::warn Watch for the `# type: ignore` reflex
Agents under pressure to make a check pass will reach for `# type: ignore`, `cast(Any, x)` and `Optional[...]` widening. Add `warn_unused_ignores = true`, and grep the diff for `type: ignore` before you accept it. An ignore with no comment explaining it is a defect.
:::

## Property tests: the check that is hard to game

Example-based tests have a structural weakness with generated code: the model can see the examples. Given `assert total([1,2,3]) == 6`, a sufficiently cornered model will special-case the input. It is not being malicious; it is minimising the distance to a passing state.

Property tests remove that option, because there is no specific input to special-case.

```python tests/test_pricing_properties.py
from decimal import Decimal
from hypothesis import given, strategies as st

money = st.decimals(min_value=0, max_value=10_000, places=2)


@given(lines=st.lists(st.tuples(st.integers(1, 100), money), min_size=1))
def test_total_never_exceeds_undiscounted_sum(lines):
    cart = Cart([Line(sku="x", qty=q, unit_price=p) for q, p in lines])
    undiscounted = sum(q * p for q, p in lines)
    assert Decimal("0") <= cart.total() <= undiscounted


@given(lines=st.lists(st.tuples(st.integers(1, 100), money), min_size=1))
def test_total_is_order_independent(lines):
    a = Cart([Line("x", q, p) for q, p in lines]).total()
    b = Cart([Line("x", q, p) for q, p in reversed(lines)]).total()
    assert a == b
```

Two properties, and a whole family of plausible wrong implementations becomes unreachable. Hypothesis also shrinks failures to a minimal case, which is exactly the input an agent needs to fix the bug rather than guess at it.

:::note Where property tests pay off most
Parsers, serialisers, money and unit arithmetic, date handling, sorting and dedup, anything with an inverse (`encode`/`decode`, `to_dict`/`from_dict`). If your function has a round-trip property, one `hypothesis` test is worth twenty examples.
:::

## Checks agents reliably game

Every check has a cheap way to satisfy it that is not the intended way. Know yours.

| Check | The cheap way out | Defence |
|---|---|---|
| A single failing test | Hardcode the expected value | Always add the boundary case and one negative case |
| Coverage percentage | Tests that execute but never assert | Never set a coverage target as the goal |
| `mypy` | `# type: ignore`, `Any`, `cast` | `warn_unused_ignores`, grep the diff |
| "Make the tests pass" | Edit the test | Say "without changing tests"; review test diffs separately |
| Lint | `# noqa` | `ruff` with `--extend-select` and no blanket noqa |
| "Handle errors" | `except Exception: pass` | Ban bare and broad excepts in lint config |

That last one is worth enforcing mechanically:

```toml pyproject.toml
[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B", "S", "SIM", "RUF", "ASYNC"]
# B902/BLE001 broad-except, S110 try-except-pass, S101 assert-in-prod
extend-select = ["BLE", "T20"]   # no broad excepts, no stray print()
ignore = ["E501"]                # formatter owns line length

[tool.ruff.lint.per-file-ignores]
"tests/*" = ["S101"]             # assert is fine in tests
```

## Close the loop with a hook

The strongest version of this is not asking the agent to run checks — it is making the checks run whether it asks or not. Claude Code supports hooks that fire on tool events; other tools have watchers or pre-commit.

```json .claude/settings.json
{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          { "type": "command", "command": "$CLAUDE_PROJECT_DIR/.claude/hooks/lint.sh" }
        ]
      }
    ]
  }
}
```

```bash .claude/hooks/lint.sh
#!/usr/bin/env bash
set -Eeuo pipefail
# The hook receives a JSON object on stdin. Read the path from tool_input —
# that is the form that is stable across releases and identical for every event.
file=$(jq -r '.tool_input.file_path // empty')
[[ "$file" == *.py && -f "$file" ]] || exit 0

uv run ruff check --fix "$file" 2>&1 | tail -20
if out=$(uv run mypy "$file" 2>&1); then exit 0; fi
echo "$out" | tail -20 >&2
exit 2      # exit 2 feeds the errors back to the model to fix
```

Now every edit is immediately followed by real feedback the model did not choose to request, and exit code 2 hands the errors back to it as something to fix rather than to you as a notification. There is a full treatment of the hook system in [harness hooks](https://codelearningdojo.com/harness-hooks/). In practice this is the single change that most improves output quality on a Python repo, because it removes the "I'll check at the end" failure mode entirely.

:::promo digitalocean
:::

## A loop worth having

```text
edit  ->  ruff (instant)  ->  mypy on changed files (~1s)
      ->  fast unit tests (<5s)  ->  property tests on core (~2s)
      ->  integration suite (on demand, before PR)
```

Fast, honest, and hard to satisfy dishonestly. Everything else — better prompts, bigger models, more detailed instructions — is a smaller effect than this.

## Common questions

### My test suite takes six minutes. Where do I start?

Split it before you optimise it. Mark everything that touches a database, the network or the filesystem as `integration`, exclude those by default, and add `pytest-xdist`. Most Python suites are dominated by a small number of slow tests, and getting the default run under five seconds is usually an afternoon's work.

### Should the agent be allowed to edit tests?

Yes, but review test diffs separately and with more suspicion than source diffs. A green suite where the test changed is not evidence of anything. In practice: read the test diff first, then the source diff.

### Are property tests worth the learning curve?

For code with an invariant or a round-trip, yes, and the curve is about an hour. For CRUD glue code, no. Start with the three or four functions in your codebase where a subtle wrongness would be expensive, and leave the rest on examples.

### Does any of this replace reading the diff?

No. The loop raises the floor — it stops obvious wrongness reaching you — but it cannot tell you that the code solves the wrong problem, duplicates something that already exists, or takes an approach you will regret. That judgement is still yours, and it is where your time is now best spent.