AI-Native Updated 2026-09 11 min read View as Markdown

The verification loop: giving an agent something it cannot fake

An agent is only as good as the signal it gets back. Here is how to build a Python feedback loop that is fast, honest, and hard to game.

The quality of agent output is mostly a function of one thing: how good is the signal it gets after each edit, and how fast does it arrive?

A model asked to write code with no way to run it is doing creative writing. The same model with a four-second test suite is doing engineering, because it can be wrong five times in a row and you will never see the first four attempts.

So the interesting question is not "which model is best at Python". It is "what can I hand it that tells the truth quickly".

Speed is a correctness feature#

There is a threshold effect here that is easy to underestimate.

Nobody instructs an agent to stop running slow tests; it just happens, the same way it happens to humans. If your suite takes four minutes, the highest-leverage thing you can do for agent output quality is not a better prompt — it is splitting the suite.

[tool.pytest.ini_options]
addopts = "-q --strict-markers -p no:cacheprovider"
markers = [
  "integration: needs a database or network. Excluded by default.",
  "slow: over one second. Excluded by default.",
]
# the default run is the fast one
addopts = "-q --strict-markers -m 'not integration and not slow'"

test:            # the loop. must stay under ~5 seconds.
	uv run pytest -q -m "not integration and not slow"

test-all:        # pre-push and CI
	uv run pytest -q

Types are the cheapest signal you have#

A type checker catches an entire class of generated-code error — wrong argument order, None where a value is required, a method that does not exist on that object — at a fraction of the cost of a test, with no test to write.

The problem is that most existing Python codebases produce thousands of errors on day one, so the check is turned off, so the signal is lost.

Baseline it instead. Turn strictness on for new code only, then ratchet.

[tool.mypy]
python_version = "3.12"
strict = true
warn_unreachable = true
files = ["src", "tests"]

# Existing debt: opted out module by module, deleted as it is paid down.
[[tool.mypy.overrides]]
module = ["myapp.legacy.*", "myapp.reporting.old_exports"]
ignore_errors = true

Now make typecheck is green, so it can go in make check, so the agent gets the signal on everything it writes — and the overrides list is a visible, shrinking to-do list rather than an invisible surrender.

Property tests: the check that is hard to game#

Example-based tests have a structural weakness with generated code: the model can see the examples. Given assert total([1,2,3]) == 6, a sufficiently cornered model will special-case the input. It is not being malicious; it is minimising the distance to a passing state.

Property tests remove that option, because there is no specific input to special-case.

from decimal import Decimal
from hypothesis import given, strategies as st

money = st.decimals(min_value=0, max_value=10_000, places=2)


@given(lines=st.lists(st.tuples(st.integers(1, 100), money), min_size=1))
def test_total_never_exceeds_undiscounted_sum(lines):
    cart = Cart([Line(sku="x", qty=q, unit_price=p) for q, p in lines])
    undiscounted = sum(q * p for q, p in lines)
    assert Decimal("0") <= cart.total() <= undiscounted


@given(lines=st.lists(st.tuples(st.integers(1, 100), money), min_size=1))
def test_total_is_order_independent(lines):
    a = Cart([Line("x", q, p) for q, p in lines]).total()
    b = Cart([Line("x", q, p) for q, p in reversed(lines)]).total()
    assert a == b

Two properties, and a whole family of plausible wrong implementations becomes unreachable. Hypothesis also shrinks failures to a minimal case, which is exactly the input an agent needs to fix the bug rather than guess at it.

Checks agents reliably game#

Every check has a cheap way to satisfy it that is not the intended way. Know yours.

That last one is worth enforcing mechanically:

[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B", "S", "SIM", "RUF", "ASYNC"]
# B902/BLE001 broad-except, S110 try-except-pass, S101 assert-in-prod
extend-select = ["BLE", "T20"]   # no broad excepts, no stray print()
ignore = ["E501"]                # formatter owns line length

[tool.ruff.lint.per-file-ignores]
"tests/*" = ["S101"]             # assert is fine in tests

Close the loop with a hook#

The strongest version of this is not asking the agent to run checks — it is making the checks run whether it asks or not. Claude Code supports hooks that fire on tool events; other tools have watchers or pre-commit.

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          { "type": "command", "command": "$CLAUDE_PROJECT_DIR/.claude/hooks/lint.sh" }
        ]
      }
    ]
  }
}

#!/usr/bin/env bash
set -Eeuo pipefail
# The hook receives a JSON object on stdin. Read the path from tool_input —
# that is the form that is stable across releases and identical for every event.
file=$(jq -r '.tool_input.file_path // empty')
[[ "$file" == *.py && -f "$file" ]] || exit 0

uv run ruff check --fix "$file" 2>&1 | tail -20
if out=$(uv run mypy "$file" 2>&1); then exit 0; fi
echo "$out" | tail -20 >&2
exit 2      # exit 2 feeds the errors back to the model to fix

Now every edit is immediately followed by real feedback the model did not choose to request, and exit code 2 hands the errors back to it as something to fix rather than to you as a notification. There is a full treatment of the hook system in harness hooks. In practice this is the single change that most improves output quality on a Python repo, because it removes the "I'll check at the end" failure mode entirely.

A loop worth having#

edit  ->  ruff (instant)  ->  mypy on changed files (~1s)
      ->  fast unit tests (<5s)  ->  property tests on core (~2s)
      ->  integration suite (on demand, before PR)

Fast, honest, and hard to satisfy dishonestly. Everything else — better prompts, bigger models, more detailed instructions — is a smaller effect than this.

Common questions#

My test suite takes six minutes. Where do I start?#

Split it before you optimise it. Mark everything that touches a database, the network or the filesystem as integration, exclude those by default, and add pytest-xdist. Most Python suites are dominated by a small number of slow tests, and getting the default run under five seconds is usually an afternoon's work.

Should the agent be allowed to edit tests?#

Yes, but review test diffs separately and with more suspicion than source diffs. A green suite where the test changed is not evidence of anything. In practice: read the test diff first, then the source diff.

Are property tests worth the learning curve?#

For code with an invariant or a round-trip, yes, and the curve is about an hour. For CRUD glue code, no. Start with the three or four functions in your codebase where a subtle wrongness would be expensive, and leave the rest on examples.

Does any of this replace reading the diff?#

No. The loop raises the floor — it stops obvious wrongness reaching you — but it cannot tell you that the code solves the wrong problem, duplicates something that already exists, or takes an approach you will regret. That judgement is still yours, and it is where your time is now best spent.

Disclosure: some links on this page are affiliate links. If you buy something through one, we earn a commission at no extra cost to you. We only list tools we would tell a friend to use, and we say so when we have not used something ourselves. This is how the site stays free and ad-light.