AI-Native Updated 2026-09 11 min read View as Markdown

Testing Python code that calls a language model

Your function is now non-deterministic, slow, costs money per call and fails in ways an assertion cannot express. Here is a test strategy that still works.

More and more Python code is a thin layer around a model call. That code has properties ordinary Python does not: the same input gives different output, each test costs money and a second or two, and "correct" is often a judgement rather than an equality.

The instinct is to give up and test nothing, or to mock the model and test nothing meaningful. There is a better middle, and it looks like a pyramid.

The pyramid#

      few, slow, expensive
   ┌──────────────────────────┐
   │  4. live evals (nightly) │   real model, scored dataset
   ├──────────────────────────┤
   │  3. cassette tests (CI)  │   recorded responses, real shapes
   ├──────────────────────────┤
   │  2. contract tests       │   parsing, validation, retries, errors
   ├──────────────────────────┤
   │  1. pure logic tests     │   prompt building, chunking, routing
   └──────────────────────────┘
      many, fast, free

The mistake most codebases make is having only layer 4, running it rarely because it is expensive, and therefore having no signal at all during development.

Layer 1: make most of it deterministic#

The majority of an LLM application is not the model call. It is prompt assembly, chunking, retrieval ranking, routing, output post-processing, cost accounting. All of that is ordinary Python and should be tested as ordinary Python.

This only works if you have separated it. The single most valuable structural decision here:

class Completion(Protocol):
    async def __call__(self, *, system: str, messages: list[Message]) -> str: ...

One narrow interface, one implementation that talks to the API, one that does not. Everything above it is testable without a network.

def test_prompt_includes_only_top_k_chunks():
    prompt = build_prompt(query="q", chunks=make_chunks(20), k=5)
    assert prompt.count("<chunk>") == 5


def test_chunks_are_ordered_by_score_descending():
    ...

Unglamorous, fast, and it catches a surprising share of real bugs.

Layer 2: contract tests against a fake#

Test what your code does with a response, not what the model says. Every one of these is a real production failure:

@pytest.mark.parametrize("payload", [
    '{"category": "billing"}',           # happy path
    '```json\n{"category":"billing"}\n```',  # fenced — models do this constantly
    '{"category": "Billing"}',           # wrong case
    '{"categorie": "billing"}',          # misspelled key
    '{"category": "refunds"}',           # not in your enum
    'I think this is a billing issue.',  # ignored the format entirely
    '',                                  # empty
    '{"category": "billing"',            # truncated at the token limit
])
def test_parser_never_raises_and_never_invents(payload):
    result = parse_category(payload)
    assert result is None or result in Category

The assertion is the point: never raise, never invent. A parser that returns a valid-looking wrong answer on malformed input is worse than one that returns None.

Then test the behaviour around it — retries, timeouts, rate limit backoff, what happens on a 500 — with a fake that returns those conditions on demand. None of this needs a real model.

Layer 3: cassettes in CI#

Record real responses once, replay them forever. You get real response shapes — including the weird ones — at zero cost and zero flakiness.

import pytest

@pytest.fixture(scope="module")
def vcr_config():
    return {
        "filter_headers": ["authorization", "x-api-key"],
        "record_mode": "once",
        "match_on": ["method", "scheme", "host", "port", "path", "body"],
    }

@pytest.mark.vcr
async def test_classifies_a_refund_request(client):
    result = await classify("I want my money back")
    assert result.category is Category.REFUNDS

Two rules that make this work rather than rot:

1. Scrub credentials before committing cassettes. filter_headers above, and read the first one you commit.
2. Re-record on a schedule, not never. A cassette from eighteen months ago is testing a model that no longer exists. Monthly is usually right, and the diff when you re-record is genuinely informative.

Layer 4: evals with a scored dataset#

Here you accept non-determinism and measure it instead of asserting on it.

{"input": "my card was charged twice", "expect": "billing"}
{"input": "how do I export my data?", "expect": "support"}
{"input": "cancel and refund please",  "expect": "refunds"}

import asyncio, json, statistics

async def main() -> None:
    cases = [json.loads(l) for l in open("evals/dataset.jsonl")]
    results = await asyncio.gather(*(classify(c["input"]) for c in cases))
    hits = [r.category.value == c["expect"] for r, c in zip(results, cases)]
    accuracy = statistics.mean(hits)
    print(f"accuracy {accuracy:.1%} on {len(cases)} cases")
    for hit, c, r in zip(hits, cases, results):
        if not hit:
            print(f"  MISS {c['input']!r}: got {r.category} want {c['expect']}")
    raise SystemExit(0 if accuracy >= 0.90 else 1)

asyncio.run(main())

Nightly, not per-commit. A threshold, not an assertion. And look at the misses — the list of failures is worth more than the number, because it is where the next prompt change comes from.

When the output is free text#

For summarisation, drafting, or explanation there is no equality to assert. Three approaches, in order of how much you should trust them:

Assert on properties, not content. Does the summary mention every entity in the source? Is it under the length limit? Does it avoid the words we banned? Is it in the requested language? These are deterministic, cheap, and catch most real regressions.

def test_summary_is_grounded(article, summary):
    """Every number in the summary must appear in the source."""
    assert set(re.findall(r"\d[\d,.]*", summary)) <= set(re.findall(r"\d[\d,.]*", article))

That one test catches fabricated figures, which is the failure that matters most in summarisation.

Assert on invariants across runs. Same input twice at temperature 0 should give similar output. A large divergence is a signal even without a ground truth.

LLM-as-judge, carefully. A second model scores the output against a rubric. It works, and it has known biases — position, verbosity, self-preference. Use it for relative comparisons (is variant B better than A?) rather than absolute scores, keep the rubric short and specific, and calibrate it against fifty human-labelled examples before you trust a number from it.

Cost control in CI#

• Layers 1–3 use no tokens. They are the ones on every commit.
• Layer 4 runs nightly and on changes to prompts/ — path filters do most of the work.
• Set a hard spend cap on the CI key, not just an alert.
• Cache aggressively; use the cheapest model that discriminates for the judge.

on:
  push:
  pull_request:
jobs:
  fast:
    steps:
      - run: uv run pytest -q -m "not eval"      # free, every push
  evals:
    if: github.event_name == 'schedule' || contains(github.event.head_commit.modified, 'prompts/')
    steps:
      - run: uv run python evals/run.py

Common questions#

Should I mock the model in unit tests?#

Mock the transport, not the model. Put a narrow protocol between your code and the API, and give it a fake implementation. Mocking the SDK's internals ties your tests to a library version and tests nothing you care about.

How much will running these cost?#

Layers 1-3 use no tokens at all, which is why they are the ones on every commit. Layer 4 is where the money goes, and the levers are batch APIs, a cheap model for the judge, and running nightly rather than per-push. There is a full treatment in tracking and cutting token costs.

How many eval cases do I need?#

Start with twenty real, awkward ones and grow the set from production failures. Statistical power matters less than coverage of the cases that actually bite you, and a small curated set you look at beats a large synthetic one you only read the mean of.

Is LLM-as-judge trustworthy?#

For relative comparisons between two variants, reasonably. For absolute quality scores, not without calibration — judges show position bias, favour longer answers, and prefer output from their own model family. Label fifty examples by hand and check that the judge agrees with you before you let a number gate a deploy.

Temperature 0 makes it deterministic, so can I just assert equality?#

No. Temperature 0 is greedy sampling, not determinism — batching, hardware and provider-side changes all move the output, and any model version change moves it substantially. Assert on properties, or use cassettes.

Disclosure: some links on this page are affiliate links. If you buy something through one, we earn a commission at no extra cost to you. We only list tools we would tell a friend to use, and we say so when we have not used something ourselves. This is how the site stays free and ad-light.