# Testing Python code that calls a language model

> Source: https://learn-python.com/ai/evals/
> Part of Learn Python, free to read.

More and more Python code is a thin layer around a model call. That code has properties ordinary Python does not: the same input gives different output, each test costs money and a second or two, and "correct" is often a judgement rather than an equality.

The instinct is to give up and test nothing, or to mock the model and test nothing meaningful. There is a better middle, and it looks like a pyramid.

## The pyramid

```text
      few, slow, expensive
   ┌──────────────────────────┐
   │  4. live evals (nightly) │   real model, scored dataset
   ├──────────────────────────┤
   │  3. cassette tests (CI)  │   recorded responses, real shapes
   ├──────────────────────────┤
   │  2. contract tests       │   parsing, validation, retries, errors
   ├──────────────────────────┤
   │  1. pure logic tests     │   prompt building, chunking, routing
   └──────────────────────────┘
      many, fast, free
```

The mistake most codebases make is having only layer 4, running it rarely because it is expensive, and therefore having no signal at all during development.

## Layer 1: make most of it deterministic

The majority of an LLM application is not the model call. It is prompt assembly, chunking, retrieval ranking, routing, output post-processing, cost accounting. All of that is ordinary Python and should be tested as ordinary Python.

This only works if you have separated it. The single most valuable structural decision here:

```python src/app/llm.py
class Completion(Protocol):
    async def __call__(self, *, system: str, messages: list[Message]) -> str: ...
```

One narrow interface, one implementation that talks to the API, one that does not. Everything above it is testable without a network.

```python
def test_prompt_includes_only_top_k_chunks():
    prompt = build_prompt(query="q", chunks=make_chunks(20), k=5)
    assert prompt.count("<chunk>") == 5


def test_chunks_are_ordered_by_score_descending():
    ...
```

Unglamorous, fast, and it catches a surprising share of real bugs.

## Layer 2: contract tests against a fake

Test what your code does with a response, not what the model says. Every one of these is a real production failure:

```python
@pytest.mark.parametrize("payload", [
    '{"category": "billing"}',           # happy path
    '```json\n{"category":"billing"}\n```',  # fenced — models do this constantly
    '{"category": "Billing"}',           # wrong case
    '{"categorie": "billing"}',          # misspelled key
    '{"category": "refunds"}',           # not in your enum
    'I think this is a billing issue.',  # ignored the format entirely
    '',                                  # empty
    '{"category": "billing"',            # truncated at the token limit
])
def test_parser_never_raises_and_never_invents(payload):
    result = parse_category(payload)
    assert result is None or result in Category
```

The assertion is the point: **never raise, never invent**. A parser that returns a valid-looking wrong answer on malformed input is worse than one that returns `None`.

Then test the behaviour around it — retries, timeouts, rate limit backoff, what happens on a 500 — with a fake that returns those conditions on demand. None of this needs a real model.

:::tip Use structured output, then test the fallback anyway
Constrained decoding and JSON-schema modes remove most parse failures and you should absolutely use them. Test the fallback path regardless: providers have outages, you will change models, and the day the schema mode fails is the day you find out whether your parser was defensive.
:::

## Layer 3: cassettes in CI

Record real responses once, replay them forever. You get real response shapes — including the weird ones — at zero cost and zero flakiness.

```python conftest.py
import pytest

@pytest.fixture(scope="module")
def vcr_config():
    return {
        "filter_headers": ["authorization", "x-api-key"],
        "record_mode": "once",
        "match_on": ["method", "scheme", "host", "port", "path", "body"],
    }
```

```python
@pytest.mark.vcr
async def test_classifies_a_refund_request(client):
    result = await classify("I want my money back")
    assert result.category is Category.REFUNDS
```

Two rules that make this work rather than rot:

1. **Scrub credentials before committing cassettes.** `filter_headers` above, and read the first one you commit.
2. **Re-record on a schedule**, not never. A cassette from eighteen months ago is testing a model that no longer exists. Monthly is usually right, and the diff when you re-record is genuinely informative.

## Layer 4: evals with a scored dataset

Here you accept non-determinism and measure it instead of asserting on it.

```python evals/dataset.jsonl
{"input": "my card was charged twice", "expect": "billing"}
{"input": "how do I export my data?", "expect": "support"}
{"input": "cancel and refund please",  "expect": "refunds"}
```

```python evals/run.py
import asyncio, json, statistics

async def main() -> None:
    cases = [json.loads(l) for l in open("evals/dataset.jsonl")]
    results = await asyncio.gather(*(classify(c["input"]) for c in cases))
    hits = [r.category.value == c["expect"] for r, c in zip(results, cases)]
    accuracy = statistics.mean(hits)
    print(f"accuracy {accuracy:.1%} on {len(cases)} cases")
    for hit, c, r in zip(hits, cases, results):
        if not hit:
            print(f"  MISS {c['input']!r}: got {r.category} want {c['expect']}")
    raise SystemExit(0 if accuracy >= 0.90 else 1)

asyncio.run(main())
```

Nightly, not per-commit. A threshold, not an assertion. And **look at the misses** — the list of failures is worth more than the number, because it is where the next prompt change comes from.

:::note Building the dataset is the actual work
Fifty real, awkward examples beat five hundred synthetic ones. Take them from production logs, from support tickets, from the bug reports you get. Every time something goes wrong in production, the input becomes a case. That is the flywheel; the harness is trivial by comparison.
:::

## When the output is free text

For summarisation, drafting, or explanation there is no equality to assert. Three approaches, in order of how much you should trust them:

**Assert on properties, not content.** Does the summary mention every entity in the source? Is it under the length limit? Does it avoid the words we banned? Is it in the requested language? These are deterministic, cheap, and catch most real regressions.

```python
def test_summary_is_grounded(article, summary):
    """Every number in the summary must appear in the source."""
    assert set(re.findall(r"\d[\d,.]*", summary)) <= set(re.findall(r"\d[\d,.]*", article))
```

That one test catches fabricated figures, which is the failure that matters most in summarisation.

**Assert on invariants across runs.** Same input twice at temperature 0 should give similar output. A large divergence is a signal even without a ground truth.

**LLM-as-judge, carefully.** A second model scores the output against a rubric. It works, and it has known biases — position, verbosity, self-preference. Use it for *relative* comparisons (is variant B better than A?) rather than absolute scores, keep the rubric short and specific, and calibrate it against fifty human-labelled examples before you trust a number from it.

## Cost control in CI

- Layers 1–3 use no tokens. They are the ones on every commit.
- Layer 4 runs nightly and on changes to `prompts/` — path filters do most of the work.
- Set a hard spend cap on the CI key, not just an alert.
- Cache aggressively; use the cheapest model that discriminates for the judge.

```yaml .github/workflows/test.yml
on:
  push:
  pull_request:
jobs:
  fast:
    steps:
      - run: uv run pytest -q -m "not eval"      # free, every push
  evals:
    if: github.event_name == 'schedule' || contains(github.event.head_commit.modified, 'prompts/')
    steps:
      - run: uv run python evals/run.py
```

:::promo digitalocean
:::

## Common questions

### Should I mock the model in unit tests?

Mock the *transport*, not the model. Put a narrow protocol between your code and the API, and give it a fake implementation. Mocking the SDK's internals ties your tests to a library version and tests nothing you care about.

### How much will running these cost?

Layers 1-3 use no tokens at all, which is why they are the ones on every commit. Layer 4 is where the money goes, and the levers are batch APIs, a cheap model for the judge, and running nightly rather than per-push. There is a full treatment in [tracking and cutting token costs](/ai/tokenomics/).

### How many eval cases do I need?

Start with twenty real, awkward ones and grow the set from production failures. Statistical power matters less than coverage of the cases that actually bite you, and a small curated set you look at beats a large synthetic one you only read the mean of.

### Is LLM-as-judge trustworthy?

For relative comparisons between two variants, reasonably. For absolute quality scores, not without calibration — judges show position bias, favour longer answers, and prefer output from their own model family. Label fifty examples by hand and check that the judge agrees with you before you let a number gate a deploy.

### Temperature 0 makes it deterministic, so can I just assert equality?

No. Temperature 0 is greedy sampling, not determinism — batching, hardware and provider-side changes all move the output, and any model version change moves it substantially. Assert on properties, or use cassettes.