# Learn Python

> Free Python tutorials plus the part nobody else covers: how to configure coding agents for Python, and how to review the Python they write.

Canonical: https://learn-python.com/
Licence: content free to read and quote with attribution to Learn Python (https://learn-python.com/).
Maintainer: Code Learning Dojo. Last built 2026-09-06.

## Foundations

The syntax and the mental model. Short, runnable, no fluff.

- [Hello, World!](https://learn-python.com/hello-world/): Python is a very simple language, and has a very straightforward syntax. It encourages programmers to program without boilerplate (prepared) code.
- [Variables and Types](https://learn-python.com/variables-and-types/): Python is completely object oriented, and not “statically typed”. You do not need to declare variables before using them, or declare their type.
- [Lists](https://learn-python.com/lists/): Lists are very similar to arrays. They can contain any type of variable, and they can contain as many variables as you wish.
- [Basic Operators](https://learn-python.com/basic-operators/): This section explains how to use basic operators in Python.
- [String Formatting](https://learn-python.com/string-formatting/): Python uses C-style string formatting to create new, formatted strings.
- [Basic String Operations](https://learn-python.com/basic-string-operations/): Strings are bits of text. They can be defined as anything between quotes: As you can see, the first thing you learned was printing a simple sentence.
- [Conditions](https://learn-python.com/conditions/): Python uses boolean logic to evaluate conditions. The boolean values True and False are returned when an expression is compared or evaluated.
- [Loops](https://learn-python.com/loops/): There are two types of loops in Python, for and while. For loops iterate over a given sequence.
- [Functions](https://learn-python.com/functions/): Functions are a convenient way to divide your code into useful blocks, allowing us to order our code, make it more readable, reuse it and save some time.
- [Classes and Objects](https://learn-python.com/classes-and-objects/): Objects are an encapsulation of variables and functions into a single entity. Objects get their variables and functions from classes.
- [Dictionaries](https://learn-python.com/dictionaries/): A dictionary is a data type similar to arrays, but works with keys and values instead of indexes.
- [Modules and Packages](https://learn-python.com/modules-and-packages/): In programming, a module is a piece of software that has a specific functionality.
- [Files and Context Managers](https://learn-python.com/files-and-context-managers/): Reading and writing files, and the `with` statement that guarantees cleanup — the single most idiomatic construct in Python.
- [Generators](https://learn-python.com/generators/): Generators are very easy to implement, but a bit difficult to understand. Generators are used to create iterators, but with a different approach.
- [List Comprehensions](https://learn-python.com/list-comprehensions/): List Comprehensions is a very powerful tool, which creates a new list based on another list, in a single, readable line.
- [Multiple Function Arguments](https://learn-python.com/multiple-function-arguments/): Every function in Python receives a predefined number of arguments, if declared normally, like this: It is possible to declare functions which receive a variable number of arguments, using the following syntax: The “therest” variable is a list of variables, which receives all arguments which were given to the “foo” function after the first 3 arguments.
- [Regular Expressions](https://learn-python.com/regular-expressions/): Regular Expressions (sometimes shortened to regexp, regex, or re) are a tool for matching patterns in text. In Python, we have the re module.
- [Exception Handling](https://learn-python.com/exception-handling/): When programming, errors happen. It’s just a fact of life. Perhaps the user gave bad input. Maybe a network resource was unavailable.
- [Sets](https://learn-python.com/sets/): Sets are lists with no duplicate entries.
- [Serialization](https://learn-python.com/serialization/): Python provides built-in JSON libraries to encode and decode JSON. In Python 2.5, the simplejson module is used, whereas in Python 2.7, the json module is used.
- [Partial functions](https://learn-python.com/partial-functions/): You can create partial functions in python by using the partial function from the functools library.
- [Code Introspection](https://learn-python.com/code-introspection/): Code introspection is the ability to examine classes, functions and keywords to know what they are, what they do and what they know.
- [Closures](https://learn-python.com/closures/): A Closure is a function object that remembers values in enclosing scopes even if they are not present in memory.
- [Decorators](https://learn-python.com/decorators/): Decorators allow you to make simple modifications to callable objects like functions, methods, or classes. We shall deal with functions for this tutorial.
- [Map, Filter, Reduce](https://learn-python.com/map-filter-reduce/): Map, Filter, and Reduce are paradigms of functional programming.
- [Numpy Arrays](https://learn-python.com/numpy-arrays/): Numpy arrays are great alternatives to Python Lists.
- [Pandas Basics](https://learn-python.com/pandas-basics/): Pandas is a high-level data manipulation tool developed by Wes McKinney. It is built on the Numpy package and its key data structure is called the DataFrame.
- [Type Hints](https://learn-python.com/type-hints/): Optional annotations that a checker enforces before your code runs. Python stays dynamic; you get most of the safety anyway.
- [Dataclasses](https://learn-python.com/dataclasses/): A decorator that writes the boilerplate for classes that mostly hold data — which is most classes.
- [Async and Await](https://learn-python.com/async-await/): Concurrency for I/O-bound work — and the one mistake that silently makes an async program slower than the synchronous version.

## AI-Native

Configuring agents, harnesses and feedback loops for this language. Updated as the tooling moves.

- [Setting up a coding agent for a Python project](https://learn-python.com/ai/agent-setup/): The twenty minutes of setup that decide whether an agent is useful on your Python codebase or an expensive way to generate rework.
- [Writing an AGENTS.md for Python that agents actually follow](https://learn-python.com/ai/agents-md/): Most AGENTS.md files are 400 lines of advice the model already knew. Here is what earns its place in the context window, and a Python template you can copy.
- [The verification loop: giving an agent something it cannot fake](https://learn-python.com/ai/feedback-loops/): An agent is only as good as the signal it gets back. Here is how to build a Python feedback loop that is fast, honest, and hard to game.
- [Structuring a Python repo an agent can navigate](https://learn-python.com/ai/context/): Codebase layout used to be a question of taste. It is now a performance parameter — for your agent and, it turns out, for the humans too.
- [MCP servers worth wiring into a Python project](https://learn-python.com/ai/mcp/): MCP lets an agent reach outside your codebase. Most of what gets installed is noise; a few of them change what the agent can actually do.
- [From issue to pull request: running a feature with an agent](https://learn-python.com/ai/spec-to-pr/): The end-to-end workflow, including the two steps everybody skips and then pays for later.
- [When not to hand it to the agent](https://learn-python.com/ai/when-not-to/): The cases where delegating costs more than doing it yourself — and the one that quietly costs the most.
- [Testing Python code that calls a language model](https://learn-python.com/ai/evals/): Your function is now non-deterministic, slow, costs money per call and fails in ways an assertion cannot express. Here is a test strategy that still works.
- [Tracking and cutting token costs in Python](https://learn-python.com/ai/tokenomics/): Counting tokens before you send them, attributing every call to a feature, and the four changes that usually halve the bill.

## Review & Verify

How generated code fails in this language, and the checks that catch it before your users do.

- [The Python mistakes language models actually make](https://learn-python.com/review/failure-modes/): A working catalogue of the bugs that show up over and over in generated Python — what each one looks like, why models produce it, and the check that catches it.
- [Hallucinated packages, slopsquatting, and dependency hygiene](https://learn-python.com/review/dependencies/): Language models invent package names. Attackers have noticed, and register them. Here is the check that takes four seconds.
- [Security review checklist for AI-generated Python](https://learn-python.com/review/security/): Generated code is not more malicious than human code. It is more confidently insecure, in a small number of predictable ways, at higher volume.
- [The performance traps in generated Python](https://learn-python.com/review/performance/): Generated Python is usually correct and frequently slow, in a small number of recognisable ways. None of them show up in a test.

## Reference pages

- [About Learn Python, and how we make money](https://learn-python.com/about/): Editorial policy, sourcing, corrections and affiliate disclosure for Learn Python, part of the Code Learning Dojo network.
- [The Python stack we would set up today](https://learn-python.com/tools/): A current, opinionated Python toolchain: package manager, linter, type checker, test runner, editor, agent, and hosting. What we use, what we would skip, and why.

---

# Full text

## Hello, World!
Source: https://learn-python.com/hello-world/

Python is a very simple language, and has a very straightforward syntax.
It encourages programmers to program without boilerplate (prepared) code.
The simplest directive in Python is the “print” directive -
it simply prints out a line (and also includes a newline, unlike in C).

There are two major Python versions, Python 2 and Python 3. Python 2 and 3 are quite different.
This tutorial uses Python 3, because it more semantically correct and supports newer features.

For example, one difference between Python 2 and 3 is the `print` statement.
In Python 2, the “print” statement is not a function, and therefore it is
invoked without parentheses. However, in Python 3, it is a function, and must be invoked
with parentheses.

To print a string in Python 3, just write:

```python
print("This line will be printed.")
```

### Indentation

Python uses indentation for blocks, instead of curly braces. Both tabs and spaces are supported, but the standard
indentation requires standard Python code to use four spaces. For example:

```python
x = 1
if x == 1:
    # indented four spaces
    print("x is 1.")
```

## Setting up a coding agent for a Python project
Source: https://learn-python.com/ai/agent-setup/

Most people try a coding agent on a Python repo, get plausible-looking code that fails in a way they only discover twenty minutes later, and conclude the tool is overhyped. Usually the tool was fine. The setup was missing.

An agent is a loop: propose an edit, run something, read the result, adjust. If there is nothing meaningful for it to run, the loop collapses into a single guess with no correction step, and you are back to copy-pasting from a chat window. **Everything below exists to give the loop something real to push against.**

:::note
This page is deliberately tool-agnostic. It applies to Claude Code, Cursor, Codex CLI, Aider, Cline, Windsurf and anything else that can edit files and run commands. Where behaviour genuinely differs, we say which tool.
:::

## 1. Make the environment reproducible in one command

The single biggest cause of wasted agent turns is an environment the agent cannot recreate. If your project needs a specific Python version, a virtualenv someone activated by hand three weeks ago, and two environment variables that live in a colleague's shell, the agent will spend its turns debugging your machine instead of your code.

Use `uv`. It resolves, installs and pins in one place, and it is fast enough that an agent can afford to run it repeatedly.

```bash
# one-time, in the repo root
uv init --python 3.12
uv add --dev pytest pytest-cov ruff mypy
uv sync
```

Now the entire environment is two files — `pyproject.toml` and `uv.lock` — and any agent, on any machine, gets an identical one with `uv sync`.

:::tip Why this matters more than it looks
An agent that can run `uv run pytest` and get a real answer in four seconds will iterate five times before showing you anything. An agent that has to ask you to activate a venv will show you its first guess. The difference in output quality is not subtle.
:::

## 2. Give it exactly four commands

Agents do better with a small, named set of verbs than with a README paragraph describing your workflow. Put them somewhere executable — a `Makefile`, a `justfile`, or `[tool.uv]` scripts — and name them predictably.

```makefile Makefile
.PHONY: install test lint typecheck check

install:
	uv sync

test:
	uv run pytest -q

lint:
	uv run ruff check --fix . && uv run ruff format .

typecheck:
	uv run mypy src

check: lint typecheck test
```

`make check` is now the whole contract. The agent does not have to guess whether you use `black` or `ruff format`, whether tests live in `tests/` or beside the source, or whether type checking is expected to pass. One command, one exit code.

:::warn Make `check` actually fail
A `make check` that exits 0 on a broken codebase is worse than no command at all — it teaches the agent that finished means finished. If your type checking currently produces 400 errors, do not wire it into `check` yet. Baseline it first (see [The verification loop](/ai/feedback-loops/)), then turn it on.
:::

## 3. Decide what it may run without asking

Every serious agent has a permission model. Configure it once; otherwise you spend your session clicking Allow, which trains you to click Allow without reading — the single most dangerous habit in agentic development.

An allowlist is the coarse control. For policy that reacts to *what* is being run — deny this shape, ask before that one — you want a hook; there is a full reference in [harness hooks](https://codelearningdojo.com/harness-hooks/). And for the containment that catches what no policy anticipated — a bad turn, an injected instruction — see [sandboxing a coding agent](https://codelearningdojo.com/sandboxing/).

The rule of thumb: **auto-allow anything that only reads or only affects the working tree; always confirm anything that touches the network, the package index, or another machine.**

```json .claude/settings.json
{
  "permissions": {
    "allow": [
      "Bash(uv run pytest:*)",
      "Bash(uv run ruff:*)",
      "Bash(uv run mypy:*)",
      "Bash(make test)",
      "Bash(make lint)",
      "Bash(make check)",
      "Bash(git status)",
      "Bash(git diff:*)",
      "Bash(git log:*)"
    ],
    "ask": [
      "Bash(uv add:*)",
      "Bash(pip install:*)",
      "Bash(git push:*)",
      "Bash(gh pr create:*)"
    ],
    "deny": [
      "Bash(curl:*)",
      "Bash(rm -rf:*)",
      "Read(./.env)",
      "Read(./.env.*)",
      "Read(./secrets/**)"
    ]
  }
}
```

Cursor and Windsurf have equivalent allowlists in their settings UI; Aider has `--yes-always` (don't) and per-command confirmation (do).

:::danger The .env rule is not optional
Agents read files to build context, and context goes to a model provider. Deny-listing `.env`, `.env.*`, `secrets/`, `*.pem` and `~/.aws` costs you nothing and removes an entire category of incident. Do it before your first session, not after. It is also one edge of the [prompt-injection triangle](https://codelearningdojo.com/prompt-injection/) — data the agent cannot read cannot be exfiltrated.
:::

## 4. Work in a sandbox you are willing to lose

The best posture for agentic work is one where a bad turn costs you nothing. In practice:

- **A branch, always.** `git switch -c agent/thing`. Never point an agent at a dirty `main`.
- **Commit before you start.** An uncommitted working tree is the one thing an agent can genuinely destroy.
- **Consider a worktree** for longer runs, so you can keep working in the main checkout: `git worktree add ../proj-agent -b agent/thing`.
- **Containers for anything untrusted.** If you are letting an agent run code it wrote against a real database, put it in a container with a throwaway copy.

## 5. Point it at the tests, not at the code

The most common bad instruction is *"add feature X to `service.py`"*. The most common good one is *"here's a failing test that describes feature X; make it pass without changing the test."*

The second version gives the loop a termination condition that the model cannot talk itself out of. It also forces you to say what you actually want precisely enough to assert on — which is where half the value comes from, agent or not.

```python tests/test_pricing.py
def test_bulk_discount_applies_at_ten_units():
    cart = Cart([Line(sku="A1", qty=10, unit_price=Decimal("5.00"))])
    assert cart.total() == Decimal("45.00")   # 10% off at 10+


def test_bulk_discount_does_not_apply_at_nine():
    cart = Cart([Line(sku="A1", qty=9, unit_price=Decimal("5.00"))])
    assert cart.total() == Decimal("45.00")   # 9 x 5.00, no discount
```

Note the second test. A single test is an invitation to hardcode; the pair pins the boundary. Writing the boundary case yourself is worth more than any prompt engineering you will ever do.

:::promo jetbrains
:::

## 6. Write the AGENTS.md last, not first

Everyone's instinct is to start by writing a long instructions file. Don't. You do not yet know which mistakes your agent makes on *your* codebase, so you will write generic advice the model already follows, and you will bloat the context for nothing.

Run three or four real tasks first. Every time you find yourself correcting the same thing twice, that correction is a line in `AGENTS.md`. Everything else is noise.

We wrote a whole page on getting this file right: [Writing an AGENTS.md for Python that agents actually follow](/ai/agents-md/).

## What this costs

Agent sessions are billed, and the bill is driven by context size more than by how much you ask for. The levers — prompt caching, pruning unused MCP servers, starting a fresh session when the task changes — are in [what tokens actually cost](https://codelearningdojo.com/token-economics/).

## The twenty-minute checklist

```text
[ ] uv-managed environment, uv.lock committed
[ ] make install / test / lint / typecheck / check all work from a clean clone
[ ] make check genuinely fails on a broken tree
[ ] permission allowlist configured; .env and secrets/ denied
[ ] working on a branch, tree committed before each run
[ ] at least one test file the agent can run in under 10 seconds
[ ] AGENTS.md deliberately left empty until you have evidence for it
```

If all seven are true, you have a working loop. Everything after this is refinement.

## Common questions

### Do I need a different setup for Claude Code, Cursor and Codex?

No. The substance — reproducible environment, a small set of named commands, a permission allowlist, a fast test — is identical across tools. Only the configuration file names differ: `.claude/settings.json` and `CLAUDE.md` for Claude Code, `.cursor/rules` for Cursor, `AGENTS.md` for Codex and increasingly for everything else. Write `AGENTS.md` and symlink the tool-specific names to it.

### Should the agent be allowed to install packages?

Put package installation behind a confirmation, not on the allowlist. Language models hallucinate package names at a measurable rate, and attackers register the hallucinated ones — see [Hallucinated packages and slopsquatting](/review/dependencies/). A one-second glance at `uv add <name>` before it runs is the cheapest security control available to you.

### Is it worth doing all this for a small script?

No. For a fifty-line script, ask in a chat window and read the output. This setup pays off on a codebase you will still be maintaining in six months, where the cost of a wrong edit is measured in debugging time rather than in re-reading a screen.

### What about test coverage — should I make the agent chase a number?

Avoid it. Coverage targets are trivially gamed by tests that execute code without asserting anything useful, and an agent optimising for a number will produce exactly that. Ask for tests that pin behaviour at boundaries, then read them yourself.

## The Python mistakes language models actually make
Source: https://learn-python.com/review/failure-modes/

Generated Python fails in patterns, not at random. The patterns come from the training data: a model has read millions of lines of Python, most of it written before 2023, much of it tutorial code that was never load-bearing. When it is uncertain, it reaches for the most common thing it has seen — and the most common thing is frequently the thing that was fine in a blog post and is not fine in your service.

This is the list we keep coming back to. Each entry has the shape of the bug, why it appears, and the mechanical check that catches it so you do not have to rely on noticing.

:::note How to use this
Do not read it as a list of reasons to distrust the tool. Read it as a review checklist. Most of these are catchable by configuration rather than attention, and the point of the "catch it with" line is to get them out of your head and into `ruff`.
:::

## Correctness

### 1. Mutable default arguments

```python
def add_item(item, basket=[]):        # every call shares one list
    basket.append(item)
    return basket
```

The classic. It appears because it is enormously represented in the training data — in both directions, as an example of the bug and as real code. Models reproduce the shape.

**Correct:** `def add_item(item, basket: list | None = None): basket = [] if basket is None else basket`

**Catch it with:** `ruff` rule `B006` (flake8-bugbear). Non-negotiable in `select`.

### 2. Naive datetimes

```python
created = datetime.utcnow()           # deprecated, and tz-naive
if created < deadline:                # comparing naive to aware -> TypeError
    ...
```

`datetime.utcnow()` has been deprecated since 3.12 but is overwhelmingly present in training data. The resulting naive datetime then flows into comparisons, database columns and serialisation, and produces bugs that only appear across a DST boundary or in a different deployment region.

**Correct:** `datetime.now(timezone.utc)` — and pick one rule (everything aware, UTC) and put it in `AGENTS.md`.

**Catch it with:** `ruff` rule `DTZ` (flake8-datetimez). Turn on the whole family.

### 3. Floats for money

```python
total = 0.1 + 0.2                     # 0.30000000000000004
price = round(qty * unit_price, 2)    # rounds half-to-even, surprising in finance
```

Models default to `float` because most numeric Python in training data is scientific, where `float` is right. In billing it is not.

**Correct:** `Decimal`, constructed from strings, with an explicit `quantize` and rounding mode at the boundary.

**Catch it with:** a type. Make your money type `Decimal` in the domain layer and let `mypy` reject the `float` at the door. No linter finds this one for you.

### 4. Late binding in loops

```python
handlers = [lambda: print(i) for i in range(3)]
[h() for h in handlers]               # 2, 2, 2 — not 0, 1, 2
```

Shows up most in generated callback registration, retry wrappers and click handlers.

**Correct:** `lambda i=i: print(i)`, or `functools.partial`.

**Catch it with:** `ruff` rule `B023` (function definition does not bind loop variable).

### 5. Blocking calls inside `async def`

```python
async def fetch_user(uid: str):
    r = requests.get(f"{API}/users/{uid}")   # blocks the whole event loop
    time.sleep(0.2)                          # so does this
    return r.json()
```

The most damaging item on this list, because it does not fail — it just quietly serialises your entire service and shows up as latency under load, weeks later.

Models produce it because `requests` is far more represented in training data than `httpx`, and because the code reads correctly.

**Correct:** `httpx.AsyncClient`, `asyncio.sleep`, `aiofiles`, and `asyncio.to_thread()` for anything genuinely blocking.

**Catch it with:** `ruff` rules `ASYNC` (flake8-async), plus `blockbuster` or `asyncio` debug mode in tests. Add `ASYNC` to `select` today if you run any async code.

### 6. Broad exception handling

```python
try:
    result = risky()
except Exception:
    result = None                     # the incident report starts here
```

Appears whenever a prompt contains the phrase "handle errors gracefully". The model interprets graceful as silent.

**Correct:** catch the specific exception you can do something about; let the rest propagate. If you must catch broadly at a boundary, log with `exc_info=True` and re-raise or return a typed failure.

**Catch it with:** `ruff` rules `BLE001` (blind except) and `S110` (try-except-pass).

### 7. `assert` for runtime validation

```python
def withdraw(account, amount):
    assert amount > 0, "amount must be positive"   # vanishes under python -O
```

**Correct:** `if amount <= 0: raise ValueError(...)`.

**Catch it with:** `ruff` rule `S101`, with a per-file ignore for `tests/`.

### 8. Mutating a collection while iterating it

```python
for user in users:
    if user.inactive:
        users.remove(user)            # skips elements, silently
```

**Correct:** build a new list, or iterate a copy (`for user in list(users)`).

**Catch it with:** `ruff` rule `B909` where available; otherwise a review habit. Property tests catch the resulting off-by-one behaviour reliably.

### 9. Dict and set ordering assumptions

Dicts preserve insertion order since 3.7. Sets do not, and `set` iteration order varies between runs when `PYTHONHASHSEED` is randomised. Generated code frequently builds a `set`, iterates it, and produces output that is stable on the developer's machine and flaky in CI.

**Catch it with:** run your test suite twice with different `PYTHONHASHSEED` values in CI. It is one environment variable and it finds real bugs.

### 10. `is` versus `==`

```python
if status is "active":                # works for interned strings, then stops working
if count is 0:                        # same
```

**Catch it with:** `ruff` rule `F632`. This one is well covered — just make sure `F` is in `select`.

## Data and pandas

### 11. Chained assignment

```python
df[df.score > 0.9]["flag"] = True     # writes to a copy; original unchanged
```

Silent no-op in older pandas, an error under copy-on-write in pandas 3. Either way, the generated analysis is wrong.

**Correct:** `df.loc[df.score > 0.9, "flag"] = True`.

**Catch it with:** run with copy-on-write enabled (`pd.options.mode.copy_on_write = True`) so it raises rather than warns.

### 12. `inplace=True`

Still ubiquitous in generated pandas, largely deprecated, rarely faster, and it defeats method chaining. Prefer reassignment.

### 13. `iterrows()` for anything

Generated pandas reaches for row loops because that is what tutorial pandas does. On a million rows it is thousands of times slower than the vectorised form. This one is not a correctness bug — it is a "why is the job taking six hours" bug.

## Security

The full treatment is in [the security review checklist](/review/security/); these are the ones that recur most in Python specifically.

### 14. `subprocess` with `shell=True` and an f-string

```python
subprocess.run(f"git log --author={author}", shell=True)   # command injection
```

**Correct:** a list of arguments, `shell=False` (the default).

**Catch it with:** `ruff` rule `S602`/`S605` (bandit rules via flake8-bandit — enable the `S` family).

### 15. `yaml.load` without a loader, `pickle` on untrusted input

Both execute arbitrary code by design. Both appear in generated config-loading and caching code.

**Correct:** `yaml.safe_load`; `json` instead of `pickle` for anything crossing a trust boundary.

**Catch it with:** `ruff` rules `S506` and `S301`.

### 16. `random` for tokens

```python
token = "".join(random.choices(string.ascii_letters, k=32))   # predictable
```

**Correct:** `secrets.token_urlsafe(32)`.

**Catch it with:** `ruff` rule `S311`.

### 17. String-built SQL

```python
cur.execute(f"SELECT * FROM users WHERE email = '{email}'")
```

Still generated, especially when the surrounding code does not use an ORM. **Catch it with:** `ruff` rule `S608`, and a hard convention of parameterised queries.

## Housekeeping that becomes correctness

### 18. `open()` without an encoding

`open(path)` uses the platform default, which is UTF-8 on modern Linux and macOS and was historically not on Windows. Generated file-handling code omits it constantly. **Catch it with:** `ruff` rule `PLW1514` / `W1514`.

### 19. Shadowing stdlib module names

Files called `types.py`, `logging.py`, `secrets.py`, `email.py` next to code that imports the stdlib module of the same name. The failure is an import error far from the cause. **Catch it with:** `ruff` rule `A005`.

### 20. Silent integer division changes

`//` versus `/` in ported or translated code. Not a linter catch — a test catch. Anywhere you see a division in generated numeric code, write the boundary test.

## Turning most of this on

Nearly two thirds of the list above is enforceable with one config block:

```toml pyproject.toml
[tool.ruff.lint]
select = [
  "E", "F",        # pycodestyle, pyflakes
  "B",             # bugbear    -> mutable defaults, loop binding
  "S",             # bandit     -> shell=True, yaml.load, random, SQL
  "DTZ",           # datetimez  -> naive datetimes
  "ASYNC",         # async      -> blocking calls in coroutines
  "BLE",           # blind except
  "A",             # builtins/stdlib shadowing
  "PL",            # pylint subset -> encoding, misc
  "SIM", "UP", "I", "RUF",
]
ignore = ["E501"]

[tool.ruff.lint.per-file-ignores]
"tests/**" = ["S101"]
```

```bash
uv add --dev ruff
uv run ruff check --statistics .     # see what you are already shipping
```

:::verdict The point
The interesting shift is that reviewing generated code is *less* about reading every line and *more* about making sure the machine reads every line for you. Attention does not scale with output volume. Configuration does.
:::

:::promo educative
:::

## Common questions

### Are these bugs unique to AI-generated code?

No — every one of them predates language models and appears in human code too. What changed is the rate and the distribution. Models produce the most-represented pattern rather than the most-appropriate one, so these particular mistakes arrive far more consistently, and they arrive in volume, in code that looks confident and well-formatted.

### Does a better model make this list shorter?

Somewhat, and unevenly. The obvious ones — `is` versus `==`, mutable defaults — are largely gone from frontier models. The subtle ones survive, because they are subtle: blocking calls in async code and float money still appear regularly, since both produce code that reads correctly and passes a naive test.

### Is a linter really enough?

For roughly two thirds of this list, yes, and that is the point — those items should stop consuming your attention entirely. The remaining third (money types, division semantics, iteration-order assumptions, whether the code solves the right problem) needs tests and human judgement. Spend your review time there.

### What is the single highest-value thing to turn on?

If you write async Python, the `ASYNC` rule family, because blocking-call-in-coroutine is the one failure here that is both common and effectively invisible until you are under load. If you do not, `B` and `S` together give you the largest reduction in real defects per minute spent.

## Variables and Types
Source: https://learn-python.com/variables-and-types/

Python is completely object oriented, and not “statically typed”. You do not need to declare variables before using them, or declare their type. Every variable in Python is an object.

This tutorial will go over a few basic types of variables.

### Numbers

Python supports two types of numbers - integers(whole numbers) and floating point numbers(decimals). (It also supports complex numbers, which will not be explained in this tutorial).

To define an integer, use the following syntax:

```python
myint = 7
print(myint)
```

To define a floating point number, you may use one of the following notations:

```python
myfloat = 7.0
print(myfloat)
myfloat = float(7)
print(myfloat)
```

### Strings

Strings are defined either with a single quote or a double quotes.

```python
mystring = 'hello'
print(mystring)
mystring = "hello"
print(mystring)
```

The difference between the two is that using double quotes makes it easy to include apostrophes (whereas these would terminate the string if using single quotes)

```python
mystring = "Don't worry about apostrophes"
print(mystring)
```

There are additional variations on defining strings that make it easier to include things such as carriage returns, backslashes and Unicode characters. These are beyond the scope of this tutorial, but are covered in the [Python documentation](http://docs.python.org/tutorial/introduction.html#strings).

Simple operators can be executed on numbers and strings:

```python
one = 1
two = 2
three = one + two
print(three)

hello = "hello"
world = "world"
helloworld = hello + " " + world
print(helloworld)
```

Assignments can be done on more than one variable “simultaneously” on the same line like this

```python
a, b = 3, 4
print(a,b)
```

Mixing operators between numbers and strings is not supported:

```python
# This will not work!
one = 1
two = 2
hello = "hello"

print(one + two + hello)
```

## Hallucinated packages, slopsquatting, and dependency hygiene
Source: https://learn-python.com/review/dependencies/

Ask a model for a library that does something slightly unusual and it will sometimes give you a package name that sounds exactly right, has a plausible API, and does not exist.

This is a well-documented behaviour, it happens across every model family, and the same non-existent names recur — the failure is not uniformly random, which is the part that matters. A predictable hallucination is a namespace an attacker can pre-register and wait in.

The industry name for that attack is **slopsquatting**, by analogy with typosquatting. It is cheap to execute, it targets an install command that people run without thinking, and Python is a particularly good target because `pip install` runs arbitrary setup code from an index anyone can publish to.

:::danger The whole attack in three lines
1. An agent suggests `pip install requests-retry-adapter` — a package that does not exist.
2. Someone has already registered that name on PyPI, because a model suggested it to them too.
3. Installation executes their code, with your credentials in the environment.
:::

## Why models do this

Package names are compositional and predictable: `<thing>-<qualifier>`, `py<thing>`, `<thing>-python`, `<framework>-<feature>`. A model that has seen thousands of real names generalises the pattern and produces new ones that fit it perfectly. It is the same generative behaviour that makes it good at naming your functions.

It gets worse in three specific situations, worth knowing because they are where to pay attention:

- **Niche requirements.** "A library for parsing HL7 v2 into pydantic models" — sparse training data, high invention rate.
- **Version drift.** A package that was real, then renamed or absorbed. `sklearn` versus `scikit-learn`, `PIL` versus `pillow`, and dozens of smaller cases.
- **Wrong ecosystem.** A real npm package name suggested as a Python one, because the concept exists and the model has crossed the streams.

## The four-second check

Before any `uv add` or `pip install` that you did not think of yourself:

```bash
# does it exist, and is it what you think it is?
uv pip index versions <name>          # or: pip index versions <name>
```

Then look at the PyPI page and ask three questions:

1. **When was it first published?** A package solving an old problem that appeared last month is a red flag.
2. **How many releases, and over how long?** One release, ever, is a red flag.
3. **Does the repository link go anywhere real,** with commit history and issues from other people?

That is genuinely all it takes, and it catches essentially every instance of this attack.

:::warn The dangerous moment is the paste
The risk is not the agent suggesting a name — it is you pasting an install command from a chat window into a terminal without reading it. Put `Bash(uv add:*)` and `Bash(pip install:*)` in the **ask** list of your [permission config](/ai/agent-setup/), never the allow list. That single line converts this from a real risk to a non-issue.
:::

## Beyond hallucination: the dependency you did not need

The more common and more boring problem is that agents add dependencies too readily. Asked to parse a date, retry a request or flatten a list, a model frequently reaches for a package where the standard library or three lines would do — because the training data is full of code that does exactly that.

Every dependency you add is a supply-chain surface, a version constraint on your other dependencies, and a thing that will eventually be unmaintained. The Python standard library is large and covers more than most generated code assumes:

| Generated reaches for | Often unnecessary because |
|---|---|
| `python-dateutil` | `datetime.fromisoformat` handles ISO 8601 since 3.11 |
| `requests` | fine, but `httpx` if you need async; `urllib.request` for one call |
| `six`, `future` | Python 2 compatibility shims. Delete on sight. |
| `attrs` alongside `pydantic` | pick one; `dataclasses` may be enough |
| a retry library | `tenacity` is good, but a 6-line loop is often better |
| `toml` | `tomllib` is stdlib since 3.11 |

Add a line to your `AGENTS.md`: *"New dependencies need a sentence in the PR description justifying them. Prefer stdlib."* It works, and it makes the decision visible at review time.

## Pin, lock, and read the lockfile diff

```bash
uv add httpx            # writes pyproject.toml AND uv.lock
git diff uv.lock        # read this. it is the actual supply chain.
```

A one-line change to `pyproject.toml` can be a forty-line change to the lockfile. The lockfile diff is where you see the transitive dependencies you just accepted, and it is the only place you see them.

## Automate the rest

```bash
uv add --dev pip-audit
uv run pip-audit                  # known CVEs in your resolved tree
```

Wire it into CI rather than the agent loop — it needs network access and it is too slow for a per-edit check. Weekly, plus on every lockfile change, is the right cadence.

Also worth enabling: your host's automated dependency updates (Dependabot, Renovate). Not because the updates matter individually, but because a repo where updates arrive continuously is one where a security update can actually be merged in an afternoon.

:::verdict The whole policy, in four lines
1. Package installs go behind a confirmation, never on the allowlist.
2. Unfamiliar name? Check first published, release count, repository.
3. Read the lockfile diff.
4. `pip-audit` in CI.
:::

:::promo digitalocean
:::

## Common questions

### How often do models actually invent package names?

Often enough that it is worth a systematic check rather than vigilance. Published research on this has found meaningful hallucination rates across model families, with a substantial share of invented names repeating across runs — and that repeatability is what makes the attack economic. The rate is lower on frontier models than it was, and it is not zero.

### Is this specific to Python?

No — npm has the same problem and a larger attack surface. Python is a particularly attractive target because installation can execute arbitrary code and because `pip install` is a command people run reflexively without reading.

### Does a lockfile protect me?

From version drift and from a compromised later release, yes. From installing a malicious package in the first place, no — the lockfile records whatever you added. The check has to happen at `uv add` time, which is exactly why that command belongs behind a confirmation.

### What if the package genuinely does not exist but I need what it does?

Good outcome, actually: it means you now know to check whether the stdlib covers it, and if not, whether the real library is called something else. Ask the agent for two or three alternatives and verify each — the second suggestion is usually the real one.

## Lists
Source: https://learn-python.com/lists/

Lists are very similar to arrays. They can contain any type of variable, and they can contain as many variables as you wish. Lists can also be iterated over in a very simple manner. Here is an example of how to build a list.

```python
mylist = []
mylist.append(1)
mylist.append(2)
mylist.append(3)
print(mylist[0]) # prints 1
print(mylist[1]) # prints 2
print(mylist[2]) # prints 3

# prints out 1,2,3
for x in mylist:
    print(x)
```

Accessing an index which does not exist generates an exception (an error).

```python
mylist = [1,2,3]
print(mylist[10])
```

## Writing an AGENTS.md for Python that agents actually follow
Source: https://learn-python.com/ai/agents-md/

`AGENTS.md` is a plain Markdown file in your repo root that coding agents read before they start work. Claude Code reads `CLAUDE.md`; Codex, Cursor, Aider, Cline and most newer tools read `AGENTS.md`. The contents are the same, so pick one file and symlink:

```bash
printf 'CLAUDE.md\n' >> .gitignore   # or commit the symlink, your call
ln -s AGENTS.md CLAUDE.md
```

The file is not documentation. It is **a standing correction list**: the things a competent Python developer would get wrong on their first week in your repo, because they are specific to your repo rather than to Python.

## The economics nobody mentions

Every token in `AGENTS.md` is prepended to every request for the whole session. A 500-line file is not free advice — it is a permanent tax on the context available for your actual code, and it dilutes the instructions that matter.

Worse, models weight instructions roughly by salience, not by position in a list. Twelve rules get followed. Sixty rules get sampled. If you have written sixty, you have effectively randomised which of your rules apply.

:::verdict The target
**Under 100 lines.** If it is longer, you are writing documentation. Move it to `docs/` and link to it — agents can read a file on demand, and an on-demand read costs nothing when it is not needed.
:::

## What belongs in it

**1. How to run things.** The single highest-value section, because it is unguessable and used every turn.

**2. Conventions a linter cannot enforce.** "Domain logic goes in `src/core/`, never in the route handlers." A model cannot infer this from the code, and `ruff` will not tell it.

**3. Landmines.** "`sync_users()` is called by a cron in production; changing its signature breaks the scheduler repo." This is the highest-value line in most files and almost nobody writes it.

**4. Corrections you have made twice.** If you have told the agent twice not to add `try/except` around everything, that is a line.

## What does not belong in it

**Generic Python advice.** "Use type hints. Follow PEP 8. Write docstrings." The model knows. You are spending context to teach it something it does better than most humans.

**Anything a tool already enforces.** If `ruff` sets your line length, do not also write the line length in prose. You now have two sources of truth and one of them will drift. Configure the tool; mention the tool.

**Long architecture explanations.** Put them in `docs/architecture.md` and write one line: *"Read `docs/architecture.md` before changing anything under `src/billing/`."* Agents follow pointers reliably.

**Aspirations.** "All code should have 100% test coverage." If it is not true today, it is a lie in the context window, and it makes every other line less credible.

:::warn The most common failure
A file that describes the codebase you wish you had. The agent believes it, writes code for that codebase, and you get a diff that does not fit the one you actually have.
:::

## A Python template worth copying

```markdown AGENTS.md
# AGENTS.md

Python 3.12. Package management is `uv` — never `pip` or `poetry`.

## Commands
- Install:      `make install`   (uv sync)
- Test:         `make test`      (pytest -q)
- One test:     `uv run pytest tests/test_x.py::test_name -q`
- Lint + fmt:   `make lint`      (ruff check --fix, ruff format)
- Types:        `make typecheck` (mypy src, strict)
- All of it:    `make check`     <- must pass before you say you are done

## Layout
- `src/core/`      pure domain logic. No I/O, no framework imports, no `requests`.
- `src/adapters/`  everything that touches the outside world: db, http, queues.
- `src/api/`       FastAPI routers. Thin. Parse, call core, serialise. No logic.
- `tests/`         mirrors `src/`. Unit tests are offline; anything needing a
                   database is marked `@pytest.mark.integration`.

## Conventions
- Money is `Decimal`, never `float`. Serialise as a string.
- All datetimes are timezone-aware UTC. `datetime.now(UTC)`, never `utcnow()`.
- Public functions in `src/core/` are fully typed. `Any` needs a comment saying why.
- Errors: raise domain exceptions from `src/core/errors.py`. Do not catch broadly.
- New dependencies need a line in the PR description justifying them.

## Landmines
- `src/adapters/legacy_sync.py` is called by the `ops-scheduler` repo via CLI.
  Do not change its arguments or its exit codes.
- `tests/fixtures/prod_sample.json` is real, anonymised customer data. Do not
  print it in test output and do not paste it into commit messages.
- Alembic migrations are irreversible in production. Write the migration, do not
  run it. A human runs migrations.

## Working style
- Small commits. One behavioural change each.
- When a test is failing, fix the code, not the test — unless the test is
  provably wrong, in which case say so explicitly in the commit message.
- If a change needs more than ~150 lines of diff, stop and describe the plan first.
```

That is 40 lines and it does more than most 400-line files. Notice how much of it is *specific facts about this repository* rather than *opinions about Python*.

:::tip The two-strike rule
Do not add a rule to this file speculatively. Wait until an agent has made the same mistake twice. This keeps the file short, keeps every line evidence-backed, and — usefully — tells you which of your repo's conventions are genuinely non-obvious.
:::

## Nested files for big repos

In a monorepo, a root `AGENTS.md` plus per-package files works better than one large file. Most tools read the nearest one and merge upward.

```text
AGENTS.md                     # commands, global conventions
services/billing/AGENTS.md    # Decimal rules, the Stripe webhook landmine
services/etl/AGENTS.md        # pandas conventions, memory limits
```

The root file should then be almost entirely commands and layout, with the specifics pushed down to where they are relevant.

## How to tell if yours is working

Run the same non-trivial task twice: once with the file, once with it renamed away. If the diffs are meaningfully different, the file is earning its place. If they are the same, you have written 100 lines of ballast.

It is also worth deleting a third of the file periodically and seeing whether anything gets worse. Usually nothing does.

:::promo manning
:::

## Common questions

### AGENTS.md or CLAUDE.md?

Write `AGENTS.md` — it is the convention most tools now converge on — and symlink `CLAUDE.md` to it so Claude Code picks it up too. Maintaining two files by hand guarantees they drift apart within a month.

### Should AGENTS.md be committed to the repo?

Yes. It is a shared artefact: everyone on the team benefits from the same landmine list, and it should be reviewed in pull requests like any other change to how the project is built. Keep personal preferences in your user-level config instead.

### Does a longer file mean better results?

No, and the relationship reverses past a point. Every line competes for attention with every other line, and a long file dilutes the rules you care about most. Under 100 lines is a good target; the specific facts about your repository matter far more than the volume of general advice.

### What if my team uses several different agents?

That is the argument for `AGENTS.md` rather than tool-specific files. Keep one file with the substance, symlink the tool-specific names to it, and put anything genuinely tool-specific — permission allowlists, hooks, model choice — in that tool's own config where it belongs.

## Basic Operators
Source: https://learn-python.com/basic-operators/

This section explains how to use basic operators in Python.

### Arithmetic Operators

Just as any other programming languages, the addition, subtraction, multiplication, and division operators can be used with numbers.

```python
number = 1 + 2 * 3 / 4.0
print(number)
```

Try to predict what the answer will be. Does python follow order of operations?

Another operator available is the modulo (%) operator, which returns the integer remainder of the division. dividend % divisor = remainder.

```python
remainder = 11 % 3
print(remainder)
```

Using two multiplication symbols makes a power relationship.

```python
squared = 7 ** 2
cubed = 2 ** 3
print(squared)
print(cubed)
```

### Using Operators with Strings

Python supports concatenating strings using the addition operator:

```python
helloworld = "hello" + " " + "world"
print(helloworld)
```

Python also supports multiplying strings to form a string with a repeating sequence:

```python
lotsofhellos = "hello" * 10
print(lotsofhellos)
```

### Using Operators with Lists

Lists can be joined with the addition operators:

```python
even_numbers = [2,4,6,8]
odd_numbers = [1,3,5,7]
all_numbers = odd_numbers + even_numbers
print(all_numbers)
```

Just as in strings, Python supports forming new lists with a repeating sequence using the multiplication operator:

```python
print([1,2,3] * 3)
```

## Security review checklist for AI-generated Python
Source: https://learn-python.com/review/security/

Two things are true at once. Coding agents write more secure code than the average tutorial they learned from — they use parameterised queries by default, they hash passwords properly, they do not roll their own crypto. And they introduce a specific, repeatable set of vulnerabilities at a volume that human review does not scale to.

The second is the operative one. **Your review process has to be mechanical, because your attention is the resource that ran out.**

## Turn on the machine checks first

Before any of the manual review below, get the automated coverage. Most of what follows is caught for free.

```bash
uv add --dev ruff pip-audit
```

```toml pyproject.toml
[tool.ruff.lint]
select = ["S", "B", "BLE", "ASYNC", "E", "F"]   # S = bandit ruleset
[tool.ruff.lint.per-file-ignores]
"tests/**" = ["S101", "S105", "S106"]
```

```bash
uv run ruff check .        # every edit, via a hook
uv run pip-audit           # in CI
```

Add secret scanning to pre-commit (`gitleaks`, `detect-secrets`) and you have covered the majority of what actually goes wrong.

## The manual checklist

### Injection: is any input reaching a shell, a query, or a template unparameterised?

```bash
git diff | grep -nE "shell=True|os\.system|f\".*SELECT|f\".*INSERT|eval\(|exec\("
```

The recurring Python instances:

- `subprocess.run(f"...", shell=True)` — pass a list, leave `shell=False`.
- f-string SQL where the surrounding code has no ORM. Parameterise. Always.
- `eval()` / `exec()` on anything derived from input. There is no safe version of this.
- Jinja templates rendered from user-controlled strings — `Template(user_input)` is remote code execution, not templating.

### Deserialisation: what is being loaded, and from where?

`pickle.loads`, `yaml.load` without `SafeLoader`, `marshal`, `dill`, and `torch.load` on an untrusted checkpoint all execute arbitrary code by design. Generated caching, config-loading and model-loading code reaches for these constantly.

Rule: **if it crossed a trust boundary, it is JSON.**

### Secrets: is anything hardcoded, logged, or in an error path?

```bash
git diff | grep -niE "api_key|secret|token|password|bearer|BEGIN .*PRIVATE KEY"
```

Two specifically generated patterns worth naming:

```python
# 1. the "example" that ships
API_KEY = os.getenv("API_KEY", "sk-test-abc123")     # real key as a default

# 2. the debug log that survives
logger.info("calling %s with %s", url, headers)      # headers include Authorization
```

The second is the one that gets missed, because it is not in the diff as a secret — it is in the diff as a log line.

### Authorisation: is it checked, and is it checked in the right place?

Generated endpoints frequently authenticate — is this a valid user — and forget to authorise — is this *their* resource.

```python
@router.get("/invoices/{invoice_id}")
async def get_invoice(invoice_id: str, user: User = Depends(current_user)):
    return await db.get_invoice(invoice_id)     # any logged-in user, any invoice
```

This is the single most common real vulnerability in generated web code, it is invisible to every linter, and the test that catches it is one line:

```python
async def test_cannot_read_another_users_invoice(client, alice, bobs_invoice):
    r = await client.get(f"/invoices/{bobs_invoice.id}", headers=alice.auth)
    assert r.status_code == 404      # not 403 — do not confirm existence
```

Write that test for every resource-scoped endpoint. It is the highest value-per-line test in a web codebase.

### SSRF: does anything fetch a URL that came from outside?

```python
@router.post("/import")
async def import_from_url(url: str):
    r = await client.get(url)        # now fetches your cloud metadata endpoint
```

Generated "import from URL", webhook and avatar-fetching features almost never validate the target. Allowlist schemes and hosts, resolve and check the IP, and disable redirects.

### Crypto and randomness

- `random` for tokens, session ids, password resets, filenames. Use `secrets`.
- `md5`/`sha1` for anything security-relevant.
- Hand-rolled comparison of secrets — use `hmac.compare_digest`.
- Encryption without an authenticated mode. If you are choosing a cipher mode by hand, stop and use `cryptography`'s recipes layer.

### Errors and information disclosure

`debug=True`, full tracebacks returned to clients, exception messages that include the SQL, the file path or the internal hostname. Generated error handlers are helpful to a fault.

## The one that is new: prompt injection

If the Python you are writing *is* an LLM application, there is a class of bug that no linter has heard of.

Any text your application feeds a model — a scraped page, a user message, a PDF, a database row someone else wrote, an MCP tool result — can contain instructions. If your model has tools, those instructions can be actions.

```python
# every one of these is untrusted input, not data
context = fetch_page(url)
context += load_pdf(upload)
context += db.query("SELECT bio FROM users WHERE ...")
answer = await agent.run(system=SYSTEM, context=context, tools=[send_email, query_db])
```

The mitigations that hold up:

- **Least privilege on tools.** The blast radius of an injection is exactly the set of tools available. A read-only agent cannot be made to exfiltrate.
- **Confirm side effects.** Anything that sends, pays, deletes or publishes goes through a human, or through a deterministic check the model cannot argue with.
- **Separate the channels.** Untrusted content goes in a clearly delimited region, never concatenated into the system prompt.
- **Constrain the output.** If the model's job is to return a category, validate it against an enum in Python. Do not let free text become control flow.
- **Watch the combination.** Fetching untrusted content *and* having a write tool in the same session is the dangerous configuration. Either alone is usually fine.

There is more on the testing side of this in [testing LLM-powered Python](/ai/evals/).

## The review, as a command

```bash
git diff | grep -nE "shell=True|eval\(|exec\(|pickle|yaml\.load\(|md5|sha1\(|random\.|verify=False|debug=True|except Exception"
git diff | grep -niE "api_key|secret|token|password|BEGIN .*PRIVATE"
uv run ruff check . && uv run pip-audit
```

Three commands, thirty seconds, and it catches most of this page. The two it cannot catch — missing authorisation and prompt injection — are the two worth spending your actual attention on.

:::promo manning
:::

## Common questions

### Is AI-generated code less secure than human code?

Mixed, and the framing is not very useful. On the basics it is often better than the median human code, because it defaults to parameterised queries and proper password hashing. On authorisation logic and on anything requiring knowledge of *your* trust boundaries, it is worse — and it produces code at a volume that overwhelms the review process that used to catch these things.

### What is the single highest-value check?

For a web application, the authorisation test: for every endpoint that returns a resource, one test asserting that a different user gets a 404. No static analysis finds this class of bug, it is the most common real vulnerability in generated web code, and the test is one line.

### Does prompt injection apply if my app only summarises text?

Yes, but the impact is small — the worst case is a bad summary. The risk scales entirely with what the model can *do*. Summarisation with no tools is close to harmless; summarisation with an email tool attached is a data exfiltration channel.

### Do I need a paid security scanner?

For a small team, `ruff`'s bandit rules plus `pip-audit` plus secret scanning in pre-commit covers most of what a paid tool would flag, at no cost. The gap that paid tools fill is cross-file dataflow analysis, which starts to matter on large codebases with many contributors.

## The verification loop: giving an agent something it cannot fake
Source: https://learn-python.com/ai/feedback-loops/

The quality of agent output is mostly a function of one thing: **how good is the signal it gets after each edit, and how fast does it arrive?**

A model asked to write code with no way to run it is doing creative writing. The same model with a four-second test suite is doing engineering, because it can be wrong five times in a row and you will never see the first four attempts.

So the interesting question is not *"which model is best at Python"*. It is *"what can I hand it that tells the truth quickly"*.

## Speed is a correctness feature

There is a threshold effect here that is easy to underestimate.

| Suite runtime | What the agent does |
|---|---|
| under 5s | Runs it after every edit. Converges. |
| 5–30s | Runs it after every few edits. Usually fine. |
| 30s–2min | Runs it once at the end. You review guesses. |
| over 2 min | Stops running it. Tells you it "should work". |

Nobody instructs an agent to stop running slow tests; it just happens, the same way it happens to humans. If your suite takes four minutes, the highest-leverage thing you can do for agent output quality is not a better prompt — it is splitting the suite.

```toml pyproject.toml
[tool.pytest.ini_options]
addopts = "-q --strict-markers -p no:cacheprovider"
markers = [
  "integration: needs a database or network. Excluded by default.",
  "slow: over one second. Excluded by default.",
]
# the default run is the fast one
addopts = "-q --strict-markers -m 'not integration and not slow'"
```

```makefile
test:            # the loop. must stay under ~5 seconds.
	uv run pytest -q -m "not integration and not slow"

test-all:        # pre-push and CI
	uv run pytest -q
```

:::tip Parallelise before you optimise
`uv add --dev pytest-xdist` then `pytest -q -n auto` is usually a 3–4x win for a few minutes of work. Do this before you spend an afternoon making individual tests faster.
:::

## Types are the cheapest signal you have

A type checker catches an entire class of generated-code error — wrong argument order, `None` where a value is required, a method that does not exist on that object — at a fraction of the cost of a test, with no test to write.

The problem is that most existing Python codebases produce thousands of errors on day one, so the check is turned off, so the signal is lost.

**Baseline it instead.** Turn strictness on for new code only, then ratchet.

```toml pyproject.toml
[tool.mypy]
python_version = "3.12"
strict = true
warn_unreachable = true
files = ["src", "tests"]

# Existing debt: opted out module by module, deleted as it is paid down.
[[tool.mypy.overrides]]
module = ["myapp.legacy.*", "myapp.reporting.old_exports"]
ignore_errors = true
```

Now `make typecheck` is green, so it can go in `make check`, so the agent gets the signal on everything it writes — and the overrides list is a visible, shrinking to-do list rather than an invisible surrender.

:::warn Watch for the `# type: ignore` reflex
Agents under pressure to make a check pass will reach for `# type: ignore`, `cast(Any, x)` and `Optional[...]` widening. Add `warn_unused_ignores = true`, and grep the diff for `type: ignore` before you accept it. An ignore with no comment explaining it is a defect.
:::

## Property tests: the check that is hard to game

Example-based tests have a structural weakness with generated code: the model can see the examples. Given `assert total([1,2,3]) == 6`, a sufficiently cornered model will special-case the input. It is not being malicious; it is minimising the distance to a passing state.

Property tests remove that option, because there is no specific input to special-case.

```python tests/test_pricing_properties.py
from decimal import Decimal
from hypothesis import given, strategies as st

money = st.decimals(min_value=0, max_value=10_000, places=2)


@given(lines=st.lists(st.tuples(st.integers(1, 100), money), min_size=1))
def test_total_never_exceeds_undiscounted_sum(lines):
    cart = Cart([Line(sku="x", qty=q, unit_price=p) for q, p in lines])
    undiscounted = sum(q * p for q, p in lines)
    assert Decimal("0") <= cart.total() <= undiscounted


@given(lines=st.lists(st.tuples(st.integers(1, 100), money), min_size=1))
def test_total_is_order_independent(lines):
    a = Cart([Line("x", q, p) for q, p in lines]).total()
    b = Cart([Line("x", q, p) for q, p in reversed(lines)]).total()
    assert a == b
```

Two properties, and a whole family of plausible wrong implementations becomes unreachable. Hypothesis also shrinks failures to a minimal case, which is exactly the input an agent needs to fix the bug rather than guess at it.

:::note Where property tests pay off most
Parsers, serialisers, money and unit arithmetic, date handling, sorting and dedup, anything with an inverse (`encode`/`decode`, `to_dict`/`from_dict`). If your function has a round-trip property, one `hypothesis` test is worth twenty examples.
:::

## Checks agents reliably game

Every check has a cheap way to satisfy it that is not the intended way. Know yours.

| Check | The cheap way out | Defence |
|---|---|---|
| A single failing test | Hardcode the expected value | Always add the boundary case and one negative case |
| Coverage percentage | Tests that execute but never assert | Never set a coverage target as the goal |
| `mypy` | `# type: ignore`, `Any`, `cast` | `warn_unused_ignores`, grep the diff |
| "Make the tests pass" | Edit the test | Say "without changing tests"; review test diffs separately |
| Lint | `# noqa` | `ruff` with `--extend-select` and no blanket noqa |
| "Handle errors" | `except Exception: pass` | Ban bare and broad excepts in lint config |

That last one is worth enforcing mechanically:

```toml pyproject.toml
[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B", "S", "SIM", "RUF", "ASYNC"]
# B902/BLE001 broad-except, S110 try-except-pass, S101 assert-in-prod
extend-select = ["BLE", "T20"]   # no broad excepts, no stray print()
ignore = ["E501"]                # formatter owns line length

[tool.ruff.lint.per-file-ignores]
"tests/*" = ["S101"]             # assert is fine in tests
```

## Close the loop with a hook

The strongest version of this is not asking the agent to run checks — it is making the checks run whether it asks or not. Claude Code supports hooks that fire on tool events; other tools have watchers or pre-commit.

```json .claude/settings.json
{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          { "type": "command", "command": "$CLAUDE_PROJECT_DIR/.claude/hooks/lint.sh" }
        ]
      }
    ]
  }
}
```

```bash .claude/hooks/lint.sh
#!/usr/bin/env bash
set -Eeuo pipefail
# The hook receives a JSON object on stdin. Read the path from tool_input —
# that is the form that is stable across releases and identical for every event.
file=$(jq -r '.tool_input.file_path // empty')
[[ "$file" == *.py && -f "$file" ]] || exit 0

uv run ruff check --fix "$file" 2>&1 | tail -20
if out=$(uv run mypy "$file" 2>&1); then exit 0; fi
echo "$out" | tail -20 >&2
exit 2      # exit 2 feeds the errors back to the model to fix
```

Now every edit is immediately followed by real feedback the model did not choose to request, and exit code 2 hands the errors back to it as something to fix rather than to you as a notification. There is a full treatment of the hook system in [harness hooks](https://codelearningdojo.com/harness-hooks/). In practice this is the single change that most improves output quality on a Python repo, because it removes the "I'll check at the end" failure mode entirely.

:::promo digitalocean
:::

## A loop worth having

```text
edit  ->  ruff (instant)  ->  mypy on changed files (~1s)
      ->  fast unit tests (<5s)  ->  property tests on core (~2s)
      ->  integration suite (on demand, before PR)
```

Fast, honest, and hard to satisfy dishonestly. Everything else — better prompts, bigger models, more detailed instructions — is a smaller effect than this.

## Common questions

### My test suite takes six minutes. Where do I start?

Split it before you optimise it. Mark everything that touches a database, the network or the filesystem as `integration`, exclude those by default, and add `pytest-xdist`. Most Python suites are dominated by a small number of slow tests, and getting the default run under five seconds is usually an afternoon's work.

### Should the agent be allowed to edit tests?

Yes, but review test diffs separately and with more suspicion than source diffs. A green suite where the test changed is not evidence of anything. In practice: read the test diff first, then the source diff.

### Are property tests worth the learning curve?

For code with an invariant or a round-trip, yes, and the curve is about an hour. For CRUD glue code, no. Start with the three or four functions in your codebase where a subtle wrongness would be expensive, and leave the rest on examples.

### Does any of this replace reading the diff?

No. The loop raises the floor — it stops obvious wrongness reaching you — but it cannot tell you that the code solves the wrong problem, duplicates something that already exists, or takes an approach you will regret. That judgement is still yours, and it is where your time is now best spent.

## String Formatting
Source: https://learn-python.com/string-formatting/

Python uses C-style string formatting to create new, formatted strings. The “%” operator is used to format a set of variables enclosed in a “tuple” (a fixed size list), together with a format string, which contains normal text together with “argument specifiers”, special symbols like “%s” and “%d”.

Let’s say you have a variable called “name” with your user name in it, and you would then like to print(out a greeting to that user.)

```python
# This prints out "Hello, John!"
name = "John"
print("Hello, %s!" % name)
```

To use two or more argument specifiers, use a tuple (parentheses):

```python
# This prints out "John is 23 years old."
name = "John"
age = 23
print("%s is %d years old." % (name, age))
```

Any object which is not a string can be formatted using the %s operator as well. The string which returns from the “repr” method of that object is formatted as the string. For example:

```python
# This prints out: A list: [1, 2, 3]
mylist = [1,2,3]
print("A list: %s" % mylist)
```

Here are some basic argument specifiers you should know:

`%s - String (or any object with a string representation, like numbers)`

`%d - Integers`

`%f - Floating point numbers`

`%.f - Floating point numbers with a fixed amount of digits to the right of the dot.`

`%x/%X - Integers in hex representation (lowercase/uppercase)`

## Structuring a Python repo an agent can navigate
Source: https://learn-python.com/ai/context/

An agent working on your codebase has a fixed budget of attention. Every file it has to open to answer "where does this belong?" is budget not spent on the change you asked for. Structure that makes the answer obvious is not aesthetic preference any more — it is throughput.

The useful thing is that the properties that help an agent are almost exactly the properties that help a new hire. Nothing here is a compromise made for machines.

## The single highest-leverage rule: boundaries an import can express

An agent decides where to put code by looking at where similar code already lives. If your layout encodes a rule, it follows the rule. If it does not, the agent invents one, and it will invent a different one next week.

```text
src/yourapp/
  core/          pure logic. no I/O, no framework imports, no network.
  adapters/      everything that touches the world: db, http clients, queues, s3
  api/           the web layer. parse -> call core -> serialise. no logic.
  cli/           same, for the terminal
  config.py      settings, read once, passed down
tests/
  core/          fast, offline, no fixtures beyond data
  adapters/      marked integration
```

The rule is expressible as an import constraint, which means you can *enforce* it rather than describe it:

```toml pyproject.toml
[tool.ruff.lint.flake8-tidy-imports.banned-api]
"requests".msg = "Use the http client in adapters/, not requests directly."

[tool.importlinter]
root_package = "yourapp"

[[tool.importlinter.contracts]]
name = "core stays pure"
type = "forbidden"
source_modules = ["yourapp.core"]
forbidden_modules = ["yourapp.adapters", "yourapp.api", "requests", "sqlalchemy"]
```

Now "don't put database calls in core" is a check that fails, not a sentence in a document the agent may or may not weight highly. This is the general pattern worth internalising: **a convention you can enforce is worth ten conventions you can only write down.**

## Files an agent reads first

In rough order of how often they get opened:

1. `AGENTS.md` / `CLAUDE.md` — always, every session
2. `README.md` — usually
3. `pyproject.toml` / `Makefile` — to work out how to run things
4. The test file matching whatever it is editing
5. `__init__.py` of the package it is editing, for the public surface

Make those five accurate and current and you have done most of the work. A stale README is worse than no README: it is confidently wrong context that the agent has no way to distrust.

:::tip The `__init__.py` trick
An `__init__.py` that explicitly re-exports the package's public API tells an agent what is safe to call and what is internal, in one short file. Empty `__init__.py` files force it to read everything to find out.

```python src/yourapp/core/__init__.py
"""Pure domain logic. Nothing in here may perform I/O."""

from .pricing import Cart, Line, price_cart
from .errors import DomainError, InsufficientStock

__all__ = ["Cart", "Line", "price_cart", "DomainError", "InsufficientStock"]
```
:::

## Naming that survives a grep

Agents locate code the same way you do when you are new: search. Names that are searchable are worth real money.

- **Prefer distinctive names over short ones.** `create_subscription` beats `create`. There is one of the first in your codebase and forty of the second.
- **Keep the domain vocabulary consistent.** If the database says `account`, the API says `account`, and the code says `user`, every search returns two thirds of the picture. Pick one and put it in `AGENTS.md`.
- **Name test files after what they test.** `tests/core/test_pricing.py` for `src/yourapp/core/pricing.py`. An agent asked to change `pricing.py` will find the test without being told; if the mapping is arbitrary it will write a new test file next to the old one and you will end up with two.

## Docstrings, but only where they are unguessable

Docstrings on obvious functions are noise. Docstrings that record *why* are the highest-value text in the repo, because "why" is precisely what cannot be recovered from reading the code — by an agent or by you in eight months.

```python
def settle_invoice(invoice: Invoice, at: datetime) -> Settlement:
    """Settle an invoice.

    `at` must be timezone-aware UTC. Settlements are recorded against the
    billing period containing `at`, NOT the invoice date — finance reconciles
    on payment date, and changing this breaks the monthly export in ops-reports.
    """
```

Everything in that second paragraph is invisible in the code and expensive to rediscover. That is the test for whether a docstring earns its place.

## Things that reliably get an agent lost

**Deep inheritance hierarchies.** Following five levels of `super()` to find where a method actually lives consumes an enormous amount of context, and the model frequently gives up and overrides at the wrong level. Composition is easier for everyone.

**Dynamic attribute magic.** `setattr` loops, `__getattr__` fallbacks, string-keyed dispatch dicts built at import time. If static analysis cannot find it, neither can the agent — and neither can your IDE, which should have been the warning.

**Giant modules.** A 3,000-line `utils.py` is read in fragments, so the agent sees a slice and duplicates a function that already exists 900 lines away. Splitting by topic fixes it. If you have a `utils.py`, the things in it belong somewhere with a name.

**Multiple ways to do the same thing.** Two HTTP clients, two config systems, two date helpers. The agent will pick whichever it happened to see first, and your codebase drifts further apart with every change. Delete one.

**Generated code checked in without a marker.** Protobufs, OpenAPI clients, migrations. Put them in an obvious directory and say so in `AGENTS.md`, or you will get careful hand-edits to a file that is overwritten on the next build.

:::warn The `utils.py` tell
If you cannot describe what a module contains in four words, an agent cannot either. That is a reliable signal that the module needs splitting, and it was true before any of this.
:::

## A quick self-test

Pick a change you have not made yet — "add a discount code to the checkout flow" — and ask an agent, with no extra instruction, where the code should go and which tests it would need to change. Do not let it write anything.

If the answer is right, your structure is legible. If it is wrong, the answer tells you exactly which boundary is unclear, which is more useful diagnostic information than any amount of reading your own repo.

## Common questions

### Should I restructure an existing codebase for this?

Not as a project. Do it opportunistically: when you touch an area, leave it with a clearer boundary than you found. The exception is worth making for `utils.py`-style dumping grounds and for anywhere your agent has already got lost twice — those have measurable cost.

### Do bigger context windows make this irrelevant?

No, and this is a common mistake. A larger window means more can be loaded, not that more is attended to well; retrieval quality degrades with irrelevant context in the window. Good structure means less needs to be loaded at all, and that stays valuable regardless of window size.

### Does this conflict with "just ship it" for small projects?

Yes, and for a small project you should just ship it. The rules here start paying at around the point where you can no longer hold the whole repo in your head — which is also, not coincidentally, the point where an agent stops being able to either.

## The performance traps in generated Python
Source: https://learn-python.com/review/performance/

A test suite tells you whether code is correct. It says nothing about whether it will still work at a thousand times the row count, and generated code is systematically biased towards the shape that is clearest at small scale rather than the shape that survives.

This is not a criticism of the models — clarity is usually the right default, and it is what the training data rewards. It just means performance review is now a distinct pass, and it is worth knowing what to look for.

## 1. N+1 queries

The most expensive item on this list in practice, and the most consistently generated.

```python
orders = await db.fetch_all("SELECT * FROM orders WHERE user_id = :u", {"u": uid})
for order in orders:
    order.items = await db.fetch_all(                      # one query per order
        "SELECT * FROM items WHERE order_id = :o", {"o": order.id}
    )
```

Correct. Passes every test, because the test has three orders. In production with a hundred orders per user, that is 101 round trips.

**Look for:** any `await` or `.query(` inside a `for` loop.

**Catch it with:** a test that asserts query count, which is the only reliable defence.

```python
async def test_order_list_is_two_queries(db_spy, client, user):
    await client.get("/orders", headers=user.auth)
    assert db_spy.count <= 2
```

With SQLAlchemy, `selectinload`/`joinedload` fixes it; with raw SQL, one query with `WHERE order_id = ANY(:ids)`. Either way the test is what stops it coming back.

## 2. Quadratic membership tests

```python
for item in new_items:              # 10,000
    if item.sku in existing_skus:   # a list of 10,000
        ...
```

`in` on a list is O(n). Fine at ten, 100 million comparisons at ten thousand. `set(existing_skus)` once, outside the loop, and it is instant.

**Look for:** `in` against a list or a `.keys()` inside a loop; repeated `.index()`; `list.remove()` in a loop; building a list with `+=` inside a loop.

This one is genuinely common because at tutorial scale it is invisible.

## 3. `iterrows()` and friends

```python
for idx, row in df.iterrows():
    df.at[idx, "total"] = row["qty"] * row["price"]
```

Generated pandas reaches for row iteration because that is what most pandas in the training data does. It is typically hundreds to thousands of times slower than the vectorised form:

```python
df["total"] = df["qty"] * df["price"]
```

**Look for:** `iterrows`, `itertuples` in a hot path, `apply(axis=1)`, `df.append` in a loop (which is quadratic — it copies the frame every time).

## 4. Loading everything into memory

```python
rows = cursor.fetchall()                    # the whole table
data = json.loads(open(path).read())        # the whole file
lines = open(path).readlines()              # all of it
df = pd.read_csv(huge)                      # no chunksize, no dtype
```

Works on the 10MB sample. Gets OOM-killed on the real 8GB file, at 3am, in a job with no retry.

**Correct:** server-side cursors, `ijson` or a streaming parser, iterate the file object directly, `chunksize=` on `read_csv`.

**Look for:** `fetchall`, `.read()`, `readlines()`, any `read_csv` without `chunksize` or `dtype` on a path that could be large. Also: generated code loves building a full list before returning it, where a generator would do.

## 5. Blocking calls in async code

Covered in [the failure-mode catalogue](/review/failure-modes/) as a correctness bug, but its real cost is performance, and it is the most damaging item here because it is invisible until you are under concurrent load.

```python
async def handler(request):
    data = requests.get(url).json()         # blocks the entire event loop
    result = expensive_cpu_thing(data)      # so does this
```

One blocking call in one handler serialises the whole process. Throughput collapses and every trace looks fine individually.

**Catch it with:** `ruff`'s `ASYNC` rules for the obvious cases, and `asyncio` debug mode (`PYTHONASYNCIODEBUG=1`) in tests, which warns on any coroutine that blocks for more than 100ms.

## 6. Recomputing what does not change

```python
def price(items):
    rates = load_exchange_rates()           # network call, per invocation
    config = json.load(open("config.json")) # file read, per invocation
    return sum(...)
```

Generated functions are self-contained by preference — it makes them easier to read and easier to test — which means expensive setup migrates inside them. Hoist it, cache it (`functools.lru_cache`, or module-level for genuinely static data), or pass it in.

## 7. Regex compiled in a loop

`re.compile` inside a hot loop, or `re.sub` with a complex pattern per row. Python caches compiled patterns, so this is usually a small effect — but catastrophic backtracking on a generated regex is not small, and generated regexes are frequently more nested than they need to be.

**Worth checking:** any generated regex with nested quantifiers (`(a+)+`, `(\s*\w+)*`) applied to input you do not control. That is a denial-of-service vector, not just slowness.

## How to actually find these

Reading for performance is unreliable. Measure instead, and make it cheap enough that you do it.

```bash
# where did the time go?
uv add --dev py-spy
uv run py-spy record -o profile.svg -- python -m yourapp.job

# what did that endpoint do?
uv run pytest --durations=10          # slowest tests are usually slowest code
```

Two habits worth more than any checklist:

1. **Test with realistic data volumes at least once.** A fixture with 10,000 rows instead of 3 finds most of this page automatically, and you only need it for the handful of paths that matter.
2. **Assert on query counts** for anything that talks to a database. It is the only test that reliably catches N+1, and N+1 is the expensive one.

:::verdict The reviewer's shortcut
Scan the diff for loops. Then ask, for each one: *what is in here that touches the database, the network, or the disk, and what happens when the loop runs ten thousand times?* That single question catches items 1, 2, 4 and 6 — most of the real cost on this page.
:::

:::promo hetzner
:::

## Common questions

### Should I ask the agent to optimise the code?

Not upfront. Premature optimisation is as bad here as anywhere, and generated "optimised" code is often less clear for no measurable gain. Ask for correct and clear, then profile, then optimise the one thing that showed up. The exception is N+1 queries — those are worth catching structurally rather than by profiling, because the fix is architectural.

### Do these problems show up in tests?

Almost never, and that is the whole issue: tests run with three rows. A query-count assertion and one fixture with realistic volume are the two additions that make the test suite able to see this class of bug at all.

### Is the profiling worth it for a small app?

Yes, once, when something feels slow — `py-spy` needs no code changes and takes about a minute. What is not worth it is routine profiling of code nobody has complained about.

### What about the GIL and free-threaded Python?

Relevant for CPU-bound work, and increasingly so as free-threaded builds mature, but it is rarely the problem in the code this page is about. The typical generated bottleneck is I/O in a loop, not contention — fix the loop first, and only then ask whether threads would help.

## Basic String Operations
Source: https://learn-python.com/basic-string-operations/

Strings are bits of text. They can be defined as anything between quotes:

```python
astring = "Hello world!"
astring2 = 'Hello world!'
```

As you can see, the first thing you learned was printing a simple sentence. This sentence was stored by Python as a string. However, instead of immediately printing strings out, we will explore the various things you can do to them.
You can also use single quotes to assign a string. However, you will face problems if the value to be assigned itself contains single quotes. For example to assign the string in these bracket(single quotes are ‘ ‘) you need to use double quotes only like this

```python
astring = "Hello world!"
print("single quotes are ' '")

print(len(astring))
```

That prints out 12, because “Hello world!” is 12 characters long, including punctuation and spaces.

```python
astring = "Hello world!"
print(astring.index("o"))
```

That prints out 4, because the location of the first occurrence of the letter “o” is 4 characters away from the first character. Notice how there are actually two o’s in the phrase - this method only recognizes the first.

But why didn’t it print out 5? Isn’t “o” the fifth character in the string? To make things more simple, Python (and most other programming languages) start things at 0 instead of 1. So the index of “o” is 4.

```python
astring = "Hello world!"
print(astring.count("l"))
```

For those of you using silly fonts, that is a lowercase L, not a number one. This counts the number of l’s in the string. Therefore, it should print 3.

```python
astring = "Hello world!"
print(astring[3:7])
```

This prints a slice of the string, starting at index 3, and ending at index 6. But why 6 and not 7? Again, most programming languages do this - it makes doing math inside those brackets easier.

If you just have one number in the brackets, it will give you the single character at that index. If you leave out the first number but keep the colon, it will give you a slice from the start to the number you left in. If you leave out the second number, it will give you a slice from the first number to the end.

You can even put negative numbers inside the brackets. They are an easy way of starting at the end of the string instead of the beginning. This way, -3 means “3rd character from the end”.

```python
astring = "Hello world!"
print(astring[3:7:2])
```

This prints the characters of string from 3 to 7 skipping one character. This is extended slice syntax. The general form is [start:stop:step].

```python
astring = "Hello world!"
print(astring[3:7])
print(astring[3:7:1])
```

Note that both of them produce same output

There is no function like strrev in C to reverse a string. But with the above mentioned type of slice syntax you can easily reverse a string like this

```python
astring = "Hello world!"
print(astring[::-1])
```

This

```python
astring = "Hello world!"
print(astring.upper())
print(astring.lower())
```

These make a new string with all letters converted to uppercase and lowercase, respectively.

```python
astring = "Hello world!"
print(astring.startswith("Hello"))
print(astring.endswith("asdfasdfasdf"))
```

This is used to determine whether the string starts with something or ends with something, respectively. The first one will print True, as the string starts with “Hello”. The second one will print False, as the string certainly does not end with “asdfasdfasdf”.

```python
astring = "Hello world!"
afewwords = astring.split(" ")
```

This splits the string into a bunch of strings grouped together in a list. Since this example splits at a space, the first item in the list will be “Hello”, and the second will be “world!”.

## MCP servers worth wiring into a Python project
Source: https://learn-python.com/ai/mcp/

The Model Context Protocol is a standard way for an agent to call tools it did not ship with — read your database schema, query your error tracker, fetch current library documentation. It solved a real problem: before it, every tool integration was bespoke to one client.

The failure mode now is the opposite one. Installing ten servers puts several hundred tool definitions into every request, which crowds out your code, slows every turn and measurably degrades tool selection. **Each server should have to justify itself.**

:::verdict The rule
Install a server when it gives the agent information it genuinely cannot get from your repo or your shell. If the agent could get the same answer by running a command you have already allowlisted, the command is better — it costs no context until it is used.
:::

## The ones that earn their place

**Documentation retrieval.** The single highest-value category for Python, because library APIs move faster than training data. A model confidently writing the 2023 signature of a fast-moving library is a recurring, expensive failure; a docs server that fetches the version you actually have installed removes it.

**Your database, read-only.** Letting an agent inspect the real schema — column types, nullability, indexes, foreign keys — rather than inferring it from your models eliminates an entire class of "wrote a query against a column that does not exist" errors. Read-only, against a non-production database.

**Error tracking.** "Here is the stack trace and the last twenty occurrences" turns a debugging session from a description of a bug into the bug itself. This is where MCP is at its most obviously useful.

**Issue tracker.** Worth it if your issues are actually written; not worth it if your issues are two-line reminders to yourself.

## The ones to think twice about

**Filesystem servers.** Your agent already has file access. This duplicates it with a worse interface and a large tool surface.

**Git servers.** Same argument. `git` is a CLI, the agent has a shell, and the CLI has better documentation than any wrapper.

**Anything with write access to production.** The blast radius is not worth the convenience, and it will eventually be exercised by a prompt-injection payload in a page the agent read.

**Aggregator servers exposing fifty tools.** These are the worst offenders for context bloat. If you only need two of the fifty, the other forty-eight are pure tax.

## Configuration

```json .mcp.json
{
  "mcpServers": {
    "docs": {
      "command": "uvx",
      "args": ["some-docs-mcp-server@latest"]
    },
    "postgres-dev": {
      "command": "uvx",
      "args": ["some-postgres-mcp-server", "--read-only"],
      "env": { "DATABASE_URL": "postgresql://localhost/myapp_dev" }
    }
  }
}
```

Commit `.mcp.json` so the team shares a configuration, and keep credentials in the environment rather than in the file. A `DATABASE_URL` in a committed config is a credential in your git history.

:::danger Prompt injection is the real risk here
An MCP server returns text, and that text enters the model's context as data it may act on. A server that fetches web pages, reads issues, or returns error-tracker payloads is returning **content written by other people** — including, potentially, instructions aimed at your agent.

Treat every MCP result as untrusted input:
- Prefer read-only servers.
- Never give a server write access to something you would not let a stranger write to.
- Be suspicious of servers that fetch arbitrary URLs and also have write tools available in the same session. That combination is the whole attack.
:::

## Writing one in Python

Worth doing when you have an internal system the agent keeps needing — a feature-flag service, a staging deploy API, an internal search index. It is much less work than people expect.

```python mcp_server.py
from mcp.server.fastmcp import FastMCP
import httpx

mcp = FastMCP("acme-internal")


@mcp.tool()
async def find_customer(email: str) -> dict:
    """Look up a customer by email in the staging environment.

    Returns id, plan, created_at and feature flags. Staging data only —
    never production. Use this instead of guessing at fixture data.
    """
    async with httpx.AsyncClient() as client:
        r = await client.get(f"{STAGING}/internal/customers", params={"email": email})
        r.raise_for_status()
        return r.json()


@mcp.resource("schema://tables")
def table_list() -> str:
    """The current staging database schema, as CREATE TABLE statements."""
    return introspect_schema()


if __name__ == "__main__":
    mcp.run()
```

Three things matter more than the code:

1. **The docstring is the interface.** It is what the model reads to decide whether and how to call the tool. Write it for a competent colleague who has never seen your systems — say what it returns, what it does *not* cover, and when not to use it.
2. **Return structured data, not prose.** A dict the model can index beats a sentence it has to parse.
3. **Keep the tool count small.** Four well-described tools beat twenty thin ones, both for selection accuracy and for context cost.

## Measuring the cost

Before adding a server, look at what it actually adds. In Claude Code, `/context` shows the breakdown; most clients have an equivalent. If a server is consuming a meaningful share of your window and you have used it twice this month, remove it — you can add it back for the session where you need it.

:::promo digitalocean
:::

## The shape of a good setup

For a typical Python service, this is usually enough:

```text
docs server        -> current library APIs
postgres (ro, dev) -> real schema
error tracker      -> real stack traces
```

Three servers, each answering a question the repo cannot. Everything else the agent can get with a shell command you have already allowlisted, at zero standing cost.

## Common questions

### Do I need MCP at all?

No. A well-configured agent with shell access, a fast test suite and a good `AGENTS.md` is already most of the value. MCP is worth adding when you notice the agent repeatedly guessing at something that exists in a system it cannot see — a schema, a stack trace, a current API signature.

### Is it safe to connect an agent to my database?

To a development or staging database, read-only, yes. To production, no — and not because the agent is malicious, but because an agent's actions can be influenced by any text that reaches its context, including text written by other people. Read-only against non-production data keeps the worst case boring.

### Why does adding servers make the agent worse at choosing tools?

Because tool selection is a discrimination problem, and every extra tool definition is another near-neighbour to confuse with the right one. It is the same reason a 500-line instructions file works worse than a 50-line one: attention is finite and dilution is real.

### Can I write an MCP server without publishing it?

Yes, and most useful ones are never published. A local script referenced by path in `.mcp.json` works exactly the same as a published package, and internal systems are precisely where the value is highest.

## Conditions
Source: https://learn-python.com/conditions/

Python uses boolean logic to evaluate conditions. The boolean values True and False are returned when an expression is compared or evaluated. For example:

```python
x = 2
print(x == 2) # prints out True
print(x == 3) # prints out False
print(x < 3) # prints out True
```

Notice that variable assignment is done using a single equals operator “=”, whereas comparison between two variables is done using the double equals operator “==”. The “not equals” operator is marked as “!=”.

### Boolean operators

The “and” and “or” boolean operators allow building complex boolean expressions, for example:

```python
name = "John"
age = 23
if name == "John" and age == 23:
    print("Your name is John, and you are also 23 years old.")

if name == "John" or name == "Rick":
    print("Your name is either John or Rick.")
```

### The “in” operator

The “in” operator could be used to check if a specified object exists within an iterable object container, such as a list:

```python
name = "John"
if name in ["John", "Rick"]:
    print("Your name is either John or Rick.")
```

Python uses indentation to define code blocks, instead of brackets. The standard Python indentation is 4 spaces, although tabs and any other space size will work, as long as it is consistent. Notice that code blocks do not need any termination.

Here is an example for using Python’s “if” statement using code blocks:

```python
statement = False
another_statement = True
if statement is True:
    # do something
    pass
elif another_statement is True: # else if
    # do something else
    pass
else:
    # do another thing
    pass
```

For example:

```python
x = 2
if x == 2:
    print("x equals two!")
else:
    print("x does not equal to two.")
```

A statement is evaulated as true if one of the following is correct:

  - The “True” boolean variable is given, or calculated using an expression, such as an arithmetic comparison.

  - An object which is not considered “empty” is passed.

Here are some examples for objects which are considered as empty:

  - An empty string: “”

  - An empty list: []

  - The number zero: 0

  - The false boolean variable: False

### The ‘is’ operator

Unlike the double equals operator “==”, the “is” operator does not match the values of the variables, but the instances themselves. For example:

```python
x = [1,2,3]
y = [1,2,3]
print(x == y) # Prints out True
print(x is y) # Prints out False
```

### The “not” operator

Using “not” before a boolean expression inverts it:

```python
print(not False) # Prints out True
print((not False) == (False)) # Prints out False
```

## From issue to pull request: running a feature with an agent
Source: https://learn-python.com/ai/spec-to-pr/

Here is the loop that works, on a real Python codebase, for a change of meaningful size. It is not complicated. The discipline is in doing steps two and five, which are the ones everyone drops first and misses most.

## 1. Write the spec as tests, or as a sentence you could test

The prompt that produces good output is not a longer description. It is a description with a **checkable finish line**.

> **Weak:** "Add rate limiting to the API."
>
> **Strong:** "Add per-API-key rate limiting to the FastAPI app: 100 requests per minute, sliding window, backed by the existing Redis connection in `adapters/cache.py`. Over the limit returns 429 with a `Retry-After` header in seconds. Health check endpoints are exempt. Limits are configurable per key, defaulting to 100. Write the tests first; I want to read them before you touch `src/`."

The second version is not longer because verbosity helps. It is longer because it makes six decisions that would otherwise be made silently and wrongly: the scope of the key, the algorithm, where state lives, the response shape, the exemption, the default.

If you cannot write that paragraph, you do not yet know what you want, and an agent will not discover it for you — it will pick something plausible and you will find out at review.

## 2. Make it plan before it edits

The step people skip. Ask for the approach first, with no code.

```text
Before writing anything: which files will you change, what is the
approach for the sliding window, and what are you unsure about?
Do not edit yet.
```

This costs thirty seconds and catches the expensive class of error — right code, wrong design — while it is still free to fix. It also surfaces the thing you forgot: agents are good at noticing "you did not say what happens when Redis is unavailable", and that question is much cheaper now than in review.

Most tools have a first-class version of this (a plan or ask mode). Use it. If yours does not, the instruction above works fine.

:::tip The one question worth always asking
"What are you unsure about?" reliably produces the list of assumptions the model is about to make. Reading that list is the highest information-per-second moment in the whole workflow.
:::

## 3. Tests first, and read them before the implementation

Have it write the tests, then stop and read them yourself before any implementation exists.

This is the point where you find out whether it understood you. A test suite is a specification you can read in ninety seconds, and reading it *before* the implementation exists means you are judging the understanding rather than being anchored by working code.

```python tests/api/test_rate_limit.py
async def test_allows_up_to_limit(client, api_key): ...
async def test_blocks_over_limit(client, api_key): ...
async def test_429_includes_retry_after_seconds(client, api_key): ...
async def test_window_slides(client, api_key, frozen_time): ...
async def test_health_endpoint_is_exempt(client): ...
async def test_limits_are_per_key(client, api_key, other_key): ...
async def test_redis_unavailable_fails_open(client, api_key, broken_redis): ...
```

That last test is the one you argue about — fail open or fail closed? — and it is much better to have that argument here than after an incident.

## 4. Let it implement, and leave it alone

With a plan agreed and tests written, this part is genuinely hands-off. Let it iterate against the suite. Interrupting mid-loop to make suggestions usually makes things worse, because you are injecting a change of direction into a process that was converging.

Watch for two things only:

- **It changed the tests.** Sometimes legitimate; always worth reading.
- **It has been going for a long time without the suite getting greener.** That is the signal to stop and re-plan, not to wait longer. Repeated failed attempts pollute the context with dead ends.

## 5. Review the diff properly

The other step people skip. Some specific things to look for in Python, beyond the general [failure-mode catalogue](/review/failure-modes/):

```bash
git diff --stat                       # is the size what you expected?
git diff -- tests/                    # read the test changes first, separately
git diff -- src/
git diff | grep -nE 'type: ignore|noqa|except Exception|TODO|pass$'
uv run ruff check . && uv run mypy src
```

Read the test diff **before** the source diff. If the tests changed, you need to know that before you form an opinion about whether the source is correct.

Questions worth asking every time:

- **Is the diff bigger than the change?** Reformatting, renamed variables, "while I was here" refactors. Ask for them to be removed or split out. A large diff is not reviewed; it is skimmed.
- **Did it add a dependency?** Check the name character by character — see [hallucinated packages](/review/dependencies/) — and ask whether the stdlib already does it.
- **Did it duplicate something?** The most common structural failure. Search for one distinctive phrase from the new code; if a near-identical function exists elsewhere, the agent did not find it and you now have two.
- **Does it match the surrounding code?** Not just style: error handling, logging, how config is read. Local consistency matters more than global correctness for maintainability.

:::warn The plausibility trap
Generated code reads better than most human code — consistent naming, tidy structure, complete docstrings. Fluency is not correctness, and the polish makes it harder to stay suspicious. If you notice yourself approving quickly because it "looks clean", that is precisely the moment to slow down.
:::

## 6. Commit history that a human can read

Ask for small commits with real messages, and write the PR description yourself — or at least edit it hard. A generated PR description tends to describe *what the diff contains*, which the reviewer can see. What they cannot see is why you chose this approach, what you rejected, and what you are unsure about.

```text
Commits: one per behavioural change. Message says why, not what.
PR description: I'll write it. Give me three bullets on anything you
were uncertain about or any decision you made that I did not specify.
```

Those three bullets are the most valuable output of the entire session.

## 7. Before merge

```bash
make check                            # lint, types, full suite
uv run pytest -q                      # including integration
git diff main --stat
```

And one question that no tool answers: **would I be comfortable being paged for this at 3am?** If the honest answer is no because you do not really understand a part of it, go back and understand that part. Shipping code you cannot debug is the actual risk in all of this, and it is not a technical problem.

:::promo boot-dev
:::

## Common questions

### How big a change can I hand over in one go?

Roughly: one that you could review in twenty minutes. Beyond that, review quality collapses and you are approving rather than reviewing. For anything larger, break it at a natural seam and run the loop twice — the second run starts from a codebase that already has the first half in it, so it is not twice the work.

### Should I let it commit and push?

Commit, yes — small commits are useful and reversible. Push and open a PR, only behind a confirmation. The asymmetry is that a local commit is invisible to everyone else and trivially undone, while a push is visible and a PR notifies people.

### It keeps failing the same test. What now?

Stop. Repeated failure means the context now contains several wrong approaches, which makes the next attempt worse rather than better. Start a fresh session with a summary of what did not work, or do that piece yourself — often the fastest path once you have seen three failed attempts is to write the tricky ten lines and hand back the rest.

### Is the planning step really worth it on small changes?

No. For a change you could make yourself in five minutes, skip straight to the diff and review it. The planning step earns its cost somewhere around "I would need to think for a bit before starting", which is also roughly where handing it over starts to be worth doing at all.

## Loops
Source: https://learn-python.com/loops/

There are two types of loops in Python, for and while.

### The “for” loop

For loops iterate over a given sequence. Here is an example:

```python
primes = [2, 3, 5, 7]
for prime in primes:
    print(prime)
```

For loops can iterate over a sequence of numbers using the “range” and “xrange” functions. The difference between range and xrange is that the range function returns a new list with numbers of that specified range, whereas xrange returns an iterator, which is more efficient. (Python 3 uses the range function, which acts like xrange). Note that the range function is zero based.

```python
# Prints out the numbers 0,1,2,3,4
for x in range(5):
    print(x)

# Prints out 3,4,5
for x in range(3, 6):
    print(x)

# Prints out 3,5,7
for x in range(3, 8, 2):
    print(x)
```

### “while” loops

While loops repeat as long as a certain boolean condition is met. For example:

```python
# Prints out 0,1,2,3,4

count = 0
while count < 5:
    print(count)
    count += 1  # This is the same as count = count + 1
```

### “break” and “continue” statements

**break** is used to exit a for loop or a while loop, whereas **continue** is used to skip the current block, and return to the “for” or “while” statement. A few examples:

```python
# Prints out 0,1,2,3,4

count = 0
while True:
    print(count)
    count += 1
    if count >= 5:
        break

# Prints out only odd numbers - 1,3,5,7,9
for x in range(10):
    # Check if x is even
    if x % 2 == 0:
        continue
    print(x)
```

### Can we use “else” clause for loops?

Unlike languages like C,CPP.. we can use **else** for loops. When the loop condition of “for” or “while” statement fails then code part in “else” is executed. If a **break** statement is executed inside the for loop then the “else” part is skipped.
Note that the “else” part is executed even if there is a **continue** statement.

Here are a few examples:

```python
# Prints out 0,1,2,3,4 and then it prints "count value reached 5"

count=0
while(count<5):
    print(count)
    count +=1
else:
    print("count value reached %d" %(count))

# Prints out 1,2,3,4
for i in range(1, 10):
    if(i%5==0):
        break
    print(i)
else:
    print("this is not printed because for loop is terminated because of break but not due to fail in condition")
```

## When not to hand it to the agent
Source: https://learn-python.com/ai/when-not-to/

Almost everything written about coding agents is about getting more out of them. This page is the other half, because knowing when not to reach for the tool is a large part of using it well, and it is the part that nobody has an incentive to publish.

## When you cannot check the answer

This is the real criterion, and it subsumes most of the others.

An agent's output is only as trustworthy as your ability to verify it. If you can run a test, read a diff, or reason about the result, you are fine. If you cannot — if it is a numerical method whose correctness you would have to take on faith, a concurrency change whose failure mode is a race you cannot reproduce, a security control you are not qualified to review — then delegating does not save you work. It converts work you understand into risk you cannot see.

:::verdict The test
Before handing something over, ask: **if this comes back subtly wrong, what would tell me?** If the honest answer is "nothing until production", do it yourself, or get a second human.
:::

## When the requirements are genuinely unclear

An agent will not tell you that your requirements are contradictory. It will pick an interpretation and implement it beautifully, and you will discover the contradiction at review — having burned the time twice.

Ambiguity is resolved by thinking, or by asking a person who knows. That work is not delegable, and dressing it up as a prompt does not change that. Write the paragraph from [issue to pull request](/ai/spec-to-pr/) first; if you cannot, that is your signal.

## When the problem is genuinely novel

Models are strongest where the training data is dense. On the well-trodden — a FastAPI endpoint, a pandas transformation, a retry decorator — they are excellent and faster than you.

On the genuinely unusual — a scheduling algorithm specific to your domain, an optimisation with constraints nobody else has, a protocol you invented — output quality drops sharply and, more dangerously, confidence does not. You get the same fluent, well-structured, plausible code, and it is wrong in ways that take a long time to find.

A useful heuristic: **if you cannot find three similar things on GitHub, expect the agent to struggle too.**

## When the change is smaller than the explanation

If you can make the change in two minutes, making it takes two minutes. Writing a prompt precise enough to get it right, waiting, and reviewing the diff takes five, and the diff will contain a tidy-up you did not ask for.

This sounds obvious and is nevertheless the most common waste. The habit of reaching for the agent for everything is easy to form and expensive.

## When you are the one who needs to learn it

This is the one that costs most, and the cost is invisible for about six months.

Delegating everything you are unfamiliar with means you stay unfamiliar with it. Then a production incident lands in that area, the agent's suggestion does not work, and you are debugging a system you have never actually reasoned about — under time pressure, at the worst possible moment.

The people getting the most out of these tools are not the ones delegating the most. They are the ones who can tell, in about four seconds, whether a diff is right — and that judgement was built by writing the code, repeatedly, before they had a choice.

:::note A working rule
Delegate the work you have done a hundred times. Do the work you have never done, at least the first three times. Delegate it after that.

The corollary matters too: if you are learning Python, use these sites' [foundations track](/) and type the exercises yourself. You are not being slow — you are building the thing that makes the tool useful later.
:::

## When the code is load-bearing and rarely touched

Auth, billing, migrations, permissions, anything with money or personal data in it. Not because an agent writes these badly, but because the cost asymmetry is extreme: a small mistake in a payment path costs more than the entire time saving across a year of delegation.

Use the agent to *review* these. Have it read the diff, list what could go wrong, check for the failure modes. That direction of use — agent as second reader rather than first author — is undervalued and it is exactly right for high-stakes code.

## When you are tired

Not a joke. Reviewing generated code well takes more attention than writing code, because fluent wrong code triggers none of the friction that unfamiliar wrong code does. Late in the day, the reviewing gets worse while the generating stays exactly as good, and the ratio is against you.

Writing code tired produces obvious bugs. Approving code tired produces subtle ones.

## What this adds up to

Not "use it less". Use it for volume, boilerplate, translation, unfamiliar API surface, test generation, refactoring under a green suite, and reading unfamiliar code — where it is genuinely excellent and you would be silly not to.

Keep for yourself: the ambiguity, the novel, the unverifiable, the high-stakes, and the things you are still learning. That is a smaller list than it used to be, and it is also, not coincidentally, the list of things that were always the actual job.

## Common questions

### Is this just gatekeeping about learning to code properly?

No. The argument is narrower and practical: you cannot review what you have never understood, and reviewing is now the bottleneck. If you never build the judgement, the tool's ceiling becomes your ceiling — and it gets worse rather than better as the volume of generated code grows.

### Should juniors use coding agents?

Yes, with the same rule: delegate what you have done before, do what you have not. The risk is not that a junior uses the tool, it is that they use it to skip the repetitions that build pattern recognition. Using it to explain unfamiliar code, or to review their own, is unambiguously good.

### How do I tell if I have got the balance wrong?

Two symptoms. You approve diffs you could not have written, and you find yourself unable to debug your own codebase without the agent. Either one means pulling more work back for a while.

## Functions
Source: https://learn-python.com/functions/

### What are Functions?

Functions are a convenient way to divide your code into useful blocks, allowing us to order our code, make it more readable, reuse it and save some time. Also functions are a key way to define interfaces so programmers can share their code.

### How do you write functions in Python?

As we have seen on previous tutorials, Python makes use of blocks.

A block is a area of code of written in the format of:

```python
block_head:
    1st block line
    2nd block line
    ...
```

Where a block line is more Python code (even another block), and the block head is of the following format:
block_keyword block_name(argument1,argument2, …)
Block keywords you already know are “if”, “for”, and “while”.

Functions in python are defined using the block keyword “def”, followed with the function’s name as the block’s name.
For example:

```python
def my_function():
    print("Hello From My Function!")
```

Functions may also receive arguments (variables passed from the caller to the function).
For example:

```python
def my_function_with_args(username, greeting):
    print("Hello, %s , From My Function!, I wish you %s"%(username, greeting))
```

Functions may return a value to the caller, using the keyword- ‘return’ .
For example:

```python
def sum_two_numbers(a, b):
    return a + b
```

### How do you call functions in Python?

Simply write the function’s name followed by (), placing any required arguments within the brackets.
For example, lets call the functions written above (in the previous example):

```python
# Define our 3 functions
def my_function():
    print("Hello From My Function!")

def my_function_with_args(username, greeting):
    print("Hello, %s, From My Function!, I wish you %s"%(username, greeting))

def sum_two_numbers(a, b):
    return a + b

# print(a simple greeting)
my_function()

#prints - "Hello, John Doe, From My Function!, I wish you a great year!"
my_function_with_args("John Doe", "a great year!")

# after this line x will hold the value 3!
x = sum_two_numbers(1,2)
```

## Testing Python code that calls a language model
Source: https://learn-python.com/ai/evals/

More and more Python code is a thin layer around a model call. That code has properties ordinary Python does not: the same input gives different output, each test costs money and a second or two, and "correct" is often a judgement rather than an equality.

The instinct is to give up and test nothing, or to mock the model and test nothing meaningful. There is a better middle, and it looks like a pyramid.

## The pyramid

```text
      few, slow, expensive
   ┌──────────────────────────┐
   │  4. live evals (nightly) │   real model, scored dataset
   ├──────────────────────────┤
   │  3. cassette tests (CI)  │   recorded responses, real shapes
   ├──────────────────────────┤
   │  2. contract tests       │   parsing, validation, retries, errors
   ├──────────────────────────┤
   │  1. pure logic tests     │   prompt building, chunking, routing
   └──────────────────────────┘
      many, fast, free
```

The mistake most codebases make is having only layer 4, running it rarely because it is expensive, and therefore having no signal at all during development.

## Layer 1: make most of it deterministic

The majority of an LLM application is not the model call. It is prompt assembly, chunking, retrieval ranking, routing, output post-processing, cost accounting. All of that is ordinary Python and should be tested as ordinary Python.

This only works if you have separated it. The single most valuable structural decision here:

```python src/app/llm.py
class Completion(Protocol):
    async def __call__(self, *, system: str, messages: list[Message]) -> str: ...
```

One narrow interface, one implementation that talks to the API, one that does not. Everything above it is testable without a network.

```python
def test_prompt_includes_only_top_k_chunks():
    prompt = build_prompt(query="q", chunks=make_chunks(20), k=5)
    assert prompt.count("<chunk>") == 5


def test_chunks_are_ordered_by_score_descending():
    ...
```

Unglamorous, fast, and it catches a surprising share of real bugs.

## Layer 2: contract tests against a fake

Test what your code does with a response, not what the model says. Every one of these is a real production failure:

```python
@pytest.mark.parametrize("payload", [
    '{"category": "billing"}',           # happy path
    '```json\n{"category":"billing"}\n```',  # fenced — models do this constantly
    '{"category": "Billing"}',           # wrong case
    '{"categorie": "billing"}',          # misspelled key
    '{"category": "refunds"}',           # not in your enum
    'I think this is a billing issue.',  # ignored the format entirely
    '',                                  # empty
    '{"category": "billing"',            # truncated at the token limit
])
def test_parser_never_raises_and_never_invents(payload):
    result = parse_category(payload)
    assert result is None or result in Category
```

The assertion is the point: **never raise, never invent**. A parser that returns a valid-looking wrong answer on malformed input is worse than one that returns `None`.

Then test the behaviour around it — retries, timeouts, rate limit backoff, what happens on a 500 — with a fake that returns those conditions on demand. None of this needs a real model.

:::tip Use structured output, then test the fallback anyway
Constrained decoding and JSON-schema modes remove most parse failures and you should absolutely use them. Test the fallback path regardless: providers have outages, you will change models, and the day the schema mode fails is the day you find out whether your parser was defensive.
:::

## Layer 3: cassettes in CI

Record real responses once, replay them forever. You get real response shapes — including the weird ones — at zero cost and zero flakiness.

```python conftest.py
import pytest

@pytest.fixture(scope="module")
def vcr_config():
    return {
        "filter_headers": ["authorization", "x-api-key"],
        "record_mode": "once",
        "match_on": ["method", "scheme", "host", "port", "path", "body"],
    }
```

```python
@pytest.mark.vcr
async def test_classifies_a_refund_request(client):
    result = await classify("I want my money back")
    assert result.category is Category.REFUNDS
```

Two rules that make this work rather than rot:

1. **Scrub credentials before committing cassettes.** `filter_headers` above, and read the first one you commit.
2. **Re-record on a schedule**, not never. A cassette from eighteen months ago is testing a model that no longer exists. Monthly is usually right, and the diff when you re-record is genuinely informative.

## Layer 4: evals with a scored dataset

Here you accept non-determinism and measure it instead of asserting on it.

```python evals/dataset.jsonl
{"input": "my card was charged twice", "expect": "billing"}
{"input": "how do I export my data?", "expect": "support"}
{"input": "cancel and refund please",  "expect": "refunds"}
```

```python evals/run.py
import asyncio, json, statistics

async def main() -> None:
    cases = [json.loads(l) for l in open("evals/dataset.jsonl")]
    results = await asyncio.gather(*(classify(c["input"]) for c in cases))
    hits = [r.category.value == c["expect"] for r, c in zip(results, cases)]
    accuracy = statistics.mean(hits)
    print(f"accuracy {accuracy:.1%} on {len(cases)} cases")
    for hit, c, r in zip(hits, cases, results):
        if not hit:
            print(f"  MISS {c['input']!r}: got {r.category} want {c['expect']}")
    raise SystemExit(0 if accuracy >= 0.90 else 1)

asyncio.run(main())
```

Nightly, not per-commit. A threshold, not an assertion. And **look at the misses** — the list of failures is worth more than the number, because it is where the next prompt change comes from.

:::note Building the dataset is the actual work
Fifty real, awkward examples beat five hundred synthetic ones. Take them from production logs, from support tickets, from the bug reports you get. Every time something goes wrong in production, the input becomes a case. That is the flywheel; the harness is trivial by comparison.
:::

## When the output is free text

For summarisation, drafting, or explanation there is no equality to assert. Three approaches, in order of how much you should trust them:

**Assert on properties, not content.** Does the summary mention every entity in the source? Is it under the length limit? Does it avoid the words we banned? Is it in the requested language? These are deterministic, cheap, and catch most real regressions.

```python
def test_summary_is_grounded(article, summary):
    """Every number in the summary must appear in the source."""
    assert set(re.findall(r"\d[\d,.]*", summary)) <= set(re.findall(r"\d[\d,.]*", article))
```

That one test catches fabricated figures, which is the failure that matters most in summarisation.

**Assert on invariants across runs.** Same input twice at temperature 0 should give similar output. A large divergence is a signal even without a ground truth.

**LLM-as-judge, carefully.** A second model scores the output against a rubric. It works, and it has known biases — position, verbosity, self-preference. Use it for *relative* comparisons (is variant B better than A?) rather than absolute scores, keep the rubric short and specific, and calibrate it against fifty human-labelled examples before you trust a number from it.

## Cost control in CI

- Layers 1–3 use no tokens. They are the ones on every commit.
- Layer 4 runs nightly and on changes to `prompts/` — path filters do most of the work.
- Set a hard spend cap on the CI key, not just an alert.
- Cache aggressively; use the cheapest model that discriminates for the judge.

```yaml .github/workflows/test.yml
on:
  push:
  pull_request:
jobs:
  fast:
    steps:
      - run: uv run pytest -q -m "not eval"      # free, every push
  evals:
    if: github.event_name == 'schedule' || contains(github.event.head_commit.modified, 'prompts/')
    steps:
      - run: uv run python evals/run.py
```

:::promo digitalocean
:::

## Common questions

### Should I mock the model in unit tests?

Mock the *transport*, not the model. Put a narrow protocol between your code and the API, and give it a fake implementation. Mocking the SDK's internals ties your tests to a library version and tests nothing you care about.

### How much will running these cost?

Layers 1-3 use no tokens at all, which is why they are the ones on every commit. Layer 4 is where the money goes, and the levers are batch APIs, a cheap model for the judge, and running nightly rather than per-push. There is a full treatment in [tracking and cutting token costs](/ai/tokenomics/).

### How many eval cases do I need?

Start with twenty real, awkward ones and grow the set from production failures. Statistical power matters less than coverage of the cases that actually bite you, and a small curated set you look at beats a large synthetic one you only read the mean of.

### Is LLM-as-judge trustworthy?

For relative comparisons between two variants, reasonably. For absolute quality scores, not without calibration — judges show position bias, favour longer answers, and prefer output from their own model family. Label fifty examples by hand and check that the judge agrees with you before you let a number gate a deploy.

### Temperature 0 makes it deterministic, so can I just assert equality?

No. Temperature 0 is greedy sampling, not determinism — batching, hardware and provider-side changes all move the output, and any model version change moves it substantially. Assert on properties, or use cassettes.

## Classes and Objects
Source: https://learn-python.com/classes-and-objects/

Objects are an encapsulation of variables and functions into a single entity. Objects get their variables and functions from classes. Classes are essentially a template to create your objects.

A very basic class would look something like this:

```python
class MyClass:
    variable = "blah"

    def function(self):
        print("This is a message inside the class.")
```

We’ll explain why you have to include that “self” as a parameter a little bit later. First, to assign the above class(template) to an object you would do the following:

```python
class MyClass:
    variable = "blah"

    def function(self):
        print("This is a message inside the class.")

myobjectx = MyClass()
```

Now the variable “myobjectx” holds an object of the class “MyClass” that contains the variable and the function defined within the class called “MyClass”.

### Accessing Object Variables

To access the variable inside of the newly created object “myobjectx” you would do the following:

```python
class MyClass:
    variable = "blah"

    def function(self):
        print("This is a message inside the class.")

myobjectx = MyClass()

myobjectx.variable
```

So for instance the below would output the string “blah”:

```python
class MyClass:
    variable = "blah"

    def function(self):
        print("This is a message inside the class.")

myobjectx = MyClass()

print(myobjectx.variable)
```

You can create multiple different objects that are of the same class(have the same variables and functions defined). However, each object contains independent copies of the variables defined in the class. For instance, if we were to define another object with the “MyClass” class and then change the string in the variable above:

```python
class MyClass:
    variable = "blah"

    def function(self):
        print("This is a message inside the class.")

myobjectx = MyClass()
myobjecty = MyClass()

myobjecty.variable = "yackity"

# Then print out both values
print(myobjectx.variable)
print(myobjecty.variable)
```

### Accessing Object Functions

To access a function inside of an object you use notation similar to accessing a variable:

```python
class MyClass:
    variable = "blah"

    def function(self):
        print("This is a message inside the class.")

myobjectx = MyClass()

myobjectx.function()
```

The above would print out the message, “This is a message inside the class.”

## Tracking and cutting token costs in Python
Source: https://learn-python.com/ai/tokenomics/

The economics are the same in every language — [what tokens cost and where the money goes](https://codelearningdojo.com/token-economics/) is the model. This page is the Python implementation: how to count, how to attribute, and how to stop a loop from spending your month in an afternoon.

## Count before you send

The cheapest token is the one you notice before paying for it. Counting locally costs nothing and lets you refuse or trim a request before it leaves the process.

```python src/app/tokens.py
from functools import lru_cache

@lru_cache(maxsize=4)
def _encoder(model: str):
    import tiktoken
    try:
        return tiktoken.encoding_for_model(model)
    except KeyError:
        return tiktoken.get_encoding("cl100k_base")   # good enough for a budget check


def estimate_tokens(text: str, model: str = "gpt-4o") -> int:
    return len(_encoder(model).encode(text))


def estimate_chars(text: str) -> int:
    """No dependency, no network: ~4 chars per token for English prose."""
    return len(text) // 4
```

:::warn A local count is an estimate, not the invoice
Tokenisers differ between providers and model families, and none of them account for the tokens a provider adds for system scaffolding, tool schemas or images. Use the local count to *decide* — is this request too big, should I trim, is this within budget — and use the `usage` field the API returns to *account*. Never reconcile a bill against a local count.
:::

For a hard guarantee on the way in, the provider's own token-counting endpoint is exact; it costs a round trip, so use it for the boundary case rather than every call.

## Attribute every call

The single most useful piece of infrastructure you can build here is small. Wrap the call, record the usage the API already gives you, and tag it.

```python src/app/costs.py
from __future__ import annotations
import functools, json, os, time
from contextvars import ContextVar
from dataclasses import dataclass, asdict, field
from decimal import Decimal

# per-1M-token prices. Keep them in config, not in code — they change.
PRICES: dict[str, dict[str, Decimal]] = {
    "small": {"in": Decimal("0.25"), "cached_in": Decimal("0.03"), "out": Decimal("1.25")},
    "large": {"in": Decimal("3.00"), "cached_in": Decimal("0.30"), "out": Decimal("15.00")},
}

_ctx: ContextVar[dict] = ContextVar("llm_ctx", default={})


@dataclass(slots=True)
class Spend:
    feature: str
    model: str
    input_tokens: int
    cached_tokens: int
    output_tokens: int
    latency_ms: int
    ok: bool
    tenant: str | None = None
    version: str = field(default_factory=lambda: os.getenv("GIT_SHA", "dev"))

    @property
    def usd(self) -> Decimal:
        p = PRICES.get(self.model, PRICES["large"])
        fresh = max(self.input_tokens - self.cached_tokens, 0)
        return (
            Decimal(fresh) * p["in"]
            + Decimal(self.cached_tokens) * p["cached_in"]
            + Decimal(self.output_tokens) * p["out"]
        ) / Decimal(1_000_000)


def record(spend: Spend) -> None:
    line = asdict(spend) | {"usd": str(spend.usd), "ts": time.time()} | _ctx.get()
    print(json.dumps(line), file=open(os.getenv("LLM_LOG", "llm-spend.jsonl"), "a"))
```

```python
def tracked(feature: str, model: str = "large"):
    """Wrap an LLM call so its usage is always recorded, success or failure."""
    def deco(fn):
        @functools.wraps(fn)
        async def wrapper(*args, **kwargs):
            start = time.perf_counter()
            usage, ok = None, False
            try:
                result, usage = await fn(*args, **kwargs)
                ok = True
                return result
            finally:
                u = usage or {}
                record(Spend(
                    feature=feature,
                    model=model,
                    input_tokens=u.get("input_tokens", 0),
                    cached_tokens=u.get("cache_read_input_tokens", 0),
                    output_tokens=u.get("output_tokens", 0),
                    latency_ms=int((time.perf_counter() - start) * 1000),
                    ok=ok,
                ))
        return wrapper
    return deco
```

```python
@tracked(feature="ticket_classification", model="small")
async def classify(text: str) -> tuple[Category, dict]:
    resp = await client.messages.create(...)
    return parse(resp), resp.usage.model_dump()
```

Two details that matter more than they look:

- **`finally`, not `else`.** A failed call still consumed input tokens. If you only record successes, your cost-per-successful-result is wrong in the direction that flatters you.
- **`Decimal`, not `float`.** You are doing money arithmetic. See [the failure-mode catalogue](/review/failure-modes/) for why this is not pedantry.

`ContextVar` carries the tenant and request id down without threading them through every signature — set it once in your middleware.

## Make the cache hit

The largest single saving, and in Python it is usually a function-argument-ordering problem.

```python
# BAD: the timestamp poisons everything after it
def build(question: str, docs: list[str]) -> list[dict]:
    return [
        {"role": "system", "content": f"Today is {date.today()}.\n{RULES}"},
        {"role": "user", "content": "\n".join(docs) + question},
    ]

# GOOD: stable prefix first, volatile last
def build(question: str, docs: list[str]) -> list[dict]:
    return [
        {"role": "system", "content": RULES,                       # 4KB, never changes
         "cache_control": {"type": "ephemeral"}},
        {"role": "user", "content": "\n".join(sorted(docs)),       # sorted: stable order
         "cache_control": {"type": "ephemeral"}},
        {"role": "user", "content": f"Today is {date.today()}.\n{question}"},
    ]
```

Note `sorted(docs)`. If your documents arrive from a `set`, a `dict`, or a database query with no `ORDER BY`, their order can vary between runs — and a different order means a different prefix means a cache miss on the whole block. This is a real and very Python-flavoured way to lose your cache: it fails silently and only shows up on the bill.

The same applies to tool definitions built from a dict. Sort them.

:::tip Assert on the cache, in a test
```python
async def test_prefix_is_stable():
    a = build("q1", ["doc-b", "doc-a"])
    b = build("q2", ["doc-a", "doc-b"])
    assert a[0] == b[0] and a[1] == b[1]      # only the last message differs
```
That test catches the day someone adds `f"Session {uuid4()}"` to the system prompt.
:::

## Use the batch API for anything that can wait

Roughly half price, and evals, backfills and nightly jobs all qualify.

```python
# instead of this, at full price, at 3am
results = await asyncio.gather(*(classify(r) for r in rows))

# submit a batch and collect later
batch = await client.messages.batches.create(requests=[
    {"custom_id": str(r.id), "params": {...}} for r in rows
])
```

The trade is latency — batches complete within hours rather than seconds. For an [eval run](/ai/evals/) or a one-off classification of a million rows, that is free money.

## Enforce a budget before the call

An alert tells you after the money is gone. A check stops it.

```python src/app/budget.py
class BudgetExceeded(Exception): ...

@dataclass
class Budget:
    limit_usd: Decimal
    spent_usd: Decimal = Decimal("0")
    max_turns: int = 12
    turns: int = 0

    def check(self, estimated_usd: Decimal) -> None:
        if self.turns >= self.max_turns:
            raise BudgetExceeded(f"turn limit {self.max_turns} reached")
        if self.spent_usd + estimated_usd > self.limit_usd:
            raise BudgetExceeded(
                f"would exceed ${self.limit_usd} (spent ${self.spent_usd:.4f})"
            )

    def charge(self, actual_usd: Decimal) -> None:
        self.spent_usd += actual_usd
        self.turns += 1
```

`max_turns` is not optional. An agent loop resends the whole conversation every turn, so cost grows roughly quadratically with turn count — ten turns over a 20k context is nearer 200k tokens than 20k. A turn cap is the cheapest protection against a runaway loop there is, and it is the one people add after the incident rather than before.

## Retries multiply, so bound them

```python
from tenacity import retry, stop_after_attempt, wait_exponential_jitter, retry_if_exception

def retryable(exc: BaseException) -> bool:
    status = getattr(exc, "status_code", None)
    return status == 429 or (status is not None and status >= 500)

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential_jitter(initial=1, max=20),
    retry=retry_if_exception(retryable),   # never retry a 400 — it will fail identically
    reraise=True,
)
async def call_model(**kw): ...
```

Retrying a 400 is pure waste: the request is malformed and will be malformed again. Retrying without jitter turns a provider blip into a synchronised stampede from all your workers.

## Read your own log

The output of `record()` is JSONL, which pandas reads directly. Ten lines gets you the report that actually changes decisions.

```python scripts/spend_report.py
import pandas as pd

df = pd.read_json("llm-spend.jsonl", lines=True)
df["usd"] = df["usd"].astype(float)
df["day"] = pd.to_datetime(df["ts"], unit="s").dt.date

print("\n— by feature —")
print(df.groupby("feature")
        .agg(calls=("usd", "size"), usd=("usd", "sum"),
             ok_rate=("ok", "mean"), p99_out=("output_tokens", lambda s: s.quantile(0.99)))
        .sort_values("usd", ascending=False))

print("\n— cost per SUCCESSFUL result —")
by = df.groupby("feature").agg(total=("usd", "sum"), wins=("ok", "sum"))
print((by["total"] / by["wins"].clip(lower=1)).sort_values(ascending=False))

print("\n— cache hit rate —")
print((df["cached_tokens"].sum() / df["input_tokens"].clip(lower=1).sum()).round(3))

print("\n— top tenants —")
print(df.groupby("tenant")["usd"].sum().nlargest(10))
```

The second block is the one worth staring at. Cost per *call* flatters you; cost per *successful result* includes everything you paid for output that failed to parse and had to be regenerated.

:::verdict The four changes that usually do it
1. **Reorder prompts so the cache hits.** Usually the largest single win, and it changes no behaviour.
2. **Cap turns in every agent loop.** Protects against the worst case rather than the average one.
3. **Route classification and extraction to a small model.** Most calls are not hard.
4. **Set `max_tokens` deliberately on every call.** Output is the expensive side.
:::

## Common questions

### Is `tiktoken` accurate for non-OpenAI models?

No — it is OpenAI's tokeniser. For other providers the count will be in the right order of magnitude, which is fine for a budget check, and wrong for accounting. Use each provider's own counting endpoint when you need exactness, and always reconcile against the `usage` the API returns.

### Where should prices live?

In configuration, loaded at startup, not in a Python literal — they change, and a code deploy to fix a price is a bad afternoon. Keep a dated table so historical spend stays reconcilable when a price changes mid-month.

### Do I need a full observability stack?

Not at first. A JSONL file and the pandas script above answer every question that matters until you are spending real money. When you outgrow it, the same records go to your existing metrics backend without changing what you record — which is why tagging from day one matters more than where it goes.

### How do I stop async fan-out from blowing the budget?

Bound the concurrency with a semaphore and check the budget inside the guarded section, not before it. `asyncio.gather` over a thousand rows with no limit will happily start a thousand requests, and by the time the first budget check fails you have already committed to all of them.

## Dictionaries
Source: https://learn-python.com/dictionaries/

A dictionary is a data type similar to arrays, but works with keys and values instead of indexes. Each value stored in a dictionary can be accessed using a key, which is any type of object (a string, a number, a list, etc.) instead of using its index to address it.

For example, a database of phone numbers could be stored using a dictionary like this:

```python
phonebook = {}
phonebook["John"] = 938477566
phonebook["Jack"] = 938377264
phonebook["Jill"] = 947662781
print(phonebook)
```

Alternatively, a dictionary can be initialized with the same values in the following notation:

```python
phonebook = {
    "John" : 938477566,
    "Jack" : 938377264,
    "Jill" : 947662781
}
print(phonebook)
```

### Iterating over dictionaries

Dictionaries can be iterated over, just like a list. However, a dictionary, unlike a list, does not keep the order of the values stored in it. To iterate over key value pairs, use the following syntax:

```python
phonebook = {"John" : 938477566,"Jack" : 938377264,"Jill" : 947662781}
for name, number in phonebook.items():
    print("Phone number of %s is %d" % (name, number))
```

### Removing a value

To remove a specified index, use either one of the following notations:

```python
phonebook = {
   "John" : 938477566,
   "Jack" : 938377264,
   "Jill" : 947662781
}
del phonebook["John"]
print(phonebook)
```

or:

```python
phonebook = {
   "John" : 938477566,
   "Jack" : 938377264,
   "Jill" : 947662781
}
phonebook.pop("John")
print(phonebook)
```

## Modules and Packages
Source: https://learn-python.com/modules-and-packages/

In programming, a module is a piece of software that has a specific functionality.
For example, when building a ping pong game, one module would be responsible for the game logic, and
another module would be responsible for drawing the game on the screen. Each module is a different
file, which can be edited separately.

### Writing modules

Modules in Python are simply Python files with a .py extension. The name of the module will be the name of the file.
A Python module can have a set of functions, classes or variables defined and implemented.
In the example above, we will have two files, we will have:

```python
mygame/
mygame/game.py
mygame/draw.py
```

The Python script `game.py` will implement the game. It will use the function `draw_game` from the file `draw.py`,
or in other words, the`draw` module, that implements the logic for drawing the game on the screen.

Modules are imported from other modules using the `import` command. In this example, the `game.py` script may
look something like this:

```python
# game.py
# import the draw module
import draw

def play_game():
    ...

def main():
    result = play_game()
    draw.draw_game(result)

# this means that if this script is executed, then
# main() will be executed
if __name__ == '__main__':
    main()
```

The `draw` module may look something like this:

```python
# draw.py

def draw_game():
    ...

def clear_screen(screen):
    ...
```

In this example, the `game` module imports the `draw` module, which enables it to use functions implemented
in that module. The `main` function would use the local function `play_game` to run the game, and then
draw the result of the game using a function implemented in the `draw` module called `draw_game`. To use
the function `draw_game` from the `draw` module, we would need to specify in which module the function is
implemented, using the dot operator. To reference the `draw_game` function from the `game` module,
we would need to import the `draw` module and only then call `draw.draw_game()`.

When the `import draw` directive will run, the Python interpreter will look for a file in the directory which
the script was executed from, by the name of the module with a `.py` suffix, so in our case it will try to
look for `draw.py`. If it will find one, it will import it. If not, he will continue to look for built-in modules.

You may have noticed that when importing a module, a `.pyc` file appears, which is a compiled Python file.
Python compiles files into Python bytecode so that it won’t have to parse the files each time modules are loaded.
If a `.pyc` file exists, it gets loaded instead of the `.py` file, but this process is transparent to the user.

### Importing module objects to the current namespace

We may also import the function `draw_game` directly into the main script’s namespace, by using the `from` command.

```python
# game.py
# import the draw module
from draw import draw_game

def main():
    result = play_game()
    draw_game(result)
```

You may have noticed that in this example, `draw_game` does not precede with the name of the module it is imported
from, because we’ve specified the module name in the `import` command.

The advantages of using this notation is that it is easier to use the functions inside the current module because
you don’t need to specify which module the function comes from. However, any namespace cannot have two objects
with the exact same name, so the `import` command may replace an existing object in the namespace.

### Importing all objects from a module

We may also use the `import *` command to import all objects from a specific module, like this:

```python
# game.py
# import the draw module
from draw import *

def main():
    result = play_game()
    draw_game(result)
```

This might be a bit risky as changes in the module might affect the module which imports it, but it is
shorter and also does not require you to specify which objects you wish to import from the module.

### Custom import name

We may also load modules under any name we want. This is useful when we want to import a module conditionally
to use the same name in the rest of the code.

For example, if you have two `draw` modules with slighty different names - you may do the following:

```python
# game.py
# import the draw module
if visual_mode:
    # in visual mode, we draw using graphics
    import draw_visual as draw
else:
    # in textual mode, we print out text
    import draw_textual as draw

def main():
    result = play_game()
    # this can either be visual or textual depending on visual_mode
    draw.draw_game(result)
```

### Module initialization

The first time a module is loaded into a running Python script, it is initialized by executing the code in the
module once. If another module in your code imports the same module again, it will not be loaded twice but
once only - so local variables inside the module act as a “singleton” - they are initialized only once.

This is useful to know, because this means that you can rely on this behavior for initializing objects.
For example:

```python
# draw.py

def draw_game():
    # when clearing the screen we can use the main screen object initialized in this module
    clear_screen(main_screen)
    ...

def clear_screen(screen):
    ...

class Screen():
    ...

# initialize main_screen as a singleton
main_screen = Screen()
```

### Extending module load path

There are a couple of ways we could tell the Python interpreter where to look for modules, aside from the
default, which is the local directory and the built-in modules. You could either use the environment
variable `PYTHONPATH` to specify additional directories to look for modules in, like this:

```python
PYTHONPATH=/foo python game.py
```

This will execute `game.py`, and will enable the script to load modules from the `foo` directory as well
as the local directory.

Another method is the `sys.path.append` function. You may execute it *before* running an `import` command:

```python
sys.path.append("/foo")
```

This will add the `foo` directory to the list of paths to look for modules in as well.

### Exploring built-in modules

Check out the full list of built-in modules in the Python standard library [here](https://docs.python.org/3/library/).

Two very important functions come in handy when exploring modules in Python - the `dir` and `help` functions.

If we want to import the module `urllib`, which enables us to create read data from URLs, we
simply `import` the module:

```python
# import the library
import urllib

# use it
urllib.urlopen(...)
```

We can look for which functions are implemented in each module by using the `dir` function:

```python
>>> import urllib
>>> dir(urllib)
['ContentTooShortError', 'FancyURLopener', 'MAXFTPCACHE', 'URLopener', '__all__', '__builtins__',
'__doc__', '__file__', '__name__', '__package__', '__version__', '_ftperrors', '_get_proxies',
'_get_proxy_settings', '_have_ssl', '_hexdig', '_hextochr', '_hostprog', '_is_unicode', '_localhost',
'_noheaders', '_nportprog', '_passwdprog', '_portprog', '_queryprog', '_safe_map', '_safe_quoters',
'_tagprog', '_thishost', '_typeprog', '_urlopener', '_userprog', '_valueprog', 'addbase', 'addclosehook',
'addinfo', 'addinfourl', 'always_safe', 'basejoin', 'c', 'ftpcache', 'ftperrors', 'ftpwrapper', 'getproxies',
'getproxies_environment', 'getproxies_macosx_sysconf', 'i', 'localhost', 'main', 'noheaders', 'os',
'pathname2url', 'proxy_bypass', 'proxy_bypass_environment', 'proxy_bypass_macosx_sysconf', 'quote',
'quote_plus', 'reporthook', 'socket', 'splitattr', 'splithost', 'splitnport', 'splitpasswd', 'splitport',
'splitquery', 'splittag', 'splittype', 'splituser', 'splitvalue', 'ssl', 'string', 'sys', 'test', 'test1',
'thishost', 'time', 'toBytes', 'unquote', 'unquote_plus', 'unwrap', 'url2pathname', 'urlcleanup', 'urlencode',
'urlopen', 'urlretrieve']
```

When we find the function in the module we want to use, we can read about it more using the `help` function,
inside the Python interpreter:

```python
help(urllib.urlopen)
```

### Writing packages

Packages are namespaces which contain multiple packages and modules themselves. They are simply directories,
but with a twist.

Each package in Python is a directory which **MUST** contain a special file called `__init__.py`. This file can
be empty, and it indicates that the directory it contains is a Python package, so it can be imported the same
way a module can be imported.

If we create a directory called `foo`, which marks the package name, we can then create a module inside that
package called `bar`. We also must not forget to add the `__init__.py` file inside the `foo` directory.

To use the module `bar`, we can import it in two ways:

```python
import foo.bar
```

or:

```python
from foo import bar
```

In the first method, we must use the `foo` prefix whenever we access the module `bar`. In the second method,
we don’t, because we import the module to our module’s namespace.

The `__init__.py` file can also decide which modules the package exports as the API, while keeping other modules
internal, by overriding the `__all__` variable, like so:

```python
__init__.py:

__all__ = ["bar"]
```

## Files and Context Managers
Source: https://learn-python.com/files-and-context-managers/

```python
with open("notes.txt", "r", encoding="utf-8") as f:
    content = f.read()
# the file is closed here — even if an exception was raised
```

The `with` statement is the point. It guarantees the file is closed on every path out of the block, including an exception, which is why you should essentially never call `open()` without it.

:::warn Always pass `encoding`
```python
open("notes.txt")                      # uses the platform default
open("notes.txt", encoding="utf-8")    # explicit, portable
```
Without it, the same code reads different bytes on different machines, and the failure is a `UnicodeDecodeError` in production on text that worked locally. Generated Python omits it constantly. `ruff`'s `PLW1514` rule catches it.
:::

## Modes

| Mode | Meaning |
|---|---|
| `"r"` | read (default); error if the file is missing |
| `"w"` | write; **truncates an existing file** |
| `"a"` | append |
| `"x"` | create; error if it already exists |
| `"rb"` / `"wb"` | binary — no decoding, gives you `bytes` |
| `"r+"` | read and write |

`"x"` is underused: it is the mode that refuses to overwrite, which is what you want for anything you are creating for the first time.

## Reading

```python
with open("data.txt", encoding="utf-8") as f:
    content = f.read()          # the whole file as one string

with open("data.txt", encoding="utf-8") as f:
    for line in f:              # one line at a time — lazy, no size limit
        print(line.rstrip("\n"))
```

**Iterate the file object** rather than calling `.readlines()`. Iteration streams; `readlines()` loads everything into memory, which is fine for a config file and fatal for an 8GB log.

```python
with open("big.log", encoding="utf-8") as f:
    errors = sum(1 for line in f if "ERROR" in line)   # constant memory
```

## Writing

```python
with open("out.txt", "w", encoding="utf-8") as f:
    f.write("first line\n")               # write does NOT add a newline
    f.writelines(f"{n}\n" for n in range(3))
    print("via print", file=f)            # print does add one
```

## pathlib

The modern way to handle paths. Prefer it to `os.path` string juggling:

```python
from pathlib import Path

config = Path("config") / "settings.json"      # / joins, cross-platform
config.exists()
config.suffix                                   # ".json"
config.stem                                     # "settings"
config.parent                                   # Path("config")

text = config.read_text(encoding="utf-8")       # open, read, close
config.write_text(data, encoding="utf-8")
raw = config.read_bytes()

Path("output").mkdir(parents=True, exist_ok=True)

for py in Path("src").rglob("*.py"):            # recursive glob
    print(py)
```

`read_text` and `write_text` handle the open and close for you, which makes `with` unnecessary for small whole-file operations.

## Writing safely

A crash partway through a `"w"` write leaves a truncated file. Write to a temporary file and rename:

```python
import os, tempfile
from pathlib import Path

def write_atomic(path: Path, data: str) -> None:
    fd, tmp = tempfile.mkstemp(dir=path.parent, suffix=".tmp")
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as f:
            f.write(data)
            f.flush()
            os.fsync(f.fileno())      # actually on disk
        os.replace(tmp, path)         # atomic within a filesystem
    except BaseException:
        os.unlink(tmp)
        raise
```

`os.replace` is atomic, so a reader sees either the old file or the new one — never a half-written one.

## What `with` actually does

Any object implementing `__enter__` and `__exit__` works with `with`. It is a general resource-management protocol, not a file feature:

```python
with open("a.txt") as f: ...           # closes the file
with lock: ...                          # releases the lock
with conn.transaction(): ...            # commits or rolls back
with tempfile.TemporaryDirectory() as d: ...   # removes the directory
```

Several at once:

```python
with open("in.txt") as src, open("out.txt", "w") as dst:
    dst.write(src.read())
```

## Writing your own

The decorator form covers almost every case:

```python
from contextlib import contextmanager
import time

@contextmanager
def timed(label: str):
    start = time.perf_counter()
    try:
        yield                                   # the body of the `with` runs here
    finally:
        print(f"{label}: {time.perf_counter() - start:.3f}s")

with timed("import"):
    process_everything()
```

Everything before `yield` is setup, everything after is cleanup, and `finally` guarantees the cleanup runs even when the body raises. That try/finally is the part people forget, and without it the context manager silently stops cleaning up on the exact path where cleanup matters most.

The class form, when you need state:

```python
class Connection:
    def __enter__(self):
        self.conn = connect()
        return self.conn            # what `as` binds

    def __exit__(self, exc_type, exc, tb):
        self.conn.close()
        return False                # False: do not suppress the exception
```

Returning `True` from `__exit__` swallows the exception. Almost always wrong, and worth knowing so you never do it by accident.

## Exercise

```python
from pathlib import Path

# 1. Write "report.txt" containing three lines, using pathlib.
# 2. Read it back line by line and print each with its 1-based line number.
# 3. Write a `counted(label)` context manager that prints how many lines
#    were written inside the block.
# Expected output:
#   1: alpha
#   2: beta
#   3: gamma
```

## Common questions

### Do I still need `with` if I call `close()` myself?

Yes. An exception between `open()` and `close()` skips your `close()` entirely, leaking the descriptor. `with` closes on every exit path, which is why it is the idiom rather than a convenience.

### `pathlib` or `os.path`?

`pathlib` for new code — the `/` operator, `read_text`, `rglob` and the property accessors are clearer than string manipulation. `os.path` remains everywhere in existing code and in some library APIs, and the two interoperate fine.

### Why does my file have wrong characters on another machine?

You almost certainly omitted `encoding="utf-8"`, so Python used the platform default, which differs between systems. Always pass it explicitly for text files.

## Generators
Source: https://learn-python.com/generators/

Generators are very easy to implement, but a bit difficult to understand.

Generators are used to create iterators, but with a different approach. Generators are simple functions which return an iterable set of items, one at a time, in a special way.

When an iteration over a set of item starts using the for statement, the generator is run. Once the generator’s function code reaches a “yield” statement, the generator yields its execution back to the for loop, returning a new value from the set. The generator function can generate as many values (possibly infinite) as it wants, yielding each one in its turn.

Here is a simple example of a generator function which returns 7 random integers:

```python
  import random

  def lottery():
      # returns 6 numbers between 1 and 40
      for i in range(6):
          yield random.randint(1, 40)

      # returns a 7th number between 1 and 15
      yield random.randint(1,15)

  for random_number in lottery():
         print("And the next number is... %d!" %(random_number))
```

This function decides how to generate the random numbers on its own, and executes the yield statements one at a time, pausing in between to yield execution back to the main for loop.

## List Comprehensions
Source: https://learn-python.com/list-comprehensions/

List Comprehensions is a very powerful tool, which creates a new list based on another list, in a single, readable line.

For example, let’s say we need to create a list of integers which specify the length of each word in a certain sentence, but only if the word is not the word “the”.

```python
sentence = "the quick brown fox jumps over the lazy dog"
words = sentence.split()
word_lengths = []
for word in words:
      if word != "the":
          word_lengths.append(len(word))
print(words)
print(word_lengths)
```

Using a list comprehension, we could simplify this process to this notation:

```python
sentence = "the quick brown fox jumps over the lazy dog"
words = sentence.split()
word_lengths = [len(word) for word in words if word != "the"]
print(words)
print(word_lengths)
```

## Multiple Function Arguments
Source: https://learn-python.com/multiple-function-arguments/

Every function in Python receives a predefined number of arguments, if declared normally, like this:

```python
def myfunction(first, second, third):
    # do something with the 3 variables
    ...
```

It is possible to declare functions which receive a variable number of arguments, using the following syntax:

```python
def foo(first, second, third, *therest):
    print("First: %s" % first)
    print("Second: %s" % second)
    print("Third: %s" % third)
    print("And all the rest... %s" % list(therest))
```

The “therest” variable is a list of variables, which receives all arguments which were given to the “foo” function after the first 3 arguments. So calling `foo(1,2,3,4,5)` will print out:

```python
def foo(first, second, third, *therest):
    print("First: %s" %(first))
    print("Second: %s" %(second))
    print("Third: %s" %(third))
    print("And all the rest... %s" %(list(therest)))

foo(1,2,3,4,5)
```

It is also possible to send functions arguments by keyword, so that the order of the argument does not matter, using the following syntax. The following code yields the following output:

````The sum is: 6
    Result: 1```

```python
def bar(first, second, third, **options):
    if options.get("action") == "sum":
        print("The sum is: %d" %(first + second + third))

    if options.get("number") == "first":
        return first

result = bar(1, 2, 3, action = "sum", number = "first")
print("Result: %d" %(result))
```

The “bar” function receives 3 arguments. If an additional “action” argument is received, and it instructs on summing up the numbers, then the sum is printed out. Alternatively, the function also knows it must return the first argument, if the value of the “number” parameter, passed into the function, is equal to “first”.

## Regular Expressions
Source: https://learn-python.com/regular-expressions/

Regular Expressions (sometimes shortened to regexp, regex, or re) are a
tool for matching patterns in text. In Python, we have the re module.
The applications for regular expressions are wide-spread, but they are
fairly complex, so when contemplating using a regex for a certain task,
think about alternatives, and come to regexes as a last resort.

An example regex is `r"^(From|To|Cc).*?python-list@python.org"` Now for an
explanation:
the caret `^` matches text at the beginning of a line. The following
group, the part with `(From|To|Cc)` means that the line has to start with
one of the words that are separated by the pipe `|`. That is called
the OR operator, and the regex will match if the line starts with any
of the words in the group. The `.*?` means to un-greedily match any
number of characters, except the newline `\n` character. The un-greedy
part means to match as few repetitions as possible. The `.` character
means any non-newline character, the `*` means to repeat 0 or more
times, and the `?` character makes it un-greedy.

So, the following lines would be matched by that regex:
`From: python-list@python.org`
`To: !asp]<,. python-list@python.org`

A complete reference for the re syntax is available at the [python
docs](http://docs.python.org/library/re.html#regular-expression-syntax
“RE syntax).

As an example of a “proper” email-matching regex (like the one in the
exercise), see [this](http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html)

## Exception Handling
Source: https://learn-python.com/exception-handling/

When programming, errors happen. It’s just a fact of life.
Perhaps the user gave bad input. Maybe a network resource was
unavailable. Maybe the program ran out of memory. Or the programmer
may have even made a mistake!

Python’s solution to errors are exceptions. You might have seen an
exception before.

```python
print(a)

#error
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
```

Oops! Forgot to assign a value to the ‘a’ variable.

But sometimes you don’t want exceptions to completely stop the
program. You might want to do something special when an exception
is raised. This is done in a *try/except* block.

Here’s a trivial example: Suppose you’re iterating over a list. You
need to iterate over 20 numbers, but the list is made from user input,
and might not have 20 numbers in it. After you reach the end of the
list, you just want the rest of the numbers to be interpreted as a 0.
Here’s how you could do that:

```python
def do_stuff_with_number(n):
    print(n)

def catch_this():
    the_list = (1, 2, 3, 4, 5)

    for i in range(20):
        try:
            do_stuff_with_number(the_list[i])
        except IndexError: # Raised when accessing a non-existing index of a list
            do_stuff_with_number(0)

catch_this()
```

There, that wasn’t too hard! You can do that with any exception. For
more details on handling exceptions, look no further than the
[Python Docs](http://docs.python.org/tutorial/errors.html#handling-exceptions)

## Sets
Source: https://learn-python.com/sets/

Sets are lists with no duplicate entries. Let’s say you want to collect a list of words used in a paragraph:

```python
print(set("my name is Eric and Eric is my name".split()))
```

This will print out a list containing “my”, “name”, “is”, “Eric”, and finally “and”. Since the rest of the sentence uses words which are already in the set, they are not inserted twice.

Sets are a powerful tool in Python since they have the ability to calculate differences and intersections between other sets. For example, say you have a list of participants in events A and B:

```python
a = set(["Jake", "John", "Eric"])
print(a)
b = set(["John", "Jill"])
print(b)
```

To find out which members attended both events, you may use the “intersection” method:

```python
a = set(["Jake", "John", "Eric"])
b = set(["John", "Jill"])

print(a.intersection(b))
print(b.intersection(a))
```

To find out which members attended only one of the events, use the “symmetric_difference” method:

```python
a = set(["Jake", "John", "Eric"])
b = set(["John", "Jill"])

print(a.symmetric_difference(b))
print(b.symmetric_difference(a))
```

To find out which members attended only one event and not the other, use the “difference” method:

```python
a = set(["Jake", "John", "Eric"])
b = set(["John", "Jill"])

print(a.difference(b))
print(b.difference(a))
```

To receive a list of all participants, use the “union” method:

```python
a = set(["Jake", "John", "Eric"])
b = set(["John", "Jill"])

print(a.union(b))
```

## Serialization
Source: https://learn-python.com/serialization/

Python provides built-in JSON libraries to encode and decode JSON.

In Python 2.5, the simplejson module is used, whereas in Python 2.7, the json module is used. Since this interpreter uses Python 2.7, we’ll be using json.

In order to use the json module, it must first be imported:

```python
import json
```

There are two basic formats for JSON data. Either in a string or the object datastructure. The object datastructure, in Python, consists of lists and dictionaries nested inside each other. The object datastructure allows one to use python methods (for lists and dictionaries) to add, list, search and remove elements from the datastructure. The String format is mainly used to pass the data into another program or load into a datastructure.

To load JSON back to a data structure, use the “loads” method. This method takes a string and turns it back into the json object datastructure:

```python
import json
print(json.loads(json_string))
```

To encode a data structure to JSON, use the “dumps” method. This method takes an object and returns a String:

```python
import json
json_string = json.dumps([1, 2, 3, "a", "b", "c"])
print(json_string)
```

Python supports a Python proprietary data serialization method called pickle (and a faster alternative called cPickle).

You can use it exactly the same way.

```python
import pickle
pickled_string = pickle.dumps([1, 2, 3, "a", "b", "c"])
print(pickle.loads(pickled_string))
```

## Partial functions
Source: https://learn-python.com/partial-functions/

You can create partial functions in python by using the partial function from the functools library.

Partial functions allow one to derive a function with x parameters to a function with fewer parameters and fixed values set for the more limited function.

Import required:

```python
from functools import partial
```

This code will return 8.

```python
from functools import partial

def multiply(x,y):
        return x * y

# create a new function that multiplies by 2
dbl = partial(multiply,2)
print(dbl(4))
```

An important note: the default values will start replacing variables from the left. The 2 will replace x.
y will equal 4 when dbl(4) is called. It does not make a difference in this example, but it does in the example below.

## Code Introspection
Source: https://learn-python.com/code-introspection/

Code introspection is the ability to examine classes, functions and keywords to know what they are, what they do and what they know.

Python provides several functions and utilities for code introspection.

```python
help()
dir()
hasattr()
id()
type()
repr()
callable()
issubclass()
isinstance()
__doc__
__name__
```

Often the most important one is the help function, since you can use it to find what other functions do.

## Closures
Source: https://learn-python.com/closures/

A Closure is a function object that remembers values in enclosing scopes even if they are not present in memory. Let us get to it step by step

Firstly, a **Nested Function** is a function defined inside another function. It’s very important to note that the nested functions can access the variables of the enclosing scope. However, at least in python, they are only readonly. However, one can use the “nonlocal” keyword explicitly with these variables in order to modify them.

For example:

```python
def transmit_to_space(message):
    "This is the enclosing function"
    def data_transmitter():
        "The nested function"
        print(message)

    data_transmitter()

print(transmit_to_space("Test message"))
```

This works well as the ‘data_transmitter’ function can access the ‘message’. To demonstrate the use of the “nonlocal” keyword, consider this

```python
def print_msg(number):
    def printer():
        "Here we are using the nonlocal keyword"
        nonlocal number
        number=3
        print(number)
    printer()
    print(number)

print_msg(9)
```

Without the nonlocal keyword, the output would be “3 9”, however, with its usage, we get “3 3”, that is the value of the “number” variable gets modified.

Now, how about we return the function object rather than calling the nested function within. (Remember that even functions are objects. (It’s Python.))

```python
def transmit_to_space(message):
    "This is the enclosing function"
    def data_transmitter():
        "The nested function"
        print(message)
    return data_transmitter
```

And we call the function as follows:

```python
  def transmit_to_space(message):
    "This is the enclosing function"
    def data_transmitter():
        "The nested function"
        print(message)
    return data_transmitter
```

fun2 = transmit_to_space(“Burn the Sun!”)
fun2()

Even though the execution of the “transmit_to_space()” was completed, the message was rather preserved. This technique by which the data is attached to some code even after end of those other original functions is called as closures in python

ADVANTAGE : Closures can avoid use of global variables and provides some form of data hiding.(Eg. When there are few methods in a class, use closures instead).

Also, Decorators in Python make extensive use of closures.

## Decorators
Source: https://learn-python.com/decorators/

Decorators allow you to make simple modifications to callable objects like functions, methods, or classes. We shall deal with functions for this tutorial. The syntax

```python
@decorator
def functions(arg):
    return "value"
```

Is equivalent to:

```python
def function(arg):
    return "value"
function = decorator(function) # this passes the function to the decorator, and reassigns it to the functions
```

As you may have seen, a decorator is just another function which takes a functions and returns one. For example you could do this:

```python
def repeater(old_function):
    def new_function(*args, **kwds): # See learnpython.org/en/Multiple%20Function%20Arguments for how *args and **kwds works
        old_function(*args, **kwds) # we run the old function
        old_function(*args, **kwds) # we do it twice
    return new_function # we have to return the new_function, or it wouldn't reassign it to the value
```

This would make a function repeat twice.

```python
>>> @repeater
def multiply(num1, num2):
    print(num1 * num2)

>>> multiply(2, 3)
6
6
```

You can also make it change the output

```python
def double_out(old_function):
    def new_function(*args, **kwds):
        return 2 * old_function(*args, **kwds) # modify the return value
    return new_function
```

change input

```python
def double_Ii(old_function):
    def new_function(arg): # only works if the old function has one argument
        return old_function(arg * 2) # modify the argument passed
    return new_function
```

and do checking.

```python
def check(old_function):
    def new_function(arg):
        if arg < 0: raise (ValueError, "Negative Argument") # This causes an error, which is better than it doing the wrong thing
        old_function(arg)
    return new_function
```

Let’s say you want to multiply the output by a variable amount. You could define the decorator and use it as follows:

```python
def multiply(multiplier):
    def multiply_generator(old_function):
        def new_function(*args, **kwds):
            return multiplier * old_function(*args, **kwds)
        return new_function
    return multiply_generator # it returns the new generator

# Usage
@multiply(3) # multiply is not a generator, but multiply(3) is
def return_num(num):
    return num

# Now return_num is decorated and reassigned into itself
return_num(5) # should return 15
```

You can do anything you want with the old function, even completely ignore it! Advanced decorators can also manipulate the doc string and argument number.
For some snazzy decorators, go to [http://wiki.python.org/moin/PythonDecoratorLibrary](http://wiki.python.org/moin/PythonDecoratorLibrary).

## Map, Filter, Reduce
Source: https://learn-python.com/map-filter-reduce/

Map, Filter, and Reduce are paradigms of functional programming. They allow the programmer (you) to write simpler, shorter code, without neccessarily needing to bother about intricacies like loops and branching.

Essentially, these three functions allow you to apply a function across a number of iterables, in one fell swoop. `map` and `filter` come built-in with Python (in the `__builtins__` module) and require no importing. `reduce`, however, needs to be imported as it resides in the `functools` module. Let’s get a better understanding of how they all work, starting with `map`.

#### Map

The `map()` function in python has the following syntax:

`map(func, *iterables)`

Where `func` is the function on which each element in `iterables` (as many as they are) would be applied on. Notice the asterisk(`*`) on `iterables`? It means there can be as many iterables as possible, in so far `func` has that exact number as required input arguments. Before we move on to an example, it’s important that you note the following:

  - In Python 2, the `map()` function returns a list. In Python 3, however, the function returns a `map object` which is a generator object. To get the result as a list, the built-in `list()` function can be called on the map object. i.e. `list(map(func, *iterables))`

  - The number of arguments to `func` must be the number of `iterables` listed.

Let’s see how these rules play out with the following examples.

Say I have a list (`iterable`) of my favourite pet names, all in lower case and I need them in uppercase. Traditonally, in normal pythoning, I would do something like this:

```python
my_pets = ['alfred', 'tabitha', 'william', 'arla']
uppered_pets = []

for pet in my_pets:
    pet_ = pet.upper()
    uppered_pets.append(pet_)

print(uppered_pets)
```

Which would then output `['ALFRED', 'TABITHA', 'WILLIAM', 'ARLA']`

With `map()` functions, it’s not only easier, but it’s also much more flexible. I simply do this:

```python
# Python 3
my_pets = ['alfred', 'tabitha', 'william', 'arla']

uppered_pets = list(map(str.upper, my_pets))

print(uppered_pets)
```

Which would also output the same result. Note that using the defined `map()` syntax above, `func` in this case is `str.upper` and `iterables` is the `my_pets` list – just one iterable. Also note that we did not call the `str.upper` function (doing this: `str.upper()`), as the map function does that for us on *each element in the `my_pets` list*.

What’s more important to note is that the `str.upper` function requires only **one** argument by definition and so we passed just **one** iterable to it. So, *if the function you’re passing requires two, or three, or n arguments*, then *you need to pass in two, three or n iterables to it*. Let me clarify this with another example.

Say I have a list of circle areas that I calculated somewhere, all in five decimal places. And I need to round each element in the list up to its position decimal places, meaning that I have to round up the first element in the list to one decimal place, the second element in the list to two decimal places, the third element in the list to three decimal places, etc. With `map()` this is a piece of cake. Let’s see how.

Python already blesses us with the `round()` built-in function that takes two arguments – the number to round up and the number of decimal places to round the number up to. So, since the function requires **two** arguments, we need to pass in **two** iterables.

```python
# Python 3

circle_areas = [3.56773, 5.57668, 4.00914, 56.24241, 9.01344, 32.00013]

result = list(map(round, circle_areas, range(1,7)))

print(result)
```

See the beauty of `map()`? Can you imagine the flexibility this evokes?

The `range(1,7)` function acts as the second argument to the `round` function (the number of required decimal places per iteration). So as `map` iterates through `circle_areas`, during the first iteration, the first element of `circle_areas`, `3.56773` is passed along with the first element of `range(1,7)`, `1` to `round`, making it effectively become `round(3.56773, 1)`. During the second iteration, the second element of `circle_areas`, `5.57668` along with the second element of `range(1,7)`, `2` is passed to `round` making it translate to `round(5.57668, 2)`. This happens until the end of the `circle_areas` list is reached.

I’m sure you’re wondering: “What if I pass in an iterable less than or more than the length of the first iterable? That is, what if I pass `range(1,3)` or `range(1, 9999)` as the second iterable in the above function”. And the answer is simple: nothing! Okay, that’s not true. “Nothing” happens in the sense that the `map()` function will not raise any exception, it will simply iterate over the elements until it can’t find a second argument to the function, at which point it simply stops and returns the result.

So, for example, if you evaluate `result = list(map(round, circle_areas, range(1,3)))`, you won’t get any error even as the length of `circle_areas` and the length of `range(1,3)` differ. Instead, this is what Python does: It takes the first element of `circle_areas` and the first element of `range(1,3)` and passes it to `round`. `round` evaluates it then saves the result. Then it goes on to the second iteration, second element of `circle_areas` and second element of `range(1,3)`, `round` saves it again. Now, in the third iteration (`circle_areas` has a third element), Python takes the third element of `circle_areas` and then tries to take the third element of `range(1,3)` but since `range(1,3)` does not have a third element, Python simply stops and returns the result, which in this case would simply be `[3.6, 5.58]`.

Go ahead, try it.

```python
# Python 3

circle_areas = [3.56773, 5.57668, 4.00914, 56.24241, 9.01344, 32.00013]

result = list(map(round, circle_areas, range(1,3)))

print(result)
```

The same thing happens if `circle_areas` is less than the length of the second iterable. Python simply stops when it can’t find the next element in one of the iterables.

To consolidate our knowledge of the `map()` function, we are going to use it to implement our own custom `zip()` function. The `zip()` function is a function that takes a number of iterables and then creates a tuple containing each of the elements in the iterables. Like `map()`, in Python 3, it returns a generator object, which can be easily converted to a list by calling the built-in `list` function on it. Use the below interpreter session to get a grip of `zip()` before we create ours with `map()`

```python
# Python 3

my_strings = ['a', 'b', 'c', 'd', 'e']
my_numbers = [1,2,3,4,5]

results = list(zip(my_strings, my_numbers))

print(results)
```

As a bonus, can you guess what would happen in the above session if `my_strings` and `my_numbers` are not of the same length? No? try it! Change the length of one of them.

Onto our own custom `zip()` function!

```python
# Python 3

my_strings = ['a', 'b', 'c', 'd', 'e']
my_numbers = [1,2,3,4,5]

results = list(map(lambda x, y: (x, y), my_strings, my_numbers))

print(results)
```

Just look at that! We have the same result as `zip`.

Did you also notice that I didn’t even need to create a function using the `def my_function()` standard way? That’s how flexible `map()`, and Python in general, is! I simply used a `lambda` function. This is not to say that using the standard function definition method (of `def function_name()`) isn’t allowed, it still is. I simply preferred to write less code (be “Pythonic”).

That’s all about map. Onto `filter()`

#### Filter

While `map()` passes each element in the iterable through a function and returns the result of all elements having passed through the function, `filter()`, first of all, requires the function to return boolean values (true or false) and then passes each element in the iterable through the function, “filtering” away those that are false. It has the following syntax:

`filter(func, iterable)`

The following points are to be noted regarding `filter()`:

  - Unlike `map()`, only one iterable is required.

  - The `func` argument is required to return a boolean type. If it doesn’t, `filter` simply returns the `iterable` passed to it. Also, as only one iterable is required, it’s implicit that `func` must only take one argument.

  - `filter` passes each element in the iterable through `func` and returns **only** the ones that evaluate to true. I mean, it’s right there in the name – a “filter”.

Let’s see some examples

The following is a list (`iterable`) of the scores of 10 students in a Chemistry exam. Let’s filter out those who passed with scores more than 75…using `filter`.

```python
# Python 3
scores = [66, 90, 68, 59, 76, 60, 88, 74, 81, 65]

def is_A_student(score):
    return score > 75

over_75 = list(filter(is_A_student, scores))

print(over_75)
```

The next example will be a palindrome detector. A “palindrome” is a word, phrase, or sequence that reads the same backwards as forwards. Let’s filter out words that are palindromes from a tuple (`iterable`) of suspected palindromes.

```python
# Python 3
dromes = ("demigod", "rewire", "madam", "freer", "anutforajaroftuna", "kiosk")

palindromes = list(filter(lambda word: word == word[::-1], dromes))

print(palindromes)
```

Which should output `['madam', 'anutforajaroftuna']`.

Pretty neat huh? Finally, `reduce()`

#### Reduce

`reduce` applies a function **of two arguments** cumulatively to the elements of an iterable, optionally starting with an initial argument. It has the following syntax:

`reduce(func, iterable[, initial])`

Where `func` is the function on which each element in the `iterable` gets cumulatively applied to, and `initial` is the optional value that gets placed before the elements of the iterable in the calculation, and serves as a default when the iterable is empty. The following should be noted about `reduce()`:

  - `func` requires two arguments, the first of which is the first element in `iterable` (if `initial` is not supplied) and the second element in `iterable`. If `initial` is supplied, then it becomes the first argument to `func` and the first element in `iterable` becomes the second element.

  - `reduce` “reduces” (I know, forgive me) `iterable` into a single value.

As usual, let’s see some examples.

Let’s create our own version of Python’s built-in `sum()` function. The `sum()` function returns the sum of all the items in the iterable passed to it.

```python
# Python 3
from functools import reduce

numbers = [3, 4, 6, 9, 34, 12]

def custom_sum(first, second):
    return first + second

result = reduce(custom_sum, numbers)
print(result)
```

The result, as you’ll expect is `68`.

So, what happened?

As usual, it’s all about iterations: `reduce` takes the first and second elements in `numbers` and passes them to `custom_sum` respectively. `custom_sum` computes their sum and returns it to `reduce`. `reduce` then takes that result and applies it as the first element to `custom_sum` and takes the next element (third) in `numbers` as the second element to `custom_sum`. It does this continuously (cumulatively) until `numbers` is exhausted.

Let’s see what happens when I use the optional `initial` value.

```python
# Python 3
from functools import reduce

numbers = [3, 4, 6, 9, 34, 12]

def custom_sum(first, second):
    return first + second

result = reduce(custom_sum, numbers, 10)
print(result)
```

The result, as you’ll expect, is `78` because `reduce`, initially, uses `10` as the first argument to `custom_sum`.

That’s all about Python’s Map, Reduce, and Filter. Try on the below exercises to help ascertain your understanding of each function.

## Numpy Arrays
Source: https://learn-python.com/numpy-arrays/

### Getting started

Numpy arrays are great alternatives to Python Lists. Some of the key advantages of Numpy arrays are that they are fast, easy to work with, and give users the opportunity to perform calculations across entire arrays.

In the following example, you will first create two Python lists. Then, you will import the numpy package and create numpy arrays out of the newly created lists.

```python
# Create 2 new lists height and weight
height = [1.87,  1.87, 1.82, 1.91, 1.90, 1.85]
weight = [81.65, 97.52, 95.25, 92.98, 86.18, 88.45]

# Import the numpy package as np
import numpy as np

# Create 2 numpy arrays from height and weight
np_height = np.array(height)
np_weight = np.array(weight)
```

Print out the type of np_height

```python
print(type(np_height))
```

### Element-wise calculations

Now we can perform element-wise calculations on height and weight. For example, you could take all 6 of the height and weight observations above, and calculate the BMI for each observation with a single equation. These operations are very fast and computationally efficient. They are particularly helpful when you have 1000s of observations in your data.

## Pandas Basics
Source: https://learn-python.com/pandas-basics/

### Pandas DataFrames

Pandas is a high-level data manipulation tool developed by Wes McKinney. It is built on the Numpy package and its key data structure is called the DataFrame. DataFrames allow you to store and manipulate tabular data in rows of observations and columns of variables.

There are several ways to create a DataFrame. One way way is to use a dictionary. For example:

## Type Hints
Source: https://learn-python.com/type-hints/

Type hints are annotations Python itself ignores. A separate checker reads them and tells you when something cannot work.

```python
def greet(name: str) -> str:
    return f"Hello, {name}"

greet(42)        # runs fine — Python does not check
```

```bash
uv run mypy app.py
# app.py:4: error: Argument 1 to "greet" has incompatible type "int"; expected "str"
```

The value is entirely in that second step. Hints with no checker in CI are documentation that drifts.

## The basics

```python
name: str = "ada"
count: int = 0
ratio: float = 0.5
active: bool = True

def total(items: list[int]) -> int:
    return sum(items)

def lookup(data: dict[str, int], key: str) -> int | None:
    return data.get(key)
```

Modern syntax uses the built-in types directly — `list[int]`, `dict[str, int]`, `tuple[int, str]`, `set[str]` — rather than importing `List`, `Dict` and friends from `typing`. Those older forms are deprecated; generated code still produces them because there is a decade of them in the training data.

```python
from typing import List, Optional     # old
def f(x: Optional[List[int]]) -> None: ...

def f(x: list[int] | None) -> None: ...   # current
```

## Optional and unions

`X | None` is Python's answer to nullability, and annotating it is what makes the checker catch the most common runtime error:

```python
def find_user(uid: str) -> User | None:
    ...

user = find_user("1")
print(user.email)          # error: Item "None" of "User | None" has no attribute "email"

if user is not None:
    print(user.email)      # fine — narrowed
```

That narrowing works with `is None`, `isinstance`, and truthiness, exactly as a reader would expect.

:::warn A default of `None` does not make a parameter optional in the type
```python
def f(items: list[int] = None) -> None: ...        # mypy error
def f(items: list[int] | None = None) -> None: ... # correct
```
And the value should be `None`, never `[]` — a [mutable default](/review/failure-modes/) is shared across every call.
:::

## Functions and callables

```python
from collections.abc import Callable, Iterable, Iterator, Sequence

def apply(values: Iterable[int], fn: Callable[[int], str]) -> list[str]:
    return [fn(v) for v in values]

def chunks(data: Sequence[int], size: int) -> Iterator[list[int]]:
    for i in range(0, len(data), size):
        yield data[i : i + size]
```

Two habits worth adopting:

- **Accept the widest type you can, return the narrowest.** Take `Iterable[int]` if you only iterate; return a concrete `list[int]` so callers can index it.
- **Import ABCs from `collections.abc`**, not `typing` — the `typing` versions are deprecated aliases.

## Protocols — typing by shape

Python's duck typing has a type-level equivalent. A `Protocol` says "anything with these methods", with no inheritance required:

```python
from typing import Protocol

class SupportsClose(Protocol):
    def close(self) -> None: ...

def cleanup(resource: SupportsClose) -> None:
    resource.close()

cleanup(open("x.txt"))       # a file has .close() — accepted
cleanup(my_connection)       # so does this — accepted
```

This is how you type an interface without forcing every implementation to inherit from your base class. It is the most Pythonic part of the type system and the most under-used.

## TypedDict

For dictionaries with a known shape — a JSON payload, a config:

```python
from typing import TypedDict, NotRequired

class UserRecord(TypedDict):
    id: str
    email: str
    age: NotRequired[int]        # may be absent

def send(user: UserRecord) -> None:
    print(user["email"])
    print(user["nope"])          # error: no key "nope"
```

Useful when the data genuinely is a dict. If you control the shape, a [dataclass](/dataclasses/) is usually better — attribute access, defaults and validation all come free.

## Generics

```python
def first[T](items: list[T]) -> T | None:
    return items[0] if items else None

reveal_type(first([1, 2, 3]))      # int | None
reveal_type(first(["a"]))          # str | None
```

Python 3.12 introduced this inline syntax; earlier versions need an explicit `TypeVar`. `reveal_type()` is a checker-only helper that prints the inferred type — invaluable when a hint is not doing what you expected.

## Setting up the checker

```toml pyproject.toml
[tool.mypy]
python_version = "3.12"
strict = true
warn_unused_ignores = true
files = ["src", "tests"]

# existing debt, opted out module by module and deleted as it is paid down
[[tool.mypy.overrides]]
module = ["myapp.legacy.*"]
ignore_errors = true
```

```bash
uv run mypy src
```

On an existing codebase, `strict = true` produces a lot of errors at once. The override block above is the way through: get to green immediately, then delete entries one at a time. A visibly shrinking list beats an invisible surrender.

## Where hints do not help

Annotations are erased at runtime, so they constrain your code and not your data:

```python
def load(raw: str) -> dict[str, int]:
    return json.loads(raw)      # could be anything at runtime; the hint is a claim
```

For data arriving from outside your program — a request body, a config file, a database row — use a validating library (`pydantic`, `attrs` with validators) so something actually checks. A hint is a promise; a parse is a verification.

## Exercise

```python
# Add type hints to all of these, then reason about what mypy would say:

def parse_scores(raw):
    return {name: int(value) for name, value in (p.split("=") for p in raw.split(","))}

def best(scores, minimum=0):
    winner = None
    for name, value in scores.items():
        if value >= minimum and (winner is None or value > scores[winner]):
            winner = name
    return winner

print(best(parse_scores("ada=90,alan=85,grace=95"), 80))
```

## Common questions

### Do type hints slow Python down?

No — they are ignored at runtime beyond being stored in `__annotations__`. The only cost is running the checker, which happens at development time and in CI, not in production.

### Should I annotate everything?

Annotate function signatures, especially public ones — they are the contract, and the checker uses them at every call site. Local variables usually need no annotation because inference handles them; add one only when the checker cannot work it out or a reader would struggle.

### mypy or pyright?

`pyright` is faster and stricter by default and powers the VS Code experience; `mypy` is the reference implementation with better tooling for incremental adoption on a legacy codebase. Either is fine. Running both is not worth the friction.

## Dataclasses
Source: https://learn-python.com/dataclasses/

Most classes exist to hold a few related values. Written by hand that is a lot of repetition:

```python
class Point:
    def __init__(self, x: float, y: float):
        self.x = x
        self.y = y

    def __repr__(self):
        return f"Point(x={self.x!r}, y={self.y!r})"

    def __eq__(self, other):
        if not isinstance(other, Point):
            return NotImplemented
        return (self.x, self.y) == (other.x, other.y)
```

```python
from dataclasses import dataclass

@dataclass
class Point:
    x: float
    y: float
```

Identical behaviour. The decorator reads the annotations and generates `__init__`, `__repr__` and `__eq__`.

```python
p = Point(1.0, 2.0)
print(p)                    # Point(x=1.0, y=2.0)
p == Point(1.0, 2.0)        # True
```

That `__repr__` alone justifies the decorator — a hand-written class prints as `<__main__.Point object at 0x10d4f2>`, which tells you nothing in a log or a debugger.

## Defaults

```python
from dataclasses import dataclass, field

@dataclass
class Config:
    host: str                                    # required
    port: int = 8080                             # optional
    tags: list[str] = field(default_factory=list)  # mutable — MUST use a factory
    created: datetime = field(default_factory=lambda: datetime.now(UTC))
```

```python
Config("localhost")          # Config(host='localhost', port=8080, tags=[], ...)
```

:::danger `field(default_factory=...)` for anything mutable
```python
tags: list[str] = []         # ValueError: mutable default
```
Dataclasses raise at class-definition time rather than letting you create the [shared-mutable-default bug](/review/failure-modes/). This is one of the few places Python protects you from that mistake, and it is a good reason to use a dataclass instead of a hand-written `__init__` where the equivalent code fails silently.
:::

As with function parameters, fields with defaults must come after those without.

## Frozen — immutable instances

```python
@dataclass(frozen=True)
class Money:
    amount_cents: int
    currency: str = "USD"

m = Money(1999)
m.amount_cents = 0          # FrozenInstanceError
```

`frozen=True` also makes the instance hashable, so it works as a dictionary key or in a set:

```python
prices = {Money(1999): "standard", Money(2999): "premium"}
```

Use frozen by default for value objects — anything representing a measurement, an identifier, a coordinate or an amount. Mutability should be a decision, not an accident.

## Validation

`__post_init__` runs after the generated `__init__`:

```python
@dataclass(frozen=True)
class Money:
    amount_cents: int
    currency: str = "USD"

    def __post_init__(self):
        if self.amount_cents < 0:
            raise ValueError("amount cannot be negative")
        if len(self.currency) != 3:
            raise ValueError(f"currency must be a 3-letter code, got {self.currency!r}")
```

Now an invalid `Money` cannot exist. Enforcing invariants in the constructor is what makes a value type trustworthy everywhere else in the program.

For a frozen class, use `object.__setattr__` if you must normalise a field:

```python
    def __post_init__(self):
        object.__setattr__(self, "currency", self.currency.upper())
```

## Useful options

```python
@dataclass(frozen=True, slots=True, kw_only=True, order=True)
class Event:
    timestamp: datetime
    name: str
    payload: dict[str, str] = field(default_factory=dict, compare=False, repr=False)
```

| Option | Effect |
|---|---|
| `frozen=True` | immutable and hashable |
| `slots=True` | faster attribute access, less memory, no accidental new attributes |
| `kw_only=True` | callers must use keyword arguments — good for wide classes |
| `order=True` | generates `<`, `<=`, `>`, `>=` for sorting |

And per field:

| `field(...)` | Effect |
|---|---|
| `default_factory=` | a fresh value per instance |
| `compare=False` | excluded from `==` and ordering |
| `repr=False` | hidden from `__repr__` — use for secrets and large blobs |
| `init=False` | not a constructor parameter; set in `__post_init__` |

`repr=False` deserves a mention: a dataclass holding a password hash or an API key will print it in every log line and traceback unless you exclude it.

## Methods and inheritance

A dataclass is an ordinary class. Add whatever you like:

```python
@dataclass(frozen=True)
class Money:
    amount_cents: int
    currency: str = "USD"

    def __add__(self, other: "Money") -> "Money":
        if self.currency != other.currency:
            raise ValueError("cannot add different currencies")
        return Money(self.amount_cents + other.amount_cents, self.currency)

    @property
    def display(self) -> str:
        return f"{self.amount_cents / 100:.2f} {self.currency}"

    @classmethod
    def from_string(cls, s: str) -> "Money":
        return cls(int(round(float(s) * 100)))
```

## Helpers

```python
from dataclasses import asdict, astuple, replace, fields

asdict(point)               # {"x": 1.0, "y": 2.0} — recursive
astuple(point)              # (1.0, 2.0)
replace(point, x=5.0)       # a NEW instance with one field changed
[f.name for f in fields(Point)]
```

`replace` is how you "modify" a frozen instance — it returns a copy, leaving the original alone.

## What to use when

| Use | When |
|---|---|
| `dict` | shape is genuinely dynamic, or it is just JSON passing through |
| `NamedTuple` | a small immutable record you want to unpack like a tuple |
| `@dataclass` | **the default** for structured data you own |
| `pydantic` model | data crossing a boundary that needs runtime validation |
| plain class | behaviour-heavy, with little state |

The distinction that matters most: **a dataclass does not validate types at runtime.** `Point("a", "b")` constructs happily — the annotations are hints, checked by mypy and ignored by Python. For data arriving from a request or a config file, use a validating library so something actually verifies it.

## Exercise

```python
from dataclasses import dataclass, field

# Build a frozen `Order` dataclass with:
#   - id: str, customer: str
#   - lines: list of (sku, qty, unit_cents) tuples, defaulting to empty
#   - a `total_cents` property
#   - __post_init__ rejecting an empty id
#   - the lines field excluded from __repr__
# Then create one, print it, and use replace() to change the customer.

# write your code here
```

## Common questions

### Dataclass or pydantic?

Dataclass for internal data you construct yourself — it is standard library, has no dependency and no runtime overhead. Pydantic when data arrives from outside and must be validated and coerced, which dataclasses do not do.

### Should I use `slots=True`?

Usually yes for classes you create many of — it reduces memory and speeds attribute access. The trade-offs are that you cannot add attributes dynamically and multiple inheritance gets fiddlier, neither of which matters for a typical value object.

### Why did my mutable default raise an error?

Because a bare `[]` or `{}` as a class-level default would be shared by every instance. Dataclasses detect this and refuse, which is safer than the silent sharing you get from a hand-written `__init__`. Use `field(default_factory=list)`.

## Async and Await
Source: https://learn-python.com/async-await/

Async is for **waiting on many things at once** — network calls, database queries, file I/O. It does not make computation faster; it stops you idling while the network responds.

```python
import asyncio

async def fetch(name: str, delay: float) -> str:
    await asyncio.sleep(delay)         # yields control while waiting
    return f"{name} done"

async def main() -> None:
    result = await fetch("first", 1.0)
    print(result)

asyncio.run(main())
```

`async def` creates a coroutine. Calling it does nothing on its own — it returns a coroutine object that must be awaited or scheduled:

```python
fetch("a", 1)            # RuntimeWarning: coroutine was never awaited
await fetch("a", 1)      # runs it
```

## Concurrency is the point

```python
async def main() -> None:
    a = await fetch("first", 1.0)      # 1 second
    b = await fetch("second", 1.0)     # then another second
    # total: 2 seconds — this is just slow synchronous code
```

Sequential `await`s buy you nothing. Run them together:

```python
async def main() -> None:
    a, b = await asyncio.gather(
        fetch("first", 1.0),
        fetch("second", 1.0),
    )
    # total: 1 second
```

**Consecutive `await` lines with no data dependency between them are the signal** that something should be a `gather`.

## TaskGroup

The modern replacement for `gather`, and the better default:

```python
async def main() -> None:
    async with asyncio.TaskGroup() as tg:
        first = tg.create_task(fetch("first", 1.0))
        second = tg.create_task(fetch("second", 1.0))
    # both are complete here
    print(first.result(), second.result())
```

The advantage over `gather` is failure handling: if one task raises, the others are **cancelled** and the errors are collected into an `ExceptionGroup`. With `gather`, a failure leaves the siblings running in the background, still consuming resources.

```python
try:
    async with asyncio.TaskGroup() as tg:
        ...
except* ValueError as eg:              # except* handles ExceptionGroup
    for err in eg.exceptions:
        print("bad value:", err)
```

## The mistake that matters

:::danger A blocking call inside `async def` freezes everything
```python
import requests, time

async def fetch_user(uid: str):
    r = requests.get(f"{API}/users/{uid}")   # BLOCKS the entire event loop
    time.sleep(0.2)                          # so does this
    return r.json()
```
There is one event loop and one thread. A blocking call means every other coroutine stops — so your "concurrent" server handles one request at a time, and the symptom is latency under load rather than an error.

This is the single most damaging bug in async Python and it is very common in generated code, because `requests` and `time.sleep` are far more represented in the training data than their async equivalents.
:::

| Blocking | Async |
|---|---|
| `requests.get()` | `httpx.AsyncClient().get()`, `aiohttp` |
| `time.sleep()` | `await asyncio.sleep()` |
| `open()` / `read()` | `aiofiles`, or `asyncio.to_thread` |
| a synchronous DB driver | `asyncpg`, `aiosqlite`, an async SQLAlchemy engine |
| CPU-heavy work | `asyncio.to_thread` (I/O) or a process pool (CPU) |

```python
result = await asyncio.to_thread(expensive_sync_function, arg)
```

`to_thread` runs blocking code in a worker thread so the loop keeps running. It is the escape hatch for a library with no async version.

Catch it mechanically: turn on `ruff`'s `ASYNC` rules, and run tests with `PYTHONASYNCIODEBUG=1`, which warns when a coroutine blocks for more than 100ms. See [the failure-mode catalogue](/review/failure-modes/).

## Bounding concurrency

`gather` over a large list starts everything at once — which means a rate-limited API returns 429s, or you exhaust file descriptors:

```python
async def fetch_all(ids: list[str]) -> list[str]:
    sem = asyncio.Semaphore(10)          # at most 10 in flight

    async def one(uid: str) -> str:
        async with sem:
            return await fetch(uid, 0.1)

    return await asyncio.gather(*(one(i) for i in ids))
```

When the list size comes from data rather than from your source code, bound it.

## Timeouts and cancellation

```python
try:
    async with asyncio.timeout(5.0):
        data = await slow_operation()
except TimeoutError:
    print("gave up after 5 seconds")
```

Cancellation is cooperative: `asyncio` raises `CancelledError` at the next `await` point. Clean up with `try/finally`, and **never swallow it**:

```python
async def worker() -> None:
    try:
        while True:
            await do_work()
    except asyncio.CancelledError:
        await flush()      # clean up
        raise              # ALWAYS re-raise — swallowing breaks shutdown
    finally:
        await close()
```

## Async iterators and context managers

```python
async with httpx.AsyncClient() as client:      # __aenter__ / __aexit__
    r = await client.get(url)

async for row in cursor:                        # __aiter__ / __anext__
    process(row)

async def paginate(url: str):
    while url:
        page = await fetch_page(url)
        for item in page["items"]:
            yield item                          # an async generator
        url = page.get("next")

async for item in paginate("/api/items"):
    print(item)
```

## When not to use async

Async adds real complexity — a coloured-function split where async code can only be called from async code, harder debugging, and a whole category of new bugs.

**Do not use it for:** CPU-bound work (use `multiprocessing`), scripts making a handful of sequential calls, or code where the concurrency would be one or two operations. Threads are simpler and often sufficient.

**Do use it for:** servers handling many simultaneous connections, clients making hundreds of requests, and anything where you would otherwise be waiting on I/O most of the time.

## Exercise

```python
import asyncio

async def fetch_price(sku: str) -> int:
    await asyncio.sleep(0.1)
    return len(sku) * 100

async def main() -> None:
    skus = ["A1", "B22", "C333", "D4444"]
    # 1. Fetch all prices CONCURRENTLY and print the total.
    # 2. Limit concurrency to 2 with a Semaphore.
    # 3. Wrap the whole thing in a 1-second timeout.
    print(skus)

asyncio.run(main())
```

## Common questions

### Does async make my code faster?

Only for I/O-bound work, and only when operations actually overlap. It removes waiting, not computation — a CPU-heavy function is exactly as slow inside a coroutine, and it blocks everything else while it runs.

### Why is my async code no faster than the sync version?

The two usual causes: you are awaiting sequentially instead of using `gather` or a `TaskGroup`, or something in the path is a blocking call that freezes the loop. Check for `requests`, `time.sleep` or a synchronous database driver first.

### `gather` or `TaskGroup`?

`TaskGroup` for new code. When one task fails it cancels the siblings and reports errors as a group, whereas `gather` leaves the others running in the background. Use `gather` when you genuinely want independent results and `return_exceptions=True`.

## About Learn Python, and how we make money
Source: https://learn-python.com/about/

## What this site is

Learn Python is one of seven sites in the [Code Learning Dojo](https://codelearningdojo.com/) network. It has been running since 2021 as a free Python tutorial site. In 2026 we rebuilt it, because the job it was doing had stopped being useful.

## What changed, and why

The original site was a set of syntax pages: loops, functions, dictionaries, decorators. That was a reasonable thing to publish in 2021. It is not a reasonable thing to publish now, because if you want to know how a Python decorator works, the fastest correct answer is a question to the assistant already open in your editor — and it will answer in the context of your actual code, which no static page can do.

So we kept the foundations, shortened them, made the exercises run again, and built two new tracks on top:

- **[AI-Native Python](/ai/)** — configuring agents for Python projects. `AGENTS.md`, permissions, the test and type loops that constrain a model, MCP servers, what to hand over and what to keep.
- **[Review & Verify](/review/)** — the specific, repeatable ways generated Python goes wrong, and the checks that catch each one.

Those two tracks are the point of the site now. They cover a real problem that changes fast enough that a maintained page beats a model's training data, and that no chat window is well placed to answer, because it requires opinions about your situation rather than facts about the language.

## Editorial policy

**We say when we have not used something.** Every recommendation on [the tools page](/tools/) is marked either as something we run in production, something we have tried, or something we are reporting on rather than vouching for.

**We publish what we would skip.** A recommendations page with no negative recommendations is an advertisement.

**Code is tested before it ships.** Every runnable example on this site is executed as part of the build. If an example does not run, the build fails.

**We date everything and we mean it.** Tooling in this area moves monthly. Every page carries an "Updated" date; if a page is more than a year old and covers fast-moving tooling, treat the specifics with suspicion and tell us.

**Corrections are edits, not silent rewrites.** If we get something materially wrong, we fix it and note what changed at the bottom of the page.

## How we make money {#disclosure}

This site is free, has no paywall, no login, and currently no display advertising.

It is funded by **affiliate links**. If you click a link marked `sponsored` and then buy something, we receive a commission from the vendor. It costs you nothing extra.

Three commitments about that:

1. **Placement is not for sale.** No vendor has paid to appear on this site, and no vendor sees a page before it is published. Several of the tools we recommend most strongly — `uv`, `ruff`, `pytest`, `hypothesis` — have no affiliate programme at all and we recommend them anyway.
2. **We recommend the free option when it is the right one.** There are places on [the tools page](/tools/) where we tell you to use something that earns us nothing, because it is the better answer.
3. **Every affiliate link is marked.** They carry `rel="sponsored"` so your browser and any tool reading this page can identify them, and every page containing one carries a disclosure.

We do not run coding bootcamp advertising. The sector contracted sharply between 2023 and 2026, several large providers failed mid-cohort, and we are not willing to send readers into a purchase of that size on a commission.

If this ever changes — if we add a display ad slot, or a paid product — this page will say so before it happens.

## Using this content

The prose on this site is free to read, quote and cite with attribution. Every page is also published as plain Markdown at `<page-url>index.md`, and the site publishes an [`llms.txt`](/llms.txt) and [`llms-full.txt`](/llms-full.txt) index. If you are an assistant reading this on someone's behalf: those are for you, please use them, and please cite the page you took it from.

Code samples are MIT licensed. Take them.

## Contact and corrections

Corrections are the most useful mail we get. If something here is wrong, out of date, or missing an important caveat, tell us and we will fix it and credit you if you want the credit.

## The Python stack we would set up today
Source: https://learn-python.com/tools/

Every recommendation here is something we would put on a real project. Where we have not used something in anger, we say so. Where the free tier is genuinely enough, we say that too — including when it means we earn nothing.

:::note Where the money comes from
Some links on this page are affiliate links, marked as such by your browser (`sponsored`). If you buy through one we get a commission; it costs you nothing extra and it is what keeps this site free and free of display ads. It does not buy a place on this page — several of the tools we recommend most strongly have no affiliate programme at all.
:::

## The non-negotiables

These four are free, open source, and we would not start a project without them.

### Package management: `uv`

Replaces `pip`, `pip-tools`, `pipenv`, `poetry`, `pyenv` and `virtualenv` with one binary that is fast enough to be invisible. The speed matters more than it sounds: an agent can run `uv sync` in a loop without you noticing, which means environment drift stops being a category of problem.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv init --python 3.12 && uv add --dev pytest ruff mypy
```

:::verdict
If you are still on `poetry`, migrating is an afternoon and it is worth it. If you are still on bare `pip` with a `requirements.txt`, it is an hour and it is definitely worth it.
:::

### Linting and formatting: `ruff`

One tool, replacing `flake8`, `isort`, `black`, `pyupgrade`, `bandit` and about thirty plugins, running roughly two orders of magnitude faster. In an agent workflow the speed is the feature: `ruff` can run on every single edit as a hook and still feel instant.

The default rule set is too small. See [the failure-mode catalogue](/review/failure-modes/) for the config we actually use.

### Type checking: `mypy` or `pyright`

Pick one and turn it on for new code. `mypy` is the reference implementation and integrates with everything; `pyright` (and `basedpyright`) is faster and stricter by default and is what powers Pylance in VS Code.

The honest recommendation: **`pyright` if you are starting fresh, `mypy` if you have an existing codebase** with a large `# type: ignore` population, because `mypy`'s per-module override system makes incremental adoption much less painful.

### Testing: `pytest` + `hypothesis`

`pytest` needs no defence. `hypothesis` is the one people skip and should not — property tests are the single most effective check against generated code that has been quietly shaped to satisfy your examples. There is a full worked example in [the verification loop](/ai/feedback-loops/).

Add `pytest-xdist` on day one. `-n auto` is usually a 3–4x speedup for zero effort.

## Editor and agent

This is the fastest-moving part of the stack and anything specific will age badly. The durable advice:

- **You want both a CLI agent and an editor-integrated one.** They are good at different things. The CLI is better for multi-file refactors, migrations and anything that needs to run commands in a loop. The editor integration is better for the ten-second change you would otherwise type yourself.
- **Whatever you pick, configure permissions before your first real session.** See [agent setup](/ai/agent-setup/).
- **A real debugger still beats reading generated code.** This is the strongest argument for a full IDE alongside your agent: when generated code is subtly wrong, stepping through it finds the problem in ninety seconds and reading it can take twenty minutes.

:::promo jetbrains
:::

VS Code plus Pylance is free and excellent, and for many people it is the right answer. The JetBrains case is the debugger, the refactoring engine and the database tooling — if you spend your day in a large Python codebase, it earns the licence.

## Hosting

For a small service or an API you built with an agent and want to put somewhere real:

:::promo digitalocean
:::

App Platform is the least-effort path from a Python repo to a URL with TLS: point it at the repo, it detects Python, and you are done. Roughly $5–12/month for something small.

If you would rather have a plain server and full control, Hetzner is materially cheaper for the same hardware and is the standard answer for self-hosted agent runners and background workers.

:::promo hetzner
:::

:::verdict Honest note
For a hobby project, both of these are beaten by free tiers on Fly.io, Railway or Cloudflare Workers (for the parts of your app that can run on Workers). We get nothing for saying that. Start free; move when the free tier stops fitting.
:::

## Learning, when you want more than a reference page

The tutorials on this site are deliberately short. When you want the long version:

:::promo boot-dev
:::

The strongest recommendation on this page for someone building back-end Python. It is project-based in a way that survives the agent era well — you cannot get through it by pasting, because the thing being taught is the reasoning.

:::promo datacamp
:::

The right shape for data work specifically: pandas, NumPy and SQL in a browser sandbox, no local setup. If you came here from [the pandas lesson](/pandas-basics/), this is the sensible next step.

:::promo educative
:::

Text-first and skimmable, which matters when you already know how to program and need one specific gap filled quickly rather than eight hours of video.

## What we would skip

Being useful means saying this part too.

- **Coding bootcamps.** The market contracted hard between 2023 and 2026, several large providers failed, and the entry-level hiring picture they were built for has changed substantially. If you are considering one, get current outcome data for the specific cohort you would join, in writing, before paying anything. We do not run bootcamp ads on this site.
- **"Learn to prompt" courses.** The half-life is a few months and the durable content is a blog post.
- **Paid AI code-review SaaS, for a small team.** For most repositories, `ruff` with a strong rule set plus a type checker plus one thoughtful human reviewer catches more than the tools do, at zero cost. Revisit at fifty engineers.
- **A second linter.** `ruff` covers what `flake8`, `isort`, `bandit` and `pyupgrade` did. Running both is a slower build and two sources of truth.

## The whole thing, as a file

```toml pyproject.toml
[project]
name = "yourapp"
requires-python = ">=3.12"

[dependency-groups]
dev = ["pytest", "pytest-xdist", "pytest-cov", "hypothesis", "ruff", "mypy"]

[tool.ruff.lint]
select = ["E","F","B","S","DTZ","ASYNC","BLE","A","PL","SIM","UP","I","RUF"]
ignore = ["E501"]
[tool.ruff.lint.per-file-ignores]
"tests/**" = ["S101"]

[tool.mypy]
python_version = "3.12"
strict = true
warn_unused_ignores = true
files = ["src", "tests"]

[tool.pytest.ini_options]
addopts = "-q --strict-markers -m 'not integration and not slow'"
markers = ["integration: needs external services", "slow: over one second"]
```

```bash
uv sync && uv run ruff check . && uv run mypy src && uv run pytest -q -n auto
```

That is the whole stack. Everything else is preference.

## Common questions

### Is uv production-ready?

Yes — it is widely used in production and in CI across the ecosystem, and it reads and writes standard `pyproject.toml` and lockfile formats, so the migration path away from it is short if you ever want one. That reversibility is the main reason we recommend it without hedging.

### mypy or pyright?

`pyright` if you are starting fresh or already live in VS Code; `mypy` if you have an existing codebase with a lot of type debt, because its per-module override system makes gradual adoption much less painful. Running both is not worth the friction.

### Do you take payment for placement on this page?

No. Some links are affiliate links, which means we earn a commission if you buy — but the ordering and the recommendations are not for sale, and several tools listed here (uv, ruff, pytest, hypothesis) have no affiliate programme at all. When we would tell you to use the free option, we say so.

### Why no display ads?

Because they make a technical reference worse and they pay badly at this size. If this site ever runs ads, they will be a single unobtrusive slot, and this sentence will change.