Claude Fable 5.1 & GPT-6 Astra packages are live

Pytest

Free

A pytest suite that stays fast and trustworthy — layout, fixtures with the right scope, parametrize instead of loops, markers that mean something,…

231 lines8.9 KB Grok Testing
targetModels
Grok 4.6Grok 4.5Grok 4 FamilyGrok Code FastFuture Grok Models
name
pytest
category
Testing
description
A pytest suite that stays fast and trustworthy — layout, fixtures with the right scope, parametrize instead of loops, markers that mean something, mocking only at boundaries you own, coverage as a diagnostic, async tests, and keeping the whole run under a minute.
license
MIT
author
Agent.md maintainers
last-verified
reviewed-by
unreviewed
<!-- Generated from models/_canonical by scripts/build-model-variants.js. Edit the canonical source, not this file. Behavioural profile for Grok: scripts/model-profiles.json -->

#Non-negotiable

The constraints hoisted below override anything later in this document. Read them first; the rest is rationale.


#Purpose

Rules for organising and writing tests with pytest. What to assert is Testing/unit; this package is about the harness: how tests are found, shared, isolated and kept fast.


#Layout and configuration

bash
tests/
  conftest.py            # shared fixtures, no tests
  unit/test_money.py
  integration/conftest.py  test_orders_repo.py
  e2e/test_checkout.py
toml
[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-q --strict-markers --strict-config -ra"
xfail_strict = true
filterwarnings = ["error"]
markers = ["slow: takes >1s", "integration: needs the database"]
  • --strict-markers: a typo'd @pytest.mark.integraton is an error, not a silently unselected test.
  • filterwarnings = ["error"]: deprecation warnings fail the suite while they are cheap to fix.
  • xfail_strict: an xfail that starts passing fails, so fixed bugs get their marker removed.
  • src/ layout with the package installed editable, so import orders in tests is the installed package. → Backend/python-conventions

#Fixtures and scope

python
@pytest.fixture(scope="session")
def engine():                          # expensive: once per run
    e = create_engine(TEST_URL); yield e; e.dispose()

@pytest.fixture
def session(engine):                   # cheap and isolating: once per test
    conn = engine.connect(); tx = conn.begin()
    s = Session(bind=conn, join_transaction_mode="create_savepoint")
    yield s
    s.close(); tx.rollback(); conn.close()

@pytest.fixture
def order(session) -> Order:
    return OrderFactory(session=session)
  • Default scope is function. Widen it only for things that are expensive to build and immutable (an engine, a compiled schema, a loaded model). A session-scoped fixture that holds mutable state is order-dependent flakiness.
  • Fixtures compose: order depends on session, which depends on engine. Ask for what you need by name; do not build the world in one mega-fixture.
  • yield fixtures for teardown; the code after yield runs even when the test fails.
  • conftest.py per directory for fixtures that belong to that layer; a 600-line root conftest.py is a sign the layers are not separated.
  • autouse=True is a global; use it for things every test genuinely needs (freezing the clock, disabling network) and nothing else.

#Parametrize, don't loop

python
@pytest.mark.parametrize(("qty", "discount"), [
    (0, 0), (1, 0), (9, 0), (10, 100), (11, 110),
], ids=["zero", "one", "below", "threshold", "above"])
def test_bulk_discount(qty: int, discount: int) -> None:
    assert discount_for(qty) == discount
  • One case per parameter set, each reported and re-runnable by id (-k threshold). A for loop inside a test stops at the first failure and hides the rest.
  • ids= for readable failures; the default qty0-discount0 tells you nothing.
  • pytest.param(..., marks=pytest.mark.xfail(reason="#123")) for a known failing case inside the table, instead of deleting it.

#Markers and selection

bash
pytest -m "not slow and not integration"     # the pre-commit run
pytest -m integration                        # CI job with the database
pytest --lf                                  # only what failed last time
pytest -x --ff                               # stop at first, run failures first
  • Declare every marker in config. Markers are the contract between the suite and CI; undeclared ones are noise.
  • @pytest.mark.skipif(sys.platform == "win32", reason="...") with a reason, always. A bare skip is a test that quietly stopped existing.

#Mocking at boundaries you own

python
def test_sends_receipt(order, mailer_spy):                 # fake of OUR interface
    send_receipt(order, mailer=mailer_spy)
    assert mailer_spy.sent == [Receipt(order.id)]

def test_retries_on_timeout(monkeypatch):
    monkeypatch.setattr(payments, "charge", AsyncMock(side_effect=[TimeoutError, "ok"]))
  • Fake the interface you defined (Mailer, PaymentGateway), not the vendor SDK. Mocking stripe.Charge.create tests your guess about Stripe's API.
  • monkeypatch over unittest.mock.patch decorators: it undoes itself, works on env vars and attributes, and reads top-to-bottom.
  • AsyncMock for async callables; a Mock returns a non-awaitable and the test passes for the wrong reason.
  • Block real network in the suite (pytest-socket, or an autouse fixture that raises on socket.connect). A test that hits a real API is an outage waiting for CI. → Testing/unit

#Async tests

toml
asyncio_mode = "auto"      # pytest-asyncio: every `async def test_*` just runs
python
async def test_fetch_times_out(monkeypatch):
    with pytest.raises(TimeoutError):
        async with asyncio.timeout(0.01):
            await fetch_slow()

Await events and futures; never await asyncio.sleep(0.2) to let something finish. Use pytest.raises with the specific exception, not Exception. → Backend/python-async


#Coverage and speed

bash
pytest --cov=src --cov-report=term-missing --cov-fail-under=80   # a floor, not a target
pytest -n auto                                                    # pytest-xdist
pytest --durations=10                                             # find the slow ones
  • Coverage is a diagnostic for untested code; 100% with assertion-free tests is worse than 70% with real assertions. Set a floor to stop regressions, then ignore the number.
  • The unit run should finish in seconds. Anything slower is I/O leaking in: find it with --durations, mark it slow or integration, move the real work behind a fake.
  • -n auto requires tests that do not share state — which the fixture rules above already guarantee.
  • Freeze time (freezegun/time-machine) and seed randomness; a test that depends on the wall clock fails at midnight.

#Anti-patterns

Anti-patternWhy it failsFix
Undeclared markersTypos silently unselect tests--strict-markers
Warnings allowedDeprecations pile up until removalfilterwarnings = ["error"]
Session-scoped mutable fixtureOrder-dependent flakesFunction scope, or immutable
One giant root conftest.pyLayers tangledPer-directory conftest.py
autouse for convenienceHidden global behaviourExplicit fixture args
for loop over casesFirst failure hides the restparametrize with ids
Bare @pytest.mark.skipTest silently vanishesskipif with a reason
Mocking the vendor SDKTests an assumptionFake your own interface
mock.patch decorators stacked five deepUnreadable, order-sensitivemonkeypatch
Mock() for async codeReturns non-awaitableAsyncMock
Real network in testsFlaky, slow, outagesBlock sockets
asyncio.sleep() to waitSlow and still flakyAwait an event
pytest.raises(Exception)Passes on the wrong errorSpecific exception
Coverage as a targetAssertion-free testsFloor + mutation testing
Unit run over a minuteI/O leaked in--durations, mark and move
Wall clock in testsFails at midnight/DSTFreeze time

#Checklist

  • Verify: pyproject.toml sets --strict-markers, --strict-config, filterwarnings = ["error"], xfail_strict
  • Verify: Every marker declared and used to split unit from integration runs
  • Verify: Fixtures default to function scope; wider scope only for immutable, expensive resources
  • Verify: Database tests run inside a rolled-back transaction fixture
  • Verify: conftest.py per layer; autouse reserved for clock and network guards
  • Verify: Tables of cases use parametrize with ids
  • Verify: Every skip has a reason; every xfail references an issue
  • Verify: Fakes target interfaces we own; vendor SDKs are never mocked directly
  • Verify: monkeypatch and AsyncMock used; real network blocked
  • Verify: asyncio_mode = "auto"; no real sleeps in async tests
  • Verify: Coverage floor enforced; number treated as diagnostic
  • Verify: Unit run finishes in seconds; -n auto works
  • Verify: Time and randomness are controlled