Essential Python Tools #0: Pytest

Tags: Python, Software Engineering, Testing, pytest

This is the first in an ongoing series of essential tools that I use day to day as a python developer which I couldn’t be as productive without. First episode is on pytest.

What is pytest?

This should be pretty obvious by the name, it’s a testing framework for Python. Unlike the standard libraries unittest it doesn’t wholesale inherit ideas from other languages, it reads, looks and works like you’d expect in Python. In my view, there’s 3 main reasons to use pytest:

1. Concise syntax that’s clear without being verbose. Pytest overrides the existing assert keyword so you can use all of the normal language terms like in, not, is and comparisons without having to remember any extra method names.
2. Integrated test runnner that enables test discovery that works accross almost all of the common Python test types. The test runner can find and execute unittest, pytest, doctest and behavoural tests all from one entrypoint.
3. Fixture ecosystem is HUGE and will save you lots of time.

What do tests look like?

Simple…

sky = True

def test_sky_is_true() -> None:
    assert sky is True

Clear and concise without being verbose.

What output do you get?

The runner output is much the same, simple and informative without being cluttered.

% pytest tests

Honestly that’s not very interesting though, really what you want to see is what pytest offers when things don’t go right.

% pytest --showlocals -vv tests

--showlocals and -vv are two handy extra that return the variables in the local scope plus return some extra details on the failure.

How does test discovery work?

Pytest will recursively check for tests from the directory argument, or if one isn’t given then your current working directory. It does this by:

1. Search for files that match the test_*.py or *_test.py name.
2. In those files match any function OR class prefixed with test

Full details on test collection here.

What batteries does it come with out of the box?

A full list of fixtures are in the docs, most are really focused on building on pytest but there’s a few that are very useful when writing common tests.

You can setup logging on a per test level, I like to use this when I end up stuck and need some more context on why my test is failing outside of what I can get from pdb.

import logging

def test_callable(caplog) -> None:
    caplog.set_level(logging.DEBUG)

Dealing with files is a pain and remembering the library methods for the stdlib TempDirectory is a pain. There are fixtures available to create one time use files that are cleaned up once the test is run:

from pathlib import Path

def write_to_file(path: Path, body: str) -> None:
    with open(str(path), 'w') as f:
        f.write(body)

def test_function_that_writes_to_file(tmp_path: Path) -> None:
    test_string = 'potato'
    # Write to file.
    write_to_file(tmp_path, test_string)

    with open(tmp_path, 'r') as f:
        assert f.read() == test_string

Sometimes even the best of us need to use mocks as an escape hatch. While unittest.Mock is undoutably the king of mocks, sometimes you don’t need all of that. Monkeypatch has enough built in utility functions for the common tasks you might need

import os
from _pytest.monkeypatch import MonkeyPatch

def in_development() -> bool:
    return True if os.environ.get('development') else False

def test_environment_discovery(monkeypatch: MonkeyPatch) -> None:
    assert in_development() is False
    monkeypatch.setenv("development", "True")

    assert in_development() is True

A full tutorial on this is here.

What batteries come from pypi?

Now this is the meat of the matter, available tooling is a huge part of why people use Python. A few initial ones to look at would be:

• pytest-cov incorporates the coverage utility tool into your test flow. Output coverage metrics every time your tests run, set minimum coverage limits that your tests fail after.
• pytest-freezegun incorporates the freezegun utility for mocking time within your tests. For anyone that has ever mocked out the datetime library in python and knows how much of a pain it is, this is for you.
• pytest-vcr mock out your external HTTP calls with ease using the VCR library. This can REALLY save your time with complex chains of HTTP calls, it’ll usually work with libraries like boto3 as well.
• pytest-random-order is a small library that randomises the order of your tests. It does this by patching the randomness of your python runtime and then shuffling the order in which your tests are run. The patching of randomness is essential as it allows you to offer repeatable randomness, the library offers a seed with each run that can be used to rerun the test suite in that same order. This makes it very useful to uncover race conditions or places where your logic is interdependant.

What don’t I like?

Well. Glad you asked as there’s a couple of things…

The magic is hard to explain when you’re explaining the framework to someone new to the language. While the use of decorators to define and mark fixtures along with their use in function arguments can be highly consise yet clear it does require you to have understanding of how all these advanced parts of the language work. Without this knowledge they just appear as magic.

It’s not beginner friendly, although it’s quite simple to get started and start testing your application you can quite easily get lost. Some of the most powerful tools in pytest can be very easily abused and CAN work initially, but can stop working weeks, months and years down the line. The fixture system is very easy to write race conditions into