Sync calls in asyncio.

Asyncio in Python is a double edged sword, if you’re making lots of calls out to IO with very little actual compute happening locally within the app it’s a godsend for performance. Honestly though, is that something that you’re doing regularly? If you are great, keep doing your thing, if not then using vanilla sync python is probably going to make your life easier.

I’ve inherited quite a bit of async code recently that doesn’t really need to be async. In a couple of places I’ve seen code like this:

Essential Python Tools #2: Factoryboy

This is the first in an ongoing series of essential tools that I use day to day as a python developer which I couldn’t be as productive without. This episode is on factoryboy

What do you use for fixtures?

Traditionally I built database fixtures one of two ways:

  1. Manually crafted artisan SQL commands which populated required fields into the database.
  2. ‘Live’ or environmental data that was dumped from the DB into a SQL file and is imported into the test database at test run startup.

I normally would use method 1 for testing repository or service methods where I needed a specific example which had some required properties, and method 2 for general testing where I was more testing rough results rather than specific outputs.

Essential Python Tools #1: On demand pytest fixtures

I think the dependencies you use for development should differ from the ones that you use for testing. It’s quite easy to introduce dependencies in your code that are based off of data or configuration that you have created outside of your codebase. To get around this we can use an entirely separate set of test fixtures which are created before each test run, configured, tested against and torn down when the test run is over.

Python Logging.

Two things I keep seeing:

  1. People using f strings in logging.
  2. People calling expensive functions eagerly in the log event to fetch debugging data.

I see it so much at this point that I’m not even correcting it when I see it in the most of the codebases I use. It seems too controversial of a thing to bring up, and honestly it’s not worth spending some of your capital with your colleagues on. While I do love fstrings, there are some great reasons outside of the above to not use them.

Shaking out test interdependence

I’ve been battling bugs recently, this came up and it seemed like a fun exercise in test suite hygiene. I take the view that test hygiene is everyones responsability regardles of the level of the engineer, this is such a simple subject that everyone regardless of their skill level can implement the ideas here.

Flaky tests are the bane of many a developers existence, even worse than this however is test inter-dependencies. Let’s say we’ve got the following code, some function that marks a data structure as being completed.

Static Duck Typing.

I’ve been using writing libraries and tooling for others recently and one of the most useful tools in this has been gradual typing. In the libraries that have been built we’ve fully typed them, allowing our users to make the decision of opting into type checking or not depending upon their needs.

As we don’t know what objects are users are going to be passing into our consuming functions we need a way of defining what properties our objects need ahead of time without knowing what the object will be. This is static duck typing; or to use it’s correct name as per the PEP, structural subtyping.

Essential Python Tools #0: Pytest

This is the first in an ongoing series of essential tools that I use day to day as a python developer which I couldn’t be as productive without. First episode is on pytest.

What is pytest?

This should be pretty obvious by the name, it’s a testing framework for Python. Unlike the standard libraries unittest it doesn’t wholesale inherit ideas from other languages, it reads, looks and works like you’d expect in Python. In my view, there’s 3 main reasons to use pytest:

Line coverage is not branch coverage.

I’ve spent the past couple of weeks using line coverage to search out weakly tested sections of code in a large codebase I’m working on. While this has been quick at identifying areas that have room for improvement I think it’s worth highlighting that line coverage isn’t always the best indicator of quality of your tests.

Take this highly contrived example…

Hypothetical big bank requirement

We’re a new developer and we’ve been tasked with turning a new product requirement on a mortgage service into reality. We’re not actually checking whether a mortgage is deemed as affordable by some regulatory checks.

Security and the normalisation of deviance

I’ve been involved in quite a few post mortems in my time, a running them across almost all of them was that they weren’t down to a single cause but a series of failures that combined to result in a damaging event. The most recent post mortem was a data synchronisation service that uses an unstable API. It was built by one team and inherited by another, errors have grown to be expected and it’s mostly worked. Concerns have been raised but a working system is not often prioritised. The dissociation between the engineers who built the code and those now running it has led to a certain acceptance that some error conditions can be ignored.

Decorators

I love decorators. Python is a pretty concise language but all too often you can end up with cluttered code that is full of monitoring and debugging logic. Let’s take some pretty typical code and see if we can improve it with some decorators.

This post uses timing as a contrived example, in the real world you’ll probably just want to using a pre-existing timing decorator and be done with it. Realistically you’ll find decorators useful for things like authentication, metrics, logging, tracing and really anything where you want to do something before or after a callable.

Varnish Debugging

I love Varnish. As far as the world of HTTP goes, in environments where you can cache it’s one of the easiest quick wins there is. However this does somewhat come at a price to debugging. Often you’ll sit back and your black box of magic doesn’t quite produce the result you were expect from it.

Normally I end up adding a sleuth of headers cache status headers to help identify where things are going wrong. The first and most useful of these to add is X-Grace to show if Varnish thinks it has served a graced object. Graced objects are those that are outside of their TTL but are delivered anyway as no backend was available to service the request.