At a fast-paced startup, where many commits are pushed frequently across various projects and multiple merge requests are created daily, reviewing them and maintaining code quality standards like code compliance with PEP8 all at the same time is a huge challenge.
Making peeps correct their code format or telling them to make changes to follow the pep8 standard is irritating for both.
Sometimes comments on merge requests may sound very nitpicky, like “can you please put two blank lines between two function definition?", “can you please remove extra spaces?", “can you please fix imports in place on top?".
We can delegate this task to tools for maintaining code formatting and styling. So, instead of humans correcting the linting mistakes, we let the tool report devs for their errors. And the “tool” here is the pre-commit hooks. Also, it let us fail fast in the software development cycle.
Workflow with pre-commit hooks
When I commit staged Python files, before committing pre-commit hooks are executed - isort sorts imports then black formats the code and at last flake8 checks code compliance with PEP8. If every check passes, the commit is made else, code is automatically formatted and sent back for review. I then just review the changes made (and perform any other changes if required) and commit again. So, less time is spent on code formatting and more on code logic & implementation.
In this blog post, I’ll discuss how I set up the pre-commit hooks’ pipeline using the pre-commit framework.
First I’ll describe in brief the tool/utility used in the pipeline:
isort — sorts all your imports!
seed-isort-config — populates the
black — The Uncompromising Code Formatter
flake8 — a wrapper around PyFlakes, pycodestyle and Ned Batchelder’s McCabe script
When all hooks combined in the pipeline the workflow look like this:
pre-commit hooks pipeline for checking python files
First, we’ll set up the pre-commit framework then add
flake8 in the pipeline.
Note: All dot or config files mentioned below should be placed in the root directory of the project.
The pre-commit framework
Recently, I came to know about the pre-commit framework for managing & maintaining multi-language pre-commit git hooks. This framework itself is written in python and comes along with some pre-commit hooks out of the box. It is configured with a
Perform the following steps to install it:
pip install pre-commit.
.pre-commit-config.yamlwith hooks needed in the pipeline.
At last, execute
pre-commit installthis will install git hooks in
.git/directory of the project.
.pre-commit-config.yaml, we define the source (repo) from where the hooks will be downloaded, some high-level options and hooks from each source. Below is a sample
.pre-commit-config.yaml file that I used in one of the projects.
repos: - repo: https://github.com/asottile/seed-isort-config rev: v1.9.3 hooks: - id: seed-isort-config - repo: https://github.com/pre-commit/mirrors-isort rev: v4.3.21 hooks: - id: isort - repo: https://github.com/ambv/black rev: stable hooks: - id: black language_version: python3.6 - repo: https://github.com/pre-commit/pre-commit-hooks rev: v2.3.0 hooks: - id: flake8
Now, we’ll discuss all hooks added in the pipeline one-by-one with its config.
isort: sorts imports
This sorts all the imports in a python file alphabetically and automatically separated into sections. Manually keeping imports sorted alphabetically is tedious when you have so many imports, like in views of Django projects one may have multiple import statements for internal or external packages.
Along with isort we used seed-isort-config hook which statically populates the
known_third_part parameter for the isort configuration setting, which otherwise we’ll manually need to populate. Automate everything!
Here’s a sample
.isort.cfg file for configuring isort that I used:
[settings] line_length = 88 multi_line_output = 3 include_trailing_comma = True known_third_party = celery,django,environ,pyquery,pytz,redis,requests,rest_framework
Black: the code formatter
Black is an uncompromising Python code formatter. I’ve been using it for a long time even without pre-commit hooks, it’s super nice!
With Black, we get code formatted with PEP 8 compliance, plus some other style choices that the Black team sees as helpful for making maintainable code. Some notable style choices by black are:
Unlike in PEP8, characters per line is 88, not 79.
Use of double-quotes than single-quotes in strings.
If there are many function arguments, each argument will be wrapped per line.
One can always override the default styling choices, Black uses
pyproject.toml as a configuration file, which is PEP 518 standard.
pyproject.toml file for configuring black:
[tool.black] line-length = 88 include = '\.pyi?$' exclude = ''' /( \.git | \.hg | \.mypy_cache | \.tox | \.venv | _build | buck-out | build | dist )/ '''
flake8: code style checker
flake8 is a wrapper around PyFlakes, pycodestyle and Ned Batchelder’s McCabe script for checking cyclomatic complexity. For black to work along with flake8 we need to keep some configs in sync, like
max-line-length and also need to mention some error & warning codes to ignore in the configuration file (
.flake8). The error/warning may be from PyFlake error codes or pycodestyle error codes list.
I used the following
[flake8] max-line-length = 88 max-complexity = 18 select = B,C,E,F,W,T4,B9 ignore = E203, E266, E501, W503, F403, F401
When added all the above config we good to go! Also, when added these configs to an existing project you may want to format code base for the start, so for that, you just need to run
pre-commit run --all-files. ✌️
In this post, I discussed how we can automate the workflow of formatting wrongly formatted python code by creating pre-commit hooks pipeline with various tools/utilities. Though here I was specific with python language, you can use the same steps for other languages by using appropriate linting utility hook for that language. Setting up pre-commit is fairly easy for the team when all configurations are there in the project, they just need to run
pre-commit install! Also, it’s better to have these checks also in CI (Continuous Integration) pipeline, because one can always skip these pre-commit checks by passing the flag
--no-verify while committing.