Python code snippet photo by <a href="https://unsplash.com/@cdr6934">Chris Ried</a> on <a href="https://unsplash.com">Unsplash</a> Python code snippet photo by Chris Ried on Unsplash

At a fast-paced startup, where many commits are pushed frequently across various projects and multiple merge requests are created daily, reviewing them and maintaining code quality standards like code compliance with PEP8 all at the same time is a huge challenge.

Making peeps correct their code format or telling them to make changes to follow the pep8 standard is irritating for both.

Sometimes comments on merge requests may sound very nitpicky, like “can you please put two blank lines between two function definition?”, “can you please remove extra spaces?”, “can you please fix imports in place on top?”.

We can delegate this task to tools for maintaining code formatting and styling. So, instead of humans correcting the linting mistakes, we let the tool report devs for their errors. And the “tool” here is the pre-commit hooks. Also, it let us fail fast in the software development cycle.

Workflow with pre-commit hooks

When I commit staged Python files, before committing pre-commit hooks are executed - isort sorts imports then black formats the code and at last flake8 checks code compliance with PEP8. If every check passes, the commit is made else, code is automatically formatted and sent back for review. I then just review the changes made (and perform any other changes if required) and commit again. So, less time is spent on code formatting and more on code logic & implementation.

In this blog post, I’ll discuss how I set up the pre-commit hooks’ pipeline using the pre-commit framework.

First I’ll describe in brief the tool/utility used in the pipeline:

  • isort — sorts all your imports!

  • seed-isort-config — populates the known_third_party of isort setting

  • black — The Uncompromising Code Formatter

  • flake8 — a wrapper around PyFlakes, pycodestyle and Ned Batchelder’s McCabe script

When all hooks combined in the pipeline the workflow look like this:

pre-commit hooks pipeline for checking python files pre-commit hooks pipeline for checking python files

First, we’ll set up the pre-commit framework then add isort, black, and flake8 in the pipeline.

Note: All dot or config files mentioned below should be placed in the root directory of the project.

The pre-commit framework

Recently, I came to know about the pre-commit framework for managing & maintaining multi-language pre-commit git hooks. This framework itself is written in python and comes along with some pre-commit hooks out of the box. It is configured with a YAML file.

Perform the following steps to install it:

  • Install pre-commit pip install pre-commit.

  • Add pre-commit to requirements.txt or Pipfile.

  • Create .pre-commit-config.yaml with hooks needed in the pipeline.

  • At last, execute pre-commit install this will install git hooks in .git/ directory of the project.

In .pre-commit-config.yaml, we define the source (repo) from where the hooks will be downloaded, some high-level options and hooks from each source. Below is a sample .pre-commit-config.yaml file that I used in one of the projects.

repos:
-   repo: https://github.com/asottile/seed-isort-config
    rev: v1.9.3
    hooks:
    - id: seed-isort-config
-   repo: https://github.com/pre-commit/mirrors-isort
    rev: v4.3.21
    hooks:
    - id: isort
-   repo: https://github.com/ambv/black
    rev: stable
    hooks:
    - id: black
      language_version: python3.6
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.3.0
    hooks:
    - id: flake8

Now, we’ll discuss all hooks added in the pipeline one-by-one with its config.

isort: sorts imports

This sorts all the imports in a python file alphabetically and automatically separated into sections. Manually keeping imports sorted alphabetically is tedious when you have so many imports, like in views of Django projects one may have multiple import statements for internal or external packages.

Along with isort we used seed-isort-config hook which statically populates the known_third_part parameter for the isort configuration setting, which otherwise we’ll manually need to populate. Automate everything! Here’s a sample .isort.cfg file for configuring isort that I used:

[settings]
line_length = 88
multi_line_output = 3
include_trailing_comma = True
known_third_party = celery,django,environ,pyquery,pytz,redis,requests,rest_framework

Black: the code formatter

Black is an uncompromising Python code formatter. I’ve been using it for a long time even without pre-commit hooks, it’s super nice!

DEP 0008 (‘Formatting Code with Black’) was accepted by the Django board. Which proposes to enforce code formatting with Black in Django.

With Black, we get code formatted with PEP 8 compliance, plus some other style choices that the Black team sees as helpful for making maintainable code. Some notable style choices by black are:

  • Unlike in PEP8, characters per line is 88, not 79.

  • Use of double-quotes than single-quotes in strings.

  • If there are many function arguments, each argument will be wrapped per line.

One can always override the default styling choices, Black uses pyproject.toml as a configuration file, which is PEP 518 standard.

Here’s pyproject.toml file for configuring black:

[tool.black]
line-length = 88
include = '\.pyi?$'
exclude = '''
/(
    \.git
  | \.hg
  | \.mypy_cache
  | \.tox
  | \.venv
  | _build
  | buck-out
  | build
  | dist
)/
'''

Black formats only python files, for other languages you can give a try to prettier used by over million projects on GitHub itself.

flake8: code style checker

flake8 is a wrapper around PyFlakes, pycodestyle and Ned Batchelder’s McCabe script for checking cyclomatic complexity. For black to work along with flake8 we need to keep some configs in sync, like max-line-length and also need to mention some error & warning codes to ignore in the configuration file (.flake8). The error/warning may be from PyFlake error codes or pycodestyle error codes list. I used the following .flake8 configuration:

[flake8]
max-line-length = 88
max-complexity = 18
select = B,C,E,F,W,T4,B9
ignore = E203, E266, E501, W503, F403, F401

When added all the above config we good to go! Also, when added these configs to an existing project you may want to format code base for the start, so for that, you just need to run pre-commit run --all-files. ✌️

Conclusion

In this post, I discussed how we can automate the workflow of formatting wrongly formatted python code by creating pre-commit hooks pipeline with various tools/utilities. Though here I was specific with python language, you can use the same steps for other languages by using appropriate linting utility hook for that language. Setting up pre-commit is fairly easy for the team when all configurations are there in the project, they just need to run pre-commit install! Also, it’s better to have these checks also in CI (Continuous Integration) pipeline, because one can always skip these pre-commit checks by passing the flag --no-verify while committing.

Resources