🤤 ❎ 🤚🏿 Integrating .pre-commit hook into a Django project 🕵️ 💺 ⛎

Good day!

My name is Andrey Sobolev and today I will tell you how we prepared .pre-commit hook on our project.

Introduction

To begin with, a few words about what hooks are in general and why they may be needed. Git out of the box provides a tool that can run your scripts when an event occurs (for example, pushing to a server, etc.).

Pre-commit is a convenient add-on to the default git pre-commit hook that runs the scripts described in .pre-commit-config.yaml before committing. In theory, it sounds simple, let's move on to practice.

Installation

Set the necessary dependencies:

pre-commit
#   https://pre-commit.com/

autoflake
#     (  )
black
#  
pyupgrade
#     
reorder-python-imports
#   
yesqa
#   noqa  ( )

# 
flake8
flake8-annotations
flake8-annotations-coverage
flake8-bandit
flake8-broken-line
flake8-bugbear
flake8-builtins
flake8-commas
flake8-comprehensions
flake8-debugger
flake8-eradicate
flake8-executable
flake8-fixme
flake8-future-import
flake8-pyi
flake8-pytest
flake8-pytest-style
flake8-mutable
flake8-string-format
flake8-todo
flake8-unused-arguments

# 
pytest

I will express my opinion about flake-8 and linter in general. If you already have a large project with a bunch of legacy code, then you can safely delete linters. The costs that will be spent on "bringing to the ideal", the authorities will not appreciate. We put linters for new (and small) projects. I repeat, this is my personal opinion, I do not impose it on anyone.

Environment Integration

We go to the root directory of the development environment and execute the following commands

$ pre-commit install
pre-commit installed at .git/hooks/pre-commit
$ pre-commit --version
pre-commit 2.4.0

If .pre-commit will swear on sqlite, then you will need to install it (for example $ yum install sqlite) and build python again

Setting up the .pre-commit-config.yaml file

In the root directory of the environment, create the file .pre-commit-config.yaml

- repo: https://github.com/pre-commit/pre-commit-hooks
  rev: "v2.5.0"
  hooks:
    - id: check-merge-conflict
    - id: debug-statements

- repo: local

  hooks:
    - id: black
      name: black
      entry: black
      language: system
      types: [python]
      args: [--line-length=200, --target-version=py37]

    - id: autoflake
      name: autoflake
      entry: autoflake
      language: system
      types: [python]
      args: [--in-place, --remove-all-unused-imports, --remove-duplicate-keys]

    # -   id: flake8
    #     name: flake8
    #     entry: flake8
    #     language: system
    #     types: [python]
    #     args: [
    #         "--ignore=E203,W503,FI10,FI11,FI12,FI13,FI14,FI15,FI16,FI17,FI58,PT013",
    #         # black
    #             # E203 whitespace before ':'
    #             # W503 line break before binary operator
    #         # flake8-future-import
    #             # FI10 __future__ import "division" missing
    #             # FI11 __future__ import "absolute_import" missing
    #             # FI12 __future__ import "with_statement" missing
    #             # FI13 __future__ import "print_function" missing
    #             # FI14 __future__ import "unicode_literals" missing
    #             # FI15 __future__ import "generator_stop" missing
    #             # FI16 __future__ import "nested_scopes" missing
    #             # FI17 __future__ import "generators" missing
    #             # FI58 __future__ import "annotations" present
    #         # flake8-pytest-style
    #             # PT013 found incorrect import of pytest, use simple 'import pytest' instead
    #         "--max-line-length=110",
    #         "--per-file-ignores=tests/*.py:S101"
    #         # S101 Use of assert detected
    #     ]

    - id: pyupgrade
      name: pyupgrade
      entry: pyupgrade
      language: system
      types: [python]
      args: [--py37-plus]

    - id: reorder-python-imports
      name: reorder-python-imports
      entry: reorder-python-imports
      language: system
      types: [python]
      args: [--py37-plus]

    - id: yesqa
      name: yesqa
      entry: yesqa
      language: system
      types: [python]

    - id: tests
      name: Run tests
      entry: "bash tests.sh"
      language: system
      verbose: true

Tests

In addition to checking and formatting the code, we will perform tests at the stage of creating the commit. To do this, we will use pytest (https://docs.pytest.org/en/latest/) and configure it for our needs.

In the root directory of the environment, create the tests folder and put the following files
test_example_without_db.py, test_example_with_db.py there

for convenience, configure the tests so that you can use the current database (for example, a copy of the database from the battle server), and not create a new one every time .

Simple test test_example_without_db.py

def inc(x):
    return x + 1

def test_answer():
    assert inc(3) == 4

In simple tests, we can connect for example webbot and bypass the nodes of our system to automate the manual work of the tester.

Test using test_example_with_db.py database

import pytest
from chat.models import ChatRoom
from settings import POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD, \
                    POSTGRES_HOST, POSTGRES_PORT

@pytest.fixture(scope='session')
def django_db_setup():
    settings.DATABASES['default'] = {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': POSTGRES_DB,
        'USER': POSTGRES_USER,
        'PASSWORD': POSTGRES_PASSWORD,
        'HOST': POSTGRES_HOST,
        'PORT': POSTGRES_PORT,
    }   
    
    
@pytest.fixture
def db_access_without_rollback_and_truncate(request, django_db_setup, django_db_blocker):
    django_db_blocker.unblock()
    request.addfinalizer(django_db_blocker.restore)

def chat():
    return ChatRoom.objects.all().count()

@pytest.mark.django_db
def test_chat():
    assert chat() > 0

The example is quite artificial and created exclusively for this note, but nevertheless it allows us to access the current database and conduct complex tests that go beyond manual testing.

We include tests in .pre-commit

To connect the tests, we need a shell script in the root directory of the environment, which we will call tests.sh:

source ../../python38_env/bin/activate && python -m pytest -v tests

Its content is very obvious, but you may notice that the activation of the virtual environment is explicitly written in the code. This can be inconvenient if your team is developing on different workstations (for example, who deployed the environment on a local machine, and someone develops on a server).

You can solve this problem through variables in .env Implementation

example:

github.com/Sobolev5/starlette-vue-backend/blob/master/.env.example (note the variable ENV_ACTIVATE)

github.com/Sobolev5/starlette-vue-backend /blob/master/tests.sh (parse ENV_ACTIVATE and activate the environment)

Create a commit

Now it remains to create a commit and see how it works

$ git add .
$ git commit -m Sobolev:TestPreCommitHook
Check for merge conflicts................................................Passed
Debug Statements (Python)................................................Passed
black....................................................................Failed
- hook id: black
- files were modified by this hook

reformatted /var/www/file.py
All done!   
1 file reformatted, 2 files left unchanged.

autoflake................................................................Passed
pyupgrade................................................................Passed
reorder-python-imports...................................................Failed
- hook id: reorder-python-imports
- exit code: 1
- files were modified by this hook

Reordering imports in file.py

yesqa....................................................................Passed
Run tests................................................................Passed
- hook id: tests
- duration: 2.85s

tests/test_example_with_db.py::test_chat PASSED                          [ 66%]
tests/test_example_without_db.py::test_answer PASSED                     [100%]

A commit is now created in two steps. At the first stage, hooks format the code, so after they work, we just need to “repeat” the commands.

It turns out the following sequence.

$ git add .
$ git commit -m Sobolev:TestPreCommitHook
$ git add .
$ git commit -m Sobolev:TestPreCommitHook

That's all, thanks for your attention.

Sitelinks

→ Full list of hooks

Integrating .pre-commit hook into a Django project