Get best of the week

Finding the Perfect Toolkit: Analyzing Popular Python Project Templates

The material, the translation of which we publish today, is dedicated to the story about the tools used to create Python applications. It is designed for those programmers who have already left the category of beginners, but have not yet reached the category of experienced Python developers. For those who are eager to start practicing, the author suggests using Flake8 , pytest and Sphinx in existing Python projects . He also recommends a look at pre-commit , Black, and Pylint . Those who plan to start a new project, he advises to pay attention to Poetry and Dependabot .





Overview


It has always been difficult for me to form an objective opinion about the “best practices” of Python development. In the world of technology, some popular trends are constantly emerging, often not existing for long. This complicates the extraction of the “useful signal” from the information noise.

The freshest tools are often only good, so to speak, on paper. Can they really help the practical programmer with something? Or their application only leads to the introduction of something new in the project, the efficiency of which must be maintained, which carries more difficulties than benefits?

I did not have a clear understanding of what exactly I considered the "best practices" of development. I suppose I found something useful, based mainly on episodic evidence of "utility", and on occasional mention of this in conversations. In the last couple of weeks, I decided to put things in order in this matter. To do this, I began to analyze all the templates of Python projects that I could find (we are talking about templates used by the cookiecutter command-line utility to create Python projects based on them).

It seemed to me that it was very interesting to learn about what auxiliary tools the template authors consider worthy of getting these tools into new Python projects created on the basis of these templates.

I analyzed and compared the 18 most popular template projects (from 76 to 6300 stars on GitHub), paying particular attention to what kind of auxiliary tools they use. The results of this analysis can be found in this table.

Below I want to share the main conclusions that I have made while analyzing popular templates.

De facto standards


The tools discussed in this section are included in more than half of the templates. This means that they are perceived as standard in a large number of real Python projects.

▍Flake8


I have been using Flake8 for quite some time , but I did not know about the dominant position in the field of linting that this tool occupies. I thought that it exists in a kind of competition, but the vast majority of project templates use it.

Yes, this is not surprising. It is difficult to oppose something to the convenience of linting the entire code base of the project in a matter of seconds. Those who want to use cutting-edge developments can recommend a look at wemake-python-styleguide . This is something like "Flake8 on steroids." This tool may well contribute to the transfer to the category of obsolete other similar tools (like Pylint).

▍Pytest and coverage.py


The vast majority of templates use pytest . This reduces the use of the standard unittest test framework. Pytest looks even more attractive when combined with tox . That's exactly what was done in about half of the templates. Code coverage with tests is most often checked using coverage.py .

▍Sphinx


Most templates use Sphinx to generate documentation . To my surprise, MkDocs is rarely used for this purpose.

As a result, we can say that if you do not use Flake8, pytest and Sphinx in your current project, then you should consider introducing them.

Promising tools


In this section, I have collected those tools and techniques of work, the use of which in the templates suggested some trends. The point is that although all this does not appear in most project templates, it is found in many fairly recent templates. So - all this is worth paying attention to.

▍Pyproject.toml


File usage is pyproject.tomlsuggested in PEP 518 . This is a modern mechanism for specifying project assembly requirements. It is used in most fairly young templates.

▍Poetry


Although the Python ecosystem isn’t doing well in terms of a good tool for managing dependencies, I cautiously optimistic that Poetry could be the equivalent of npm from the JavaScript world in the Python world .

The youngest (but, nevertheless, popular) project templates seem to agree with this idea of ​​mine. True, it is worth saying that if someone is working on some kind of library that he can plan to distribute through PyPI , then he still has to use setuptools . (It should be noted that after the publication of this material I was informed that this, apparently, is no longer a problem).

Also, be careful if your project (the same goes for dependencies) relies on Conda. In this case, Poetry will not suit you, since this tool, in its current form, binds the developer to pip and to virtualenv .

▍Dependabot


Dependabot regularly checks project dependencies for obsolescence and tries to help the developer by automatically opening PR.

Personally, I have recently seen this tool more often than before. It seems to me that it is an excellent tool, the addition of which to the project affects the project very positively. Dependabot helps reduce security risks by pushing developers to keep dependencies up to date.

As a result, I advise you not to lose sight of Poetry and Dependabot. Consider introducing these tools into your next project.

Personal recommendations


The analysis of project templates gave me a somewhat ambivalent perception of the tools that I will list in this section. In any case, I want to use this section to talk about them based on my own experience. At one time they were very useful to me.

▍Pre-commit


Even if you are extremely disciplined - do not waste your energy on performing simple routine actions such as additional code run through the linter before sending the code to the repository. Similar tasks can be passed to Pre-commit . And it's better to spend your energy on TDD and on team work on code.

▍Pylint


Although Pylint is criticized for being slow, although this tool is criticized for the features of its settings, I can say that it allowed me to grow above myself in the field of programming.

He gives me specific instructions on those parts of the code that I can improve, tells me how to make the code better comply with the rules. For a free tool, this alone is already very much. Therefore, I am ready to put up with the inconvenience associated with Pylint.

▍Black


Black at the root of the debate over where to put spaces in the code. This protects our teams from empty talk and from meaningless differences in files caused by different editors settings.

In my case, this brightens up what I personally dislike in Python (the need to use a lot of spaces). Moreover, it should be noted that the Black project in 2019 joined the Python Software Foundation, which indicates the seriousness and quality of this project.

As a result, I want to say that if you still do not use pre-commit, Black and Pylint - think about whether these tools can benefit your team.

Subtotals


Twelve of the eighteen investigated templates were created using the cookiecutter framework . Some of those templates where this framework is not used have interesting qualities.

But given the fact that cookiecutter is the leading framework for creating project templates, those who decide to use a template that did not use a cookiecutter should have very good reasons for this. Such a decision should be very well justified.

Those who are looking for a template for their project should choose a template that most closely matches their own view of things. If you, when creating projects according to a certain template, constantly have to reconfigure them in the same way, think about making a fork of such a template and finalizing it, inspired by examples of templates from my list.

And if you are attracted to adventure - create your own template from scratch. Cookiecutter is a great feature of the Python ecosystem, and the simple process of creating Jinja templates allows you to quickly and easily do something of your own.

Bonus: Template Recommendations


▍Django


Together with the most popular Django templates, consider using the wemake-django-template . It gives the impression of a deeply thought out product.

▍ Data processing and analysis


In most projects aimed at processing and analyzing data, the Cookiecutter Data Science template is useful . However, data scientists should also look at Kedro .

This template extends Cookiecutter Data Science with a mechanism for creating standardized data processing pipelines. It supports loading and saving data and models. These features are very likely to be able to find worthy application in your next project.

Here I would also like to express my gratitude to the creators of the shablona template for preparing very high-quality documentation. It can be useful to you even if you end up choosing something else.

▍ General purpose templates


Which general purpose template to choose in some way depends on what exactly you are going to develop based on this template - a library or a regular application. But, choosing such a template, along with the most popular projects of this kind, I would very carefully look at Jace's Python Template .

This is not a well-known pattern, but I like the fact that it has Poetry, isort , Black, pylint, and mypy .

PyScaffold is one of the most popular non-cookiecutter based templates. It has many extensions (for example, for Django, and for Data Science projects ). It also downloads version numbers from GitHub using setuptools-scm. Further, this is one of the few templates supporting Conda.

Here are a couple of templates that use GitHub Actions technology:

  1. The Python Best Practices Cookiecutter template , which, I want to note, has most of my favorite tools.
  2. The Blueprint / Boilerplate For Python Projects template , which I find pretty interesting, as the opportunity it gives them to find common security problems with Bandit, looks promising. In addition, this template has a remarkable feature, which consists in the fact that the settings of all tools are collected in a single file setup.cfg.

And finally - I recommend taking a look at the wemake-python-package template . I think it’s worth doing it anyway. In particular - if you like the Django-template of the same developer, or if you are going to use the advanced wemake-python-styleguide instead of pure Flake8.

Summary


After I published this article, I wrote Guido van Rossum about it.

Perhaps you, like me, will be interested in his comment . He said that I forgot about mypy, and that it’s easier to work not with Sphinx, but with Markdown. Regarding Black, he noted that this tool is overrated and can only benefit if team members argue a lot about styles. According to him, those who use Flake8 do not need Pylint. He had not heard of Poetry and Dependabot. In addition, he advised using a certain CI solution, like Travis-CI, to run tests.

Dear readers! What Python project templates do you use?

More articles:


All Articles