Get best of the week

We make Python and Bash friends: smart-env and python-shell libraries

Good day to all.

Today, Python is one of the most used languages ​​in the field of creating not only directly software products, but also providing their infrastructure. As a result, many devops, by their will or against it, had to learn a new language for later use as a complement to the good old Bash scripts. However, Bash and Python profess different approaches to writing code and have certain features, in view of which porting Bash scripts to the "snake language" is sometimes capacious and far from a trivial task.

To make life easier for devs, many useful libraries and utilities in Python have been created and continue to be created. This article immediately describes two new libraries created by the author of this post - smart-envand python-shell - and designed to save the devops from the need to pay much attention to the intricacies of working with Python, leaving room for more interesting tasks. The scope of the library is environment variables and the launch of external utilities.

Who are interested, please, under the cat.

New bicycles?


It would seem, why create new packages for fairly mundane operations? What prevents the direct use of os.environ and subprocess. <Method or class of your choice>?

I will give evidence in favor of each of the libraries separately.

Smart-env library


Before writing your own brainchild, it’s useful to surf the Internet and look for ready-made solutions. Of course, there is a risk of not finding what you need, but rather, it is an "insurance case". As a rule, the approach works and saves a lot of time and effort.

According to the search results , the following were revealed:

  • there are packages that actually wrap calls to os.environ, but at the same time require a bunch of distracting actions (creating an instance of the class, special parameters in calls, etc.);
  • there are good packages that are, however, tightly tied to a specific ecosystem (mainly web frameworks like Django) and therefore are not universal without a file;
  • there are rare attempts to do something new. For example, add typing and explicitly pars variable values ​​by calling methods of the form

    get_<typename>(var_name)

    Or here is another solution that, however, does not currently support disgraced Python 2 (on which, despite the official RIP , there are still mountains of written code and entire ecosystems);
  • there are school-student crafts, it’s not at all clear why they ended up in the upstream PyPI and only create problems with naming new packages (in particular, the name “smart-env” is a necessary measure).

And the list goes on and on. However, the above points were enough to catch the idea of ​​making something convenient and universal.

Requirements for smart-env:

  • The most simple scheme of use
  • Easily Configurable Data Typing Support
  • Python 2.7 compatible
  • Good test coverage

In the end, all of this was realized. Here is a usage example:

from smart_env import ENV

print(ENV.HOME)  # Equals print(os.environ['HOME'])

# assuming you set env variable MYVAR to "True"

ENV.enable_automatic_type_cast()

my_var = ENV.MY_VAR  # Equals boolean True

ENV.NEW_VAR = 100  # Sets a new environment variable

As you can see from the example, to work with the new class it is enough to import it (you do not need to create an instance - minus the extra action). Access to any environment variable is achieved by referring to it as an ENV class variable, which, in fact, makes this class an intuitive wrapper for the native system environment, simultaneously turning it into a possible variant of the configuration object of almost any system (a similar approach, for example, is achieved in Django , only there the configuration object is directly the module / settings package).

Enabling / disabling the automatic typing support mode is achieved using two methods - enable_automatic_type_cast () and disable_automatic_type_cast (). This can be convenient if the environment variable contains a serialized JSON-like object or even just a Boolean constant (explicitly setting the DEBUG variable in Django by comparing the environment variable with “valid” strings is one of the most common cases). But now there is no need to explicitly convert the lines - most of the necessary actions are already embedded in the bowels of the library and are only waiting for a signal to action. :) In general, typing works transparently and supports almost all available built-in data types (frozenset, complex, and bytes have not been tested).

The support requirement for Python 2 was implemented with virtually no sacrifice (the rejection of typing and some sugar candy from the latest versions of Python 3), in particular, thanks to the ubiquitous six (to solve the problems of using metaclasses).

But there are a few limitations:

  • Support for Python 3 implies version 3.5 and higher (their presence in your project is the result of either laziness or lack of need for improvements, because it is difficult to come up with an objective reason why you are still sitting on 3.4);
  • In Python 2.7, the library does not support deserialization of set literals. Description here . But, if someone wants to implement - welcome :);

The library also professes an exception mechanism in case of parsing errors. If a string could not be recognized by any of the available analyzers, the value remains string (rather, for reasons of convenience and backward compatibility with the usual logic of variables in Bash).

Python-shell library


Now I’ll talk about the second library (I’ll omit the description of the shortcomings of the available analogues - it is similar to that described for smart-env. Analogs - here and here ).

In general, the idea of ​​implementation and requirements for it are similar to those described for smart-env, as can be seen from the example:

from python_shell import Shell

Shell.ls('-l', '$HOME')  # Equals "ls -l $HOME"

command = Shell.whoami()  # Equals "whoami"
print(command.output)  # prints your current user name

print(command.command)  # prints "whoami"
print(command.return_code)  # prints "0"
print(command.arguments)  # prints ""

Shell.mkdir('-p', '/tmp/new_folder')  # makes a new folder

The idea is this:

  1. A single class that represents Bash in the Python world;
  2. Each Bash command is called as a function of the Shell class;
  3. The call parameters of each function are then forwarded to the call to the corresponding Bash command;
  4. Each command is executed "here and now" at the time of its call, i.e. works synchronous approach;
  5. it is possible to access the exhaust command in stdout, as well as its return code;
  6. If the command is absent in the system, an exception is thrown.

As with smart-env, support for Python 2 is provided (although it took a little more sacrificial blood) and there is no support for Python 3.0-3.4.

Library Development Plans


You can use the libraries now: both are laid out on the official PyPI. Sources are available on Github (see below).

Both libraries will be developed taking into account feedback collected from those interested. And, if in smart-env it may be difficult to come up with a variety of new features, then there is definitely something else to add in python-shell:

  • support for non-blocking calls;
  • the possibility of interactive communication with the team (work with stdin);
  • adding new properties (for example, property to get exhaust from stderr);
  • implementation of the catalog of available commands (for use with the dir () function);
  • etc.

References


  1. Smart-env library: Github and PyPI
  2. Python-shell library: github and pypi
  3. Telegram channel for library updates


UPD 02/23/2020:
* Repositories have been moved, the corresponding links have been updated
* The python-shell version == 1.0.1 is being prepared for release on 02/29/2020. Among the changes are support for the autocomplete of commands and the dir (Shell) command, the launch of commands with a Python invalid identifier, and bug fixes.

UPD 03/01/2020:
* Post on the next release.

More articles:


All Articles