👨🏼‍🏫 📕 ☕️ Automation testing software QIWI-terminals ◼️ 🧔🏽 ✊🏽

Hello, Habr!

Today we’ll talk about a specific topic: software testing automation for QIWI self-service terminals.

There are areas in the topic of testing automation that are scanned up and down several times, for example, testing web services. For such areas, there are separate tools, patterns, and best practices. You do not need to invent anything, the risks are minimal, you take and do.

There are reverse situations. The subject area is specific, no one has a peep at the ready-made solutions, there are no tools, the technological stack of the product is peculiar. You have to dive deep into the subject area, from sticks and uh ... other improvised materials to craft tools for yourself and simultaneously collect many, many rakes of different sizes and lethal force.

I wanted to talk about something like this today. The article is suitable for those involved in the development and testing of software for bank terminals or self-service machines. And also to those who want to expand their technical horizons with examples of "but it also happens like this." QIWI-terminal in 2020. In the background, you can see its filling.

Problem

The first QIWI terminal appeared in 2004 and then was able to replenish the account of a mobile phone. Since then, a lot of water has flowed, and ASO (self-service machines) have changed a lot. Now with the help of them you can replenish bank cards, make transfers, pay for services of various suppliers (providers) - housing and communal services, traffic police fines, credit organizations, our own QIWI products (QIWI wallet, interest-free installment card Conscience). QIWI terminals can also operate in the mode of postamata (automated points of delivery for online stores).

What is a terminal like?

The ASO terminal, from the point of view of iron, is an ordinary desktop with specific peripherals, such as bill acceptors and coin acceptors, receipt printers, dispensers, POS terminals (receiving plastic cards), etc.

And here is the first problem.

This famous photograph was taken in Sonoma County, California. They say that these are natural colors, without any processing. By the way, it was on the territory of Sonoma County that the southernmost Russian colony in America was located.

Yes, that's her. The operating system, which was discontinued in 2014, is Windows XP. The POSReady version was pulled right up to 2019, but ... It's not so simple.

The fact is that the terminals are not the property of QIWI. They belong to partners, the so-called Agents, individual entrepreneurs and legal entities that conclude an agreement with QIWI purchase terminals, rent space in shopping centers and receive almost passive income.

Accordingly, 90% of the fleet of QIWI terminals (and there are more than 100k in Russia and foreign countries) are running Windows XP on a variety of hardware - from 512 MB to 4 GB of RAM, a variety of CPUs (from Pentium 4 deep zero years to more- less modern Core i5) and different quality and speed of the Internet. Terminals of different ages, some of them are regularly upgraded, and some of them have not been upgraded for a very long time (it works - don’t touch it!). Our task is to regularly supply the latest terminal software updates (it is called MarATL) and maintain compatibility with all this variety of toppings and peripherals.

Expired Support Operating System

Imagine a simple test automation circuit. We have a CI server, for example, TeamCity or Jenkins. Our terminal software is collected from raws by event (commit or heap), the collected binary file is laid out somewhere. The software automatically installs, the night functional autotests are launched ... Uh, stop! Where is the software installation going? And How?

As I said above, 90 percent of the terminals are running Windows XP, which ended in 2014. This means that this OS no longer complies with security policies for a long time, no updates are released for it, there is no fresh working software for it, and even the native Microsoft Visual Studio compiles C ++ for it only after dancing with a tambourine, or rather with the version of the toolchain for the MSBuild collector.

In general, running tests and some kind of software on a terminal or virtual machine with Windows XP is a very bad idea. You cannot raise Buildagent TeamCity on XP, you cannot configure such a machine in Ansible playbook, there are no containers there either. Virtualization templates for Windows XP are also not that much. In any large company that thinks about information security, a Windows XP machine will be kept away from the corporate network, or even less the domain. That is, all interaction with the terminal (or virtual machine pretending to be it) must occur remotely, the terminal itself must have a minimum of third-party software.

Decision

Some people are surprised when they hear about a bunch of OpenSSH and Windows XP. In a modern version of Microsoft, OSH support is included directly in the system. In the not-so-modern (Windows 7), there are SSH installers using PowerShell. C Windows XP, the situation is a little worse, but only a little. There is an ancient version of the installer that runs on it without additional software.

SSH is an old, good, reliable way to remotely control a host, a well-known technology. You need to implement classes of SSH connectors that can do several things in parallel. For example, in the online mode, load fresh logs from the terminal, execute some commands (copying files, running scripts, issuing rights, obtaining a list of processes, monitoring the terminal’s system time, etc.).

Monitoring system time is especially interesting. By standard command-line tools, Windows XP displays time only in a formatted form hours-minutes-seconds, etc., no UNIX-time for you. The ambush, for example, is that XP has not received timezone updates for a long time, and our Russian government is known for constant games with the cancellation of summer or winter time.

For example, the standard Windows XP SP3 in the Moscow time zone shows the time one hour earlier than the real Moscow time. Accordingly, the timestamps of the terminal logs in any case do not coincide with the time on the test circuit.

Payments-Payments-Payments ...

The QIWI terminal interface is essentially a web that runs on a browser engine, which is usually quite old. According to the websocket protocol , it interacts with MarATL, actually a terminal software. When choosing a payment on the terminal (for example, paying for cellular communications) using a series of commands, a payment provider is selected, the amount of commissions, restrictions on accepting bills are determined, a payment is created for a specific provider, which is then sent through payment terminal processing to payments.

Interacting with the web interface remotely is an so-so idea, especially interface tests - another task. You can create a test web page that will be based on the test time instead of the standard index.html. The page runs a JS script that works with terminal software, and on the other hand creates a web socket client that looks outward towards the host on which the tests are running.

On the side of autotests, you need to implement a web socket server that waits at the start of client connection from the terminal and sends commands that are unchanged through the test page to the terminal software. Software responses are also sent to the test web socket server.

That is, you can describe the set of commands for the payment test in the form of JSON, which successively describes which commands you need to send and what kind of response to wait for them.

Interface test page. Displays the commands passing through it.

And where is the money?

The most insidious technical task is to simulate the peripheral devices of the terminal. The receipt printer is simulated trivially, a virtual printer is created in the OS that prints the data to a file. You can get this check from the terminal (or its copy, formed by the terminal software itself) and decode it, simultaneously checking, for example, that all necessary fields are present in this check.

More complicated with devices that accept money - bill acceptors and coin acceptors. It is clear that if we want to test the payment scenario, we need to somehow emulate the presence of the bill acceptor on the terminal, support the functionality of accepting bills, returning, simulating various problems (for example, returning a crumpled bill).

Most money receiving devices operate using the CashCode NET protocol. This protocol defines the scenarios of interaction between the bill acceptor and the controller (in our case, the controller is our terminal software).

An example of the interaction of the controller and the validator of banknotes using the CashCode protocol. With a certain frequency, the controller requests the status of the device.

The minimum size of the CashCode command is 6 bytes. For a description of the commands, see the CashCode NET protocol specification.

The standard bill acceptor is connected to the terminal via the COM port. There are third-party utilities that allow you to create a virtual serial port on the machine, for example, VSPE . Even scripts for forwarding this port through a TCP connection are supported.

Our case is simpler, we need a tool that can cling to the port using WinAPI and has an arbitrary convenient interface, for example, the simplest stdin / stdout, which can be hooked up using SSH.

Tula in a separate stream communicates with the controller via a serial port and, if necessary, imitates situations of receiving supposedly “money”. It is also necessary to have some kind of pool of test accounts or providers, the payments for which will not be processed for real, but will at some point be repulsed. Or, for more complex cases, the money does go through real processing, but gets to special accounts, from which then again they will be debited for payment tests.

Such is the cycle of money in nature.

Receipt printer (above) and bill acceptor (below).

24/7

Software for terminals and ATMs should be extremely resistant to various emergency situations - lack of Internet, problems with a bill acceptor, etc. Since such software runs on a terminal with a very high priority and rights, it essentially intercepts all OS control. You can’t just collapse it or close it with Alt-F4. Software can reboot itself and the terminal, incl. In a loop. It is also necessary to check such stressful scenarios, for example, the lack of communication with payment processing. The terminal responds to this with periodic reboots, you need to check that it does it correctly and correctly every time it loads the terminal software.

Installation from scratch and update

Installed terminal software is centrally updated using payment processing. Processing plans to update a specific terminal and determines the update profile for it, that is, what files and where to load from. To update, it is necessary to perform a number of procedures in the processing database, and then monitor the terminal. Check by logs that the download of necessary files has begun and that the update has been successfully completed and the terminal software has risen.

A clean installation from scratch is harder. Usually, a person does it in the fields by simply copying the installation file to the terminal and entering settings and authorization data into the installer forms, which is usually a WinAPI application with standard Windows controllers (TextBox, CheckBox, etc.).

In our automatic tests for a clean installation script, a Python script is attached to the terminal, which the installer starts, interacts with the controllers using the pywinauto library and writes its own installation log. After the first stage of the installation is completed, the script ends, and the terminal software passes authorization in processing, receives paths to the installation profile, and downloads all the necessary files. Here, in real time, you need to monitor the logs on the terminal through SSH. All abnormal situations during installation (for example, incorrect authorization data) are written to it, you need to read these logs online and, if necessary, consider the clean installation test failed.

Clean installation from scratch MarATL

Conclusion

We went over overviews on the main technical aspects of creating autotests for terminal software. Without diving deep into technology, we sorted out a big task into separate aspects. Ask questions in the comments, if the topic is interesting, you can highlight some points in a separate article in more detail. Thank you for the attention!

Automation testing software QIWI-terminals