Get best of the week

Why and how are we testing the update

image

In this article I will explain why it is so important not to forget about testing product updates and how this process works in our company. Update stability is a matter of product reputation and user confidence in your innovations. From my own experience, I can say that sometimes before starting an update, for example, on the phone, I prefer to wait at least a day and read the comments (they are always relevant only for the latest version). If the comments are abusive, then the probability that I will decide to update, tends to zero. The rating of the application due to negative comments falls, and it is not so easy to restore it, because you need to be able to interest the user in installing a new update, which he will now fear.

What are users usually unhappy with?

  • the application does not work at all after updating. For example, it simply does not know what to do with the data in the old format;
  • bonuses, discounts, user saves (games, shops, cafes) were lost;
  • the new feature for which the update is announced does not work;
  • a new feature works, but some of the old ones fell off;

All these problems are relevant both for the mobile application and for the DLP system that we are testing at home. Further about what we are dealing with.

What we release and what needs to be updated


Our company provides customers with a solution to protect against leakage of corporate information with the ability to configure separately for each organization, based on its needs. The main elements of the system are its settings (by what criteria will the intruders be searched) and events (recorded incidents).

The product consists of several parts, in this article we will consider the InfoWatch Traffic Monitor incident analysis and storage server. The product runs on Linux family OS, has its own database. The security officer uses the web console for work. Two different Linux distributions and two databases are currently supported. The system can be installed in several ways: all-in-one, when all the components on one machine; and distributed installation when system components exist on different physical machines.

In addition to the declared functionality of the system, we must guarantee an update from major versions N-1 and N-2 to N, as well as updating from all minor versions - patches and hotfixes of each version. This is due to the fact that our customers often have a rather complicated IT infrastructure, the update can take a long time, so it is important to limit the number of updates, not to carry out them too often, avoiding downtime.

All these conditions give a sufficiently large set of configurations of our product, for each of which we must guarantee a successful update. By this is meant an update, as a result of which:

  • user data is not lost
  • old features are not broken
  • new features are available for use

In this case, the priority of the requirements from the list above is in descending order of importance. For example, if you transfer priorities from the plane of working with DLP systems to the plane of user applications mentioned above, then saving the game user is probably more important than the ability to start a new game. And the user of food ordering applications doesn’t care how cute the new menu is if the “Send Order” button has stopped working.

Consider the example with our update table when major version 3 is released. For version 1.0.0, two patches and three hotfixes were released, and for version 2.0.0 there were two hotfixes.
1.0.02.0.03.0.0
1.0.12.0.1
1.0.22.0.2
1.1.0
1.1.1
1.2.0

In total, in the example we have nine versions, from which we must upgrade to the new version 3.0.0. There are two OS and two databases for each version of our product. Those. A total of 27 updated configurations are released. And if we take also different installation methods, then this number can be easily multiplied by two, as a result we get 54.

Each of our releases contains a number of serious (from the point of view of the impact on the product) features. Methods of adjusting the system can change, the analysis systems are refined, incidents are supplemented with new data, the version of the environment changes, for example, the version of the OS or database, etc.

Historically...


In time immemorial, we have developed a research approach when testing product updates. He proved himself in the conditions of a good knowledge of the product and a small number of environments and tested configurations. But time passed, the team grew and changed its composition: both testers and developers came and went, and some undocumented features were safely forgotten.

Planning research testing without good product knowledge is a thankless task, it was fraught with consequences like:

  • Testing updates each time took a different time. More often, time was allocated according to the residual principle, since the update was tested when the product is already stabilized and the final touches remain before the release.
  • Sometimes some environments were forgotten.
  • , , . , , , «» - .

Preliminary preparation was also carried out mainly at the discretion of the tester, who got the task and sometimes without taking into account all changes in the version.

In addition to the internal features of the testing process, the fact that the documentation did not describe the changes occurring with the functionality of the product added fuel to the fire. Thus, we could test not what was changing, but what was not affected at all by the current release.

As a result, it was impossible to clearly understand from the update testing report which checks were carried out, at what values, etc.

Nevertheless, for some time we were satisfied with the results of research testing: there were not so many military defects for the upgrade, and there were no resources for changes for the theoretical advantage.

Something went wrong


The situation changed when critical update defects to our customers nevertheless crept. Most often, they were found somewhere away from the main update scenarios and were associated with situations where, for example, an analysis technology element created and working in the previous version blocked the update, or some of the events in the client database were lost, or the most critical scenario when after the upgrade some of the services did not start and we received an inoperative product. Base-related defects have also begun to appear in our customers.

In this situation, the current process no longer satisfied us. Something had to be changed.
We started by formulating problems that, in our opinion, prevented us from testing the update better. Following the discussion, the following list of problems was formed:

  • ;
  • , ;
  • , , , , ;
  • , ;
  • ;
  • . ? ? ? ?


Further, for each identified problem, we tried to find a worthy solution. In addition to solving specific problems that we have formulated for ourselves, we decided in the course of discussions to put forward requirements for the testing process itself, on which we would like to work further.

The process should be made more open and transparent, for this we completely abandon research testing in favor of test cases.

To create the necessary test cases, we need information on changes between versions. This information needs to be requested from product developers and analysts.

In the new approach, we use a combination of checks on system objects (unchanged and changing during the update) + smoke-checks of the functionality (both old, new or changed old).

For the upgrade, the most difficult option to install the system will be selected - distributed installation. For all OS and DB. The simpler options are omitted as special cases.

This combination of checks will give us the opportunity to test the following system components:

  • DB
  • Web (frontend, backend);
  • Linux components

As a result, the solution for each of our problems was presented as follows:
There is not enough information about the changes in the current version. Insufficient knowledge of the system, not enough information about the system. Together with analysts and developers, we determine the changing areas of the product between the updated and current versions.

Testing updates turns into regression. Functional testing is performed, not system objects and operations on them. We will test the update on test cases in the form of smoke-tests of the functional + data verification.

image

Incomprehensible reporting. Coverage and results can be taken from test cases.

The process is opaque. After solving each individual problem, a new process was formed that was justified by our needs and also fixed in such a way that every member of the team could now familiarize himself with its principles.

New process


As a result, we built a rather effective process.

We divided testing into several stages, which gave even more unplanned bonuses, which are described below:

  1. Training.
    • We analyze the changes in the system, prepared jointly with analysts and developers for changing areas.
    • In accordance with the results of the analysis, we compile a list of system objects prepared for testing updates.
    • For each system object, the necessary set of states, statuses and parameters is determined.
    • We create stands with the necessary (updated) version of the product.
    • , .
    • ( ).
    • smoke- , .

    :

    • ;
    • (, , , );
    • , .



    • .

    • -, .
    • .
    • , .
    • Checks of operations on objects (creation, editing, deletion).
    • Checking the interaction of objects with other objects (detection).


Pros and cons of approach


We have achieved our goal, namely:

  1. The process has become transparent. We know that we are testing how, how much time it will take, and what artifacts will be output. We received objective criteria by which it was possible to give our verdict about a working or non-working product update.
  2. The reports became clear. The presence of test cases and a report on the results of their passage made it possible to quickly report to the project manager and technical director about the quality of the created assembly.
  3. Larger and more understandable coverage than with a research approach.
  4. Now we are testing a change in data and functionality. Thanks to the responsiveness of analysts and developers, we can say with a high degree of accuracy what has changed in the system and where there is a risk of catching a defect.

But running around a new testing strategy, we could not fail to encounter the obvious minus of the new approach - a significant increase in testing time.

We began to spend time not only on direct testing, as was the case with research testing. New time costs were associated primarily with preparation for testing, namely:

  • change analysis;
  • creation, filling, maintenance of stands for updating;
  • updating old test kits and creating new ones.

This minus could well become decisive for making a decision to abandon the new methodology, if not for one “but”.

All preparation, and this is the most time-consuming stage, could be carried out as soon as the final composition of the release was formed, even without a finished product. Thus, we “spread” the preparation according to the test plan, without overloading the overly saturated pre-release period. It still only had to pass the tests on the finished stabilized product.

In total, according to the results of the first stage of improvements, we received the following: we began to spend more time testing, but at the same time we have a transparent process, a clear display of the results and more coverage, which allows us to:

  • detect more defects as a result of checking product updates in your department;
  • reduce the number of defects when updating our customers by the implementation engineers;
  • reduce the number of defects received from technical support.

What's next? - about optimization


It could well have been settled on this, but time is always money. Moreover, already according to the results of the first break-in of the new methodology, ways to optimize time costs became more clearly visible.

We went in two ways:

  1. optimization based on analysis of test runs: here we were engaged in identifying obvious and implicit dependencies of test results on the environments used. This is the way functional testing has gone.
  2. test automation. Then our team of automation came to the rescue.

I’ll talk a little more about each path.

First way: optimization based on analysis of test runs.

This way we chose to optimize update testing between major versions, i.e. between those where there was a significant change in functionality.

The first thing we noticed was the tests that depend on the database and only on the database. We were able to understand what checks are enough to be done once at each base, and then to exclude them from our combinatorics with the OS completely.

The second step was the “collapse” of volumetric checks of the old functionality into a check list. In the first iterations, the functionality, regardless of whether it appeared in the current release, in the previous one or was always, relied on its own full-fledged test. Now, full-fledged tests rely only on new features, while the old functionality is checked on the checklist.

Moreover, the same checklist was very useful for us when releasing patches and hotfixes, where in addition to updates between major versions, it is also important to check for updates within the version.

The second way: test automation

First of all, I want to make a reservation that we did not plan to automate testing updates because it was accepted and generally good tone, but because updating is the type of testing that could not be excluded with any release, be it major release, patch or hotfix. We chose this path to reduce the time for testing updates for minor versions, namely patches and hotfixes, i.e. versions between which the composition of the functional does not change. In this case, the test automation looked very efficient.

At this stage, testing updates when releasing patches and hotfixes is almost entirely automated.

Testing updates when major versions are released is divided between manual and automated testing. Manually checks for changeable areas affected by features. Tests for invariable areas are automatically chased, most often it is more reuse of autotests already written for previous versions with little updating for a new one.

In order to start the process of automation of test cases, we had to further refine them, since the language of tests of a manual tester allows much more difficulties than an autotest can afford. That is, we also allocated time for the preparation of tests for automation, which really paid off almost the very first run.

Summary


We had a research process for testing updates. At some point, he ceased to satisfy us because of a noticeable drop in the quality of the update. We did not choose a replacement for research testing as a ready-made technique, but formed it based on the problems we identified. The formation of a new methodology, which we use and continue to improve, was influenced by prepared solutions to problems, as well as general wishes for the testing process.

I don’t think that in this situation we invented a bicycle when we took the path of creating our own process tailored specifically for our project and the features of our product. If we disassemble the process into its constituent parts, we obtain, in general, generally accepted and widely used techniques.

Despite the fact that the results of the work we have collected in one not too voluminous article, the whole process from understanding the problem to introducing new solutions and optimizing testing took us almost two years.

We tried to honestly describe here all the pros and cons, as well as the pitfalls of our decisions, so that the narrative does not look exclusively like a success story “it was bad - it became good”.

We hope you find our experience useful.

Material author:tryzhova (Ryzhova Tatyana).

More articles:


All Articles