Get best of the week

How to organize testing in order to accelerate and stabilize product releases. Part 1

If teamwork is not agreed upon, collisions will constantly occur between individual process participants and entire teams, and company products or microservices within the same product will interfere with each other using common resources and infrastructure. The result will be permanent breakdowns, conflicts and slowdowns. Fast and predictable releases in such conditions will be unattainable.

My name is Victoria Dezhkina, I am engaged in testing in the Department of development and maintenance of big data products X5 Retail Group. I will tell you how we changed the testing process in one of our product teams in order to speed up the preparation of releases by almost half and relieve the team of stress. Now we are introducing this approach to testing in other products of the company.



The X5 Retail Group Big Data Directorate is currently developing 13 products, 4 of them are in the monetization department, where the products are oriented to the foreign market, and any mistake, whether it is a defect in a product or a belatedly released feature, has an economic effect and leads to loss of customers . In fact, these are internal teams that earn in the foreign market and play by the rules of small business within a large company.

All our products vary significantly in their goals and therefore require different approaches to development and testing, but they have a lot in common: they use the same infrastructure (Kubernetes, Greenplum, test servers), and developers from different product teams sometimes replace each other for a while vacations.

In such a situation, the role of agreements sharply increases: if one part of the team does not know what the other is doing, and each team has its own rules, problems will inevitably arise.

For example, two products use the same load testing infrastructure, and both urgently need to conduct tests. Without notifying each other, they do this at the same time, as a result, they get false results, because the DBMS has "laid down", and because of whom it is not clear. They wanted to save time in the negotiations, as a result, they lost a lot of time without any result.
Data loss is not excluded. Suppose one of the products occupies a free test server without warning anyone about it. Officially, the hardware is considered free, so another product enters there and accidentally erases all the test data of the first. This, of course, is not user data, but just test, but still quite an unpleasant case.

There may simply not be enough workers if the work is not planned in advance. For example, now stress testing at our Directorate works in a service format, that is, we select the appropriate specialist for teams upon request. If several teams, without warning in advance, come to demand load testing in one day, there may not be enough testers at all.

It seems that the easiest way out in such a situation is to establish uniform rules for interaction for all products. But the problem is that all products are different. Some of them are intended for internal users, that is, specialists from other departments of the company, for example, services for pricing or studying demand. The other part is developed for external users - for example, for suppliers. These products have completely different architectural logic and quality criteria.

Products at different stages of readiness also do not accept the same approach: at the early stage when the product is testing ideas, the priority is to understand the functionality for the user and the absence of threats to corporate security. When a product comes to support, other requirements come first: user convenience, stability of existing functionality, speed of response to defects and full compliance with corporate security policy.

These two factors — joint work on the same territory and unique product features — set the task for us : to develop rules that allow teams not to interfere with each other, but at the same time will not bind testers and developers hand and footand they won’t reduce the development of different products to one template, hacking all the advantages of agile and a product approach in the bud.

I will say a few words about why testers play a leading role in creating standards for interaction in product teams. The fact is that, due to the specifics of our work, we see the whole product, while developers usually focus on one small area. We are the first to notice problems and offer solutions to them, but the final solution is worked out together with the developers.

We already wrote how this collective work is going on: at the initial stage, we have to literally compose a single dictionary so as not to get confused in terms. But this is only the beginning. Next, we have to agree on a mass of very different nuances.

I’ll tell you how this happened on the example of one of our products - a procurement automation system for a distribution network. Its task is to ensure the operation of all processes from the moment the store has a need for certain goods until the moment he receives them.

What processes have changed in our product


When we arrived at the product, the releases were to be released every 2 weeks, but they were almost always late for a few days, and the day of release always meant that we would definitely stay at work and until the last we would stabilize the existing version. It was necessary to change something.

It is important to note that change is a common cause. Whatever their initiator, the tester, or the developers themselves, can’t do without the consent of the entire team and strong allies. Any changes need to be discussed by the whole team in order to collect as many ideas as possible and not miss out on possible risks. The delivery and product manager of our product and before me systematically worked to improve processes from both the technical and the product side. Having come to the team, I examined the process from the testing side, and together we thought out an agreed “strategy of changes”. The first point in it was a change in the layout of the code.

Code Layout


The calculation procedure is one of the main "pains" of team development, and there are very different problems here. For example, a team has only one branch with a code, and error correction does not stop there. Or there are several branches, but tasks with overlapping functionality may appear on the test environment at the same time, as a result, testers will not be able to localize the defect or check some of the new functions blocked by the defect of one of the tasks. And about desperate developers who do not consider it something bad to quickly fix the little things on the prod without warning others, I generally will not say anything.

To avoid all this, we needed to determine the number of branches and environments, as well as agree on the procedure for putting the code to the test.

The easiest way would be to split the process into two branches with "clean" and "dirty" code. But we had to meet many conditions:

  • « », « »
  • ,
  • .

We spent 2 hours discussing a new scheme of work and came to the following: we start three branches, two of them (release and master) will be with clean code. In the "master" is the current product version, in the "release" - features that are ready for rolling out. In Dev, testing takes place, here are tasks ready for the test, development is local. Rollout to each branch occurs in agreement with the tester. Here it is:



What did this give us in terms of testing:

  • Testing time for each task individually was reduced by 20%. It was no longer necessary to start the test again, if a new task rolled out without a warning, new tasks did not block the verification of already done ones, and the time for localization of defects accelerated.
  • By the planned day of fixing the release, 1-2 tasks remained unchecked instead of 4 (that is, the time to check them was halved).
  • The time for integration testing and release candidate testing has been reduced from 6 hours to 2 (counting the retest).
  • The number of defects found at the release stage has decreased. Previously, on the first release version, we found more than 10, but now no more than 4. The time for retesting has decreased by 2 hours.
  • There was an opportunity to continue development in parallel with testing.

The total time that we spent on testing, delaying rollout to the prod, was reduced by 8 hours. Just 2 hours of discussing a new scheme for working with the team - and managed to save a whole working day, which used to be spent every two weeks. Not bad?

But the problems remained.

  • , .
  • - , , .
  • .
  • , .

Bottom line: we continued to linger on work on release day.

We gathered again. Part of the problems was resolved by refining the development process and adding CI:



We set up automatic rolling out to all environments for almost a month, but this gave an acceleration in time of almost 2 working days. The team has been going for a long time to this, the distribution manager and the developers themselves have been doing this, but so far they have not been able to achieve two things: so that the roll-out to all environments is uniform and that at the same time the version control is preserved. Automatic rollout either violated the main principle of testing “at any given time, the tester must know what is in each environment”, or slowed down developers who could not complete the development of the task without permission to roll out the test.

This is a difficult dilemma. Ignore the first principle and instead of speeding up, you get a sharp decrease in the predictability of releases and erroneously deflated tasks. For example, if a defect is corrected already in conjunction with a “clean task”, then when rolling out a fixed task, it will certainly catch the defective one. Therefore, you will have to either wait for the correction of the defect of the related task, postponing the release date, or re-fix the defect already fixed.

Automatic roll-out is not a person, you will not agree with it. Therefore, we solved the problem with the remaining errors in a different way - a special approach to planning, and then adding another stand.

Planning for development and testing


Initially, when planning, the team and I discussed whether the task was clear to the developers, how long it would take and whether we would fit into the sprint. The tester estimated how long the testing would take.

At the same time, we did not discuss at all the errors that might occur: we did not foresee them in advance and did not include them in the list of possible tasks. As a result, when these cases got out during the testing process, they were added as a separate task, they needed time to work out and increased the release volume, and sometimes they were only found in the release candidate, at the stage of joint testing of tasks, postponing roll-out indefinitely. We gathered for three - a tester, delivery and product, and thought out a new interaction scheme.

Now we pronounce all possible errors before starting development. It took me to become a boring student, who at the planning stage asks everything and everything: “What will happen if the third-party service falls off?”, “And if we get null, but not 0?”, “What if we don’t get all the data? "," And if the Pechenegs attack? " and so on, but now we were ready for everything.

We also started talking about how we will implement this or that task and how we will test it. This allowed to reduce initiative (the invention of all kinds of bicycles, for example) and gave each team member an understanding of what his colleagues were doing. This, by the way, allowed us to abandon the acceptance criteria (AC).

To prevent the discussion in the new format from being too cumbersome, we began to do this not for 2 weeks in advance, but for a week.

The new code layout and task scheduling were just the first steps. In the next part of the article, which will be released tomorrow, I will talk in detail about how we changed a whole series of processes in the team:

  • The format for setting and accepting tasks and defects: they left user stories for the “use case + technical task” hybrid more convenient for us.
  • Testing moment: 5 points were set in the release cycle at which testers actively control the process of product creation.
  • The rules of interaction of the “backend - testing - frontend” connection: we developed a scheme that allowed us to check all the data that is transmitted between the backend and the frontend.
  • Acceleration of work on fixing defects: established rules on how to prioritize debugging tasks so as not to distract developers once again.

These measures allowed us to shorten the release cycle from 2.5 weeks to 1. The increase in speed may seem small, but the main achievement was that our releases became more stable and predictable - we got the opportunity to roll out releases "on command": we can get together any day, roll out tasks and by evening everything will be ready and debugged.

More articles:


All Articles