🕺🏽 🧑🏾‍🤝‍🧑🏽 🎅🏿 How the GitLab QA Team Uses the GitLab Performance Tool 👊🏻 🕟 🤳🏽

A translation of the article was prepared ahead of the start of the DevOps Practices and Tools course .

We launched a number of initiatives to improve the performance of GitLab and we needed a new benchmark tool.

Performance testing is a complex process that differs from other types of testing. To obtain results, dedicated resources, special approaches and tools are needed here. When I joined the company and became the first member of this team, my task was to transfer the nascent efforts to improve productivity to a larger scale. To do this, we made a new tool with the simple name GitLab Performance Too l (GPT).

We are pleased to announce the public release of GPT. In this post, we will describe how we use GPT to test GitLab performance, and also how you can use it to test your environments.

However, first look at what we use it with.

Reference architectures and test data

In our experience, the most difficult part in performance testing is not in the testing itself, but in setting up the correct environments and test data.

Thus, one of the initiatives is the development of GitLab Reference Architectures , which can cope with a large number of users. We want to use these architectures as a way to standardize our recommended configurations to provide customers with high-performance, scalable, and highly accessible GitLab deployments.

For such a tool, we need realistic test data, for example, large projects with commits and merge requests. For the first iteration, we took the project of GitLab itself.

After setting up and running the test environments, we were ready to test them using the GPT.

What is the GitLab Performance tool (GPT)?

GPT can be used to run various load tests for the GitLab benchmark. All that is required is to find out what throughput the test environment can withstand (in the form of requests per second), and ensure the preparation of test data.

GPT was developed on the basis of one of the leading tools for load testing - k6 . Here are some of the GPT features:

A large set of tests out of the box, covering various GitLab endpoints, with the ability to add your own tests. See the documentation for the description of the finished tests ; new tests are often added there.
, GitLab .
.
.
, time to first byte ( ), .

The talented Load Impact team created k6 , which is the core of the GPT. We quickly realized that we did not need to reinvent the wheel, as k6 satisfies most of our requirements: it is written in Go, which is why it is very productive and open source. Thanks to the team not only for developing k6, but also for working with us.

How do we use GPT

GPT is used in several automated GitLab CI pipelines to provide quick feedback on the operation of GitLab. CI pipelines of reference architectures work with the latest pre-release and, as a rule, run daily or weekly. Immediately after testing, we analyze the results to find problems. We publish all the latest results in the GPT wiki according to our Transparency value .

GPT is also used in the benchmarking pipeline to see how GitLab's performance changes from release to release. These results are important because they show a picture of improved performance. Benchmark comparison results are also available on the GPT wiki .

Using GPT, we were able to identify several bottlenecks in GitLab's performance and, together with developers, prioritize improvements. This process has paid off, and we look forward to improving performance with every release of GitLab. For example, release 12.6 showed several noticeable improvements in all areas, in one of which we achieved a decrease of 92%! You can see the detected problems in our bug tracker .

How can you use GPT

We have long decided that we want to follow the same principles of open source as for our main product, therefore, when creating the GPT, we focused on all users, and not on making it an exclusively internal product. Therefore, we not only allow others to use it, but also encourage it! This is beneficial both for us and our clients, as we receive feedback from different points of view, which we did not think about. Some examples of this are improving guidelines for recommended bandwidth-based specifications , or improving using GPT offline for users with private clouds .

If you want to use GPT for yourself, then it’s best to start with the documentation. As noted, most of the effort to use GPT is to prepare the appropriate environments. The documentation discusses both this point and the direct use of GPT.

GPT in action

Finally, after talking about the GPT, it's time to show it in action. Here's what the load test run for the List Group Projects API with a 10k reference architecture looks like : asciinema.org/a/O96Wc5fyxvLb1IDyviTwbujj8 See the GPT documentation for more information on the results .

What's next

Our goal is to make GitLab, in terms of performance, the best in its class. And we are only at the beginning of this journey. We are pleased that we can improve the quality of customer service by providing them with additional tools.

Some of our plans for future releases include expanding the test coverage of GitLab functionality, entry points (API, Web, Git), expanding work on reference architectures, test data, and user behavior patterns to make them more representative and realistic.

Share your GPT feedback and / or suggestions in the GPT project repository! We welcome your ideas and suggestions.

Ansible: Quick Start.