Get best of the week

A man-laborer instead of a tester? Is it worth it to study Selenium in 2020?



Continuation of the interview “Who else needs Selenium? Does anyone use BDD in 2020? Machine Learning at Selenium »

The dialogue involved:

  • Vsevolod Brekelov and Daria Manukhina (program committee of the Heisenbug conference);
  • Anna Chernyshova, developer of the Akita library and the new Healenium tool ;
  • Ivan Krutov, developer of Selenoid .

At the end of the article you can find out the conclusion shared by Ivan and Anna, namely the answer to the question: “Should I study Selenium or choose something else?”. In addition, we managed to talk about:

  • Marker Man and find out who he is;
  • Valuable employees in companies;
  • Tester toolkit;
  • Infrastructure for Selenium tests;
  • Selenium competitors.

If you have not read the first part of the interview, then here are a few screenshots of what readers remembered the most: The



previous article cut short on the question of machine learning. Let's start with him.

- If machine learning is really so cool, what part will remain alive for people, testers?

Anya : At first, everything will need to be validated one way or another. This works well when we have a clear answer to the question that machine learning solves: yes or no. Depending on these yes or no, the machine makes a decision. When there is no clear answer to this question, it is already needed there ...

- A special marker man.

Ivan : Any machine learning consists of two parts.

Firstly, it is the training itself, when the algorithm is given some kind of sample of the correct data on which it learns, and then the application of the algorithm. The stage of application of the algorithm is when the machine helps us solve a large number of problems based on the correct solutions to a small number of problems. At the current stage, the training of these algorithms will in any case be on the human side.

Secondly, machine learning is primarily about search, all search engines on the Internet are very active in this. In order to improve the quality of these algorithms, one has to process huge amounts of data. The role of a person can be reduced to supporting an insanely large infrastructure that will analyze this data effectively. Perhaps the algorithm itself will do the work itself. But to ensure all conditions so that this algorithm can work and that it has good quality, it will still be human work.
To provide all the conditions so that the algorithm can work and that it has good quality, it will still be human work.
For example, to train some large number, some good algorithm, you need a distributed cluster of calculations, some kind of crazy MapReduce . For this to work, you still need people who either set it up and look for it to work normally, or who also write it specifically for this task. Machine learning is not a very simple topic. The more you want it to work better, the more you need to invest. Every next percentage of quality, this is some kind of enormous incredible increase not by percent, but by orders of magnitude.
At the initial stage, you can start it on your laptop, and then, the better you want it to work, the more you need to invest.
- That is, even with such machinery, we still will not get away from running tests on some kind of infrastructure, on some machines. This CI / CD with tests will not go anywhere. Therefore, for example, Selenoid will certainly not be a thing of the past, but the same cannot be said for the Selenium IDE.

Vanya : Selenium IDE is no longer subject to extinction, but to the fact that it will become a more niche tool, like everything I said before. He has teams to whom it is really convenient. Someone is comfortable writing autotests from scratch, someone is comfortable recording and reproducing. I am not inclined to bury everything like that, it seems to me that this is an evolutionary process.

- Well yes. “Cobol” is even in testing, it turns out.

Vanya : I’m talking about mainframes! Someone Cobol, mainframes and everything works. It seems to me that we are talking more about all sorts of technological problems, and there is another nuance. In the end, everything comes down to money when it all starts to swell.

If everything has somehow grown wildly for us, for example, the costs of machine learning, then at some point the question will generally arise: “Are these expenses generally justified?”

What is our task if we use Selenium? To test. It may turn out that at some point the costs of this machine learning development will be higher than just hiring people who sit down and call their hands.

Why do teams still exist, is there manual testing, is there automatic testing? Why didn’t manual testers disappear? Because there is a certain class of tasks that is cheaper to take and for people to test it with their hands. It's cheaper than writing autotests. Everything rests against the economy anyway, and there is no getting anywhere.

If everything has somehow grown wildly for us, for example, the costs of machine learning, then at some point the general question will arise: “Are these expenses generally justified?”


Who is a “helpful employee”?


“But how quickly a product reaches a user affects the economy.” Automation is just trying to shorten delivery time to increase profit and competitive advantage, right?

Vanya : Right. And an employee who perceives the activities of his company, as if it would be his personal company. He counts money, thinks at least on this topic, does not count them in real units, but thinks that this will probably be cheaper for us. Such an employee will always be valuable for a company that will say this at every meeting: “Look, guys, we will save now, for example, and therefore we will get more profit.” Even if he is a junior developer.

- It seems that I rarely met such people in the market.

Vanya : Well! They are few.Few people think .

- It turns out that everyone is used to solving the problem. And with the set of tools or those approaches that are already there, but few think whether they are really optimal from the point of view of business.

Anya : It depends on the development of the organization, on the culture that it promotes for its employees. I recently read the book "Discovering the Organizations of the Future" , there are different categories of organizations, they are divided by color.

Organizations of the future are turquoise companies, where every employee is sick and worried so that we do the most good for users, and everyone is included in the process. There is no strict gradation of what management is, there are lower employees, senior employees, each has its own separate area of ​​responsibility, as it were, it does not seem to go anywhere in other related fields - this is not there. Responsibility among all employees is shared. If a company begins to move in this direction, it begins to promote similar values ​​among its employees, and even if a person who does not have an idea in his head that this is how to do it, root for this business, for business, looks at his colleagues, begins to to study further, if he changes his place of work, then this too will begin to develop. This comes from the number of similar organizations that are now,and how many will appear.

Use of Selenium in 2020


- How is Selenium used now?

Vanya : There are several basic applications that I heard about. The first thing I know is functional testing. This is its intended use: to open a site, execute some user scripts, verify that everything works as it should.
The second thing I heard is the use for search engines (crawling pages on the Internet and analyzing their contents, screenshots). This allows you to understand whether this site is good or bad.

Anya : You can also use it as a UI-load testing.

Vania: It's a little expensive. There are probably people who do this, but in principle it is expensive. At least this is not one application, not only functional testing. There are still various tasks that require simply automated work with the browser. Some use this approach, for example, to send spam.

In fact, Selenium is just an API that allows code to work with browsers, that's all. Any task for which such a tool is suitable can use it, not necessarily testing, anything can be.

Tester toolkit


- To solve the problems of autotesting, what tools besides Selenium do engineers usually choose? They take Java, Allure, Selenoid, Selenide, twist it all together and get a good machine gun to solve their everyday tasks. Or is there something else?




Anya : Here you need to understand what levels this approach to automation will consist of. At each level, you can wind up some instrument.

The first level is to choose what to write on; is it Java, .NET, JS? I worked more with Java, and I will talk about it. Actually, on what to build the project - Java. You can assemble it using Maven or Gradle. Now Maven has a pom nickname that is cool in YAML, it is convenient to work with it.

Next, choose how to run these tests - these are some JUnit or TestNG. I have been working with JUnit 5 recently.

Then choose the level of interaction with the elements. This is Selenium, or some kind of wrapper over it, for example, Selenide. With it, you can shorten the time for writing tests.

Next, you need to check the test results. Here is a very large selection of tools. You can use the same Assertions from JUnit 5 , they are now quite conveniently made in it. Or the Hamcrest library , I really like it. Or AssertJ is also a convenient thing. When you select this runner to run the tests, you need to think about the parallel running of the tests, how it will be better organized. In JUnit 5, this is convenient; annotation is simply done there.

Then the test writing itself, it could be some BDD wrapper, the same Cucumber. If you choose JUnit, then additional things will be required to it.

Plus infrastructure. I’ve been working with Selenoid recently, it was the most convenient for me.

More reports.

- Well, reports are, of course, Allure ?

Anya : Well, or ReportPortal .
I can explain when Allure is better, when ReportPortal is better. Allure is good when a small project, then it ideally generally comes in. If this is some kind of large project, where 100500 thousand tests or this is an enterprise solution, or there is an understanding that there should be a lot of tests and they should all add up in some kind of report, then ReportPortal is good, it is more convenient to process the results of a large number of them tests. When there are few tests, then Allure is more convenient.

Vania: I don’t know whether you know or not that I also participated in the first Allure. Initially, Allure appeared on Yandex, as everyone knows, but in the end it turned out that the guys went separately to do it. As a result, they are just the functionality that ReportPortal has in it, all sorts of complex things: history storage, and so on, they began to do as a separate commercial product, so this does not appear in the usual Allure. And they are trying to sell it. If you want some open source report for a large number of tests, then probably ReportPortal, although I myself have never tried it. For small projects of some kind Allure, it seems to me, is for the eyes.

All these tools were written for complex tests with rather long scenarios, where it was necessary to visualize well all the steps that are done during the test. These reports are not particularly suitable for unit tests, because there is just the answer “yes / no”, fell - did not fall.

Infrastructure for Selenium Tests


- Vanya, can you tell us more about the server part and how often do people have to touch it? (to QA engineers, DevOps).


Image: Unsplash

Vanya : What is the server side? It must be some kind of web application that implements the Selenium standard. So that we can send commands according to Selenium standards to our client, and this server performs all the checks that we want to do for us. For Selenium, there are now several tools that implement it.

There is the same Selenium regular old, Selenium or Selenium Grid, or Selenium Server - this is an application written in Java, which is the oldest and easiest in terms of features. Three years ago from the Selenium standard, from the Selenium Grid, the Zalenium project budded. He already knows how to run browsers in Docker containers. This project implements the entire standard, supports the possibility of video recording, the ability to store logs, has an interface better than the standard Selenium.

We did a project from scratch called Selenoid . This is also a completely independent implementation of the Selenium protocol. It is written in Go, Java installation is not required, nothing is needed, it just starts in the binary and needs Docker.

In addition to open source, we make an implementation for Kubernetes, this is Moon . This is also a completely independent implementation, which is needed if you have Kubernetes. We emphasized that deploying the infrastructure was just easy with a couple of teams. People like it because you entered two teams and everything already works for you.

There are all sorts of online platforms for Selenium, if you don’t want to deploy Selenium itself. You can go to cloud services. They are quite expensive, but nevertheless they are, they are quite popular.

Anya : I had experience with SauceLabs , everything is also quite convenient there. You simply indicate which browser to run in, they even support mobile testing. And you launch. But they are expensive.

Cross browser and mobile browser testing


- How, from the point of view of cross-browser compatibility and mobile phones, does Selenium work, and are there any problems with the infrastructure with this? I know that some people test certain browsers on mobile phones. Fortunately, I did not test it with my hands in Selenium, I don’t know how difficult it is to set everything up.

Vanya : It's hemorrhoid and quite expensive. The goal is to test that if we open our application on some phone, in some mobile Chrome, everything works the same for us. Naturally, we want, as with desktop browsers, to do this with code.

The initial simple idea is to buy phones of different models, put them on the table. There are ready-made tools, for example, Appiumthat implement the Selenium standard. This is also an implementation of Selenium-extension, which allows you to work just with mobile. Initially, this was done just for the farm of real phones and tablets. The problem is that just the operating experience of such phone farms shows that it is very expensive. It constantly breaks down, you need to replace these phones, their batteries swell, it requires quite special systems that charge these phones, you need to put updates there, make sure nothing breaks there.

Now everything is slowly moving towards launching it all in emulators. There are special programs - emulators that show exactly the same thing that a user sees on his phone or tablet on the screen of a regular computer or server. There are emulators for Android and for iOS. The problem is that, from the point of view of Android, these are virtual machines, such emulators cannot be run on any hardware. If you want Android emulators, you need to take iron servers, it's expensive.

If you want to test on emulators for iOS, you need to take Apple hardware, that is, MacMini, MacPro, MacBook or something like that, it's also expensive. This is due to Apple’s licensing restrictions. Therefore, testing on mobile in principle is possible, it is clear how to do the infrastructure. Even in Docker, you can run Android, but it's quite expensive. If people want to do this, they need to think hard.

The main task of testing on mobile is to find bugs that play only on mobile. There are different ways to make it cheaper. It is possible to launch desktop browsers like Chrome, in which the user agent is poked, the desired screen resolution is poked. The decision on whether to test on real emulators, on phones, should be made based on whether you can catch bugs only on the emulator or on the phone.
The decision about whether to test on real emulators or on phones should be made based on whether you can catch bugs only on the emulator or on the phone.

Selenium competitors


- By the way, there are various tools like Puppeteer, Playwright, which allow you to accurately emulate and do cross-browser testing, including in mobile browsers. Maybe for a long time they all moved or transplanted?

Vanya : Frontenders, no one else moved.

Anya : These are cool things, but they have limitations in terms of cross-browser compatibility. Cypress for Firefox seems to be updating soon. You can write tests very cool very quickly, everything is convenient, but you are limited only by the Chromium.

Vanya : Let's start with Cypress . In 2004-2005, Selenium worked as follows.

The browser was launched, a special extension was loaded into it, into which commands for automating the browser were pushed. After 15 years, guys appeared who looked at it. Selenium abandoned this approach, because not everything can be done with the help of an extension. Since Javascript, it’s not possible to execute everything in the browser; you cannot access files on the file system. Selenium switched to a native approach, began to write separate binaries, separate from the browser. 15 years have passed. The JavaScript guys made a similar tool.

Puppeteer is exactly the same. Puppeteer is Google Chrome, Chromium, which implements a special protocol for working with this debug panel. Chrome has a debugging panel so that you can watch messages in the console, network requests, and so on.

- Developer tools is very cool, of course.

Vanya : Yes, it’s convenient for the developer. As it turned out, this thing interacts with the browser using a special protocol. The main idea is to not click on the mouse in this protocol, but to send commands in the same way as in Selenium. The guys just wrote a Javascript library that implements the protocol.

Anya : These tools, it seems to me, may be well suited for component testing, which frontenders themselves would cover. I know such application cases; they went very well right.

Vania: By and large, these tools are good, they are normal. They are just niche, they are well suited for front-end vendors, because they have their own tasks, they run some tests quickly locally, but the problem is that they do not cover the requirements that already exist.

There is a bunch of tests, for example, written in Java - a frantic volume, they need to be quickly run, they need to be quickly scaled and so on. This problem is not solved.

I am not saying that these tools are bad. In fact, even teams that make Cypress, or teams that make Puppeteer, say it's the same as Selenium. There is Gleb Bakhmutovwho does Cypress, he was asked: Is Cypress Selenium or not? He replied that this is a normal niche tool for front-end developers, and I agree. It seems to me that they have some kind of common functionality, solve some common problem, but still they have different areas of application.

- Guys, you have vast experience in using Selenium, if you were just starting UI testing now, what tool would you choose now? Where is it better to start?

Anya : I would start with Selenium anyway. Because it is a standard.

- Java and Selenium, right?

Anya : Well, you can not Java, you can .NET, Python, but still Selenium to them, because he lived, lives and will live.

Selenium lived, lives and will live.


Ivan : Suppose we need to cut some board and a person is a carpenter.
First, he needs to learn how to work with something standard, to learn how to saw with a saw, and then to take some kind of complex specialized machine tool.

Selenium is a simple tool that can be performed by everyone. There are 100,500 million people who will tell you how to saw a log with a saw. Something niche, on the one hand, it will be cool to cut in this niche, but for general-purpose tasks, Selenium is still the best there is at the moment.

Anya : Selenium is a base that you need to know in order to be able to develop further and add some interesting buns to it.

- Thanks you! I hope that the information received is useful to our readers.

We remind you that the Heisenbug 2020 Piter conference will be held online . There you can talk with Vanya and Anya and learn more about Healenium, Selenoid and the use of Chrome DevTools protocol in Kubernetes cluster.

For those who want to broaden their horizons and attend more than one conference, but immediately 8, we have prepared something .

More articles:


All Articles