Key Trends in Software Testing from SmartBear: Test Types and Trends

In 2019, SmartBear held its annual survey of the community of software testers for the third time . Alyona Batitskaya, front-end developer, teacher and course leader at the Programming faculty in Netology, translated the research and collected key results on development methodologies, testing tools and technologies, automation, and industry development trends over the next two years.  

Who was interviewed

The company interviewed 2,526 people from different countries and industries, among which QA-engineers and automation engineers prevail - 37%. Also among the respondents were developers (15%), QA-managers (14%), architects (8%), manual testers (7%), product / business analytics managers (5%), consultants (4%), DevOps engineers (4%).


The respondents' professions are consistently different.

There are many types of software testing, and most often several approaches are applied. The average number of types of testing in respondent teams is 2.2. This shows that the testing departments are not hyperspecialized, that is, they do not use many testing approaches, but they do not stop at the same form.  


Most respondents test software or enter the appropriate teams
 
In relation to testers who work independently, there are more QA-managers who set and control tasks. Researchers explain this by the continuous improvement of respondents' qualifications and the transition to the next steps in the career ladder.  
  
Since 2017, the size of the companies in which respondents work has changed. Previously, everyone wanted to get into a big company (10,000+ employees), but now they prefer smaller companies (less than 500 employees).  

What was tested

Most tested web applications (79%) and APIs / web services (77%). 

Since 2017, the number of testing APIs / web services and mobile web applications has increased by about 10%. At the same time, there is a downward trend in testing of desktop applications, hybrid and progressive web applications, and boxed solutions. 

Testing time

The time spent on testing has not changed much since 2018 - on average, this is 61% of the working time.

The most time-consuming task 23% of respondents consider performing tests. Creation of automated tests in second place (18%), followed by writing scripts for manual testing (17%). Reporting / analysis of the test results remained at the same level - fourth place with 12% of respondents, but this is significantly higher than 3% who said that this was the most time-consuming task in 2017. 
 
Employees of large companies are more satisfied with the built-in software testing process. The reason may be that large organizations, in contrast to small ones, have the resources, history and knowledge for better testing. 
 
The following correlation was revealed: respondents who spent less than 20% of their time testing were less satisfied with the testing process, unlike those who tested from 20 to 80%. This may reflect either greater awareness of the processes and their convenience, or a lower level of interest and satisfaction among those who test less often. 
 

Test cycles

Companies continue to shift to faster software testing cycles.

The most popular combination of development methodologies is Agile and DevOps. This suggests that companies that already use Agile continue to improve in these processes and are starting to implement DevOps processes. 

Only one approach DevOps use both small and large companies. And the use of both approaches - Agile and DevOps - is observed more in large companies. 
 

Agile and DevOps methodologies are not growing fast, slowly taking market share from Waterfall. Most of the respondents indicated that they use a combination of approaches in the process of switching to DevOps.
 

Testing tools and technologies

Among the tools, Jenkins leads the third year in a row (52%). At the same time, the number of those who do not use CI / CD tools in their work decreases from year to year - in 2019 there were only 14% of the total. 


Despite the strong leadership of Jenkins, the market for testing tools looks fragmented.
 
The most common languages ​​for writing applications are Java (49%) and JavaScript (44%). The frequency of using C # (24%) and VB.net (11%) is decreasing, but the frequency of using Python is increasing (18%). The popularity of Swift, R, and Go continues to grow, but they all retain roughly the same level of response. 

The average number of programming languages ​​for writing applications has increased: from 2.36 to 2.54. 

A similar distribution among languages ​​has been preserved in the writing of automatic tests. Unless Python swapped places with C #. 
 

Test management 

A clear pattern emerged: the smaller the company, the more they do not like to use programs to manage the testing process.  


Small companies are less likely to use testing management tools. 

Among those who still use such software, there is not one single leader program:  

Stress Testing 

From 30% to 35% the number of respondents increased that load testing is carried out before each publication of the project.  


The popularity of load testing for different types of software
 
Among the load testing tools, the open-source Apache JMeter breaks out with a confident margin - 37% of testers choose it. The open source trend is growing in the IT community, so such a leader is not surprising. 

Second place - at LoadRunner (18%). Non-designated, “other” tools are used by 17%. 
 

 

Test Automation 

The degree of software coverage by automatic tests has not practically changed since 2017:
 

Only 18% of applications are covered by autotests by more than 75%.
 
And again, there is a pattern with the size of companies. 

The larger the company, the greater the degree of application coverage with autotests. 

 Maintaining the relevance of tests as the application changes and grows is still the biggest problem ⟶ 21% of the answers. Lack of time for testing is recorded by 15%. Insufficient qualifications of employees are noted by 12%.

UI Testing 

Difficulties and main challenges

In the list of the main difficulties in testing, the UI leads stability testing (18%).

A new answer “Start to do this” [they have not tested the UI before, are now ready to start. - approx. Ed.] immediately took second place with 17%.  
 
Distribution of other answers:
 

• Determination of the test object and process control (15%).
• Testing in different environments (devices, operating systems) (13%).
• Test coverage test (11%).
• Test service (11%).
• Synthetic data creation (9%).

 

Tools

Among the tools for automated testing of UI there are many applications with low popularity. 

For the second year in a row, Selenium Webdriver remains the leader (27%). In second place is TestNG (10%). TestComplete closes the top three (8%).



 

Running tests at the same time

Running several types of tests at the same time saves time on testing, which is crucial in the conditions of a short development cycle.  

At the same time, quite a few testers use this chip: only 30% of respondents run more than 10 tests at a time.

The number of tests launched simultaneously

Another regularity: almost everyone who does not run tests in parallel does not automate the tests. 

Almost everyone who uses scripts or other UI testing methods in their work is using parallel testing - unlike those who chose Record and Repeat as the main way to test UI. 

Choosing an Automation Approach vs Parallel Tests
 

Running tests in the cloud 

The number of those who run tests in the cloud is slowly but steadily growing. In 2017, there were 45% of such answers, and in 2019 already 50%.  

Popular browsers

The three leaders have not changed since 2017. The most popular browser for testing web applications is Chrome (53%). Next comes Firefox (41%) and Internet Explorer (35%). 

Chrome, FireFox and Internet Explorer fell sharply in 2018, but the first two grew by about 5% in 2019, but Internet Explorer continues to fall.  

38% of respondents test in three or more browsers. The vast majority (51%) are tested only in the latest version of browsers, 20% - in the last two, 11% - in the last three.  

Headless browsers 

The technology of headless browsers significantly increases the number of tests run in parallel, thereby accelerating the entire testing process by almost three times. So far, only Chrome and FireFox have headless versions. 
 
According to the survey, 25% of respondents use this technology. The remaining 75% are not aware of the use of headless browsers in the company or are sure that this technology is not implemented.  

Mobile Testing

Every year, the need for businesses to test the product on mobile devices is growing: only 15% of respondents said that they do not test mobile devices at all. 
 
The leader, like last year, is iOS devices. Android is on the heels. 
 

— , , . 

 

In 2018, 45% of the tests were automated, in 2019 - 46%. Although not substantial, but stable growth. 

According to the respondents' plans, by 2021 they want to automate 69% of the tests. QA engineers have the highest expectations.  

According to the survey, those with a current automation rate of 1–25% plan to increase it to 26–50%. And those with 26–50% want to get 51–75%. That is, no one claims absolute automation, really assessing their capabilities.  

The most effective option is a weekly release of the product, about 37% of respondents chose it. In second place is the monthly development cycle (28%). Third place is occupied by the quarterly release of the product (12%). 


Respondents achieved most of the goals that they set in 2017

The graph above allows you to compare the expectations of the respondents last year's survey regarding the frequency of publications and reality. The goals for weekly and monthly publication were not only achieved, but even exceeded expectations. But every day, or to publish several times a day did not work for many - expectations were overstated. 

Trends on the horizon for two years

Respondents tried to predict the directions that will gain popularity in the field of testing in the next two years. From the answers that were mentioned more than 10 times, we collected a cloud of simplified tags.  

According to forecasts, the most popular areas will be automation, artificial intelligence and machine learning. 

the main thing 

Release frequency continues to grow as more and more companies are moving from the Waterfall methodology to a combination of Agile and DevOps methodologies.

Testing is dominated by web applications and APIs.  

There are many different testing tools in use, among which manual testing is leading. 

The most difficult task for automated testing is synchronizing test scripts with the current version of the application. 

Scripting is the most common approach to test automation, and Selenium is the most popular scripting tool. 

According to the respondents' plans, by 2021 they want to automate 69% of the tests. 

Respondents still see the need to test in different browsers and browser versions, but the number of browser versions is insignificant, but decreased. 

A quarter of testers use headless browsers, while just over a quarter are not sure whether their organizations use this technology. 

In the next two years, the most popular areas in the testing industry will be automation, artificial intelligence, and machine learning.

From the editors of Netology

We invite you to get acquainted with the profession of a tester in the open lesson " Tester: who is it and how to become the main one in terms of quality in IT " - the next one will be held on April 23 at 19:00.

And for those who already want to get this IT profession, we suggest studying the program program “ Software Tester ”.