☺️ 🤘🏼 🔚 Azure DevOps and App Center instead of test farms: simplify and speed up mobile development with cloud services 🐺 😿 🏝️

High-quality tests ensure the speed and stable operation of mobile applications, but a variety of devices, operating systems and their versions inflates test farms, increases the development cost and takes a lot of time and effort. In this article, we’ll show you how to reduce testing costs using the Visual Studio App Center cloud service, which combines more than four thousand devices with different operating systems, and simplify distribution using the Azure DevOps platform.

How to test mobile applications and how a developer resembles a farmer

One of the main problems of mobile development is the crowded market. According to research by Adweek, Sensor Tower and Think with Google, 80% of applications do not reach users due to poor quality.

The success of most champion apps is fleeting. For a month, the top 25 is updated by 74%. It turns out that three quarters of the leaders lose their positions within 30 days! One of the main reasons for disappointment is problems with speed: 68% of users leave the application and choose another if it did not load within six seconds.

But the application should not only load quickly, but also work correctly. So rolling out updates without complex tests on all possible types of devices and operating systems is an unjustified risk. But in practice, this is not so simple: the insane amount of smartphones, tablets, operating systems and their versions forces us to create huge testing matrices.

If we consider mobile OSes, then with iOS things are more or less not bad. For example, last year most users sat quietly on iOS 12. It’s good from the point of view of tests that all devices are released by a single manufacturer.

Android is more complicated: there are many types of OSes and their versions, among which it is difficult to single out the main leaders. And each manufacturer (Samsung, Xiaomi, Sony, etc.) has its own non-standard launchers, hacks, changes in the OS and services. Because of this, mobile testing farms can contain dozens to hundreds of devices. For example, in the photo - a mobile testing farm from one of the game developers.

Farms require ongoing support. Devices must be kept up to date, periodically returning to default settings, taking into account depreciation, and so on. Devices in farms are depreciated and become obsolete; you need to constantly update them to keep up with those devices that your application audience uses. And most importantly, it is difficult to provide access to such a farm to a distributed team of developers and testers located in different cities.

After conducting "home" tests, you need to send the application to test on real devices. The best way is to give early access to the beta version to your employees or external people, for example, fans of the application and fans of new products. Users get the earliest versions, developers get the opportunity to test everything they want on them. At this stage, we need complex analytics: what functions the user uses, which screens most often gets to, how much time he spends there.

App Center –– an alternative to farms

Microsoft itself faced the problem of testing mobile applications: at some point, the number of applications under development at MS exceeded one hundred. To simplify this task, a cloud-based company from Xamarin created the App Center cloud service: it contains thousands of real devices on which developers can run tests without worrying about purchasing and maintaining their own farm.

App Center can be connected to GitHub, Bitbucket, GitLab, or Azure DevOps. Visual Studio App Center covers all the popular development frameworks, allowing you to create and test applications for all common platforms.

It’s as if the device is in your infrastructure, so you can organize the process of continuous integration (Continuous integration): the developer commits the code into the project, immediately the application automatically builds, spreads to the test devices, tests are launched.

Benefits of App Center

More than four thousand devices in the cloud. These are real living devices. Each model is presented in several copies. During the tests, you can take screenshots, look at the application through the eyes of users. There is detailed information about each device: who produced it, what resources it has, what CPU load, application execution speed.

The presence of the SDK - software development kit , which is embedded in the application and collects data: how successfully the installation and launch, how many failures, crashes. From devices on which something fell, you can collect dumps, watch what happened during the application test. This greatly facilitates the troubleshooting of the application.

Detailed test reports.If the test does not work with some kind of error, then a screenshot is taken from the screen of the device on which the test is being performed, at the moment when the test stumbles about something. And we see if the application is working or not working and where it stumbles on problems as a result of a test run.

Collection of device operation logs , memory dumps, stack traces, test reports to diagnose and fix problems. You can get into devices, pump out the logs and study what was the load on the processor, how much memory was occupied. Such things will help you quickly understand what the problem was, why the application crashed or the test failed.

Application Usage Analytics: how quickly it responds, how much time it takes to switch between screens, which paths users go through within the application and get to screens. You can find out on which platforms and OS versions your audience is using the application. This helps to see the audience and devices, to understand what features these devices can easily pull. And in general, can the application work correctly on user devices. Such analytics allows you to prioritize testing for various platforms and reduce the testing matrix.

Single automated distribution process. If the application is written for several markets, for example, for iOS and Android, then you have to do a lot of manual work in the process of publishing the application. The App Center has the ability to automate distribution, and as soon as the application has passed all the tests, the resulting build is sent to stores either automatically or after approval by the developer.

Automatic updates are sent to beta testers and internal user groups after each commit. Nothing needs to be sent manually, in addition, the process allows you to receive quick feedback from testers.

How to get started with the App Center

The App Center’s web interface is simple.

The process of creating a new application is no less simple: we specify the name, platform, and operating system.

We indicate in which repository the code lies, after which automatic synchronization starts.

When developing for iOS, a MacBook is usually required to build the application. But Appcenter allows you not to have it; he deploys a special virtual machine on OSX with all the necessary tools. You can add libraries if you need something non-standard, other than a clean system.

A window with a terminal opened. It can be seen what is happening inside the virtual machine. Packages are added, installed using NuGet. As soon as the virtual machine is ready, the compilation and assembly process starts. By the way, if there is a problem with the build, then logs can be easily downloaded from this interface.

How is compilation done? Microsoft negotiated with Apple and obtained a license for virtual MacOS instances that run inside the virtual machine to make builds. So from the point of view of licenses, everything is clean, Microsoft has taken care of everything!

The build is ready, it's time to test the application. Open the Device set and select the device.

For each device, you can view the model features, specifications, version of the operating system, the number of devices simultaneously available in the cloud, and other parameters.

Once your set of test devices has been created, tests will be run in turn on each of them. By default, in the basic version, testing takes place in a single thread. You can vary the testing matrix by running some tests on a large set of devices, some on a small one.

Estimated time to complete the test is a minute per application screen. If you have 30-40 screens, then the test will take about 30-40 minutes for each device. To speed up, you can put testing in multiple threads in parallel on multiple devices. You will have to pay extra for this option.

Here you can configure automatic distribution after passing the tests: the release will be automatically laid out in application stores.

If you need to test the application on a group of users, then before releasing to the marketplace we create groups that fit employees or users who are loyal to the application. As soon as we carry out the distribution, they will all receive an email with a unique link to download the new version. After such testing, you can collect telemetry from "test users" and see how the application works on their hardware.

This is where you connect to marketplaces.

To diagnose the application, you need to connect the SDK. And all the places of the code where something may fall, surround try and catch. When the application crashes on the user device, you can receive crash dumps and carry out diagnostics.

Analytics shows how users go about applications: put the same SDK and see which events appear inside the application, in which order visitors move between screens.

An option for collecting data, metrics, all kinds of overhead information. How quickly the application opens, how it works. All this is done on the basis of Azure Cosmos DB, where you can watch quite advanced analytics.

The SDK allows users to send push notifications to attract attention or inform about new versions of the application. Not only convenient, but also beneficial: such messages are many times cheaper than SMS.

Inside the application, users can be segmented according to various criteria: device types, geography, etc., and conduct marketing activities based on the resulting groups.

Core Azure DevOps components

App Center features — devices in the clouds, builds, tests, and software distribution — are just part of the development process that Azure DevOps complements. This platform where you can create dashboards and task boards, store knowledge about the project in the wiki, use the repository for files and artifacts. And if you have to work with a complex conveyor of software manufacturing, then here you can configure pipelines, specify where the builds will be.

Boards - a set of tools for planning work, discussion, monitoring, etc. Kanban boards, dashboards, Scrum boards - all this is easily customizable for the specifics of the project and the features of the team.
Pipelines — . . Node.js, Python, Java, PHP, Ruby, C/C++, .NET, Android iOS, Kubernetes.
Repos — Git. Marketplace - REST API. .
Artifacts — - Maven, npm, NuGet Python . CI/CD.

App Center Azure DevOps

By registering at Appcenter.ms you can use all the features of the App Center for two months for free. After the trial period, one build will cost $ 40 per month, but you can run it an unlimited number of times. This is clearly more profitable than having your own mobile test farm.

Azure DevOps is a completely free service if you have a team of a maximum of five developers or you are doing any OpenSource project. Repositories, code repositories, pipelines, boards - all this is provided for free.

If you want to know more about this or start using the platform now, then you will be interested in Dev Bootcamp - our online marathon for developers. During the event to be held25-29 , , , Open Source-, OpenSource . Microsoft, . , !