+ better, but less funny cover version Sooner or later, everyone comes to AT. The situation when this happens late is understandable, and when is it early? And how to understand what is already possible?The article is based on the experience of one team: I’ll tell you about our prerequisites and reasons for the introduction of autotesting, what criteria we have selected for AT readiness and what tools we use in the end. Spoiler: at the end, some successful and not very cases with Xamarin.UITest.Intro about the product. , , , , , . offline.
-.
What the theory says
Consider the automation approach using the test pyramid as an example. As the levels of the pyramid, I will take the types of testing that we use. Namely, unit tests, integration tests and end-to-end (E2E) tests.
Unit tests (unit tests) verify the operation of individual small parts of the system (function, method). They are small, pass quickly, do not require large development and maintenance costs.Integration tests test the application's operation with external components. For example, with a database or various services. They take more time to write and support tests, as well as their runs. For such tests, most often you need to configure and support some kind of environment. In this case, they often become more fragile, because environment tend to break.End-to-end tests verify the operation of the system as a whole, emulating the actions of the end user. Most often they access the system through the UI: they press the buttons as a user, fill in the fields, etc. But they can also be used in systems without an interface. These tests are much more expensive than those considered above, because require more time to develop and support both the tests themselves and the entire environment. In addition, they are very fragile, because they are affected by changes in any part of the system, and the unexpected behavior of an emulator or browser may affect the result of a specific test. The duration of the run of such tests is also significantly higher than the rest: to perform any individual user action, you need to make many additional ones. For example, log in or go to the desired menu item.According to the principle of the pyramid, the easier, faster and cheaper the test, the more they should be in a percentage of the total number of tests. Those. a through test should not check the assignment of different values to one field (unless you need to check the UI in these cases).For all of these types of testing assigned a role. At some point in the life cycle of the application, a need arises for one or another test automation. Your product can easily do with only unit tests, and can quickly grow to the need for end-to-end testing. The main thing when planning automation is to understand what exactly your product will currently benefit from, and for which you are likely to waste time.When implementing automation, do not forget about manual testing, even with a large amount of automation, there will always be things that need to be checked manually. At a minimum, new functionality for which it is not rational to immediately write tests.But all automation, in my opinion, should primarily be aimed at reducing the volume of manual testing. And when planning manual testing, it is worth considering which cases are already automated.How are auto tests organized with us?
Each product individually has unit tests, run on each PR. They are the responsibility of the developers.
Integration tests verify the interaction of the web service with the EDMS.They simulate the operation of a client application. They turn to web service methods for obtaining and modifying EDS data.Run after each build of a new version of the server side.
End-to-end tests simulate end-user performance. They press the buttons in the interface of the mobile application, and the application works with the EDMS through a web service.Integration and end-to-end tests are now being done by testers.Why do we need end-to-end AT mobile applications?
Before embarking on automation, it is worth considering what problem you want to solve by introducing it. Unless you have specific goals, you won’t be able to explain to the supervisor or product owner why you should spend time on autotests. It will be difficult to assess the benefits of their implementation.We have long tested mobile applications manually. While this was an application with little functionality, we successfully coped. Everything that was developed was ± in one area, and manual testing could be organized in such a way as to quickly and easily test all the features of the application.But after some time, the application began to grow with new features, which were quite independent and required more attention. There was a problem with bugs in the old functionality, which we could only find in the last stages of the project.Therefore, the initial goals of introducing UI tests of mobile applications for us were:- AT coverage of the main application cases;
- reducing the number of bugs in regression before release.
Achieving the first goal should automatically lead to the achievement of the second, as regular runs of tests for basic functionality throughout the project should find bugs at earlier stages.After some time, another goal was added - reducing the number of manual regression tests.And when is it time to introduce automation into testing?
Before embarking on test automation, you should evaluate the readiness of your application for automation. And also your readiness for this process.Stable functionality
For AT to have a chance to be useful, the application must have stable functionality. And it is her worth covering autotests. Otherwise, you will spend a lot of time and effort on updating and updating tests as a result of application changes.Development plans
Automation should be implemented only if your application has a future, development plans for the coming years. Why autotests for an application that does not change?Resources
You must have resources for developing tests, and most importantly, for their further support. Those. When planning the implementation of automation, it is important to understand that resources will not only be needed to write tests. Tests will certainly fall for some reason. The results of the runs will need to be analyzed and measures taken. Including change something in tests. Well, in addition to support, do not forget about the need for their development.How to decide?
When thinking about the UI tests of mobile applications, we immediately came up with large shelves with devices or farms on which devices are rented. There were a lot of problems in the tests because of the need to check different sizes and screen resolutions. Yes, and there was not much experience in development. It was all scary and forced to put plans away.The goals formulated earlier came to the rescue, and the main one was a functional test. Therefore, we started small.As a result, our tests:- run once a day (this is enough to find the source of the problem if something happens);
- run locally on our servers;
- on two devices (iOS and Android);
- for Android there are now about 50 tests, the run takes about an hour (together with the assembly);
- for iOS - about 40 tests, the run takes about 30 minutes;
- tests are written using Xamarin.UITest;
- they are launched automatically by builds in TFS, in the same place in TFS we monitor the results of runs.
A bit about Xamarin.UITest
This is a framework for automatic UI testing for projects on Xamarin.iOS and Xamarin.Android (support for Objective-C / Swift and Java is also announced). Tests are written in C # using NUnit. Content projects can be easily integrated.The principle of the framework’s operation is based on two components: searching for an element (Queries) and performing any actions with it (Actions), for example, pressing or executing a swipe.Here, for example, is a test that checks the error display when entering an invalid login:public void ShowErrIncorrectLoginOrPassword_IfLoginIsWrong()
{
var wrongLogin = TestsSettings.UserLogin + "1";
app.EnterLoginAndPassword(wrongLogin, TestsSettings.UserPassword);
app.WaitForElement(Resources.Identifiers.ErrorMessage, "Login is incorrect, alert message wasn't shown.", TestsSettings.Delay);
Assert.AreEqual(CoreResources.ErrIncorrectLoginOrPassword, ErrorMessage);
}
private string ErrorMessage =>
app.Query(x => x.Marked(Resources.Identifiers.ErrorMessage)).First().Text;
The credential input method used in it:public static void EnterLoginAndPassword(this AndroidApp app, string login, string password)
{
app.WaitForElement(Resources.Identifiers.LoginEdit, TestsSettings.Delay);
app.EnterText(Resources.Identifiers.LoginEdit, login);
app.EnterText(Resources.Identifiers.PasswordEdit, password);
app.Tap(Resources.Identifiers.LoginButton);
}
In this example, the standard framework methods are used - waiting for an element, entering text, clicking on an element.When developing and debugging tests, it is convenient to use the built-in console utility REPL (read-eval-print-loop), which allows you to view the tree of elements located now on the screen, as well as perform standard framework methods.To be unexpected for us
Addressing interface elements sometimes did not lead to what we expected.For example, when a new window in an application is opened with a fragment in the same activity, the user sees only this new window on the screen. But in this case there will be those elements in the tree of visible elements that the user does not now see with his eyes. The search for such an "invisible" element will be successful, and you can perform an action with it. That's just a click will be executed in the window that the user sees. Those. in fact, it will be a false positive.The tree may contain several identical elements; by default, the action will be performed with the first suitable element of the tree. And if in a test you accessed an element via a label, and then you have an element with the same label somewhere, the test will begin to fall (or maybe not if the element you need is the first).We had one case when a written and debugged test fell at the very first combat launch. Because the tests ran on a device with a different screen size. And the element that was clicked on on this device appeared under another interface element.
The Outbox folder was under the Create Task button, and clicking on the folder in this case led to clicking on the button. This is a familiar situation for the user, he will simply scroll the screen. But in the tree of elements this overlay is not visible.Such problems cause minor inconvenience and make you think through some tests more carefully. And sometimes we just fit the data in the database so that nothing interferes with the test. After all, the goal in this example is to open a folder, and not check that nothing prevents its opening.Another surprise and disappointment was the inability to run and debug iOS tests from Visual Studio for Windows. This feature has not yet been implemented. I have to work in the studio for MacOS.Some captain results
Implement automation if it makes sense.Let your tests be with a minimum of configurations, let them run manually and only once a month, but if they solve your problems and help you - why not?Well, do not forget about the principle of the pyramid.