Be a CI testing superhero with Test Studio! In this post, you'll learn how to use Test Studio in a TFS build and get detailed results.
There is one immutable thing in the continuous integration/continuous delivery (CI/CD) process, no matter what build scripts are running and what tools are used, there is always a testing phase where you need to have some automation tests set.
In this article I'll walk you through the process of using Test Studio tests in a TFS build and getting detailed test results for each run. I'll focus directly on the testing phase assuming the build and deploy phases are already successfully set.
There are two basic requirements for any test tool to successfully run tests in a CI environment. The first thing is to have the CLI tools necessary for managing the test execution. The second is to have an easy way to report test results and show them in a meaningful report. From the test tool perspective, this means to be able to output test results in a commonly used format, like JUnit for example.
Fortunately, Test Studio has both capabilities. I'll show, in a few easy steps, how you can use Test Studio for testing in the TFS build process. I'll cover two approaches, one using the Test Studio CLI runner provided with each edition of Test Studio (ArtOfTest.Runner.exe), and the second with the VS test runner (vstest.console.exe).
A note as we start - for more information, feel free to refer to the CLI documentation of the Test Studio runner at any time.
Using the Test Studio (TS) CLI runner (called ArtOfTest.Runner.exe) is easy. I'll now explain in a few steps how to include Test Studio tests in the build process using the TS Runner. As I mentioned above, we'll concentrate on the test execution phase and the reporting phase.
Add a simple execute batch script step in the build process.
To run your tests, you simply need to provide some command options to the CLI tool. First you set the test project root. Next you choose whether to run a single test or a test list. Finally you decide on the result output format. For this example, I chose JUnit because this format is natively supported by TFS and the results will appear in the build summary (we'll talk more about that in the next step).
You can find more details here about the run command task.
Hint: It's very useful to set your working directory (the working directory is the folder from where the agent executing your tasks is launched). In this case my test project is in TFS and the test files are downloaded in the working directory. This way you can use relative paths as in the example. Note that the TS Runner exe is also in the system path (this can also be done runtime if it's required).
Now that we have a task that will execute your tests we need to set the Control Options to "Continue on error." This is important because if the build stops on this task it will not perform the upload results task that we use later in the process. Don't worry though, the build will still fail.
The second important step is to get the results outputted from the test run and upload them to the server. This way you can check the result history in the build summary and build a historic trend of your fail/pass rate between each build. For this reason, a publish TS Results task is needed.
Setting up the "Publish Test Results" task is simple. You just set the expected results format to JUnit and select the results output folder.
If everything is set correctly, after the build completion you can check the test results.
From the results view you can easily check the pass/failure trend, failure details (if any) and other data. You can also directly associate the results with work items in TFS or log bugs.
Hint: You can also use the publish artifact task if you need to upload any 3rd party files to the build summary. For example, your test runs may output additional data like screenshots.
If you decide to use the VS CLI test runner (vstest.console.exe) the only different step in the process is in the test execute task. What you need is to use the run VS tests task instead.
In the VS test task, you need to simply specify that you will execute tests from assemblies, then type the names of the tests you want to be executed or use a regular expression.
There are some important points to clarify here. Note the check on the "Test mix contains UI tests," as it is mandatory to enable this option. You can also choose a settings file. Settings files are useful because there you can set some options specific to Test Studio tests.You can find more info in this article about VS test settings and Test Studio tests.
In conclusion, there is no single correct way to run your tests in a CI environment. In this article I showed you the essentials on how to integrate Test Studio tests in your environment but the possibilities are endless. You can use additional scripts (PowerShell for example) that can prepare the environment, or post build scripts for enhanced reporting, or find something else that works best for you.
Want to try them out? Start a free trial today:
Try Test Studio
Miroslav Shtilianov is a Principal QA in the Telerik Mobile Testing Team. Prior to Progress, he was working as an automation engineer for VMWare.