Telerik blogs
Test StudioT2 Light_1200x303

Running automated tests on Microsoft-hosted agents in Azure Pipelines helps you eliminate testing delays and speed up software delivery.

Speed is the biggest promise of test automation. For many teams, especially the ones that already practice continuous testing, the key to speeding up their testing cycles is reducing test execution time.

In this step-by-step guide, I will demonstrate how to set up Azure pipelines for running your automated tests on Microsoft-hosted agents.

What Are the Microsoft-Hosted Agents?

Microsoft-hosted agents (MHA) are virtual machines (VMs) created from Microsoft-supported images in Azure DevOps. There are several predefined images, equipped with latest versions of the Chrome and Edge browsers, Visual Studio and other tools, and with the actual agent that automatically connects to the agent pool in the Azure project. Each VM is created when a job is started and gets discarded after the pipeline run finishes.

The Microsoft-hosted agents’ VMs are set to run with an administrator user by default. A great advantage is that these templates get regular updates by the Azure DevOps team (once per week), so you don’t to worry about having the latest browser version for example.

A key aspect is that a Microsoft-hosted agents’ machine does not have a web or a desktop interface and does not allow for a remote connection using RDP (Remote Desktop Protocol). So, for automation testing purposes, which is the topic of this guide, Microsoft recommends Microsoft-hosted agents for headless testing and self-hosted agents for “headed” UI testing. Headless testing in Test Studio also enables you to speed up test execution up to three times, which is a key aspect of integrating testing into your delivery pipeline.

Benefits of Running Your Pipeline on Microsoft-Hosted Agents

One of the key benefits of using Microsoft-hosted agents is that this technology is being supported by Microsoft, so maintenance and tooling upgrades come out of the box. On top of this, you can deploy custom software during the pipeline run—such as the Test Studio Test Runner—install it on the VM and set it up to execute tests.

You can get some performance advantages related to running builds faster, as using VMs hosted by Microsoft allows you to use their resources so you won’t put any burden on yours, not to mention the capability you get to set up ready-to-use and faster-to-deploy virtual machines.

Relying on the same environment in the expected state without additional configuration is another reason to run your pipeline on Microsoft-hosted agents. Automated tests can benefit from all the above advantages as well—running your automated tests on MHA allows you to scale up your testing setup easier, faster and more reliably.

Running Headless Tests on Microsoft-Hosted Agents Step by Step

Running Test Studio headless tests on Microsoft-hosted agents is accomplished by following a process that involves configuring and installing the prerequisites needed for testing, executing the tests and gathering feedback through the test results.

Test Studio Runtime Setup

To be able to run your tests on a MHA virtual machine, you need to install the Test Studio Test Runner on the VM once the latter is deployed and running. The recommended way of doing that is by setting the Test Studio Run-Time .msi installer as an Artifact in the Azure DevOps project. That allows you to deploy and install the .msi in the MHA as soon as it gets created.

Make sure to check out the detailed technical documentation on how to set up the Azure pipeline with a Microsoft-hosted agent.

You can use the Azure feature of publishing and downloading universal packages to add the *.msi installer in the project feed as an Artifact and make it available for the pipeline download task. To connect to a project feed, you need the latest Azure CLI installed—after you log in you can use the publish commands to upload the Test Studio Test Runner.

Test Execution on Microsoft-hosted agents

Azure Pipeline Setup for Test Studio Tests

Now that the Test Studio Run-Time is available in the project, it is time to set up the Azure Pipeline, which will run the automated tests on the MHA. Since we will be doing headless testing here, we need to employ Headless Chrome as the execution browser.

Tests that are set to run on headed-UI browsers (browsers the UI of which is rendered onscreen) are not supported in Microsoft-hosted agents. Make sure that the test list you are going to use is set to execute tests only in Headless Chrome.

  1. Start by creating a free-form pipeline. Make sure that you have your Test Studio project with tests in source control and add a task to get the source on the virtual machine. I am using GitHub repository for this demo.

    Add a task to get the source on the virtual machine

  2. Add a task to download the Run-Time installer Artifact to the MHA. The task to use is “Download Universal Packages”. Note that it is important to set the Feed and Artifact names correctly as they were used during publishing the package.

    Running tests on Microsoft-hosted agents

  3. The next task in the pipeline is to initiate the installation of the Test Studio Run-Time. Based on the details in the previous “Download Package” task, the installer is downloaded on drive C: on the MHA machine. You can use a command line task and list an msiexec command for silent install of the *.msi file—this is the “passive install” in this case. An example command is listed in the screenshot.

    Running tests on Microsoft-hosted agents - msiexec.exe /i c:\TestStudio_Runtime_2021_3_1103_1.msi /passive /le c:\errorlog.txt

  4. Once the installation is completed, the next task can initiate the test execution. You can use the Test Studio CLI runner capabilities to trigger the test list run. For this you can create a new command line task and add the proper command. The options I went with set custom name for the Test Studio result file and generate an additional junit-formatted result file.

    Initiate test execution by triggering the test list run - where am i  / dir /

    There are two important aspects when setting this task. One is to avoid using hard-coded file paths, as there is no guarantee that in future versions of the MHA templates the paths will be the same. Instead, you can rely on the Azure predefined variables, such as the $(Build.Repository.LocalPath), which will always point to the folder with the pipeline sources.

    The second key setting of this task is to set the “Continue on error” flag for it—that way, even if the test list run is failing, the pipeline will continue executing its tasks and will proceed to the test result publishing regardless of the test execution results.

  5. When the test list run completes, you can send the Test Studio result files as an Artifact in the pipeline. Тhat way you can review these at any time after the pipeline runs. Create a task to “Publish Pipeline Artifact” and include the necessary information to get the generated *.aiiresult file.

    Send the Test Studio result files as an Artifact in the pipeline

  6. In addition, you can populate the junit-formatted results from the test list run to the pipeline build summary. Use the “Publish Test Results” task.

    Populate the junit results to the pipeline build summary

With this, the pipeline is set and you can run it to trigger the test list execution in Headless Chrome mode. After the build is completed, the pipeline overall result appears in the build summary. The junit test results appear under Tests Plans/Runs, and the Test Studio results are published as pipeline Artifacts.


Running automated tests on Microsoft-hosted agents is yet another efficient and reliable way to eliminate delays and speed up testing as part of your delivery process. Especially if you are already using Azure DevOps for running pipelines. Doing it once, based on the provided step-by-step guide, will enable you to successfully integrate test execution into your CI/CD setup. Don’t believe me? Try it out for yourself.

Try Now

Miroslav Shtilianov is а senior QA in the Telerik Mobile Testing Team
About the Author

Miroslav Shtilianov

Miroslav Shtilianov is a Principal QA in the Telerik Mobile Testing Team. Prior to Progress, he was working as an automation engineer for VMWare.

Related Posts


Comments are disabled in preview mode.