Running tests against multiple servers via Microsoft Test Manager

[Shashi]

Hello,

Like several other folks on this forum, we are trying to solve the problem of running our tests against multiple deployment environments (with different URLs) on which the SUT has been deployed.  

URL of the application landing page in various environments are of the following nature:

server1:  http://<server1>/
server2:  http://<server2>/

The tests were recorded against one of these environments and run successfully against that environment (both with WebUI 2010 R3).
Further, the tests are integrated with our TFS2010 CI environment and are setup to run whenever a new build is deployed to the above environment.  In this scenario, the tests are run through the command line version of Microsoft Test Manager (tcm.exe).  The tests can also be run manually from the Test Manager GUI.  So the ability to run tests under Test Manager is an important requirement for us. 

I have reviewed several threads on this forum and see that the recommendation for this scenario is to use the BaseURL field as described in the following KB article:

http://www.telerik.com/automated-testing-tools/support/kb/running-a-test-against-multiple-urls-using-baseurl.aspx

However, this solution will not work for us for the following reason:  This article recommends creating separate test lists (QA) and testsettings files (Dev with VS 2010).  Problem is that Microsoft Test Manager does not use the testsettings files created in Visual Studio (see the first response in the following thread on MSDN:
http://social.msdn.microsoft.com/Forums/en-NZ/vsmantest/thread/229fe0b3-5ebc-4e7e-9cd5-c63bbf1f7052 ).

I was wondering if anyone on this forum or at Telerik has figured out how to adapt the BaseURL solution to work in a Microsoft Test Manager environment as described above.  If so, I would appreciate it very much if you could share the solution or point me to any references that will help me find it (I have searched this site and Googled extensively - but there is always the possibility that the answer is buried somewhere that I did not think to look). 

If it is not possible to use the BaseURL solution in an MTM environment, I would also be interested in hearing any other approaches that have worked. 

Any help in this regard would be greatly appreciated.

Thanks,
Shashi

[Daniel Djambov]

Hello Shashi,

There is one possible workaround, which can work in your case, but is time consuming, as it has to be applied to all tests affected. Besides the mentioned in the article locations, we have the ability to define Base URL in the Navigate Step of each individual test:
1. Clear the Base URL setting in Project Settings -> Recording options (as it will override the step customization)
2. Open the test and expand the Navigate to step (double-click on it or use expand button)
3. Now you see there are 2 fields - NavigateUrl and BaseUrl
Here you can change the Base Url accordingly and it will run the test against different urls.

Drawback is that you have to do update every test that has a navigate step and you will run with Test Manager.

In the mean time we will look into general solution for solving the problem.

Greetings,
Daniel Djambov
the Telerik team

[Shashi]

Daniel,

Thanks for your suggestion regarding the BaseURL.  We looked at the option and it seems to me that you would need to change the BaseURL manually in all the Navigate statements whenever we want to run the tests against a different environment - is that correct?

If so, this wouldn't work for us as our tests are kicked off automatically after a build is deployed.  So any changes to the BaseURL field in the Navigate statements (or anywhere else) have to be made automatically - without human intervention.  

Right now, our best option seems to be to make the Navigate statement a coded step which then reads the website url from a file in the output directory of the test project.  This file will be updated by the application that invokes the tests.  However, I am still open to any better solutions that you or anyone else reading this might have.

Thanks,
Shashi 

[Stoich]

Hi Shashi,
     this might be the only way to do it at the moment. I'm not sure whether you'll be able to use Test Studio's data-driven feature with Microsoft Test Manager. If you can you can use it instead of creating custom code that reads data from a file. Check out this section of our user guide:
http://www.telerik.com/automated-testing-tools/support/documentation/user-guide/data-driven-testing.aspx

Best wishes,
Stoich
the Telerik team

[Scott]

I am interested in the best way to handle this as well. Has a solution for this been completed?

Thank you,
Scott

[Mark]

Hi Daniel,
What about in case of WPF project?
I tried to change the settings in "configure WPF application path" but nothing happened.
What is supposed to be easy become sometime hard and complicated :(

[Cody]

Hello Mark and Scott,

A good alternative to the .testsettings file has not been created. We need to somehow get Microsoft to wake up and recognize they have blocked many required test environment settings by not supporting the .testsettings file in the MTM world.

For WebTests the alternative is easier... you can data bind each Navigate step to a datasource then fill in the correct data at run time.

For the WPF path the problem is more complex. You only have two methods to specify the path to the application to test:
1) Hard coded in the .tstest file
2) Use the Default path specified in a Test Studio test list. This approach is not compatible with MTM. You must use ArtOfTest.Runner.exe to take this approach.

I've created a new feature request for the problem of specifying the path to the WPF application in an MTM world.

Regards,
Cody
the Telerik team