Synchronize parallel test execution

[Konstantin]

As far as I know Test Studio supports parallel test execution. I wonder if there is a way to make tests execute on several test controllers (or several browser instances) in parallel if there are dependencies between tests?

For instance, test 2 can only execute after test 1 successfully finished. And test 4 can only run after test 3. But 1 and 3 can very well run in parallel. And all tests can run only after test zero finishes its very basic initialization of the system.

Can I specify dependencies between tests so that parallel execution obeys those?

Thank you!
Konstantin

[Cody]

Hello Konstantin,

I am sorry but Test Studio doesn't understand the concept of test dependencies (beyond a static test list running each test in sequence). There is no built-in mechanism to synchronize between multiple test execution servers.

If you don't mind writing a bit of code, it would be possible to implement this yourself i.e. create a test (or tests as needed) that does nothing more than run some code that will synchronize across multiple machines. This is a fairly advanced programming technique and beyond standard Test Studio support. We'll do our best to point you in the right direction if you wish to pursue this path.

The concept is that both machines start by running real, normal Test Studio tests, then runs another tests which is your code to synchronize, then runs more tests, and so on as needed by your testing architecture.

All the best,
Cody
the Telerik team

[Konstantin]

-- There is no built-in mechanism to synchronize between multiple test execution servers.

Good to know. At least I can stop looking for it :)

I totally don't mind to implement such distributed scheduler with dependencies and synchronizations myself (as a matter of fact I already have it for completely different purpose, so I can reuse). To use such scheduler/synchronizer I need the ability to 
1. Run any Test Studio test from my code and somehow figure when it finishes and what is the outcome (at least Pass or Fail)
2. Make this distributed run "a test", so I can report to Test Studio when I'm done and the outcome.

Could you please provide me with the code snippets to achieve that or point me to the appropriate samples?

Also, do I HAVE TO use Scheduling Server and Execution Server to run tests in parallel, if I'm going to start them in parallel on the multiple machines using my own scheduler/runner? Do I need a runtime license for every box involved in the parallel run?

Thank you!
Konstantin

[Cody]

Hello,

...do I HAVE TO use Scheduling Server and Execution Server to run tests in parallel,...

There are multiple ways of running tests. In short you do not have to use the Telerik scheduling and execution server. Besides using our scheduling server you can also run tests:
1) Using our command line runner
2) Using MSTest to run our tests
3) Call a subtest that resides within the same project from a coded step of a parent test. For example:
this.ExecuteTest("AdminTest.tstest");

4) Writing your own desktop application and take advantage of our RunHelp API. For example:
Settings runSettings = new Settings();
runSettings.Web.DefaultBrowser = BrowserType.Chrome;
ResultSummary results2 = ArtOfTest.WebAii.Design.Execution.RunHelper.Test(PathToTest, runSettings, @"c:\testprojectdir\bin\debug\testproject.dll");

Is that enough to get you started?

Kind regards,
Cody
the Telerik team

[Konstantin]

I see there are plenty of methods. Now to save me some effort could you please rank those methods from the most preferred and supported to the least recommended and suppored? 

Do all those methods provide easy way to know if the test is still running and if it is not then what's the outcome? From this point of view RunHelp API looks most flexible. Do I need a separate license to run tests through ArtOfTest.WebAii.Design.Execution.RunHelper.Test ?

Now is there a way to convey the results from my custom test running application to the "root" test studio, so that it sees my test runner as a coded test ?

[Cody]

Hi Konstantin,

...could you please rank those methods from the most preferred and supported to the least recommended and suppored? 

They are all equally supported as far as Telerik is concerned. "Preferred"? That's honestly hard to say. Generally it comes down to what architecture you'd like to use in the end and your personal coding preferences. If you're big on Microsoft tools, I might try the MSTest approach. But that forces a dependency on Microsoft tools for your architecture. Alternatively you could use our own runner and remove that dependency. If you like the idea of creating your own desktop application for managing/running tests, I'd probably go with RunHelper.Test. Does that help any?

Do I need a separate license to run tests through ArtOfTest.WebAii.Design.Execution.RunHelper.Test ?

Nope. You'll need a minimum of one full Test Studio license in order to record and design your tests. Then one Run-Time edition for each machine that will be executing the Test Studio tests. Nothing special to use RunHelper.Test as it's built into the API for both licenses.

Now is there a way to convey the results from my custom test running application to the "root" test studio, so that it sees my test runner as a coded test ?

Since RunHelper.Test returns a Results object, that's pretty easy for the local machine. The hard part is gathering the results from the remote machine. If the remote machine runs the test via our ArtOfTest.Runner you could use the Out and Result command line parameters to create the output result file (.aiiresult file) on some network share with a static name, then let the local machine pick up that file, parse it and incorporate it into the main tests results.

Kind regards,
Cody
the Telerik team

[Konstantin]

Thank you, Cody, for your answers! I think we are getting somewhere now.

I incline towards RunHelper way of doing things.

Need to clarify about Run-Time edition licenses. Let's say I have three boxes which I want to utilize for running the tests in parallel. I'm going to actually start the tests on those boxes using RunHelper API.

1. What exactly component and when is going to check that those boxes have Run-Time licenses? Is the check inside RunHelper API?

2. What shall I install on those three boxes to be able to run tests? I don't need full Test Studio there, but what DO I need there?

3. RunHelper.Test returns a Results object. Is it serializable? If it is, then there is no problem to send it back to the "root" machine, but how do I explain to the Test Studio that some test tun took place on another box and here is the result object. Is there API for that?

Konstantin

[Cody]

Hello,

1) The only license validation that happens is during initialization of full Test Studio. Our Run-Time license doesn't actually do a license check.

2) Just the Run-Time edition is all you need, retails for just $199 each. You need to purchase one license per box. Three boxes = three licenses.

3) RunHelper returns a ResultSummary object. The .Result property is a TestResult object which is marked [Serializable].

how do I explain to the Test Studio that some test tun took place on another box and here is the result object. Is there API for that?

No there is no API for that simply because Test Studio wasn't designed for multi-parallel machine execution. You'll need to figure out how to insert it into the current test results. I'm asking our developers for suggestions how to do this. I'll update you once i get their response.

Kind regards,
Cody
the Telerik team