Scheduling functional tests and CruiseControl

[Steve]

We use CruiseControl with Nant for automated integration and we want to add our TS functional test results into CruiseControl so any failures are immediately visible. The current thinking is that we have two options:

1. 1) We use the QA scheduler to handle the test execution and show the results via a new project in CruiseControl
2. 2) We use CruiseControl to execute the tests and show the results via CruiseControl

For 1) how do we configure the scheduled tests so that they automatically create .trx files? Am I right in thinking we can then simply point to a directory which contains the .trx files from within the ccnet.config file and the results will be shown in CruiseControl?

For 2) do you have a how to guide on how to configure CruiseControl so that it can excecute function tests and report the results?

Thanks,

Steve

[Ivaylo]

Hello Steve,

Оn your question about automatic creation of trx files within Test Studio UI or via our Scheduler, I'm afraid you cannot create .trx automatically. There is a manual export within Test Studio UI that you ca use. In the Results tab go to Export ribbon bar and choose VS result file button (shown in the screenshot).

 I have reviewed your request carefully and I'm positive the best option in your case is to use MSTest to run our TS tests because MSTest outputs a .trx file which integrates nicely with CruiseControl. Here is an article which will guide how to run tests via the MSTest command line tool.
Regarding publishing results you can review the following article about Cruise Control that we have.

Hope this will help.

Regards,
Ivaylo
the Telerik team

[Steve]

Hi Ivaylo -

Thanks for your reply.

It's my understanding that MSTest ships with Visual Studio and we'd need a fully licenced copy installed in order to do this. That's right isn't it? This is the reason why we're using nant builds, so we can avoid these kinds of additional costs.

We've decided to use the TS scheduling service because we need to target 2 execution servers hosted on 2 VMs. However, I'm struggling a bit on how to surface the results of the scheduled tests in Cruise Control. We have a number of projects already in Cruise Control and we use CCTray to easily show build results to our distributed team. We'd like to somehow include the test results so that we're immediately aware of any failures in regression testing.

Any help you or anyone else can give would be most appreciated.

Cheers,
Steve

[Ivaylo]

Hello Steve,

MSTest also comes with Visual Studio Agents, which is a free download so you don't need a fully licensed copy of Visual Studio installed on your CC server.
Could you please clarify on your statement “because we need to target 2 execution servers” why you are using two execution servers, what is the reason of doing this. I understand there are very valid reasons to do this. I just want to make sure I understand your needs.

Thank you for your clarification in advance.
Looking forward to hear from you.

Kind regards,
Ivaylo
the Telerik team

[Steve]

Hi Ivaylo -

Thanks for the info on Visual Studio Agents, I didn't know that.

Our set up is as follows:

We have a standalone version of Test Studio on one server, which is configured as a Scheduling Server and and Execution Server. This will execute tests against IE9, Chrome, Safari and Firefox, all of which are installed on the same server.

We also have 2 VMs, one with IE8 the other with IE9, both of which are configured as Execution Servers. They will execute tests against the respective version of IE.

So our plan is to write 2 sets of Test Lists. One will be executed against the server where the standalone version of Test Studio is installed, and it will execute against all 4 browsers. The other Test List will execute against the 2 VMs and it will execute IE only.

I attempted to create just one Test List which executed against all 4 browsers. However, if I included on of the VMs as Execution Server, the test wasn't intelligent enough to only execute IE. It actually tried to open all of the other browsers even though they aren't installed. This resulted in 100% CPU. This is the reason why we are creating 2 Test Lists.

With all this said, we couldn't think of a way to actually run the tests via Cruise Control, either with nant or MSTest, because we need to execute against specific browsers. We opted to use the Test Studio Scheduler because we can set the Execution Server in the schedule.

So now there's two questions:
1) Would we be able to execute tests with MSTest via Cruise Control and set which Execution Servers to use?
2) If we use Test Studio Scheduler to execute the tests, how could we view the results in Cruise Control?

Many thanks,

Steve

[Ivaylo]

Hello Steve,

Please excuse me for the delayed reply.

Thank you for clarifying the setup you have. Back on your questions:

1) Unfortunately it would not be possible to use MSTest and Cruise Control with our execution servers. They only work with our scheduling servers.In this case you should choose between MSTest and Cruise Control or our scheduling service.

2) If you use Test Studio to schedule and execute tests you can view the results in Cruise Control only if they are converted/exported to *.trx files. There is a manual export within Test Studio UI that you ca use. In the Results tab go to Export ribbon bar and choose VS result file button.
However there is no option to do that automatically except if you write a script using PowerShell for example which converts the result files from .aiiresult to .trx

We can continue researching a possible solution for you using MSTest but we need more time to finish our investigation since we never had similar request and this is never being tested.

Greetings,
Ivaylo
the Telerik team

[Ewin]

I would like to hear more about using powershell to convert an aiiresult to a trx file.  

Or on a similar note, is there a way to hook into the functionality to convert the aiiresult to trx without using the Test Studio application and clicking the button to TRX?

[Ivaylo]

Hеllo Ewin,

We do not have such an implementation. This was only a suggestion on how to proceed.

Thanks for your understanding.

Regards,
Ivaylo
Telerik