Scheduling functional tests and CruiseControl

8 posts, 0 answers
  1. Steve
    Steve avatar
    11 posts
    Member since:
    Jul 2012

    Posted 24 May 2012 Link to this post

    We use CruiseControl with Nant for automated integration and we want to add our TS functional test results into CruiseControl so any failures are immediately visible. The current thinking is that we have two options:

    1. 1) We use the QA scheduler to handle the test execution and show the results via a new project in CruiseControl
    2. 2) We use CruiseControl to execute the tests and show the results via CruiseControl

    For 1) how do we configure the scheduled tests so that they automatically create .trx files? Am I right in thinking we can then simply point to a directory which contains the .trx files from within the ccnet.config file and the results will be shown in CruiseControl?

    For 2) do you have a how to guide on how to configure CruiseControl so that it can excecute function tests and report the results?

    Thanks,

    Steve

  2. Ivaylo
    Admin
    Ivaylo avatar
    750 posts

    Posted 29 May 2012 Link to this post

    Hello Steve,

    Оn your question about automatic creation of trx files within Test Studio UI or via our Scheduler, I'm afraid you cannot create .trx automatically. There is a manual export within Test Studio UI that you ca use. In the Results tab go to Export ribbon bar and choose VS result file button (shown in the screenshot).

     I have reviewed your request carefully and I'm positive the best option in your case is to use MSTest to run our TS tests because MSTest outputs a .trx file which integrates nicely with CruiseControl. Here is an article which will guide how to run tests via the MSTest command line tool.
    Regarding publishing results you can review the following article about Cruise Control that we have.

    Hope this will help.

    Regards,
    Ivaylo
    the Telerik team
    Quickly become an expert in Test Studio, check out our new training sessions!
    Test Studio Trainings
  3. Steve
    Steve avatar
    11 posts
    Member since:
    Jul 2012

    Posted 29 May 2012 Link to this post

    Hi Ivaylo -

    Thanks for your reply.

    It's my understanding that MSTest ships with Visual Studio and we'd need a fully licenced copy installed in order to do this. That's right isn't it? This is the reason why we're using nant builds, so we can avoid these kinds of additional costs.

    We've decided to use the TS scheduling service because we need to target 2 execution servers hosted on 2 VMs. However, I'm struggling a bit on how to surface the results of the scheduled tests in Cruise Control. We have a number of projects already in Cruise Control and we use CCTray to easily show build results to our distributed team. We'd like to somehow include the test results so that we're immediately aware of any failures in regression testing.

    Any help you or anyone else can give would be most appreciated.

    Cheers,
    Steve
  4. Ivaylo
    Admin
    Ivaylo avatar
    750 posts

    Posted 31 May 2012 Link to this post

    Hello Steve,

    MSTest also comes with Visual Studio Agents, which is a free download so you don't need a fully licensed copy of Visual Studio installed on your CC server.
    Could you please clarify on your statement “because we need to target 2 execution servers” why you are using two execution servers, what is the reason of doing this. I understand there are very valid reasons to do this. I just want to make sure I understand your needs.

    Thank you for your clarification in advance.
    Looking forward to hear from you.

    Kind regards,
    Ivaylo
    the Telerik team
    Quickly become an expert in Test Studio, check out our new training sessions!
    Test Studio Trainings
  5. Steve
    Steve avatar
    11 posts
    Member since:
    Jul 2012

    Posted 31 May 2012 Link to this post

    Hi Ivaylo -

    Thanks for the info on Visual Studio Agents, I didn't know that.

    Our set up is as follows:

    We have a standalone version of Test Studio on one server, which is configured as a Scheduling Server and and Execution Server. This will execute tests against IE9, Chrome, Safari and Firefox, all of which are installed on the same server.

    We also have 2 VMs, one with IE8 the other with IE9, both of which are configured as Execution Servers. They will execute tests against the respective version of IE.

    So our plan is to write 2 sets of Test Lists. One will be executed against the server where the standalone version of Test Studio is installed, and it will execute against all 4 browsers. The other Test List will execute against the 2 VMs and it will execute IE only.

    I attempted to create just one Test List which executed against all 4 browsers. However, if I included on of the VMs as Execution Server, the test wasn't intelligent enough to only execute IE. It actually tried to open all of the other browsers even though they aren't installed. This resulted in 100% CPU. This is the reason why we are creating 2 Test Lists.

    With all this said, we couldn't think of a way to actually run the tests via Cruise Control, either with nant or MSTest, because we need to execute against specific browsers. We opted to use the Test Studio Scheduler because we can set the Execution Server in the schedule.

    So now there's two questions:
    1) Would we be able to execute tests with MSTest via Cruise Control and set which Execution Servers to use?
    2) If we use Test Studio Scheduler to execute the tests, how could we view the results in Cruise Control?

    Many thanks,

    Steve
  6. Ivaylo
    Admin
    Ivaylo avatar
    750 posts

    Posted 05 Jun 2012 Link to this post

    Hello Steve,

    Please excuse me for the delayed reply.

    Thank you for clarifying the setup you have. Back on your questions:

    1) Unfortunately it would not be possible to use MSTest and Cruise Control with our execution servers. They only work with our scheduling servers.In this case you should choose between MSTest and Cruise Control or our scheduling service.

    2) If you use Test Studio to schedule and execute tests you can view the results in Cruise Control only if they are converted/exported to *.trx files. There is a manual export within Test Studio UI that you ca use. In the Results tab go to Export ribbon bar and choose VS result file button.
    However there is no option to do that automatically except if you write a script using PowerShell for example which converts the result files from .aiiresult to .trx

    We can continue researching a possible solution for you using MSTest but we need more time to finish our investigation since we never had similar request and this is never being tested.


    Greetings,
    Ivaylo
    the Telerik team
    Quickly become an expert in Test Studio, check out our new training sessions!
    Test Studio Trainings
  7. Ewin
    Ewin avatar
    9 posts
    Member since:
    Jan 2012

    Posted 18 Oct 2015 in reply to Ivaylo Link to this post

    I would like to hear more about using powershell to convert an aiiresult to a trx file.  

    Or on a similar note, is there a way to hook into the functionality to convert the aiiresult to trx without using the Test Studio application and clicking the button to TRX?

  8. Ivaylo
    Admin
    Ivaylo avatar
    750 posts

    Posted 21 Oct 2015 Link to this post

    Hеllo Ewin,

    We do not have such an implementation. This was only a suggestion on how to proceed.

    Thanks for your understanding.

    Regards,
    Ivaylo
    Telerik
     
    The New Release of Telerik Test Studio Is Here! Download, install,
    and send us your feedback!
Back to Top