Display report automation test on Cruise control

[Totti]

Hello,
I run auromation test from Cruise control by MSTest.exe, then generated result test as results.trx, after that, using plug-in MsTestReport2008.xsl to display result test on Cruise Control. The Report display successfully but the view of report's not quite nice (much information's redundancy). You can see attached report here(2 attached files on single report).
So Could we do this report better ?

Many thanks.

[Cody]

Hello Totti,

You are correct, it does not look very good in CruiseControl. How does that same .TRX file look when loaded into Visual Studio? I expect it will look just fine. If so it means that the MsTestReport2008.xsl transform file, which comes with CruiseControl needs to be modified to work with with .trx files that contain WebUI test results.

Since this file is not owned or controlled by Telerik, we're not sure what we can do about it. I have logged feature request 109065 to track this.

All the best,
Cody

the Telerik team

[Totti]

Thank for your answer,
So I will investigate more and modify either Result file or MsTestReport2008.xsl  file to display report to be good view.
(I opened result test *.trx file by Visual Studio is OK  good)

[Cody]

Hi Totti,

That sounds very good. If you do come up with a modified MsTestReport2008.xsl that you are happy with, send it to us like a code sample, grant us permission to freely redistribute to our other customers and then I can grant you 2500 Telerik Points as a reward.

Best wishes,
Cody

the Telerik team

[Totti]

Hello,
I have customized xsl file completely and integrate with CruiseControl as its plug in (this xsl file will merge *.trx file of MStest to generate test report).
You can see the test report on Cruise Control in attached file.
What do you think about this ?

[Cody]

Hi Totti,

That screen shot looks perfect and beautiful! One question, how about one where a test failed? Then I think we can call it work complete.

All the best,
Cody
the Telerik team

[Totti]

Hi,
- You can see test report in case of existing some fail tests (Attached file)
- I'm having issue with CruiseControl's log file: In the XML log file of CruiseControl , in tag WebAiiTestResult's existing empty namespace : xmlns="", So for this namespace, I 'm impossible to access in child node of  WebAiiTestResult (You can see attached). So Do you know how to resolve for this? or How to avoid empty namespace ?
Thanks.

[Cody]

Hi Totti,

Yes, a report with a failure looks great! I've gone ahead and given you 2500 Telerik Points as previously discussed. Can you please attach the updated .xsl file to this forum thread (put it into a .zip file first). Thanks!

Regarding the xmlns issue, there is no way to not have it blank. Microsoft MSTest is setting it that way (rather than CruiseControl). Is there specific information or XML nodes that you are not able to access because of this? It appears in your screenshot you are able to access everything that is necessary.

Kind regards,
Cody
the Telerik team

[Totti]

Hi,
- Regarding the xmlns issue, In order to get steps information, I need to access to child node of WebAiiTestResult in XML file but for blank namespace , I cannot declare and use blank namespace in XSL file.
- I also put my xsl file in the forum.

Thanks.

[Cody]

Hello Totti,

I did a little research into this and discovered you can access the nodes that have a default namespace. You need to use an XPath like this:

<xsl:for-each select="./tt:TestRun/tt:Results/art:WebAiiTestResult/WebAiiTestResult/AutomationStepResults/Item">

Note how the last three navigation nodes have no namespace prefix unlike everything before it. This means to use the default/empty namespace.

Regards,
Cody
the Telerik team

[Totti]

Great, many thanks to Cody,
I have applied your soultion in my report file XSL, It's OK now.

Thanks and Regards,
Totti

[Cody]

Hi Totti,

I am very glad to hear it! Could you post your updated XSL for us?

Greetings,
Cody
the Telerik team

[Totti]

Cody,
You can find updated XSL file that attached in this post
BTW, I can have more points for updating this file ?

Thanks

[Cody]

Hello Totti,

Thank you for submitting the updated .xsl file. Well done! I will speak to your account manager about what may be possible regarding Telerik Points.

Kind regards,
Cody
the Telerik team

[David]

I tried using the attached XSL file to display my test results in CruiseControl but am not having much luck. My output is attached.

I am using the VS2010 plugin and the command line to build the trx results file. You can see that the actual Test Studio information does not get displayed.

I have tried accessing the WebAii result elements directly (e.g. <xsl:value-of select="./build/tt:TestRun/tt:TestDefinitions/tt:WebAiiTest/@creation" />) but that was unsuccessful.

Any help would be appreciated.

[David]

I tried using the attached XSL file to display my test results in CruiseControl but am not having much luck. My output is attached.

I am using the VS2010 plugin and the command line to build the trx results file. You can see that the actual Test Studio information does not get displayed.

I have tried accessing the WebAii result elements directly (e.g. <xsl:value-of select="./build/tt:TestRun/tt:TestDefinitions/tt:WebAiiTest/@creation" />) but that was unsuccessful.

Any help would be appreciated.

[Plamen]

Hi David,

I'm sorry for the delay getting back to you.

The XSL file you are refering to is created to parse the .trx results generated with Visual Studio 2008 distribution. I found out that there are a lot of changes in the microsoft schema in VS2010. I was able to create a new XSL file according the new schema and now the test results will be displayed as expected.

Please find attached the new AutomationTestReportDetail XSL file and let me know if you need further assistance on this.

Greetings,
Plamen
the Telerik team

Check out Telerik Trainer, the state of the art learning tool for Telerik products.

[David]

I tried the file that Plamen posted and tried a number of different iterations on it (replacing the old .xsl file, using the same name and changing the dashboard.config file) and the page did not render with the test result information. I will try to up the file Plamen sent to get this to work.

[Plamen]

Hello David,

The posted xsl file is meant to be used for displaying the results from a Unit test execution. The Unit test contains no steps, so the information about the passed/failed steps will not be displayed(see screenshot).

Please check out this video for an example of Unit test execution.
http://screencast.com/t/rPrUCw1df

In case you are running Test Studio tests(.tstest file), you can use our schema "http://artoftest.com/schemas/WebAiiDesignCanvas/1.0.0" to parse the execution results into CruiseControl.

Here's a video of how to do that:
http://screencast.com/t/nkqPmot1S7xH 
Note that when I changed the xsl file in the dashboard.config, I restarted the ccnet Application Pool Tasks from the IIS Manager. This is necessary for the changes to take effect. You can find attached the xsl file I'm using in this example (AutomationTestReportDetailTStest.xsl).

Hope this helps!

Greetings,
Plamen
the Telerik team

Check out Telerik Trainer, the state of the art learning tool for Telerik products.

[David]

Plamen,
I am just getting back to this and am still having trouble.

I know I am using the correct XSL file and it is being displayed. My dashboard.config is: 

<buildPlugins>

  <buildReportBuildPlugin>

    <xslFileNames>

      <xslFile>xsl\header.xsl</xslFile>

      <xslFile>xsl\modifications.xsl</xslFile>

      <xslFile>xsl\AutomationTestReportDetailTStest.xsl</xslFile>

      <xslFile>xsl\compile-msbuild.xsl</xslFile>

      <xslFile>xsl\MsTestSummary2008.xsl</xslFile>

    </xslFileNames>

  </buildReportBuildPlugin>

  <buildLogBuildPlugin />

<!-- <xslReportBuildPlugin description="MSTest 2008 Detail Report" actionName="MSTEST2008DetailReport" xslFileName="xsl\AutomationTestReportDetailTStest.xsl" /> -->

<xslReportBuildPlugin description="MSBuild Report" actionName="MSBuildBuildReport" xslFileName="xsl\msbuild.xsl" />

</buildPlugins>

I have restarted the ccnet app pool item in IIS to when making the dashboard.config changes.

I am still not seeing test case results. 



[Plamen]

Hi David,

Your dashboard.config looks correct to me. Could you please provide the ccnet.config file, because most probably the problem is there?

Also, check the "publishers" section of your project configuration, it should point to the same folder where your "resultsfile" is. Here's an example:

<exec>
  <!-- Call mstest to run the tests contained in the TestProject -->
  <executable>C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe</executable>
  <baseDirectory>C:\CruiseControl\MSTests\CruiseControlTest\</baseDirectory>
  <buildArgs>/testmetadata:CruiseControlTest.vsmdi /resultsfile:CruiseControlTest\testResults.trx</buildArgs>
  <buildTimeoutSeconds>60</buildTimeoutSeconds>
</exec>

Publishers:

<publishers>
  <merge>
    <files>
      <file>C:\CruiseControl\MSTests\CruiseControlTest\CruiseControlTest\testResults.trx</file>
    </files>
  </merge>
  <xmllogger />
</publishers>

Note that the path placed in the "publishers" section is the same as the combination of "baseDirectory" plus "resultsfile".

If this is not the case please provide the ccnet.config file, so we can look further into this.

Regards,
Plamen
the Telerik team

Check out Telerik Trainer, the state of the art learning tool for Telerik products.

[David]

Plamen,
I made the appropriate changes to my ccnet.config and the results are showing up.

Thanks for sticking with me on this issue.

[Randrew]

hi need your help..

what script should i do to remove this error:

The results file "TestResults\testResults.trx" already exists. Please specify a different results file or verify the existing file is no longer needed and delete it


and the list of test are names are not displaying. 

[Cody]

Hi David,

This error message is produced by MSTest (not a Telerik product) when you use a command line like this:

mstest /testsettings:TestResults\testResults.trx...

The only way to overcome this error is to delete that file prior to running mstest in your CruiseControl build script. It may be easiest to simply add a "del TestResults\testResults.trx" command right before your MSTest command in the build script.

Regards,
Cody
the Telerik team

[Randrew]

Hi Cody,

I made already a script to delete existing .trx file

My another problem was test results name don't appear only the number of test.

Here is my Config:

<cruisecontrol xmlns:cb="urn:ccnet.config.builder">
  <project name="ApplicationTestingTP" queue="ApplicationTestingTP" queuePriority="1">
    <workingDirectory>C:\TestProject</workingDirectory>
    <artifactDirectory>C:\TestProject\ApplicationTestingTP\bin\Debug</artifactDirectory>
    <webURL>localhost/ccnet</webURL>
    <modificationDelaySeconds>20</modificationDelaySeconds>
    <triggers>
       <intervalTrigger name="continuous" seconds="300" buildCondition="IfModificationExists" />
    </triggers>
    <state type="state" directory="C:\ccnetstate" />
    <labeller type="iterationlabeller">
      <prefix>1.0</prefix>
      <duration>1</duration>
      <releaseStartDate>2012/7/30</releaseStartDate>
      <separator>.</separator>
    </labeller>

    <tasks>
      <msbuild>
        <executable>C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe</executable>
        <workingDirectory>c:\TestProject</workingDirectory>
        <projectFile>ApplicationTesting.sln</projectFile>
        <buildArgs>/noconsolelogger /p:Configuration=Debug /v:Minimal</buildArgs>
        <targets>Build</targets>
        <timeout>900</timeout>
      </msbuild>

      <exec>
        <executable>c:\TestProject\delTestLog.bat</executable>
        <baseDirectory>c:\TestProject\TestResults\</baseDirectory>
      </exec>

      <exec>
        <!-- Call mstest to run the tests contained in the TestProject -->
        <executable>C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe</executable>
        <baseDirectory>C:\TestProject\</baseDirectory>
        <buildArgs>/testmetadata:ApplicationTesting.vsmdi /resultsfile:TestResults\testResults.trx</buildArgs>
        <buildTimeoutSeconds>60</buildTimeoutSeconds>
      </exec>
    </tasks>

    <publishers>
      <merge>
        <files>
          <file>C:\TestProject\TestResults\testResults.trx</file>
        </files>
      </merge>
      <xmllogger />
    </publishers>
  </project>

</cruisecontrol>

Dashboard.config

<?xml version="1.0" encoding="utf-8"?>
<dashboard>
  <remoteServices>
    <servers>
      <!-- Update this list to include all the servers you want to connect to. NB - each server name must be unique -->
      <server name="local" url="tcp://localhost:21234/CruiseManager.rem" allowForceBuild="true" allowStartStopBuild="true" backwardsCompatible="false" />
    </servers>
  </remoteServices>
  <plugins>
    <farmPlugins>
      <farmReportFarmPlugin categories="false" refreshInterval="0" successBar="Bottom"/>
      <cctrayDownloadPlugin />
      <administrationPlugin password="" />
    </farmPlugins>

    <serverPlugins>
      <serverReportServerPlugin />
    </serverPlugins>

    <projectPlugins>
      <projectReportProjectPlugin />
      <viewProjectStatusPlugin />
      <latestBuildReportProjectPlugin />
      <viewAllBuildsProjectPlugin />
    </projectPlugins>

    <buildPlugins>
      <buildReportBuildPlugin>
        <xslFileNames>
          <xslFile>xsl\header.xsl</xslFile>
          <xslFile>xsl\modifications.xsl</xslFile>
          <xslFile>xsl\AutomationTestReportDetailTStest.xsl</xslFile>
          <xslFile>xsl\compile-msbuild.xsl</xslFile>
          <xslFile>xsl\MsTestSummary2010.xsl</xslFile>
        </xslFileNames>
      </buildReportBuildPlugin>
      <buildLogBuildPlugin />
      <xslReportBuildPlugin description="MSBuild Report" actionName="MSBuildBuildReport" xslFileName="xsl\msbuild.xsl" />
    </buildPlugins>

    <securityPlugins>
      <simpleSecurity />
    </securityPlugins>
  </plugins>
</dashboard>

[Cody]

Hello,

I see this section:
<exec> 
<executable>c:\TestProject\delTestLog.bat</executable> 
<baseDirectory>c:\TestProject\TestResults\</baseDirectory> 
 </exec>

However you didn't include a copy of delTestLog.bat so I don't know what that does... I can't even confirm calling a .bat file from the <exec> section works in CruiseControl (that's beyond our CC expertise).

If it were me I'd replace the section with this:
<exec> 
<executable>del /f /q testResults.trx</executable> 
<baseDirectory>c:\TestProject\TestResults\</baseDirectory> 
 </exec> 

Bottom line the .trx file isn't getting deleted as expected prior to running MSTest. You may get better answers contact some CC.NET experts.

Kind regards,
Cody
the Telerik team

[Cody]

Hi Randrew,

My another problem was test results name don't appear only the number of test.

My apologies, I nearly overlooked this. Can you please clarify this? Can you show me a screen shot of what you are actually seeing compared to what you expected to see?

Greetings,
Cody
the Telerik team

[Randrew]

Hi Cody,

Please see attached files.

Thanks for your help.

[Cody]

Hello,

As I recall, CruiseControl uses a .XSLT file for translating the .trx output file into HTML to be displayed by the CruiseControl dashboard. Apparently this .XSLT file needs to be tweaked. If you can locate yours and send it to me, I may be able to figure out what tweaks it needs.

All the best,
Cody
the Telerik team

[Randrew]

Hi Cody,

Attached herewith my xslt files

thanks

[Cody]

Hello Randrew,

I apologize for the delay getting back to you, but I think you're going to like what I came up with for you. Instead of modifying your existing xsl files I created a new custom one just for Test Studio test results. You will now get a report similar to the attached screen shot. The custom XSL file is contained in the attached .zip file. Place it into your folder: C:\Program Files (x86)\CruiseControl.NET\webdashboard\xsl

Next modify the buildReportBuildPlugin section of your dashboard.config file to look like:

<buildReportBuildPlugin>
  <xslFileNames>
    <xslFile>xsl\header.xsl</xslFile>
    <xslFile>xsl\modifications.xsl</xslFile>
    <xslFile>xsl\compile-msbuild.xsl</xslFile>
    <xslFile>xsl\TestStudioMSTest2010Report.xsl</xslFile>
  </xslFileNames>
</buildReportBuildPlugin>

Optionally you can add a custom report entry:
<xslReportBuildPlugin description="Test Studio Report" actionName="TestStudioReport" xslFileName="xsl\TestStudioMSTest2010Report.xsl"></xslReportBuildPlugin>

The only problem I haven't worked out yet is calculating the duration of the test. The total time it took to run all of the tests is not contained in the build log. The start and finish time is there making it possible to calculate and the duration of each individual test is also stored (not currently displayed but could be) just not the grand total. I decided to give you what I have since it's 99% there and gives you the details you're looking for. I'm going to continue working on this and post an update later when I figure out how to calculate the grand total duration in XSL. Kind regards,
Cody
the Telerik team

[Randrew]

Hi Cody,

Thanks for your help,
The test names still don't appear please see attached screen shots.

Best Regards,
Randrew

[Cody]

Hello,

Please send me a complete copy of your Build Log (the XML) so I can figure out why.

All the best,
Cody
the Telerik team

[Randrew]

Hi Cody,

Attached herewith the buildlog


Thanks,
Randrew

[Cody]

Hi Randrew,

Thank you for sending that. I see now you're actually running coded unit tests. I assumed you were running Test Studio tests which I now see you are not. The XSL file I sent you only applies to running Test Studio tests via MSTest. We, the Test Studio support team, cannot assist with any report generation problems when using coded unit tests.

Regards,
Cody
the Telerik team

[Randrew]

Hi Cody,

Can you please teach me how to run my ccnet on Test Studio tests via MSTest?

Thanks,
Randrew

[Cody]

Hi Randrew,

Yes I certainly can do that. What I'm doing is updating our documentation for CruiseControl.NET to show you how. I'm nearly finished. I just need one or two more days to complete the work. Then it should have all the information you need.

Greetings,
Cody
the Telerik team

[Randrew]

Hi Cody,

Do you have the documentation already?

Many Thanks,

Randrew

[Ivaylo]

Hello Randrew,

I am taking over this ticket from Cody. I am actively working on this and as soon as it is done I will update this ticket.

Thank you for your patience and understanding regarding this matter.

Kind regards,
Ivaylo
the Telerik team