MS BUILD support

[Sanjana]

Does WEB UI Test Studio QA Edition integrate with MS BUILD so we can run our nightly smoke test plan when a new build is created?

[Cody]

Hello Sanjana,

The tests generated by the QA Edition can be run from the command line using either Microsoft's MSTest or our own ArtOfTest.Runner tool. Documentation for how to run tests with these tools can be found here, here and here.

The QA Edition doesn't directly integrate with MSBuild. You'll just have to manually modify your MSBuild to execute the command line tools. 

If you're using TFS 2010, you can have .aii tests be automatically as part of your build definition. This is documented here and not mess with how to do manually modify your MSBuild script.

Kind regards,
Cody
the Telerik team

[Jacek Karwowski]

Hello Cody,

in your reply you mention:
"The QA Edition doesn't directly integrate with MSBuild. You'll just have to manually modify your MSBuild to execute the command line tools."

Could you give more details about needed modifications?

I'm trying to run QA edition tests in CruiseControl.net build process. I already try ArtOfTest.Runner, but it not fail build process in case of broken tests.
I also fail with running QA tests with MSTest. When i run this command:
mstest /testcontainer:C:\svn\Telerik\google_search\WebTest.tstest

i get:
Loading C:\svn\Telerik\google_search\WebTest.tstest...
Starting execution...


Final Test Results:
Results               Top Level Tests
-------               ---------------
Not Executed          c:\svn\telerik\google_search\webtest.tstest
0/1 test(s) Passed, 1 Not Executed


Summary
-------
Test Run Error.
  Not Executed  1
  ---------------
  Total         1
Results file:  C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\T
estResults\bstanisz_BIES 2011-05-03 20_06_47.trx
Test Settings: Default Test Settings

Run has the following issue(s):
Failed to queue test run 'bstanisz@BIES 2011-05-03 20:06:47': Value cannot be nu
ll.
Parameter name: path1

Do you have any hint for this?
Best regards, 

Bartek

[Cody]

Hello Jacek Karwowski,

I am sorry, but no we really can't give you instructions for modifying MSBuild. Being it is a Microsoft product, we know very little about it. We can only instruct you with the correct command line syntax needed to execute tests.

You say you are using the mstest command line... is that from within a build or are you just manually running it from a Visual Studio command prompt and it's doing this? If this is actually a command contained within a TFS type build you probably need to change the "Copy To Output Director" setting of your .tstest files (see attached screenshot). It is set to "Do Not Copy" by default. As a result when the build goes to run the test, it cannot find the .tstest file in the output folder so it can run the test.

Best wishes,
Cody
the Telerik team

[Jacek Karwowski]

Hello Cody,

thank you for your response.

First - perhaps i wrongly understand this sentence:
 "The QA Edition doesn't directly integrate with MSBuild. You'll just have to manually modify your MSBuild to execute the command line tools."
I though that you mean a modification of *.tstest file or msbuild project.... So, according to your last sentence, it is not possible to run QA version tests using MSBuild tool?

Second:
"You say you are using the mstest command line... is that from within a build or are you just manually running it from a Visual Studio command prompt and it's doing this?"

I do it by running this: 
mstest /testcontainer:C:\svn\Telerik\google_search\WebTest.tstest 
from Visual Studio command prompt.
File WebTest.tstest is produced with WebTest Studio QA Edition (standalone). My goal is to make it possible that our tester will be able to prepare Silverlight tests using this edition, without usage of Visual Studio. Then such prepared tests can be run during our build process. The Build process is performed by CruiseControl.net server. To do this i need to be able to run test from windows command line.
On CI server (with cc.net) i want to have only runtime version of WebTest Studio. QA (standalone) edition will be installed on the tester workstation.


Best regards, 

Bartek

[Cody]

Hello Jacek Karwowski,

Have you checked out our documentation on how to run tests via MSBuid/MSTest with CruiseControl.NET? Pay particular attention to section 4.2. CruiseControl also has documentation for how to add any command line execution to its automated build process. This should help clear things up for you.

Regards,
Cody
the Telerik team

[Jacek Karwowski]

Dear Cody, 

in link you provide me there is a description of running tests produced with Test Studio Express (plugin for Visual Studio).
Description says that i should build up a .dll file with tests - to do this i need .csproj file. AFAIK, such file is not produced if i use only Test Studio standalone.
My problem is that i want to buy Standalone edition for testers and run tests they produce on CI.

Till now i don't see this working.

Best regards, 

Bartek

[Cody]

Hi Jacek Karwowski,

I am sorry for the confusion. What you want to accomplish is very doable. The Standalone version creates .tstest files (and also a DLL file that contains any coded steps). This documentation shows how to run Test Studio tests with MSTest from the command line. All you need to do is add these command lines to your CruiseControl build configuration.

All the best,
Cody
the Telerik team

[Jacek Karwowski]

Dear Cody,

i follow these instructions with .tstest file produced with Test Studio standalone. I got the same problem as i mention few post earlier (May 3, 2011) - when i run following command (from VS command line): c:\svn\Telerik\RunnerTest2>mstest  /testcontainer:.\WebTest.tstest
i got exception. This time i check yet .trx results file, and i found following:
 

<RunInfo computerName="BIES" outcome="Error" timestamp="2011-05-26T11:00:02.5321967+02:00"><br>
        <Text>Failed to queue test run 'bstanisz@BIES 2011-05-26 11:00:00': Value cannot be null.<br>
      Parameter name: path1</Text><br>
        <Exception>System.ArgumentNullException: Value cannot be null.<br>
   Parameter name: path1<br>   
   at System.IO.Path.Combine(String path1, String path2)<br>
   at ArtOfTest.WebAiiVSIP.WebAiiTest.UpdateDeploymentItems()<br>
   at ArtOfTest.WebAiiVSIP.WebAiiTest.get_DeploymentItems()<br>
   at Microsoft.VisualStudio.TestTools.TestManagement.DeploymentManager.CheckAndMergeImpliedDependencies(ITestElement test, DeploymentItemCollection initialItems)<br>
   at Microsoft.VisualStudio.TestTools.TestManagement.DeploymentManager.DoDeployment(TestRun run, FileCopyService fileCopyService)<br>
   at Microsoft.VisualStudio.TestTools.TestManagement.ControllerProxy.SetupTestRun(TestRun run, Boolean isNewTestRun, FileCopyService fileCopyService, DeploymentManager deploymentManager)<br>
   at Microsoft.VisualStudio.TestTools.TestManagement.ControllerProxy.SetupRunAndListener(TestRun run, FileCopyService fileCopyService, DeploymentManager deploymentManager)<br>
   at Microsoft.VisualStudio.TestTools.TestManagement.ControllerProxy.QueueTestRunWorker(Object state)</Exception>
</RunInfo>

Do you have any idea what is wrong? I use Windows 7 64bit, polish locale version.

Thank you in advance and best regards, 

Bartek

[Cody]

Hello Jacek Karwowski,

I am not 100% certain, but this may be a caused by a bug we already fixed. Please try upgrading to our latest internal build.

Regards,
Cody
the Telerik team

[Jacek Karwowski]

Hello Cody!

Unfortunately i still have same exception as previously.

Best regards, 

Bartek

[Anthony]

Hello Jacek Karwowski,

Thanks for your patience. I am still testing your scenario as I have reproduced that message locally.

In the meantime, have you tried using Test Studio's built-in execution engine? It can also be run from the command line. Please see the instructions for ArtOfTest.Runner.exe.

Best wishes,
Anthony
the Telerik team

[Cody]

Hello Jacek Karwowski,

I just wanted to let you know we just found and fixed a bug in Test Studio that can cause the error:

Failed to queue test run 'bstanisz@BIES 2011-05-26 11:00:00': Value cannot be null.
Parameter name: path1

We recently discovered it happens for tests created using Test Studio standalone that do not have any coded steps. The fix will be included in our next internal build which is due to be released on Friday of this week.

Best wishes,
Cody
the Telerik team

[David]

Cody,
I am having similar issues with running MSTEST in a CI environment.

I believe I have the correct XML written in to the ccnet.config (CruiseControl):

<!--  VSuite Automation -->
  <project name="VSuite Automation - Rapid">

    <triggers>
    <scheduleTrigger time="3:30" buildCondition="ForceBuild" name="Scheduled">
    </scheduleTrigger>
    </triggers>
    <labeller type="dateLabeller"/>
    <workingDirectory>WORKINGDIR</workingDirectory>
    <artifactDirectory>ARTIFACTDIR</artifactDirectory>
    <sourcecontrol type="svn">
      <trunkUrl>URL:8080/svn/VSuite/trunk/VSuiteAutomation/vsuiteautomation</trunkUrl>
      <username>COMPANY\cruisecontrol</username>
      <password>XXXXXXXX</password>
    </sourcecontrol>
   <tasks>

      <msbuild>
        <executable>C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe</executable>
        <projectFile>vsuiteautomation.csproj</projectFile>
        <buildArgs></buildArgs>
        <targets>Build</targets>
        <timeout>2700</timeout>
        <logger>"E:\Program Files (x86)\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MsBuild.dll"</logger>
      </msbuild>

    <!--Delete the testResults.trx first. This is required as MsTest will not create the file if it exists this could be merged with the mstest action in a single batch file
      <exec>
        <executable>$(windir)\system32\cmd</executable>
        <baseDirectory>E:\CruiseControl\Automation\VSuiteAutomation\buildlogs\buildlogs</baseDirectory>
        <buildArgs>/c if exist testResults.trx del testResults.trx /f</buildArgs>
        <buildTimeoutSeconds>30</buildTimeoutSeconds>
      </exec>-->

    <exec>
        <!-- Call mstest to run the tests contained in the TestProject -->
        <executable>E:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe</executable>
    <buildArgs>/testmetadata:E:\CruiseControl\Automation\VSuiteAutomation\vsuiteautomation\vsuiteautomation.vsmdi /resultsFile:E:\CruiseControl\Automation\VSuiteAutomation\vsuiteautomation\TestResults\testResults.trx</buildArgs>

        <!-- testcontainer: points to the DLL that contains the tests -->
        <!-- runconfig: points to solutions testrunconfig that is created by vs.net, list what test to run -->
        <!-- resultsfile: normally the test run log is written to the uniquely named testresults directory  -->
        <!-- this option causes a fixed name copy of the file to be written as well-->
        <buildTimeoutSeconds>30</buildTimeoutSeconds>
      </exec>

    </tasks>
  </project>

When building the project in CC I get an error yet the build log seems to have run the executable correctly:

- <buildresults>
  <message>Microsoft (R) Test Execution Command Line Tool Version 10.0.30319.1</message>
  <message>Copyright (c) Microsoft Corporation. All rights reserved.</message>
  <message />
  <message>Loading E:\CruiseControl\Automation\VSuiteAutomation\vsuiteautomation\vsuiteautomation.vsmdi...</message>
  <message>Starting execution...</message>
  </buildresults>

When running the entire executable in a CMD window, the error I am getting is:
Failed to queue test run 'dadams@<SERVERNAME> 2011-11-02 10:13:46': The given
path's format is not supported.

My suspicion is that the last error is thrown due to concatenation of one absolute path on to another but I would like your expert opinion and would welcome a screen share. I can make my schedule available to get this done.

Thanks.

[Cody]

Hi David,

I'm not sure what you're referring to by "the last error is thrown due to concatenation of one absolute path on to another". I don't see where that could be occurring. But one thing I do see is that your <buildArgs> section is missing both a /testlist or a /test parameter. You need one of these to tell MSTest what tests to execute. Without this I get the error "No tests to execute."

Regards,
Cody
the Telerik team

[David]

I added /test:login (a known test to exist) to the <buildargs> section. My understanding of MSTEST is that if there is no /test or /testlist, all tests found will run.

The result I get when adding /test:login is:

<buildresults>
  <message>Microsoft (R) Test Execution Command Line Tool Version 10.0.30319.1</message>
  <message>Copyright (c) Microsoft Corporation. All rights reserved.</message>
  <message>    </message>
  <message>Loading E:\CruiseControl\Automation\VSuiteAutomation\vsuiteautomation\vsuiteautomation.vsmdi...</message>
  <message>Starting execution...</message>
  <message>Final Test Results:</message>
  <message>Results               Top Level Tests</message>
  <message>-------               ---------------</message>
  <message>Not Executed          (TestTestlist/)e:\cruisecontrol\automation\vsuiteautomation\vsuiteautomation\common\login - scarlson.tstest</message>
  <message>Not Executed          (TestTestlist/)e:\cruisecontrol\automation\vsuiteautomation\vsuiteautomation\common\login.tstest</message>
  <message>0/2 test(s) Passed, 2 Not Executed</message>
  <message>Summary</message>
  <message>-------</message>
  <message>Test Run Error.</message>
  <message>  Not Executed  2</message>
  <message>  ---------------</message>
  <message>  Total         2</message>
  <message>Results file:  E:\CruiseControl\Automation\VSuiteAutomation\vsuiteautomation\TestResults\testResults.trx</message>
  <message>Test Settings: Default Test Settings</message>
  <message>Run has the following issue(s):</message>
  <message>Failed to queue test run 'SYSTEM@SERVERNAME 2011-11-03 17:25:52': The given path's format is not supported.</message>
</buildresults>

[Cody]

Hello David,

I did a little more research on this and found the results depend on what you have put into your Visual Studio test list(s). Specifying only /testmetadata will run all tests contained in all test lists you have defined in that .vsmdi file. It won't actually run all tests contained in your test project.

Adding /test to /testmetadata will only run the test by that name if it's contained in one of the test lists. I demonstrate this in this video.

Can you take a step back and just try manually executing MSTest from the command line and record what happens. I'd like to see a video of what happens on your machine. Jing is a good free screen recorder.

Best wishes,
Cody
the Telerik team

[David]

I think I have not been clear enough with my issue. The issue is not being able to identify which tests or testlists to run. The issue has been my inability to actually run ANY tests.

http://db.tt/vCtHtJLp : this is a link to a movie I recorded demonstrating my issue.

[Cody]

Hi David,

Thank you for the video. It really is strange it's not really even trying to run the tests. I'd like to look at this problem on your computer via GoToMeeting. Let me know what time zone you are in and your availability and I'll setup the meeting. I'm in Austin, TX which is Central Time (GMT -6).

All the best,
Cody
the Telerik team

[David]

Thanks, Cody.

I am in Chicago so we are in the same timezone. If we can do this meeting first thing tomorrow (8:30 AM) that would be great. If not, let me know what time(s) work for you. My schedule is very flexible to yours.

[Cody]

Hi David,

Unfortunately my Thursday morning is already full. I've got the afternoon free however. How about 1:30pm?

Regards,
Cody
the Telerik team

[David]

1:30 is great. I look forward to it.

[Cody]

Hello David,

I just sent you a GoToMeeting request.

1.  Please join my meeting.

https://www2.gotomeeting.com/join/536925042

2.  Use your microphone and speakers (VoIP) - a headset is recommended.  Or, call in using your telephone.

Dial [phone]
Access Code: 536-925-042
Audio PIN: Shown after joining the meeting
Meeting ID: 536-925-042

Best wishes,
Cody
the Telerik team

[David]

Cody,
I figured out that I have a pathing issue in my project. For example, the build is trying to execute C:\Users\David Adams\Documents\Test Studio Projects\common\click on nav\Compliance.tstest. However, the file is actually located at C:\Users\David Adams\Documents\Test Studio Projects\vsuiteautomation\common\click on nav\Compliance.tstest. The project is not getting included in the path. I added the project path to the system path but the "lost" test location is absolute.

I will investigate a little more with our developers here. Any help you can provide would be hugely helpful.

Thanks.

[Cody]

Hi David,

Because you're dealing with a Visual Studio test list, I'm not going to be any help. That is 100% Microsoft territory. Test Studio doesn't come into the picture until the test is successfully queued and launched. Since the error message is "Failed to queue test run..." this means Test Studio isn't even entering the picture yet.

Regards,
Cody
the Telerik team

[David]

I got this figured out and should have updated the thread.

The basedirectory had to be set to the bin directory. Had to figure that out through trial and error.

<exec>
        <!-- Call mstest to run the tests contained in the TestProject -->
        <executable>E:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe</executable>
        <baseDirectory>E:\CruiseControl\Automation\VSuiteAutomation\vsuiteautomation\bin</baseDirectory>
        <buildArgs>/testmetadata:..\vsuiteautomation.vsmdi /testlist:SampleList</buildArgs>

        <!-- testcontainer: points to the DLL that contains the tests -->
        <!-- runconfig: points to solutions testrunconfig that is created by vs.net, list what test to run -->
        <!-- resultsfile: normally the test run log is written to the uniquely named testresults directory  -->
        <!-- this option causes a fixed name copy of the file to be written as well-->
        <buildTimeoutSeconds>60</buildTimeoutSeconds>
</exec>

[Cody]

Hi David,

Ah yes, that makes sense. Good to know you've fully resolved this problem and are automating away with Test Studio and CruiseControl!

Regards,
Cody
the Telerik team