MS BUILD support

27 posts, 0 answers
  1. Sanjana
    Sanjana avatar
    3 posts
    Member since:
    Jan 2011

    Posted 25 Mar 2011 Link to this post

    Does WEB UI Test Studio QA Edition integrate with MS BUILD so we can run our nightly smoke test plan when a new build is created?
  2. Cody
    Admin
    Cody avatar
    3354 posts

    Posted 25 Mar 2011 Link to this post

    Hello Sanjana,

    The tests generated by the QA Edition can be run from the command line using either Microsoft's MSTest or our own ArtOfTest.Runner tool. Documentation for how to run tests with these tools can be found here, here and here.

    The QA Edition doesn't directly integrate with MSBuild. You'll just have to manually modify your MSBuild to execute the command line tools. 

    If you're using TFS 2010, you can have .aii tests be automatically as part of your build definition. This is documented here and not mess with how to do manually modify your MSBuild script.

    Kind regards,
    Cody
    the Telerik team
  3. Jacek Karwowski
    Jacek Karwowski avatar
    1 posts
    Member since:
    Aug 2008

    Posted 03 May 2011 Link to this post

    Hello Cody,

    in your reply you mention:
    "The QA Edition doesn't directly integrate with MSBuild. You'll just have to manually modify your MSBuild to execute the command line tools."

    Could you give more details about needed modifications?

    I'm trying to run QA edition tests in CruiseControl.net build process. I already try ArtOfTest.Runner, but it not fail build process in case of broken tests.
    I also fail with running QA tests with MSTest. When i run this command:
    mstest /testcontainer:C:\svn\Telerik\google_search\WebTest.tstest

    i get:
    Loading C:\svn\Telerik\google_search\WebTest.tstest...
    Starting execution...


    Final Test Results:
    Results               Top Level Tests
    -------               ---------------
    Not Executed          c:\svn\telerik\google_search\webtest.tstest
    0/1 test(s) Passed, 1 Not Executed


    Summary
    -------
    Test Run Error.
      Not Executed  1
      ---------------
      Total         1
    Results file:  C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\T
    estResults\bstanisz_BIES 2011-05-03 20_06_47.trx
    Test Settings: Default Test Settings

    Run has the following issue(s):
    Failed to queue test run 'bstanisz@BIES 2011-05-03 20:06:47': Value cannot be nu
    ll.
    Parameter name: path1

    Do you have any hint for this?
    Best regards, 

    Bartek
  4. Cody
    Admin
    Cody avatar
    3354 posts

    Posted 12 May 2011 Link to this post

    Hello Jacek Karwowski,

    I am sorry, but no we really can't give you instructions for modifying MSBuild. Being it is a Microsoft product, we know very little about it. We can only instruct you with the correct command line syntax needed to execute tests.

    You say you are using the mstest command line... is that from within a build or are you just manually running it from a Visual Studio command prompt and it's doing this? If this is actually a command contained within a TFS type build you probably need to change the "Copy To Output Director" setting of your .tstest files (see attached screenshot). It is set to "Do Not Copy" by default. As a result when the build goes to run the test, it cannot find the .tstest file in the output folder so it can run the test.

    Best wishes,
    Cody
    the Telerik team
    Do you think you know all the new features coming out in Test Studio R1 2011? Think again - we have some surprises and will share them with you in the What's New in Test Studio R1 2011 Webinar on May 12th.
    Register Today!
  5. Jacek Karwowski
    Jacek Karwowski avatar
    1 posts
    Member since:
    Aug 2008

    Posted 13 May 2011 Link to this post

    Hello Cody,

    thank you for your response.

    First - perhaps i wrongly understand this sentence:
     "The QA Edition doesn't directly integrate with MSBuild. You'll just have to manually modify your MSBuild to execute the command line tools."
    I though that you mean a modification of *.tstest file or msbuild project.... So, according to your last sentence, it is not possible to run QA version tests using MSBuild tool?

    Second:
    "You say you are using the mstest command line... is that from within a build or are you just manually running it from a Visual Studio command prompt and it's doing this?"

    I do it by running this: 
    mstest /testcontainer:C:\svn\Telerik\google_search\WebTest.tstest 
    from Visual Studio command prompt.
    File WebTest.tstest is produced with WebTest Studio QA Edition (standalone). My goal is to make it possible that our tester will be able to prepare Silverlight tests using this edition, without usage of Visual Studio. Then such prepared tests can be run during our build process. The Build process is performed by CruiseControl.net server. To do this i need to be able to run test from windows command line.
    On CI server (with cc.net) i want to have only runtime version of WebTest Studio. QA (standalone) edition will be installed on the tester workstation.


    Best regards, 

    Bartek
  6. Cody
    Admin
    Cody avatar
    3354 posts

    Posted 18 May 2011 Link to this post

    Hello Jacek Karwowski,

    Have you checked out our documentation on how to run tests via MSBuid/MSTest with CruiseControl.NET? Pay particular attention to section 4.2. CruiseControl also has documentation for how to add any command line execution to its automated build process. This should help clear things up for you.

    Regards,
    Cody
    the Telerik team
    Do you think you know all the new features coming out in Test Studio R1 2011? Think again - we have some surprises and will share them with you in the What's New in Test Studio R1 2011 Webinar on May 12th.
    Register Today!
  7. Jacek Karwowski
    Jacek Karwowski avatar
    1 posts
    Member since:
    Aug 2008

    Posted 25 May 2011 Link to this post

    Dear Cody, 

    in link you provide me there is a description of running tests produced with Test Studio Express (plugin for Visual Studio).
    Description says that i should build up a .dll file with tests - to do this i need .csproj file. AFAIK, such file is not produced if i use only Test Studio standalone.
    My problem is that i want to buy Standalone edition for testers and run tests they produce on CI.

    Till now i don't see this working.

    Best regards, 

    Bartek
  8. Cody
    Admin
    Cody avatar
    3354 posts

    Posted 25 May 2011 Link to this post

    Hi Jacek Karwowski,

    I am sorry for the confusion. What you want to accomplish is very doable. The Standalone version creates .tstest files (and also a DLL file that contains any coded steps). This documentation shows how to run Test Studio tests with MSTest from the command line. All you need to do is add these command lines to your CruiseControl build configuration.

    All the best,
    Cody
    the Telerik team
    Do you think you know all the new features coming out in Test Studio R1 2011? Think again - we have some surprises and will share them with you in the What's New in Test Studio R1 2011 Webinar on May 12th.
    Register Today!
  9. Jacek Karwowski
    Jacek Karwowski avatar
    1 posts
    Member since:
    Aug 2008

    Posted 26 May 2011 Link to this post

    Dear Cody,

    i follow these instructions with .tstest file produced with Test Studio standalone. I got the same problem as i mention few post earlier (May 3, 2011) - when i run following command (from VS command line): c:\svn\Telerik\RunnerTest2>mstest  /testcontainer:.\WebTest.tstest
    i got exception. This time i check yet .trx results file, and i found following:
     
    <RunInfo computerName="BIES" outcome="Error" timestamp="2011-05-26T11:00:02.5321967+02:00"><br>
            <
    Text>Failed to queue test run 'bstanisz@BIES 2011-05-26 11:00:00': Value cannot be null.<br>
          Parameter name: path1</
    Text><br>
            <
    Exception>System.ArgumentNullException: Value cannot be null.<br>
       Parameter name: path1<
    br>   
       at System.IO.Path.Combine(String path1, String path2)<
    br>
       at ArtOfTest.WebAiiVSIP.WebAiiTest.UpdateDeploymentItems()<
    br>
       at ArtOfTest.WebAiiVSIP.WebAiiTest.get_DeploymentItems()<
    br>
       at Microsoft.VisualStudio.TestTools.TestManagement.DeploymentManager.CheckAndMergeImpliedDependencies(ITestElement test, DeploymentItemCollection initialItems)<
    br>
       at Microsoft.VisualStudio.TestTools.TestManagement.DeploymentManager.DoDeployment(TestRun run, FileCopyService fileCopyService)<
    br>
       at Microsoft.VisualStudio.TestTools.TestManagement.ControllerProxy.SetupTestRun(TestRun run, Boolean isNewTestRun, FileCopyService fileCopyService, DeploymentManager deploymentManager)<
    br>
       at Microsoft.VisualStudio.TestTools.TestManagement.ControllerProxy.SetupRunAndListener(TestRun run, FileCopyService fileCopyService, DeploymentManager deploymentManager)<
    br>
       at Microsoft.VisualStudio.TestTools.TestManagement.ControllerProxy.QueueTestRunWorker(Object state)</
    Exception>
    </RunInfo>


    Do you have any idea what is wrong? I use Windows 7 64bit, polish locale version.

    Thank you in advance and best regards, 

    Bartek
  10. Cody
    Admin
    Cody avatar
    3354 posts

    Posted 31 May 2011 Link to this post

    Hello Jacek Karwowski,

    I am not 100% certain, but this may be a caused by a bug we already fixed. Please try upgrading to our latest internal build.

    Regards,
    Cody
    the Telerik team
    Do you think you know all the new features coming out in Test Studio R1 2011? Think again - we have some surprises and will share them with you in the What's New in Test Studio R1 2011 Webinar on May 12th.
    Register Today!
  11. Jacek Karwowski
    Jacek Karwowski avatar
    1 posts
    Member since:
    Aug 2008

    Posted 06 Jun 2011 Link to this post

    Hello Cody!

    Unfortunately i still have same exception as previously.

    Best regards, 

    Bartek
  12. Anthony
    Admin
    Anthony avatar
    19 posts

    Posted 09 Jun 2011 Link to this post

    Hello Jacek Karwowski,

    Thanks for your patience. I am still testing your scenario as I have reproduced that message locally.

    In the meantime, have you tried using Test Studio's built-in execution engine? It can also be run from the command line. Please see the instructions for ArtOfTest.Runner.exe.

    Best wishes,
    Anthony
    the Telerik team
  13. Cody
    Admin
    Cody avatar
    3354 posts

    Posted 26 Oct 2011 Link to this post

    Hello Jacek Karwowski,

    I just wanted to let you know we just found and fixed a bug in Test Studio that can cause the error:

    Failed to queue test run 'bstanisz@BIES 2011-05-26 11:00:00': Value cannot be null.
    Parameter name: path1

    We recently discovered it happens for tests created using Test Studio standalone that do not have any coded steps. The fix will be included in our next internal build which is due to be released on Friday of this week.

    Best wishes,
    Cody
    the Telerik team
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  14. David
    David avatar
    45 posts
    Member since:
    Aug 2010

    Posted 02 Nov 2011 Link to this post

    Cody,
    I am having similar issues with running MSTEST in a CI environment.

    I believe I have the correct XML written in to the ccnet.config (CruiseControl):
    <!--  VSuite Automation -->
      <project name="VSuite Automation - Rapid">
      
        <triggers>
        <scheduleTrigger time="3:30" buildCondition="ForceBuild" name="Scheduled">
        </scheduleTrigger>
        </triggers
        <labeller type="dateLabeller"/>
        <workingDirectory>WORKINGDIR</workingDirectory>
        <artifactDirectory>ARTIFACTDIR</artifactDirectory>
        <sourcecontrol type="svn">
          <trunkUrl>URL:8080/svn/VSuite/trunk/VSuiteAutomation/vsuiteautomation</trunkUrl>
          <username>COMPANY\cruisecontrol</username>
          <password>XXXXXXXX</password>
        </sourcecontrol>
       <tasks>
      
          <msbuild>
            <executable>C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe</executable>
            <projectFile>vsuiteautomation.csproj</projectFile>
            <buildArgs></buildArgs>
            <targets>Build</targets>
            <timeout>2700</timeout>
            <logger>"E:\Program Files (x86)\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MsBuild.dll"</logger>
          </msbuild>
      
        <!--Delete the testResults.trx first. This is required as MsTest will not create the file if it exists this could be merged with the mstest action in a single batch file
          <exec>
            <executable>$(windir)\system32\cmd</executable>
            <baseDirectory>E:\CruiseControl\Automation\VSuiteAutomation\buildlogs\buildlogs</baseDirectory>
            <buildArgs>/c if exist testResults.trx del testResults.trx /f</buildArgs>
            <buildTimeoutSeconds>30</buildTimeoutSeconds>
          </exec>-->
      
        <exec>
            <!-- Call mstest to run the tests contained in the TestProject -->
            <executable>E:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe</executable>
        <buildArgs>/testmetadata:E:\CruiseControl\Automation\VSuiteAutomation\vsuiteautomation\vsuiteautomation.vsmdi /resultsFile:E:\CruiseControl\Automation\VSuiteAutomation\vsuiteautomation\TestResults\testResults.trx</buildArgs>
      
            <!-- testcontainer: points to the DLL that contains the tests -->
            <!-- runconfig: points to solutions testrunconfig that is created by vs.net, list what test to run -->
            <!-- resultsfile: normally the test run log is written to the uniquely named testresults directory  -->
            <!-- this option causes a fixed name copy of the file to be written as well-->
            <buildTimeoutSeconds>30</buildTimeoutSeconds>
          </exec>
      
        </tasks>
      </project>

    When building the project in CC I get an error yet the build log seems to have run the executable correctly:
    - <buildresults>
      <message>Microsoft (R) Test Execution Command Line Tool Version 10.0.30319.1</message
      <message>Copyright (c) Microsoft Corporation. All rights reserved.</message
      <message /> 
      <message>Loading E:\CruiseControl\Automation\VSuiteAutomation\vsuiteautomation\vsuiteautomation.vsmdi...</message
      <message>Starting execution...</message
      </buildresults>

    When running the entire executable in a CMD window, the error I am getting is:
    Failed to queue test run 'dadams@<SERVERNAME> 2011-11-02 10:13:46': The given
    path's format is not supported.

    My suspicion is that the last error is thrown due to concatenation of one absolute path on to another but I would like your expert opinion and would welcome a screen share. I can make my schedule available to get this done.

    Thanks.
  15. Cody
    Admin
    Cody avatar
    3354 posts

    Posted 03 Nov 2011 Link to this post

    Hi David,

    I'm not sure what you're referring to by "the last error is thrown due to concatenation of one absolute path on to another". I don't see where that could be occurring. But one thing I do see is that your <buildArgs> section is missing both a /testlist or a /test parameter. You need one of these to tell MSTest what tests to execute. Without this I get the error "No tests to execute."

    Regards,
    Cody
    the Telerik team
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  16. David
    David avatar
    45 posts
    Member since:
    Aug 2010

    Posted 03 Nov 2011 Link to this post

    I added /test:login (a known test to exist) to the <buildargs> section. My understanding of MSTEST is that if there is no /test or /testlist, all tests found will run.

    The result I get when adding /test:login is:
    <buildresults>
      <message>Microsoft (R) Test Execution Command Line Tool Version 10.0.30319.1</message>
      <message>Copyright (c) Microsoft Corporation. All rights reserved.</message>
      <message>    </message>
      <message>Loading E:\CruiseControl\Automation\VSuiteAutomation\vsuiteautomation\vsuiteautomation.vsmdi...</message>
      <message>Starting execution...</message>
      <message>Final Test Results:</message>
      <message>Results               Top Level Tests</message>
      <message>-------               ---------------</message>
      <message>Not Executed          (TestTestlist/)e:\cruisecontrol\automation\vsuiteautomation\vsuiteautomation\common\login - scarlson.tstest</message>
      <message>Not Executed          (TestTestlist/)e:\cruisecontrol\automation\vsuiteautomation\vsuiteautomation\common\login.tstest</message>
      <message>0/2 test(s) Passed, 2 Not Executed</message>
      <message>Summary</message>
      <message>-------</message>
      <message>Test Run Error.</message>
      <message>  Not Executed  2</message>
      <message>  ---------------</message>
      <message>  Total         2</message>
      <message>Results file:  E:\CruiseControl\Automation\VSuiteAutomation\vsuiteautomation\TestResults\testResults.trx</message>
      <message>Test Settings: Default Test Settings</message>
      <message>Run has the following issue(s):</message>
      <message>Failed to queue test run 'SYSTEM@SERVERNAME 2011-11-03 17:25:52': The given path's format is not supported.</message>
    </buildresults>

  17. Cody
    Admin
    Cody avatar
    3354 posts

    Posted 04 Nov 2011 Link to this post

    Hello David,

    I did a little more research on this and found the results depend on what you have put into your Visual Studio test list(s). Specifying only /testmetadata will run all tests contained in all test lists you have defined in that .vsmdi file. It won't actually run all tests contained in your test project.

    Adding /test to /testmetadata will only run the test by that name if it's contained in one of the test lists. I demonstrate this in this video.

    Can you take a step back and just try manually executing MSTest from the command line and record what happens. I'd like to see a video of what happens on your machine. Jing is a good free screen recorder.

    Best wishes,
    Cody
    the Telerik team
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  18. David
    David avatar
    45 posts
    Member since:
    Aug 2010

    Posted 07 Nov 2011 Link to this post

    I think I have not been clear enough with my issue. The issue is not being able to identify which tests or testlists to run. The issue has been my inability to actually run ANY tests.

    http://db.tt/vCtHtJLp : this is a link to a movie I recorded demonstrating my issue.
  19. Cody
    Admin
    Cody avatar
    3354 posts

    Posted 09 Nov 2011 Link to this post

    Hi David,

    Thank you for the video. It really is strange it's not really even trying to run the tests. I'd like to look at this problem on your computer via GoToMeeting. Let me know what time zone you are in and your availability and I'll setup the meeting. I'm in Austin, TX which is Central Time (GMT -6).

    All the best,
    Cody
    the Telerik team
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  20. David
    David avatar
    45 posts
    Member since:
    Aug 2010

    Posted 09 Nov 2011 Link to this post

    Thanks, Cody.

    I am in Chicago so we are in the same timezone. If we can do this meeting first thing tomorrow (8:30 AM) that would be great. If not, let me know what time(s) work for you. My schedule is very flexible to yours.
  21. Cody
    Admin
    Cody avatar
    3354 posts

    Posted 09 Nov 2011 Link to this post

    Hi David,

    Unfortunately my Thursday morning is already full. I've got the afternoon free however. How about 1:30pm?

    Regards,
    Cody
    the Telerik team
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  22. David
    David avatar
    45 posts
    Member since:
    Aug 2010

    Posted 09 Nov 2011 Link to this post

    1:30 is great. I look forward to it.
  23. Cody
    Admin
    Cody avatar
    3354 posts

    Posted 10 Nov 2011 Link to this post

    Hello David,

    I just sent you a GoToMeeting request.

    1.  Please join my meeting.

    https://www2.gotomeeting.com/join/536925042

    2.  Use your microphone and speakers (VoIP) - a headset is recommended.  Or, call in using your telephone.

    Dial +1 (510) 201-0301
    Access Code: 536-925-042
    Audio PIN: Shown after joining the meeting
    Meeting ID: 536-925-042

    Best wishes,
    Cody
    the Telerik team
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  24. David
    David avatar
    45 posts
    Member since:
    Aug 2010

    Posted 11 Nov 2011 Link to this post

    Cody,
    I figured out that I have a pathing issue in my project. For example, the build is trying to execute C:\Users\David Adams\Documents\Test Studio Projects\common\click on nav\Compliance.tstest. However, the file is actually located at C:\Users\David Adams\Documents\Test Studio Projects\vsuiteautomation\common\click on nav\Compliance.tstest. The project is not getting included in the path. I added the project path to the system path but the "lost" test location is absolute.

    I will investigate a little more with our developers here. Any help you can provide would be hugely helpful.

    Thanks.
  25. Cody
    Admin
    Cody avatar
    3354 posts

    Posted 15 Nov 2011 Link to this post

    Hi David,

    Because you're dealing with a Visual Studio test list, I'm not going to be any help. That is 100% Microsoft territory. Test Studio doesn't come into the picture until the test is successfully queued and launched. Since the error message is "Failed to queue test run..." this means Test Studio isn't even entering the picture yet.

    Regards,
    Cody
    the Telerik team
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  26. David
    David avatar
    45 posts
    Member since:
    Aug 2010

    Posted 15 Nov 2011 Link to this post

    I got this figured out and should have updated the thread.

    The basedirectory had to be set to the bin directory. Had to figure that out through trial and error.

    <exec>
            <!-- Call mstest to run the tests contained in the TestProject -->
            <executable>E:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe</executable>
            <baseDirectory>E:\CruiseControl\Automation\VSuiteAutomation\vsuiteautomation\bin</baseDirectory>
            <buildArgs>/testmetadata:..\vsuiteautomation.vsmdi /testlist:SampleList</buildArgs>
      
            <!-- testcontainer: points to the DLL that contains the tests -->
            <!-- runconfig: points to solutions testrunconfig that is created by vs.net, list what test to run -->
            <!-- resultsfile: normally the test run log is written to the uniquely named testresults directory  -->
            <!-- this option causes a fixed name copy of the file to be written as well-->
            <buildTimeoutSeconds>60</buildTimeoutSeconds>
    </exec>
  27. Cody
    Admin
    Cody avatar
    3354 posts

    Posted 15 Nov 2011 Link to this post

    Hi David,

    Ah yes, that makes sense. Good to know you've fully resolved this problem and are automating away with Test Studio and CruiseControl!

    Regards,
    Cody
    the Telerik team
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
Back to Top