Scheduled Test List Not Running the Latest

8 posts, 0 answers
  1. Sean
    Sean avatar
    18 posts
    Member since:
    Jul 2014

    Posted 26 Sep 2014 Link to this post

    I am having a big issue with scheduled test lists.  They are not running the latest version of the tests within it regardless of if I upload latest, run remotely or schedule a one off run.  There are even times where I will delete a test case and it will still be run in the scheduled run this is most troubling and highly annoying.  I would expect that it would pull the latest and when I upload the latest that will be used and this is apparently not the case.  When I run locally it runs fine but when I try to execute remotely its low percentage that it actually runs what I expect it to run.  I have even monitored project folders while tests are running and I see weird things like old test lists in the test list folder that I have deleted weeks ago.  

    How are the scheduled runs stored?  How is it retaining old test cases to run even when the case has been deleted from the test? Why does uploading latest not actually upload the latest?

    I am using version 2014.
  2. Sean
    Sean avatar
    18 posts
    Member since:
    Jul 2014

    Posted 26 Sep 2014 in reply to Sean Link to this post

    *using version 2014.3.903.0
  3. Cody
    Admin
    Cody avatar
    3360 posts

    Posted 30 Sep 2014 Link to this post

    Hi Sean,

    There are a couple of possibilities for the symptoms you describe. Let me start with a description how the scheduling server system works.

    Tests and test lists are stored in our Storage service, which uses a SQL DB for the actual physical storage. Nothing ever gets deleted out of this DB. We admit this is a flaw and are actively working on a new design to overcome this flaw. Thus deleted tests and test lists will still show up on the remote executor when you run any test list.

    When you schedule a test list to execute, Test Studio will try to intelligently push into storage only those tests that are part of that test list and have changed. It does this by comparing the files date & time stamp to the one stored in the SQL DB for the test. If what's on disc is newer than what's in storage it will get pushed, else it will be skipped.

    Where this may break down is if you call a subtest via a coded step. Test Studio is not yet intelligent enough to scan coded steps for dependent subtests. As a result that particular sub test won't get pushed into storage and the latest version won't get used during test list execution.

    Another possible hazard is if tests were copied outside of Test Studio, such as in Windows explorer, the ID of the tests and test lists, which is stored within the file itself, will become duplicated. Now when you schedule this copied test or test list, it will conflict with what's already in storage.

    If any of the above hazards sound like they apply to you, let me know and we'll work on overcoming the problem.


     Regards,
    Cody
    Telerik
     
    The New Release of Telerik Test Studio Is Here! Download, install,
    and send us your feedback!
  4. Sean
    Sean avatar
    18 posts
    Member since:
    Jul 2014

    Posted 01 Oct 2014 in reply to Cody Link to this post

    Yes I use Test as a Step for many of my tests because I have a lot of repetitive and reusable test cases, These are also duplicated for environmental purpose for example I have a Stage version and a Production version for logins because they vary between environments. My hope was I could just enable and disable these Steps and the scheduler would pull this latest change. This duplication is not limited to test as a step either Nav URLs get the same setup for environments and enabling/disabling these single steps is not always picked up either.

    I did run into the issue of copying using windows explorer and had the issue of it running tests lists from another project so now I use the Add Existing Function to create a new project and reuse existing test.   

    To get around all this my solution has been to create a new project, add existing test 1 by 1, delete any of the environment specific test steps then create a new test list and run that.  This adds new issues as TestStudio will sometimes crash.
  5. Cody
    Admin
    Cody avatar
    3360 posts

    Posted 02 Oct 2014 Link to this post

    Hi Sean,

    Yes I use Test as a Step for many of my tests because I have a lot of repetitive and reusable test cases

    Excellent! That's exactly what we designed it to be used for.

    I did run into the issue of copying using windows explorer and had the issue of it running tests lists from another project so now I use the Add Existing Function to create a new project and reuse existing test.

    Glad to hear that. That is the right way to do it. Add Existing Test will replace the tests GUID contained in the original .tstest file with a newly generated one, guaranteed to be unique. Test steps do not have GUID's. You can duplicate those, enable/disable them all you want and this will not cause any problems.

    My hope was I could just enable and disable these Steps and the scheduler would pull this latest change

    Yes it should work that way provided:
    1) You save the changed tests to disc
    2) The changed tests are included in the test list (not only referenced by a coded step).

    Given the above 2 conditions, Test Studio should be picking up the changes, pushing the updated tests to our Storage service, which will cause them to be used during execution of the test list run. If you think you have met both of these conditions and it's still not working as expected, do you think you can demonstrate the problem to me in an online meeting?

    Regards,
    Cody
    Telerik
     
    The New Release of Telerik Test Studio Is Here! Download, install,
    and send us your feedback!
  6. Dennis
    Dennis avatar
    18 posts
    Member since:
    May 2014

    Posted 31 Aug 2015 Link to this post

    I'm facing a similar issue. I first updated a test to included a coded step, and then I ended up converting the coded step to a test as a step, but the latest version is not running when I execute the test locally as part of an established test list. I even tweaked and/or added a few comments to the test, but the list is still executing the old version of the test.

    It would be nice if the "Upload latest project files" option on the results tab would give you a count of the steps that were uploaded or some type of status message. Right now, you have no way of knowing whether the upload was successful, or if it even did anything.

  7. Ameer
    Ameer avatar
    1 posts
    Member since:
    Sep 2014

    Posted 01 Sep 2015 Link to this post

     Hi Cody

    I am experiencing very seriouse issue with our Automatic test setup which consist of 8 test agents (TS Runtime) and one controller (TS Ultimate) with scheduling . The storage is on a SQL Server and the test cases (the project) is version controlled by TFS.

    When executing test list DISTRIBUTED then all the 38 test cases in the list fails. If I target the test list to be excuted on only one Test Agent  then the test cases will pass.

     This is a show stopper for us and we need an urgent assistance.

    The controller is on windows 2008 R2 server and the test agents are mix of Win7, Win8 and win8.1.

    I am using version 2015.2.723  on all machines and and selecting Get latest from TFS.

    I appreciate your help and many thanks in advacne.

    Br.

    Ameer

     

     

     

     

     

     

     

     

  8. Ivaylo
    Admin
    Ivaylo avatar
    750 posts

    Posted 02 Sep 2015 Link to this post

    Hello Ameer,

    Let's keep the communication in the ticket you have opened for this behavior without updating other threads that are not related to the issue you are experiencing.

    Thank you for your understanding.

    Regards,
    Ivaylo
    Telerik
     
    The New Release of Telerik Test Studio Is Here! Download, install,
    and send us your feedback!
Back to Top