Hello Telerik Support.
I figured I would post here rather than use a support ticket because I would think that the answer to my question to be helpful for others trying to do the same.
I would like to understand better how the Telerik Scheduler is actually managing recurring test lists while getting the latest from TFS at each run. We are using a third party library reference in our tests and I would expect to be able to check in a new version of this assembly and have it automatically distributed with future test runs.
It appears that this is not happening as I would expect. I made changes to my assembly and needed to delete the scheduled recurring run and create a new scheduled recurring run in order to see the changes move to the execution servers and run properly there.
Can you help me understand what is the best practice for someone wanting to use this in this fashion? Develop a library for use in the tests while making ongoing changes/improvements to the library which should be picked up and consumed by existing scheduled tests. (It is expected that many of the library changes are invisible to the test - the test will not need to be updated to consume the new library, but the features of the library updates are desired to be part of the test run)
Additionally are there best practices regarding use of a third party library reference which is configurable with an app.config file? We are very interested in utilizing run time configurations from this library but are so far failing to understand how this deployment to the execution server is happening so that we might get our config details to be available to the library once deployed to the execution server.
Thanks!
8 Answers, 1 is accepted
Rank 1
Posting the details of our usage of this separately.
Using Telerik Test Studio Stand Alone
Our project is checked into TFS with the following structure:
TFSProject
-TestStudioProject
--Data
--TestLists
--TestFolders (Various other folders we organize our tests into)
--lib (library folder where we put our utility.dll)
We have created our utility.dll file and placed directly into the lib folder. Then in the Test Studio project setting we reference lib/utility.dll already within the project path, and in each coded test we add the reference to our namespace.
Please note, everything does work fine and properly. Tests run locally and remotely. Scheduled runs work fine and use the utility.dll. We are not having to manage the assembly as an installed dependency on the remote executors (which would be a deal breaker). Seems to work like we would expect and self deploys with the necessary libraries contained in the project like we would expect. But it seems the scheduled runs are not getting the latest version of this assembly from TFS when it runs.
We have noticed we need to manage the assembly directly in source control. Test Studio see's the lib folder we have checked into TFS, but it appears empty. We need to check out, update, and check in our library file manually.
Is it expected that the scheduled runs would get the latest version of the assemblies as well as the tests? Is there a best practice to this method which we are not following?
Thank you for your post.
Yes, your assumptions are correct. When you schedule a test list with checked Get Latest option Test Studio should performed that action before every recurring run.
Could you please elaborate on your workaround: We have noticed we need to manage the assembly directly in source control. Test Studio see's the lib folder we have checked into TFS, but it appears empty. We need to check out, update, and check in our library file manually.
What is your scenario? You modify the dll in Visual Studio and then check it into the TFS? After that Test Studio sees an empty folder?
Please elaborate a bit more.
Thank you!
Regards,
Boyan Boev
Telerik
Rank 1
I don't know that we have done a workaround. Very likely we have done it wrong - hence my question is how we should use this as a best practice.
If we want to include a DLL with our tests, and have that stored in TFS to enable it to deploy updates automatically to test execution machines, what are the steps to follow in adding this DLL and processing its updates in the future? Knowing that the DLL is updated outside of TestStudio and it will be updated after periods of time.
Our scenario is that we are writing our own utility class to provide us some additional functionality in our coded steps. When adding the DLL as a reference in the project it expectedly uses a file path reference to the assembly. For working development teams best practices are not to reference a direct path of where this assembly was built (example c:\users\ssudweeks\utilityproject\bin\utility.dll) but to place it within the TestStudio project and reference it with a relative path. This ensures that no matter where the developers place their workspace it will be able to find the assembly and use it.
When we have done the above and checked in the Test Studio project - I did not see any check in of the DLL into TFS. So none of my team members are going to be able to get latest and successfully run, or even build, my tests which use the DLL. (I assumed so - didn't actually test this) So I check it in manually to TFS outside of TestStudio to ensure all of our test developers have the necessary DLL now referenced. It doesn't seem that TestStudio recognizes that this DLL is part of TFS, although it does seem to download it to the developers machines, I assume by nature of doing a recurring get from TFS rather than TestStudio directly managing.
What we have noticed on the behavior of the scheduling server is that the referenced DLL is uploaded from the machine running Test Studio when the scheduled (recurring) test list is created. Updating the tests and checking into TFS changes the behavior of the tests for ongoing scheduled runs. However updating the DLL outside of Test Studio and checking into TFS (outside of Test Studio because Test Studio doesn't seem to check it in) does not update the functional behavior of the utility dll for ongoing scheduled runs. --At least from our observations, I'm not claiming we have comprehensive proof :)
Thank you for the detailed explanation.
Please allow me some time to consult this with our developers and I will update this ticket accordingly.
Thank you!
Regards,
Boyan Boev
Telerik
Please excuse me for the delayed replay.
1. Yes you are correct you should check in the dll manually not via Test Studio. This is done by design in Test Studio.
2. Regarding the dll and scheduling recurring test lists runs I have two recommendations:
2.1. Don't use a dll at all. We have now standalone code files in the latest version of Test Studio. You can implement your custom code there and avoid implementing/updating a custom dll.
2.2. If you really want to use a custom dll you should either always click on Upload Latest Project Files or check GetLatestFromTFS option. In this scenario the scheduling functionality will skip our storage database and gets always the latest files from TFS.
Hope that makes it clear.
Regards,
Boyan Boev
Telerik
Rank 1
Please excuse me for the delayed replay.
1. Yes you are correct you should check in the dll manually not via Test Studio. This is done by design in Test Studio.
2. Regarding the dll and scheduling recurring test lists runs I have two recommendations:
2.1. Don't use a dll at all. We have now standalone code files in the latest version of Test Studio. You can implement your custom code there and avoid implementing/updating a custom dll.
2.2. If you really want to use a custom dll you should either always click on Upload Latest Project Files or check GetLatestFromTFS option. In this scenario the scheduling functionality will skip our storage database and gets always the latest files from TFS.
Hope that makes it clear.
Regards,
Boyan Boev
Telerik
[/quote]
Thanks for following up Boyan. Unfortunately we are back where we started.
I've been watching the updates and keeping a close eye on things, including using the beta version. VERY excited for the new abilities with stand alone code files in Test Studio. Unfortunately that really isn't going to work well for us, because these utility libraries we are writing are not only for consumption of our UI tests through Test Studio. We are using the same utility libraries for other automated integration tests as well. Perhaps it isn't the end of the world if we maintain two copies of these libraries.
Your second option of checking GetLatestFromTFS is exactly what we have been using - and we've seen it behave in such a way a couple of times that we are reporting that this feature does not work as advertised. Hence the point of my post - am I doing it wrong, or is this a bug?
It has been a while since we've needed to make changes. Perhaps the next time we make a change we will make it a breaking change and see if the tests really pick up the latest binary file or not and report it as a bug if we can pin down what we see happening.
Thanks
~Scott
Boyan is out of the office, he will be answering this shortly.
Regards,
Ivaylo
Telerik
Thank you for being so helpful and comprehensive.
Yes if you observe an issue with GetLatestFromTFS after a major change please do not hesitate to contact us again.
Thank you again!
Regards,
Boyan Boev
Telerik