The upcoming 2013 R1 release of Test Studio comes with an entirely reworked remote execution and scheduling engine. The need for this change came from a desire to allow parallelization of test runs and to bring different test types into parity. When enterprises have many test machines, they often want to be able to run tests from a test list across these machines, rather than running them in series. We also wanted to bring load tests into parity with web tests, allowing them to be scheduled, as well as making them easier to set up. All of this is addressed by the new remote execution engine.
The prior scheduling and load services were distinct services and required set up of each to get either scheduling or load working respectively. There were no shared entry points or services and setting up one did not help you set up the other. Additionally, your data was scattered amongst different databases between the scheduling database and the load database. Finally, you could not run a load test locally; you had to run it with the load services, and load tests could not be scheduled.
Figure 1 – Old remote execution architecture
The new remote execution engine unifies all of this into one easy to use, yet highly configurable series of systems.
If you are a small, one-man test team or have individual testers that don’t share tests or results, the default architecture is perfect for you.
Figure 2 - Default installation
Merely connect your projects to scheduling use the ‘Locally’ option and start scheduling test lists that can also include load tests!
Figure 3 - Connecting your project to a Scheduling Service
Most of our customer will want to schedule test lists to run remotely across multiple machines. In this case, install the scheduling service on one of the remote machines, and fire up an execution server on each machine just like before.
Figure 4 - New remote execution architecture
Now though, when you configure your scheduling service (right there from the start menu) you will be asked where your Storage Service is.
Figure 5 - Setting up your remote Scheduling Service
This storage service is the data bus that stores all of your tests and results in the new execution environment. No more copying result files locally to each machine, etc. Everything just works with the storage service when you connect your project to a remote scheduling service. You don’t even have to tell Test Studio where the Storage Service is--the Scheduling Service knows and will inform Test Studio for you.
Selecting multiple machines to run a test list prior to this release meant that each machine ran all the tests. Well, that is fine if that is what you wanted, but many people wanted to be able to spread the tests out amongst many different machines. The old remote execution could not do that.
Figure 6 - Tests from a test list can now be distributed across many machines
Distribution is built into the new remote execution system. It is the default choice for new remote or scheduled test lists. You can still turn it off to make each machine run all of the tests in order, but one of the major benefits of the new architecture is distribution, so you should definitely use it!
Figure 7 - Distributing across multiple test machines
Note: If you are relying on test order to make your tests work, stop it! Seriously, don’t do that. See Jim’s upcoming blog post for more details. In this case your tests will likely fail if you use the distribute option, so be sure you turn it off.
Part of the reworking of the Scheduling Service and Execution Servers was enabling them to understand Load Tests. The Profiling Service, Load Agent Service, and Load Controller Services of the last release are all now just processes that the Execution Server can fire up as it needs them! This means you no longer have to install and set them up beforehand. No setup, no mess, no fuss, just load testing made easy.
When you are in the test tab for a load test you will notice that the environment screen is gone. You can still gather performance data, you just don’t have to set up any of the load service components. When you go to the run screen and hit run, it will talk to a local scheduling and execution process (it runs in your system tray when you start up test studio) and everything will be run on your machine. If you watch your processes, you will actually see a controller and agent start up locally.
The intent here (like with quick execution of web tests) is to test the test. You will not get good throughput with a single agent, but you can verify the load test runs without faults or http errors.
To do multiple agents in your load test, create a test list and include the load test in it. Select multiple machines and distribute and you will get an agent per machine. One machine will randomly be selected to also fire up as the controller.
There are exciting things coming for remote execution and scheduling, but to get there from here we had to redo a majority of the back end services. The benefits of this reworking are many: test list distribution, much easier set up, results all in one location, load tests in lists, quick run load tests, etc.
Want to see for yourself? Download Test Studio today!
Jared Twing is the Load and Performance Testing Product Manager for Telerik's Test Studio. He has 15 years experience across government contracting (including time at NASA), private software firms, and the video game industry. When not planning software releases, he is hanging out with his family or playing games. Lots and lots of games.