Thank you very much for the detailed overview of the requirements you have up in front - these will be very helpful for me to provide the necessary details as of Test Studio perspective.
The first important note for this scenario is that certainly you will need multiple machines to cover all of the listed scenarios. How many virtual users will be generated at the same time from each of the machines, depends on what resources these have - here
you can find some useful notes in regards the remote load test runs.
Designing the tests for the scenario
Next comes designing the tests - as you mentioned yourself there is no option to scale the users up and then down in the same test. So, you will need to have separate test lists with the same user scenario, but using different settings for the runs
. So, you have the designed the load test for the first hour to scale up users slowly for the whole hour. Similar to that, you need to create a separate load test, add the already existing profile
from the other test, then adjust the test to start with 15k users and ends with 8k users. Set the time to run the test for an hour as well as the user count ramp up - although the message says only 'ramp up', this will slowly decrease the amount of users when the starting amount users is larger than the ending one. See the screenshot below for reference. That way you can design the load tests for each of the 4 hours, adjusting the users and time as per the prerequisites.
Then comes the sudden burst test - it again should be a separate test adjusted as per its needs - 1 minute, starting and ending with hundred users, and executing the request in question.
So far, you should have 5 separate test lists with the different load tests - a test list for each of the four hours and the sudden stress test with 100 users.
Design the test list runs
Let me first mention some details about load test list execution as it acts a bit different in some aspects. If you have a single load test in a test list and schedule it to be executed on multiple machines, you have the option to distribute the test run among the selected machines
, or execute it in parallel on each of the machines. The difference between parallel and distributed load run
is how the virtual users are utilized - in parallel run any of the selected machine will utilize all virtual users (or he amount it is capable for) and in distributed run the virtual users will be evenly distributed among the machines.
Having this in mind and the specifics of the scenario to cover, my recommendation is to distribute the load test run among the machines.
Next comes the timing for the runs. Basically you can include the load tests to cover the 4 hours load in one single test list - that way the first test will run for an hour, then when it finishes the second will start, and so on for the 4 tests. However, in this scenario there will be some time between the test runs with no load - while finishing the test and preparing the next to start.
Therefore, I have the following suggestion - keep the four load tests in separate test lists and carefully choose the time and machines to schedule these. Here is how the setup should look like: you will need two sets of execution machines - let's say that each set has 4 machines.
- Schedule the first test list, which covers the first hour, distributed on one of the machine sets - this will utilize these machines for the next hour.
- Now schedule the second hour test list, again distributed, but on the other set of machines, which is currently not occupied.
- The tricky part here will be what start time to set for this second test list run so that you get no interruption in the load of the server. Each test list requires some time after its set start time to deploy the project files to the remote machines, build the project and prepare the run. Therefore, you will need to choose starting time shortly before the first test list end - what this time should be depends again on the specific environment.
- The third and fourth test lists should be also scheduled with the above idea in mind - shortly before the end time of the previous test list.
- The stress test list, can be scheduled at the desired time and should be executed on the set of machines, which is currently not busy with the other test lists - this should be the second set of machines if following the above suggestion.
In general, my recommendation will be to consider some time to perform initial testing in order to find out these fine timing adjustments for the specific environment. Also you can check what is the delay between test runs if the 4 hour tests are in the same test list - this shouldn't be more than some seconds and you can evaluate if this will be crucial for covering the scenario or not. So, knowing better what you can accomplish with the current environment, will help in choosing the most suitable option to cover the scenario.
The load testing results
Once you have setup the test runs and the execution is finished, you will need to explore the results. The detailed results of a test list run, which can be exported to excel
, contain information for all of the execution machines used for the run. The results will be divided per test, so basically there is no difference if the tests are in one or separate test lists.
Along with the exported results, you can use the Analyze results mode in the load test
to compare different metrics for the different machines, or overall test. Basically, the results in the Excel can also be grouped and modified to present the same graphics tailored out-of-the-box in the Test Studio UI and this is how you can combine the data for the different test runs to continue the generated data analysis.
It will certainly take you some time to setup all of the above and I hope the provided information will be of further help for you. Of course, if any other questions rise up, pleas do not hesitate to get back to me.
Thank you in advance for your cooperation.