If my understanding is correct load testing should represent simulating of more users trying to execute the same test. I know it`s just basic explanation. I`ve read the article connected to data driving of load tests - http://www.telerik.com/automated-testing-tools/support/documentation/user-guide/load-testing/designing-load-tests/load-modifying/load-data-driving.aspx. To be honest I don`t know exactly what`s this for. Here is a sample.
I have a login test where user is logged into an application. If I want to simulate more users logging in at the same time I can do it using load test. I also know I can data drive performance tests so it`s executed one by one depending on users in external file (e.g. SQL db). If I capture my login test in load test settings I can then set number of virtual users (we have only trial pack for 20 users). If I data drive it what`s this for? Does that mean that e.g. if I have 5 different users in my external file and 20 virtual users set that every virtual user will log in 5 times using a different username from the external file? What`s the main difference between load tests without data driving or with data driving?
I hope the explanation is somehow clear;-) I want to be sure I understand this correctly.
Thank you.
Petr
10 Answers, 1 is accepted
Permit me to interject here because I have a lot of experience with load test and can answer these questions for you.
The purpose to data drive a load test is to simulate multiple different users logging in instead multiple instances of the same user e.g. if you run a load test with 100 virtual users, without data driving it you would have 100 of the same user hitting your web server all at once. Many applications do not support the same user being logged on multiple times from multiple locations. By data driving your load test, or more specifically the login sequence, (with 100 rows of data in your data source) every virtual user can now login as a different user instead of them all being the same "Joe". To complete the description, if you set your test for 100 virtual users, but only have 50 rows of data, the first 50 VU's will "consume" the first 50 rows and the next VU will restart at the beginning of the data source (e.g. Excel file). The next VU would get row 2 and so on. Thus you would end up with two instances of each user defined in your data source hitting your web site at the same time.
You can data drive other transactions, but logging in as different users is what data driving load tests was originally designed for. I hope that clarifies the purpose of data driving a load test.
Cody
the Telerik team
Test Studio Trainings
Rank 1
Thanks a lot. Your explanation is clear and comprehensive. However, I still have a couple of questions so I hope I can bother you one more time;-) Here are my questions:
1. I`m not sure about workload. What does it exactly do? How are the tests influenced by the percentage proportion? Please see workload.png as a sample. How would results change if e.g. I set login for 10% and stressTest02-onlineview for 90%?
2. Data driving - If I capture a test I have an option to data drive some of the steps - please see data_drive_load_test.png. And I can also data drive the whole test in project tab - data_drive_load_project.png. Is this also different? I understand if I can`t data drive any of the steps I have to bind a data source to the test in project tab but if I have this option I`m not sure.
3. Dynamic targets - TestStudio captured my test and offered me a possibility to set dynamic targets. Again not exactly sure about the purpose...are these targets commonly used in load tests? Please see dynamic_targets.png.
4. Important metrics - As you mentioned you`ve got lots of experience with load testing. In our application we would like to test mainly some metrics at the user level, let`s put it this way. I know Telerik offers many metrics but not all of them are usable for our tests. One of the key metrics which we`d like to trace is speed and duration of queries of responses from SQL Server. Would you be so kind and recommend me any of the offered metrics which could best measure these requests?
I know many questions but if you can help me out at least with some of them I`ll be happy.
Thank you.
Petr
Workload - the purpose of work load is to more closely simulate the percentage of what the various users of your web application are doing. Usually you want to create multiple User profiles that represent the kinds of actions your users will be performing with your web application. Things like:
- Browsing product - 80% of your users will do this
- Purchasing product - 15% of your users will do this
- Checking on shipping status - 4% of your users will do this
- Requesting return - 1% (or less hopefully) of your users will do this
Notice that in the real world you have a very large percentage of users doing one thing and small percentage of users doing the rest of the things. By adjusting the work load you more accurately simulate the user type distribution. In my above example, set profile 1 to 80%, profile 2 to 15%, profile 3 to 4% and profile 4 to 1%. When our load test runs it will create VU's, assign them to the various user profile according to that distribution percentage.
Data Driving - if you want to data drive your load test, for the purpose of logging in as different users, it's better to start with a web test that is not data driven i.e. use a web test that logs in as one user does work and logs off, nothing else. If you create a load test from a data driven web test what you'll get is one long load test with all of the iterations of the web test concatenated together as one very long series of HTTP transactions. The load test will do user 1, then user 2, then user 3 and so on in series and you will not be able split up your users so they run in parallel as multiple VU's. It is virtually impossible to data drive this type of load test for multiple user login's.
Dynamic Targets - yes it is very common for load tests to use dynamic targets and most web applications require they all be turned on. The purpose of dynamic targets is to handle the unique run-time data that the web application generates at run time, sends to the client/browser and expects to be returned on subsequent HTTP requests. This includes common things like session ID, user ID, cart ID, UI language and so on. These unique values come from cookies being set by the web application and/or query string parameters. If you do not select the dynamic target during design time, Test Studio will send the value that was recorded. When you do turn on the dynamic target Test Studio will remember the value that the server sent and echo it back on the following HTTP transactions in place of the recorded value.
Important metrics such as speed and duration of queries of responses from SQL - Those specific metrics are much more difficult to get in the form you're after. Test Studio can fetch any of the Windows Performance Counters. These counters are a measure of how heavily is that particular resource is being used at that moment in time, such as %CPU load, %Memory used, GB of disk space used and so on. On SQL Server machines SQL adds a number of counters for the SQL process itself - see attached screen shot. I don't know if you'll find exactly what your looking for however (how long a SP takes).
For the SQL metrics you are after you might be better off with something like this:
http://www.simple-talk.com/sql/performance/which-of-your-stored-procedures-are-using-the-most-resources/
Here are more articles about performance counters and SQL server:
http://www.quest.com/backstage/images/promotions/SQLServer-Perfmonance-Poster.pdf
http://www.extremeexperts.com/sql/articles/sqlcounters.aspx
Cody
the Telerik team
Test Studio Trainings
Rank 1
Thanks a lot for your complex reply. I really appreciate it;-) It`s better to post on the forum so everyone else can see this discussion.
I still have some concerns about data driving. I unterstand your explanation. But how to handle this if I have a test which is composed of subtests? Please see attached test_decomposition.png which represents my sample. I`d like to use this test as a load test. This test represents logging into application and then do other stuff, e.g. pick a car, choose a print report, etc. and then log off. All the steps apart from login are in one subtest which is data driven. I`d like to simulate this test for more users at one time. So if I data drive this (I`ll need this for login purposes since I recorded login with only one value) I`m not sure if I`ll get the correct load test, i.e. logging of e.g. 20 different users at one time (see your explanation in previous comment).
And I also face one more problem. I set my test for 20 VU and duration 2 minutes but the results show only 12 users and 5 minutes. Please see test_duration_number_of_users.png. What might be causing this?
Thank you.
Petr
Rank 1
Thank you.
Petr
No, the default 20 VU's doesn't come with the Run-Time edition licenses. It comes with the full Test Studio license. The way it works is that the Load Controller you setup has a limit on the number of VU's it can spin up. With one full Test Studio license (even the trial license) you can create one Load Controller with 20 VU's and spread them across as many load agents as you like. The 20 VU's will be evenly divided and spread out across all of your load agents.
To get more than 20 VU's you must purchase one of our VU packs.
Cody
the Telerik team
Test Studio Trainings
Rank 1
Thanks for your info. What about my question below? I mean that one posted on Feb 1, the longer one;-)
Petr
"how to handle this if I have a test which is composed of subtests?"
My apologies for missing this. Keep in mind that when you are capturing a user profile from an existing functional web test, we capture any/all HTTP traffic that was exchanged between the server and the browser. It makes do difference if the web test has subtests (or how). Under the cover what is literally happening is that we insert our HTTP proxy which intercepts all HTTP transactions during capture of a user profile. You then run your functional test, with or without subtests, and Test Studio records all of the HTTP transactions as one long sequence. It will not put in any dividers on the subtests. As they execute the HTTP transactions it generates will simply be concatenated to the user profile.
"All the steps apart from login are in one subtest which is data driven."
Since Test Studio does not differentiate parent test from subtests, not even data driven sub tests, what you capture for your load test is one long sequence of HTTP transactions including all the transactions for each data driven iteration of the sub test.
Since you did not data drive the parent test or the login test, what you capture will be for one user only. You can then data drive the new captured load test in the login HTTP transactions to get multiple different users hitting your website concurrently. What was the data driven subtest will now simply be one long set of HTTP transactions all concatenated together.
"I set my test for 20 VU and duration 2 minutes but the results show only 12 users and 5 minutes."
This I cannot explain. If you set your test for two minutes it is supposed to run for almost exactly that amount and stop. At precisely two minutes (plus or minus 1 second) the load controller will send a STOP command to all of the load agents, then wait for them to signal they have aborted/stopped. For this reason we sometimes see a load test running a few seconds past the time set to run, but only a few seconds, not minutes.
I am not able to reproduce this behavior you are describing. Do you have a load test that reproduces this that we can run here to determine what's causing this behavior? Also since this is a new technical problem (instead of general load test questions) this is worthy of a new support ticket so we can track this problem separately.
Cody
the Telerik team
Test Studio Trainings
Rank 1
Hello,
I do not want to spam forum with new topics and my question is related to load testing setup.
We need to create a load test for a website. There is an upcoming event and we expect a lot of traffic based on our experience from the past. We do have a load test model in mind which would simulate the real life user behavior, but we do not know how to set it up in Telerik Test Studio.
The scenario would, in short, be that load test should run for 4 hours. Let me break down what we need to do and ask questions for each of our issues.
1.We expect the traffic to slowly increase in the 1st hour starting from zero up to around 15 000 completed user journeys (profiles) per hour with average user journey taking around 3.5 minutes to complete. We know how to do this for the first hour. But...
2. We expect the number of user journeys to drop in the second hour from 15k to 8k. How do we gradually scale down the number of virtual users? Do we do it in the same test or different one? We couldn't find that option in Test Studio, only to increase them over time. There is no way to add a breakpoint where number of users rise and then drop for the test.
3. In the 3rd hour we again expect the count to go up to 15k with sudden increase of submitting a certain request. We want to model the situation where the journey was not done at all to a sudden boost to 100 users submitting the request every second for 1 minute and then stop. How can we model this in TS?
4. In the 4th hour the user count should drop from 15k to zero.
My general questions are, for this example:
- Do we model a different test for each hour and add them to the test list or can we simulate the variation of user numbers in the same one?
- Do test lists run load tests in parallel or in series, one after another like UI web tests?
- If test lists run load tests in series do they break the load (make a pause in generating load) or do they continue w/o interruption?
- How can we start the sudden burst test in the middle of the run? How do we include that profile in the test if others should run for an hour and this one only for 1 minute in parallel?
- In the end are the results for each test separate or can we get the overall report?
- Is there a way to get unified overall result for load test regardless of how many machines we have used as load test generators?
Thanks,
Aleksandar
Thank you very much for the detailed overview of the requirements you have up in front - these will be very helpful for me to provide the necessary details as of Test Studio perspective.
The first important note for this scenario is that certainly you will need multiple machines to cover all of the listed scenarios. How many virtual users will be generated at the same time from each of the machines, depends on what resources these have - here you can find some useful notes in regards the remote load test runs.
Designing the tests for the scenario
Next comes designing the tests - as you mentioned yourself there is no option to scale the users up and then down in the same test. So, you will need to have separate test lists with the same user scenario, but using different settings for the runs. So, you have the designed the load test for the first hour to scale up users slowly for the whole hour. Similar to that, you need to create a separate load test, add the already existing profile from the other test, then adjust the test to start with 15k users and ends with 8k users. Set the time to run the test for an hour as well as the user count ramp up - although the message says only 'ramp up', this will slowly decrease the amount of users when the starting amount users is larger than the ending one. See the screenshot below for reference. That way you can design the load tests for each of the 4 hours, adjusting the users and time as per the prerequisites.
Then comes the sudden burst test - it again should be a separate test adjusted as per its needs - 1 minute, starting and ending with hundred users, and executing the request in question.
So far, you should have 5 separate test lists with the different load tests - a test list for each of the four hours and the sudden stress test with 100 users.
Design the test list runs
Let me first mention some details about load test list execution as it acts a bit different in some aspects. If you have a single load test in a test list and schedule it to be executed on multiple machines, you have the option to distribute the test run among the selected machines, or execute it in parallel on each of the machines. The difference between parallel and distributed load run is how the virtual users are utilized - in parallel run any of the selected machine will utilize all virtual users (or he amount it is capable for) and in distributed run the virtual users will be evenly distributed among the machines.
Having this in mind and the specifics of the scenario to cover, my recommendation is to distribute the load test run among the machines.
Next comes the timing for the runs. Basically you can include the load tests to cover the 4 hours load in one single test list - that way the first test will run for an hour, then when it finishes the second will start, and so on for the 4 tests. However, in this scenario there will be some time between the test runs with no load - while finishing the test and preparing the next to start.
Therefore, I have the following suggestion - keep the four load tests in separate test lists and carefully choose the time and machines to schedule these. Here is how the setup should look like: you will need two sets of execution machines - let's say that each set has 4 machines.
- Schedule the first test list, which covers the first hour, distributed on one of the machine sets - this will utilize these machines for the next hour.
- Now schedule the second hour test list, again distributed, but on the other set of machines, which is currently not occupied.
- The tricky part here will be what start time to set for this second test list run so that you get no interruption in the load of the server. Each test list requires some time after its set start time to deploy the project files to the remote machines, build the project and prepare the run. Therefore, you will need to choose starting time shortly before the first test list end - what this time should be depends again on the specific environment.
- The third and fourth test lists should be also scheduled with the above idea in mind - shortly before the end time of the previous test list.
- The stress test list, can be scheduled at the desired time and should be executed on the set of machines, which is currently not busy with the other test lists - this should be the second set of machines if following the above suggestion.
In general, my recommendation will be to consider some time to perform initial testing in order to find out these fine timing adjustments for the specific environment. Also you can check what is the delay between test runs if the 4 hour tests are in the same test list - this shouldn't be more than some seconds and you can evaluate if this will be crucial for covering the scenario or not. So, knowing better what you can accomplish with the current environment, will help in choosing the most suitable option to cover the scenario.
The load testing results
Once you have setup the test runs and the execution is finished, you will need to explore the results. The detailed results of a test list run, which can be exported to excel, contain information for all of the execution machines used for the run. The results will be divided per test, so basically there is no difference if the tests are in one or separate test lists.
Along with the exported results, you can use the Analyze results mode in the load test to compare different metrics for the different machines, or overall test. Basically, the results in the Excel can also be grouped and modified to present the same graphics tailored out-of-the-box in the Test Studio UI and this is how you can combine the data for the different test runs to continue the generated data analysis.
It will certainly take you some time to setup all of the above and I hope the provided information will be of further help for you. Of course, if any other questions rise up, pleas do not hesitate to get back to me.
Thank you in advance for your cooperation.
Regards,
Elena
Progress Telerik