Telerik blogs
Test StudioT2 Light_1200x303

It’s great that your automated tests show that your application is working correctly. But now you need a load test to see what your application does when users start showing up.

You’ve been using Telerik Test Studio to create automated tests that demonstrate that your application works “as intended” as it evolves (“functional tests”).

But now you want to test for something new: How well does your application perform (“non-functional tests”)? Particularly, you want to know how your application behaves “under load”: What happens to your application as the number of users increases or as users perform “demanding” tasks (large uploads or downloads, for example). This is the essence of load testing.

Test Studio showing the results of running a successful test

Test Studio lets you create load so you can determine how your application performs under expected loads (both typical and peak). But you can use a Test Studio load test two other ways:

  • Stress test: How does your application perform under extreme conditions? Specifically: How much load can your application address before it breaks?
  • Soak test: How does your application perform over time with the expected combination of typical and peak loads?

Because of the duration and level of demand required by these kinds of tests (potentially hundreds of users and over multiple days), the only way to perform them is to use an automated tool like Test Studio. Test Studio will not only let you run load tests (and use them for stress and soak testing), Test Studio will also let you leverage any existing functional test scripts you’ve already created to create your load test. This enables you to do some “shift left” test design and move your load tests to earlier in your testing process.

Creating a Load Test

Adding a load test to a project is easy. From Test Studio’s menu, in the Project tab, select New | Load Test. (You can also access all the code for this post here.)

The Test Studio menu bar with the Project tab selected. The first icon, labeled “New” has a dropdown arrow below that has been clicked to show a list of options. The Load Test option has been selected.

After naming your test, it will appear in your project.

The Test Studio main screen, roughly divided into three columns. The Project window on the left shows two tests – the one called Update Department Load is highlighted. On the right a pane labeled LoadTest with textboxes for holding information about the test (e.g., Owner, Priority) covers some panes below it.

Your next step is to start adding user profiles to your test. A user profile in a load test is similar to a test script in a functional test—a user profile is a specific set of actions that represents a typical user’s interaction with your application. The type of actions recorded in a user profile are different, though: While a functional test records the user’s interaction with the UI, a user profile records the browser’s interaction with the server.

To start creating your user profile, move to the User Profiles and Workload pane in the middle of the Test Studio window (you may need to unpin Test Studio’s Properties and Step Builder panes to fully expose this pane). If that pane isn’t displayed, either double-click on your load test in the Project pane on the left side of Test Studio or, on the Test Studio menu’s Tests tab, click the Design icon.

The same view of Test Studio has before but the panes on the right side of Test Studio have been closed to give more room for the User Profiles pane. To the right of the User Profiles pane are two panes stacked above each other called Test Settings and, below it, Available Users

To create your first User profile, click on the Capture User profile icon to add a user profile.

A close up view of the User Profiles pane with the first icon on the pane’s toolbar (a camera in a square) circled. A tooltip below the icon reads “Capture User Profile”

That will open the Capture Traffic dialog.

The Capture Traffic dialog. It’s divided into the three steps. Step 1 gives a choice between two capture methods (Capture from existing web test and Capture new session – Capture new session is selected). Step 2 asks to pick a traffic source and has icons for all the major browser (Chrome, Edge, Firefox, etc.)  plus two icons labeled Device. Step 3 is labeled Record steps and has a single disabled button labeled Stop Recording

In this dialog you have two choices for creating a user profile:

  • Capture from an existing web test (i.e., use an existing test script from the same project)
  • Capture a new session (i.e., record a new set of actions)

Regardless of what option you pick, you’ll need to select one of the browser icons listed in Step 2. However, what happens after you pick the browser icon depends on which option you pick in Step 1.

If, in Step 1 of the Capture Traffic dialog, you pick Capture a New Session (the default), then once you’ve selected the browser you want to use, Test Studio will start the browser so your can record the actions that make up your user profile. Just like creating a test script, you’ll work through your application performing the actions you want to be part of your user profile. Test Studio will capture your actions to create the user profile.

While this is very similar to the experience of recording a test script, you’ll notice three differences:

  • During the recording, you won’t have access to the Test Studio floating tool bar.
  • The resulting user profile just records the browser’s interactions with the server (i.e., GET and POST requests with their corresponding responses).
  • Because of that focus, after the recording, the editor for making changes to your user profile is more limited that when creating a test script.

If you’ve been recording test scripts, you’ve probably realized that recording a user profile isn’t going to be trivial. However, if you accidentally record some actions that you don’t want to be part of you user profile, you’ll have the option to delete them after you finish recording. You can also record your user profile against one server (e.g., your development server) and then, later, modify the profile to run against another server (e.g., your test server).

Still, for any user profile that involves more than a few interactions with the application, you may prefer to use an existing test script because it gives you more options for modifying your recording. In that case, in Step 1, choose Capture from an existing web test. With that option, Test Studio will display a list, just below Step 1, of all the scripts in your project.

The Capture Traffic dialog. In Step 1 the Capture from existing web test option is selected. A list of web tests is displayed below step 1, showing a test named Update Department. Step 3 with its Stop Recording button is no longer displayed in the dialog.

After selecting a test script and a browser from the icons displayed, Test Studio will open a command window and execute the test script, capturing the script’s actions to create your user profile.

 The Test Studio window with a command window overlaying it. The command window is displaying a list of messages beginning “Connecting to Controller! Connected. Executing test: “Update Department.” This is followed by information about the test.

If the test script you want isn’t part of your project, you can import it. You can also import an existing user profile from another test in the same project by using the Import user profile button at the top of the User Profile pane (that dialog will also let you create a user profile by importing a Fiddler trace).

A close up of the top of the User Profile pane showing the icons in its toolbar. The second icon from the left (and arrow curving to down and to the right) is circled.

Editing the User Profile

When you finish recording your user profile, Test Studio will display the Edit User Profile dialog and, over top of it, the Dynamic Target dialog for your user profile. With the Dynamic Targets dialog, your default action should be to select all the items and close the window (there is more you can do in this dialog).

The Dynamic Targets dialog displaying a list of targets (on the left) and (on the right) HTIML tags. In between those two columns are columns labeled From and To. There is a checkbox to the left of each row in the list. At the top of the column of checkboxes, in the column title area, is another checkbox. That checkbox has been checked, causing all of the checkboxes for the rows to be checked.

When you close the Dynamic Targets dialog, you’ll be shown the Edit User Profile dialog. This is where you can give your profile a meaningful name (if you created the profile from an existing web test, the profile’s name will default to the name of the web test).

The upper left corner of the Edit User Profile dialog box containing a textbox labeled name. The textbox contains the text Update Department. The textbox and its label are circled. Below the name textbox is a list of steps numbered 0 to 3 (the list has been cut off after step 3). The first three entries in the list are “GET” followed up a URL for a site’s web page, “GET” followed by the URL for a graphic on the page, and an entry labeled “Think time.”

While recording, you may have accidentally clicked on a link or button that you don’t want to include in this your load test. In the Edit User Profile dialog, you can use the trash can icon to delete any steps you don’t want to be part of the user profile.

A slightly shorter list of steps from the Edit User Profile dialog. To the right of the second step, the trashcan icon on the step’s row has been circled.

There’s more you can do in these dialogs, including running the profile against a different server.

Distributing Workloads

Once you’ve created your user profile, you’re ready to decide how many simultaneous users you want to have interacting with your server.

At this point, the distinction between a load test and a stress test starts to matter. If you’re doing a load test, then you want to set the number of users to what you reasonably expect to be either a typical or a peak load. If you want to test typical and peak loads, then you’ll need to create two tests, specifying a different number of virtual users in each test (here, the ability to import a user profile makes it easy for you to create the second test—just import the first test and modify the number of users). If you’re doing a stress test, then you’ll want to add a large number of users.

Setting the number of users is easy: In the User Profile pane, drag the slider in the Workload column to the right to specify what percentage of your available virtual users you want to use.

The User Profile pane showing a single user profile. The pane has fourth columns. The first column has the name of the test (“Update Department”); the second column – labelled Workload – has a slider that goes from 0% to 100%; The third column – labeled User Identity – has an empty dropdown list; The fourth unlabelled column has a pencil/edit icon.

With the Test Studio Ultimate package, you have a maximum of 100 virtual users per seat available to you (you can purchase more if that’s not enough to reflect your typical or peak loads—five TS Ultimate licenses would give you the ability to generate a load of 500 simultaneous users). But you should be careful of how you distribute all your virtual users over your test computers—a good ratio is eight virtual users per CPU core on your test computer (in fact, only some of your 100 default virtual users are assigned to be used by your Test Studio computer)

Exceeding that “eight virtual users per core” guideline will probably give you an invalid result on your load tests. If you exceed that guideline and your load test fails, it’s probably because you overwhelmed the computer running the tests and not your application’s web servers. If you want to involve more virtual users than the guideline allows (for example, to create a stress test that causes your application to fail), then you’ll need to enlist multiple test computers (all of which you can manage from your Test Studio computer).

You can see how many virtual users are assigned to your compute by clicking the Manage button on the Test Studio menu’s Tests tab to display the Manage Virtual Users dialog.

The Test Studio menu with the Tests tab selected. The second last icon, labelled Manage, is circled.

That will let you specify how many users are available to the test computer. If, for example, your computer has four cores and the default 100 virtual users, the dialog will default to 33% or 32 users (four cores times eight users). Move the slider to the left or right to specify the available users for your test.

The Manage Virtual Users dialog. The dialog lists a single computer with a slider set to 33. At the top of the dialog it reads “67 available VUs for allocation. 100 total VUs for this license account”

Setting Users and Timing

Your next step is to specify, in the Available Users pane in Test Studio’s lower right corner, your test parameters (again, you may need to unpin Test Studio’s Properties and Step Builder panes to expose this pane). In this pane, you’ll set:

  • How long you want your test to run in minutes (helpful tip: for a soak test that you want to run over multiple days, remember that there are 1,440 minutes in a day)
  • The number of virtual users present at the start of the test
  • The number of virtual users still present at the end of the test
  • How gradually your virtual users will be added to the test

The Test Studio Screen with the User Profile pane showing two user profiles (Update Department and Delete Department. The Available Users pane in the lower right corner is circled.

If, for example, you wanted to mimic a typical morning start up, you might specify a 60-minute test that begins with a single user but has all of your users present at the end of the test (up to the limit of the available users you specified). You might also specify that those additional users are to added over a 30-minute period. The Users Over Time pane right above the Available Users gives you some visual feedback on how users will be added.

A close up view of the Available Users and Test Settings pane. In the Available Users pane, several settings have been made. The test is to start with 1 user and end with 100. The users are to be ramped up in 30 minutes and the test is to run in 60 minutes. In the Test Settings pane above that a graph shows the test ramping up to 33 users over the first 30 minutes of a 60 minute test.

You’re now just about ready to run your test: Click the Run button on Test Studio’s Tests tab.

Test Studio’s menu with the Tests tab selected. The second icon from the left (a green VCR run button labelled Run) is circled.

Setting Test Options

When you click the Run button, Test Studio will pop up a textbox with some good advice. Once you click the Okay button on that dialog, Test Studio will display the Test Options dialog. In this dialog, you can specify how often test statistics are gathered (the default is every five seconds—probably a good first choice, but you might want to increase it if you find you’re getting more data than you need).

The primary purpose of this dialog, however, is to set conditions for an early end for your test.

The Test Options dialog. The first of four options has been selected (Average Response Time (ms) for Overall Test) has been selected, causes the option to expand. The Fail this goal if the Average Response Time (ms) is greater than option has been set to 3,000 milliseconds.

If you’re performing a stress test to see where your application will fail, the second and third options in this pane are useful:

  • When average response time exceeds some value
  • When the number of HTTP errors exceed some value

You can set these options so that, once either of these conditions are met, you’ve found your application’s fail point and there’s no need to continue the test.

For load tests, you’ll use the first and third options. The first option will terminate the test when the application has demonstrated that its average response time is less than a specified limit. The third option lets you stop your test when your application has demonstrated that it can process a specified number of responses or requests per second.

This dialog also allows you to specify that you don’t want any of your goals checked until your test has run for a specified period of time. In a load test, for example, you probably don’t want to bother checking the number of responses until “enough” virtual users have been added.

When you’ve made your choices, click the Run Locally button to start your test. Don’t panic when you get an initial screen that says that no results are available—remember that results are only gathered every five seconds, so you’ll need to wait for your initial results. Eventually, Test Studio will let you know how your test is doing.

The Test Studio window showing the results from running a test. The panel shows that the test passed by meeting it’s one goal around response time. It also includes graphs showing the peak average response time, the number of HTTP errors per second, and peak responses/requests per second.

When your test is done, you can use Test Studio’s Analyze pane to dig deeper into the test’s results or to compare the results from different test runs. Test Studio will also let you export the results in a variety of formats (e.g., Excel, Word, HTML).

When you return to your test, the Test Options pane is displayed by default. You can return to the Analyze pane for your test by clicking on the Analyze button in the Test Studio menu’s Project’s tab.

The Analyze your results pane showing the two primary options: Examining a single test and Compare multiple tests.

If you want to modify your test (i.e., change the workload, timing or how users are added) then, in the Test Studio menu’s Tests tab, click on the Design button.

The Test Studio menu with the Tests tab selected. The first icon (a paint brush) labeled Design is circled.

After Your First Load Test

Eventually, you’ll probably recognize that executing a single user profile isn’t a realistic test—it’s an unusual application that has all its users doing only one thing. You can create more realistic load tests by adding multiple user profiles to your test and then allocating a percentage of your virtual users to each profile. In the User Profile pane, the slide in the Workload column will let you distribute your virtual users among your profiles.

The User Profiles pane showing two projects. The Workload sliders for each project are set at 50%. The Evenly balance workload icon at the right end of the pane’s toolbar is circled. A tooltip for the icon reads “Evenly balance workload”

The easiest/fastest way to distribute your virtual users is to click the Evenly balance workload button in the upper right corner of the User Profile pane. If, for example, you have two profiles, clicking that button will allocate 50% of the users to each profile. However, an even distribution probably isn’t realistic, either, and you’ll want to play with the profiles’ sliders to get a balance you’re happy with. The only restriction is that the percentages for all the profiles can’t, of course, add up to more than 100%.

If you’ve been taking advantage of your existing functional test scripts to create your user profiles, you will also come to realize that a good functional test script won’t necessarily give you the ideal user profile for a load test. You shouldn’t let that stop you from leveraging your test scripts to create your first load tests, but you can use Test Studio’s script editor to evolve your functional test scripts into more effective non-functional load tests.

You also probably don’t want to tie up your Test Studio computer while your load test is running (especially for a soak test, which might run for many hours). You can schedule your load test to run outside of work hours by scheduling your tests.

Finally, you can run your load test run on another, remote computer controlled by your Test Studio computer. If you run a test on a remote computer, the maximum number of virtual users is controlled by the computer that’s managing the remote computer (i.e., your Test Studio computer). You still don’t want to exceed the “eight users per core” guideline on any test computer, but running your test on multiple remote computers will let you add more virtual users without overwhelming any of your test computers and let you manage that test from your Test Studio computer. An effective stress test will probably require you to use multiple remote computers (or one computer with a lot of cores).

You can check that your test is running successfully on a remote computer by opening the Test Studio Runtime Administrator on the remote computer and, on the Process History tab, checking its Load Controller entry.

The Test Studio Test Runner Administrator dialog showing the Process History tab. On the tab, the third entry – Load Controller – has a status of “Running” below its name. In the Ping Response pane on the right a message reads “Running ‘Update Department load’ test on 1 agents(s).

Start Early

Most teams leave load testing until late in the process (typically, after functional testing has shown that the application is free of known bugs). That’s not necessarily the best strategy. Load testing often reveals problems in the design or implementation of the application that can’t be fixed by throwing additional resources (e.g., more web servers) at the problem. In that scenario, code (even if it works correctly) is going to have to be rewritten. There’s not a lot of value in testing code you’re going to replace when load problems are discovered.

Test Studio lets you leverage your functional tests to get an early start on your load testing. Your initial functional tests may not be ideal load tests, but they can still tell you a lot about how your application deals with demand. It would be smart to take advantage of the ability to turn tests scripts into user profiles so that you know, as early as possible, that your application not only works, but that it works well.

Read more about starting your load test early—like, now.


Peter Vogel
About the Author

Peter Vogel

Peter Vogel is a system architect and principal in PH&V Information Services. PH&V provides full-stack consulting from UX design through object modeling to database design. Peter also writes courses and teaches for Learning Tree International.

Related Posts

Comments

Comments are disabled in preview mode.