Start Load Testing. Now. Really.

With Test Studio, you’re well positioned to start getting useful results from load testing right now (and better results in the future).

Asking how well your application works gets you into the territory of “non-functional tests.” There are a variety of non-functional tests—user experience, recovery from failure, resilience to interruptions, and more—but the tests that let people sleep night are load tests: the ones that tell you whether your application can support the expected (and unexpected) loads that users place on it.

Supporting demand may not be something that worries you. You may know that you have enough computing power lying idle and that your application, with its expected number of users, will never have a significant impact on any those computing resources. But this is something you don’t want to be wrong about: Gartner estimates that IT downtime averages $5,600 a minute (between $140,000 and $540,000 an hour).

For a load test to be fully effective, it should tell you something about how your application behaves under peak loads, typical loads and loads extreme enough to cause your application to fail. Obviously, then, it’s true that your load test mimics your real application and the demands your users place on it. That would be a perfect load test.

But let me be clear: This post is about how the perfect is the enemy of the good. That obvious truth about load tests doesn’t mean that you should defer doing load tests until you can fully impersonate your real-world application and its loads. It’s not unusual that, to solve a load problem, you need to rewrite or even re-architect your application. That being the case, you want to find that out as early as possible in development if you have a load problem.

Which means you want to start your load tests “as soon as possible”—basically, as soon as any “show-stopping” bugs have been removed. To put it another way: An early “good enough” load test is better than a perfect load test later.

Test Studio will let you do that.

Managing Timing

These days, most teams start building functional tests that prove their application is doing the right things early in the development process. One of the benefits of using Test Studio to create those functional test scripts is that you can convert those test scripts into user profiles, which are the heart of a load test. As soon as your application starts passing some of your functional tests, you can convert them to user profiles and start running load tests.

But there are two categories of tests (functional and non-functional) for a reason. In a load test, for example, you’re not interested in whether a user can successfully scroll through a long list (which is the point of a functional test) but you’re very interested in the impact of building that list on the relevant web service. So, while your functional tests position you to start load testing earlier, a good functional test isn’t quite the same as a good load test.

Therefore, before you take advantage of Test Studio’s ability to convert your functional tests into user profiles, you need to make sure that your functional test is acting like a human being. The first step is to run the functional test script and check for “unexpected” pauses.

For example, when you run your functional test, you may find a significant pause while Test Studio’s playback tool looks for an element the test script is supposed to interact with. The usual cause for this pause is that something has changed in the application since the test was recorded and the default mechanism Test Studio uses to find the element can no longer find the element. Fortunately, Test Studio uses a series of options to locate an element (HTML, screen image, others) and, after giving the default option some time to fail, Test Studio just moves on to another option.

This is a good thing—it makes your tests very reliable in the face of changes to your application (your tests are, effectively, “self-healing”). In a functional test, this pause has no impact on the validity of the test. To create a more useful load test, however, you need to fix this mechanical issue. The simplest way to eliminate these pauses is to:

1. Open the test script in Test Studio’s script editor.
2. Select the step with the pause and delete it.
3. Right-click on what was the previous step and pick the “Run to Here” option.
4. When the test script pauses and the Test Studio toolbar appears, perform the action that was formally delaying your test. Test Studio will capture your action and add it to your script in place of the older version of the step.
5. Stop your test and run your test to confirm that the mechanical pause is gone.

On the other hand, your users aren’t robots, either. Your functional tests move from step to step very quickly—not a realistic representation of how your users behave (hopefully, they’re actually looking at your application’s pages). So, once you’ve eliminated any mechanical pauses, you’re ready to make sure your profile has the right human pauses.

When you convert a functional test script into a load test user profile, Test Studio will automatically insert “think time” steps in your load test. You should review those think time steps and make sure they’re both reasonable and in the right place.

Creating User Journeys

Once you’ve got the timing right in your scripts, you can start thinking about representing your users’ typical activities—your “user journeys.” Here, again, a functional test is going to be different from the ideal non-functional test.

A typical functional test might involve surfing to a page (bypassing security on the way), changing some data and then immediately surfing to another page to check that the data was updated correctly. Unless you have a particularly paranoid set of users, I bet that test doesn’t reflect any user’s “typical” use of the application. Your functional tests also probably focus on problematic features where you expect to find bugs. Again, that probably doesn’t reflect a typical user transaction, which often involves many pages, spanning many “features” and using standard functionality that’s been in the application since its creation.

Since your load tests need to mimic how your application is actually going to be used, each user profile needs to be a typical user journey. The good news here is that you can often create those user journeys by combining multiple existing functional tests (Test Studio’s script editor facilitates merging tests or even using one script as a step in another script).

You may, occasionally, need to create a new functional test using Test Studio’s recorder just to bridge a part of the user’s journey between two functional tests.

Practical Considerations

Moving away from mimicking your real-world environment, it makes sense to consider how you’ll organize your functional and non-functional tests. It may, for example, make sense to you to keep both the functional and non-functional tests related to an application/release/feature in a single project. There is, however, a good case for creating a new project that will hold only load tests in order to reduce the overhead in managing your load tests.

As you’ve seen, over time, your load tests will evolve away from your functional tests, usually because you’ll combine multiple functional tests to mimic a typical user transaction. That alone may cause you to want to keep your load tests in a separate project. Test Studio will let you import functional tests from other projects into your load test project. You can use those functional tests to create your user profiles (and modify those functional test to make them into more effective non-functional tests).

Scheduling is also different for functional tests because load tests typically run for longer periods of time (soak tests, for example, measure how an application behaves over hours or days). Because of how long load tests typically run, while you may run your functional tests based on events (e.g., after code is checked into source control), you’re more likely to run load tests based on calendar dates (every weekend or as part of the run up to your next release).

Creating a separate project for your load tests will simplify finding, scheduling and managing those tests … which will encourage you to use those tests more.

You’ll also get more useful results from your load tests if your application’s server (and, to a certain extent, the clients) used in your load tests is more like your production server. The same is also true of any web servers and databases your application accesses. You may want to use a different set of servers for load tests than you’re using for functional tests.

Something else to consider with load tests is their impact on shared data. While functional tests tend to affect one or two records in the database, load tests can affect multiple records. As result, load tests need to avoid making changes that prevent other tests (or even themselves) from running.

For example, a load test that tries to delete the same customer every time it runs will only work correctly the first time (and, unless your users typically try to delete customers that don’t exist, that’s not a good representation of how your system is used). A profile that deletes a different customer every time, if run as a soak test over many days may eventually delete all your customers—if you intend to run this profile in conjunction with a profile that updates customers, that update customer test may not find any customers to work with.

If these considerations cause you to defer load testing, then you’re missing my point. These are issues that you should consider and adjust for when they become problems.

If, for example, you’re using your development server for a stress test and that test starts failing … well, then, that’s the moment when you should consider setting up a “better” server for your stress test. In the meantime, you can roughly approximate when your stress test will fail on your production server by calculating the differences in CPU power and memory (give yourself a wide margin for error while you’re doing this, though).

For example, if your stress test on a four-core development server fails at 50 users and your production server has 64 cores, then—while you’re waiting for a better server to test with—I bet you can live with the reasonable assumption that your app will support at least 600 users (64/4 * 50 = 800 with a 25% margin for error). If I was expecting my peak load to be about 400 users, then I’d feel pretty comfortable about saying that I don’t (currently) have a load problem … again, while waiting for that “better” test server.

There’s really no reason not to get started. Now, for example.