Using Test Oracles to Ensure Your Tests’ True Validity

Raise your hand if something similar to this has ever happened to you: You write a functional test that uses your system’s UI to create a user. You automate navigating to the proper screen, you get all the right values filled in, the Submit button gets clicked, and you do a check for the “User created!” message. You run the test a couple times to make sure everything’s working smoothly. Things look good, so you check it in to source control, where it gets wrapped in to your regular automation runs.

Shortly after a new bug gets filed stating newly created users aren’t showing up elsewhere in the system. Investigation into the bug finds the act of saving the new user to the database was buggy—that particular routine was failing without any proper error handling. The failures were ignored and the “User created!” message was being displayed regardless of the actual database action.

Whoops.

For a true end-to-end test, you shouldn’t be relying on feedback at the UI to validate the test passes; instead, you should be taking the extra step of validating down through the system’s internals.

Bringing Oracles Into Your Tests

The term “test oracles” describes a method, action, or step which helps you take this extra step. Oracles take many different forms depending on what you’re testing. You may need to check a record in the database, validate a configuration file’s status, or determine if an e-mail has been sent in the proper format.

The following test is an example of a file existence oracle. The test is part of our demonstration app hosted on GitHub. If you’re interested you can grab the latest download, or simply clone the entire repository.

This test opens logs on to the application and exports the current list of users to a PDF file.

Implementing a File-Based Oracle

Here’s what the test itself looks like:

The test executes an initialize step, logs on to the system and navigates to the home page, clicks the “Export to PDF” button, saves the file to a specific location, then validates that file actually exists.

Let’s walk through that in more detail.

The Initialize step is a simple coded step setting an extract variable pointing to the location we’ll be saving the file to:

string downloadPath = @"d:\temp\contacts.pdf";

SetExtractedValue("downloadPath", downloadPath );

if (System.IO.File.Exists(downloadPath)) {

    System.IO.File.Delete(downloadPath);

}

We also delete the file if one’s there from an earlier run. [1]

The next step uses Test Studio’s Test As Step feature to execute another entire test as one step. That other test logs us on to the system and brings us to the main grid.

Step three clicks the Export to PDF button, while step four handles the download dialog box. Note that it’s data driven and pulls in the downloadPath extracted value variable.

Once the download is complete step four, another coded step, checks for the existence of the actual file in the download path we set in step one.

Assert.IsTrue(System.IO.File.Exists(GetExtractedValue("downloadPath").ToString()));

Note we have to explicitly call ToString() on the return value of GetExtractedValue – that’s because GetExtractedValue returns a plain object, not a string.

There you have it: we’ve run through our test and taken the extra step to ensure the file is actually written. Note we’ve not confirmed the file is properly formed, contains the right information, etc. That’s a bit longer than I want to run in to in this article!

Closing Up

Taking the time to write oracles into your tests ensure you’re getting a much better test to check your system’s proper validity. Test Studio’s coded step feature lets you write great oracles right within the Test Studio UI. You can also open up Visual Studio and take advantage of that environment to write extremely powerful oracles using any software engineering steps needed.

[1] The initial value for the downloadPath is a hardwired string. This is actually a bad practice, since it locks in your configuration and will very likely cause you massive grief when you run these tests on a different system. I strongly encourage you to push this off to some sort of a settings file which you can keep clean and unique for each environment you’re running your tests in. I’ll be covering that in a future post.

About the author

Jim Holmes

Jim Holmes has around 25 years IT experience. He is co-author of "Windows Developer Power Tools" and Chief Cat Herder of the CodeMash Conference. He's a blogger and evangelist for Telerik’s Test Studio, an awesome set of tools to help teams deliver better software. Find him as @aJimHolmes on Twitter.