Telerik Testing Framework Unit Tests in Continuous Integration Environments

1    Introduction

"Continuous Integration" is a software development practice that is gaining a lot of popularity in the software industry. Continuous Integration means to nearly continuously integrate the individual developer's changes into the main source code control system (or repository); performing a new build, verifying the build, and even running automated unit tests against those builds. "Nearly continuously" is open to interpretation for each project, team and even developer.  The idea is for the developer to check-in his/her changes just as soon as he/she feels the changes are ready for an integration build. Some examples for when to check-in for continuous integration include:

  • In a C++ world, check-in once your header additions/changes have been made (assuming you don't expect them to break something else)
  • Check-in a new class as soon as it's finished, even if it's just an abstract base class, instead of waiting for all the classes to be developed.
  • Check-in each individual bug fix instead of a group of bug fixes. If you check-in a group of bug fixes all at once, it's much more difficult (impossible without code comments or some other documentation) to know which change was for which bug. Also, if your check-in breaks the main build, it's difficult to identify which bug fix broke it so you can take the appropriate corrective action.

 

Continuous integration has many advantages:

  • Integration problems are detected and fixed continuously - no last minute hiatus before release dates;
  • Constant availability of a "current" build for testing, demo, or release purposes;
  • Early warning of broken/incompatible code;
  • Early warning of conflicting changes;
  • Immediate unit testing of all changes;
  • When a unit tests fail, or a bug is discovered, developers might revert the codebase back to a bug-free state, without wasting time debugging.
  • The immediate impact of checking in incomplete or broken code acts as an incentive to developers to learn to work more incrementally with shorter feedback cycles.

 

The goals of continuous integration include:

1.1   Maintain a Code Repository

This practice advocates the use of a revision control system for the project's source code. All artifacts that are needed to build the project should be placed in the repository. In this practice and in the revision control community, the convention is that the system should be buildable from a fresh checkout and not require additional dependencies. Extreme Programming advocate Martin Fowler also mentions that where branching is supported by tools, its use should be minimized. Instead, it is preferred that changes are integrated rather than creating multiple versions of the software that are maintained simultaneously. The mainline (or trunk) should be the place for the working version of the software.

1.2   Automate the Build

The system should be buildable using a single command. Many build tools exist, such as 'make' which has existed for many years. Other more recent tools like Ant, Maven or MSBuild are frequently used in Continuous Integration environments. Automation of the build should include automating the integration, which often includes deployment into a testing environment that closely mimics production. In many cases, the build script not only compiles binaries, but also generates documentation, website pages, statistics and distribution media (such as Windows MSI files or RPM files).

1.3   Make Your Build Self-Testing

This touches on another aspect of best practice, Test-driven development. Briefly, this is the practice of writing a test that demonstrates a lack of functionality in the system, and then writing the code that makes the test pass.

Once the code is built, all the tests should be run to confirm that it behaves as the architect and developers expect it to behave.

1.4   Everyone Commits Every Day

By committing regularly, every committer can reduce the number of conflicting changes. Checking in week's worth of work runs the risk of conflicting with other features and can be very difficult to resolve. Instead by checking-in early, small conflicts in an area of the system causes team members to communicate about the change they are making and take appropriate action.

1.5   Every Commit (to Mainline) Should be Built

Every commit to the current working version should be built to verify they integrate correctly. A common practice is to use Automated Continuous Integration, although this may be done manually. In fact, James Shore prefers this approach. For many, continuous integration is synonymous with using Automated Continuous Integration where a continuous integration server or daemon monitors the version control system for changes, then automatically runs the build process.

1.6   Keep the Build Fast

The build needs to be fast, so that if there is a problem with integration, it is quickly identified and resolved.

1.7   Test in a Clone of the Production Environment

Having a test environment that doesn't mirror the production environment can lead to failures in tested systems when they are deployed to the production environment. This happens because the production environment may differ from the test environment in a significant way making it impossible for the test team to catch.

1.8   Make it Easy to Get the Latest Deliverables

Making builds readily available to stakeholders and testers can reduce the amount of rework necessary when rebuilding a feature that doesn't meet requirements. Additionally, early testing reduces the chances that defects survive until deployment. Finding issues earlier also, in some cases, reduces the amount of work necessary to resolve them.

1.9   Everyone can see the results of the latest build

It should be easy to find out whether the build is broken and who made the breaking change.

1.10 Automate Deployment

By developing an automated build process you can extend the process to automatically deploy it to your test environment. Once deployed, automated unit tests can then run and the results reported to the team as part of the build results. A good automated build process will report how many tests passed, how many failed and indentify which ones failed.

2    Running Unit Tests in Continuous Integration Environments

2.1   Testing Agent Setup

So what does Telerik Testing Framework have to do with Continuous Integration? It comes into the automated unit testing part of continuous integration. With simple unit tests; unit tests that merely check code paths and functionality; you don't care much how your automated testing agents are setup so long as they're able to execute your product code and the test code.

 

The framework actually drives and interacts with the browser, the setup of the testing agents is more sensitive. For example, if unit testing is run under the 'Local System' or 'Local Service' account; as is common on many automated build servers and testing agents; Telerik tests will fail because browser interaction is prohibited for these types of accounts.

 

Instead the testing agent (in some continuous integration systems the build server is also the testing agent) must be run in console mode (i.e. started via the command line after logging onto the testing machine). Even setting up a testing agent as a service that logs onto a real user account isn't adequate. The problem with running as a service is that desktop interaction is disabled; which is required for certain Telerik Testing Framework features.

 

One other thing to be aware of is to NOT allow your testing agent to run Telerik Testing Framework unit tests in parallel. The reason for this is twofold:

  • Telerik Testing Framework is not thread safe.
  • If multiple browser windows are opening at once, Telerik Testing Framework may mistakenly attach to the wrong browser window, or worse, connect to a browser window already attached to another Telerik Testing Framework session. 

 

The recommended approach to setting up automated testing agents is as follows:

  1. Log onto the testing agent machine using an account with necessary permissions needed to run your agent and run your unit tests.
  2. Install your testing agent software.
  3. Run the testing agent in console mode (i.e. from the command line or a shortcut link). Just make sure it's not running as a service.
  4. Install Test Studio or the Telerik Testing Framework. Don't forget to enter your license.
  5. Leave the machine running and logged into the account but locked. If your tests perform direct desktop interaction (e.g. Desktop.Mouse.Click, Window.GetBitmap, etc) instead of locking the machine leave your machine logged on and displaying the desktop all the time. This means you'll have to disable the screensaver and never lock the computer.

 

If you setup your testing agent in this fashion, you should have no problems running Telerik Testing Framework unit tests.

2.2   Using MSTest

VisualStudio comes with MSTest. This is Microsoft's method for running unit tests from the command line. Different automated build systems have different methods for running MSTest. Some have functionality built in to make it easy to configure and run MSTest as part of the build. Others have an 'exec' task allowing you to run any command line tool, which can include MSTest.

 

MSTest has many command line options and is fully documented at http://msdn.microsoft.com/en-us/library/ms182486(VS.80).aspx.

 

The key command parameters that matter to unit testing in a continuous integration environment are:

2.2.1     /testcontainer

This option tells MSTest which DLL contains your unit test code and is required. Typically the current working directory is the root of the project. You need to take this into account and specify a path that is relative to the projects root such as '/testcontainer:bin/debug/SydneyTests.dll.

2.2.2      /testlist

If you've taken the time to configure test lists, you can use this option to specify which list of tests to run by name. This option requires you also specify '/testmetadata:' which is the path to the projects metadata file (i.e. the .vsmdi file). The test lists defined in your project are kept in this metadata file and nowhere else.

 

To run multiple test lists simply add this option multiple times to the command line, each time specifying one test list. You only need to specify '/testmetadata:' once however.

 

Note: If you do not specify this option or the '/tests:' option MSTest will run all unit tests it can find in the unit test .DLL.

2.2.3      /test

This option tells MSTest to run one specific test found in the unit test .dll file. Unlike the '/testlist:' option, you do not need to specify '/testmetadata:' with this option. This is because you're not specifying a list, but a single specific test instead.

 

To run multiple tests, simply add this option multiple times to the command line, each time specifying a different test.

 

Note: If you do not specify this option or the '/testlist:' option MSTest will run all unit tests it can find in the unit test .DLL.

2.2.4      /resultsfile

Normally MSTest creates the results in a uniquely named .trx file in a 'TestResults' folder. This folder is contained in the root of the project. For example: 'TestResults\agentuser_agent-pc4 2009-04-10 11_34_07.trx'. Because the name of this file is based on the logged on users ID & the machine's name, and the date & time, it can be hard for some continuous integration systems to find this results file, especially those that run MSTest via an 'exec' task.

 

This option specifies the path and name of the results file to create. By specifying a fixed filename, it's easier for the continuous integration systems find the results file and pull it into the build report. The only drawback is that MSTest will fail (not even run tests) if the file specified already exists. Therefore you need to delete any results file that may have been left behind from previous builds. But be careful that your delete operation doesn't throw an error if the file to delete doesn't exist (e.g. a simple 'del myResults.trx' will return an error if the file doesn't exist). This can cause some automated build systems to stop if it detects the error being returned by a simple delete operation.

 

There are other command line options you may find useful in your environment. Refer to the MSDN documentation (previously listed) to learn about them and see which ones you want to take advantage of.