Test Studio Step-by-Step: Scheduling Tests

Scheduling your tests to run on your computer when you’re not around is obviously a good idea and surprisingly easy to do. But the benefits don’t stop there: Scheduling tests lets you run your tests on multiple web browsers and speed up your testing by using headless browser testing.

So you’ve created your first end-to-end (E2E) test and you’ve been running it from Test Studio. In fact, you went on to create more tests—some driven by data in a dataset or using conditional logic to handle multiple scenarios. You’re running all these tests on a regular basis to confirm that your applications continue to work “as expected” as changes are made to them.

And, quite frankly, it’s getting to be a pain. Your real job is to find problems by doing exploratory testing or to consult with the development team on quality issues or to write new tests or … well, any number of hundreds of other things. Instead, your time (and your computer’s time) are tied up running tests. It would be great to able to have your tests run automatically without your attention. Good news: Test Studio will let you do that.

In addition, having Test Studio schedule your tests to run automatically gives you two additional options that, by now, you probably want.

First: Scheduling tests gives you the ability to run your tests on multiple browsers. Making sure your application works on all the browsers you’re willing to support—cross browser testing—is obviously a good thing … if it can be done automatically.

Second: If you’ve gotten to the point where you’re considering scheduling tests, then you’re probably also at the point where running all your tests is getting to be time-consuming. Scheduling tests also lets you take advantage of headless browser testing, which will dramatically shorten the time your single-browser tests take.

Scheduling tests to run on your computer does require setting up Test Studio Services (TSS). The good news here is that, if you’ve installed Test Studio, you’ve probably installed and configured TSS—you’re already ready to run scheduled tests on your own computer.

Checking That Everything’s Installed

It’s easy to check if you’ve got TSS installed: Open a project in Test Studio (the relevant web project and Test Studio project for this blog post can be downloaded here), select the Project tab in Test Studio’s menu and, in the menu’s Scheduling section, see if the Configuration icon is enabled. If that icon is enabled, you have TSS installed and Test Studio can communicate with it.

If the icon isn’t enabled, then you probably don’t have TSS installed. Fortunately, installing TSS on a computer that already has Test Studio is a snap:

1. From the Windows menu, go to Add or Remove Programs.
2. Find the Test Studio entry and pick the Modify option.
3. When the Test Studio install dialog appears, click the Next button and then the Change button to display the Select features dialog.
4. In the Select features dialog, add the Storage Service, the Scheduling Server and the Executive Dashboard.

You can confirm that your installation is working from within Test Studio by going to Project tab on Test Studio’s menu and, in the Scheduling section, clicking the Connect icon. For the purposes of this test, you can treat your computer as if it were a remote scheduling computer by selecting the Run Remotely option and then clicking the Connect button on the dialog.

If Test Studio is able to connect to the Scheduling Server, then the dialog will update to display whether logging is enabled on the Scheduling Server and the View Log/Clear log options for the server. Once you get that feedback, you can leave the setting at Run remotely or reset back to run locally.

To check that the Executive Dashboard is set up to display your test results, start up the web browser of your choice and surf to http://localhost:8085 to see the dashboard’s default page. Later on, you’ll be able to use Test Studio to check that your storage server is set up correctly but, as long as your browser is open, you can do an initial check on the storage server by surfing to http://localhost:27017 (you should get the message: “It looks like you are trying to access MongoDB over HTTP on the native driver port”). Finally, still in your browser, you can check on the scheduling server at http://localhost:8009, which will display an XML document of valid HTTP requests.

If that all works (and it probably will), then, unless you want to get email when your scheduled test runs execute, you don’t need to configure TSS. If one or more of those URLs doesn’t work, see the configuration guide further down in this post.

Configuring for Email Reports

But if, in addition to reviewing your test results in Test Studio’s Results tab, you also want to receive emails about your scheduled test runs, you’ll need to configure Test Studio Services with your email information. To do that:

1. Start Test Studio.
2. Open a project.
3. In Test Studio’s menu, click on the Project tab.
4. In the Scheduling section of the menu, click on the Configure icon.
5. When the Configure Test Studio Services dialog appears, pick the Scheduling tab.

6. On the Scheduling tab, expand the Configure Email option and enter your email server (e.g., mymail.com), your send mail port (typically, port 25), the specific email address you want the emails sent to, and the password to be used to sign into your email server with.

Creating Scheduled Tests

Before scheduling a test run, you have to assemble the tests in a project that you want to run together into a test list (you actually schedule test lists, not individual tests). You can add to your test list either by picking individual tests from your project or by creating a set of rules that will dynamically pick tests from your project at test execution time.

Either way, to create your test list, start Test Studio, open the project with the tests that you want to schedule, and select the Test List tab from the Test Studio menu bar.

If you just want to pick individual tests to run, click on the List button at the left end of the Test List menu to open the Create a New Test List dialog. The dialog has, on the left, the list of the tests in your project and, on the right, the tests you’ve picked to be part of this test list (initially, of course, that list will be empty).

At the top of the dialog, give your test list a name and then use the arrows in between the two lists to move your tests from one list to another. You can use the up and down arrows at the bottom of the list on the right to control the order that your tests are run in.

If, instead, you want to have your tests picked at run time, then click the Dynamic List button. That will open the Add Dynamic Test List dialog, which will let you create a set of rules for selecting tests (those rules will be applied when the test list is executed so the tests you see as you set up the list may not be the tests that are run at the scheduled time).

You can create rules that select tests based on the test’s name, owner, path to the test file, or priority. You can also select tests based on the three custom properties every test has (CustomProperty1, CustomProperty2 and CustomProperty3).

Designing Projects for Scheduled Tests

As that discussion implies, there are two rules when creating test list:

1. You can create as many test lists from a project as you want.
2. All the tests in a test list come from the same project.

So one of the design questions to consider as you divide tests up among projects is: “Do I want to be able to schedule these tests to run together?”

You’ll probably find that you’ve naturally divided tests up among projects in a way that supports the schedules you want. However, you may occasionally find that you have tests in different projects that you want to run together (i.e., your projects are too “fine-grained” for the schedule you want to create). In that case, you’ll need to merge your projects’ tests using Test Studio’s Add Existing Test menu option.

On the other hand, you’ll use your test’s custom properties to let you segment the tests in a project into groups of tests that you want to run together. To set the custom properties on a test, right-click on a test in a project and then select Properties from the popup menu.

Critical Settings

You’re almost ready to start scheduling your test list’s first run. But before you do that, it’s worth looking at the options behind the Edit Settings icon on the Test Lists tab’s Edit section. There are a lot of options here for you but there are a few that you’ll want to consider, even on your first test.

For example, with web applications, it’s not unusual for a test to fail because of a transient network error rather than a real error in the application. Rerunning failed tests is one way to check to see if the problem is real or transient (if the test succeeds the second time, the odds are good that the initial failure was transient and there is no underlying bug).

You can use the Edit Settings dialog to control this option. From the Edit Settings dialog’s General tab, set the option to have all failed tests in all your test lists re-run automatically. When you created your test list, you may have noticed a checkbox at the bottom of the dialog that also causes any failed tests in the test list to be rerun automatically. Your choice as to whether you want to set the option as you create the test list or later on in the Test List’s Edit Setting dialog.

The Web tab of the Edit Settings dialog has two more useful settings.

The Executing Browsers dropdown list lets you select multiple browsers to have your tests run in all of them. One of the choices here is special, though: ChromeHeadless which runs your tests without requiring your web pages to be rendered. Headless browser tests run in far less time than tests that have the browser render its pages.

Headless browser testing doesn’t completely replace testing with multiple browsers. However, you can use it to speed up most of your test runs (e.g., your nightly test runs) and reserve testing with multiple browsers (and have them render their pages) for when you have time—your weekend test runs, for example. You’ll still be able to monitor the progress of your tests even if the browser doesn’t appear—Test Studio opens a command window that it updates with the status of your headless tests as they execute.

And, of course, while you can select multiple browsers as part of building your test list, when your test list runs, TSS will only use the browsers actually installed on the computer where the test executes.

Scheduling Tests

You can now run your test list just by clicking on the Run List button (a good way to check that the tests on your test list don’t interfere with each other). However, if you’ve gone to this much trouble, then you probably want to set up a schedule to run your tests when you’re not around. To set up a schedule for your test list, click on the Schedule List icon to start the scheduling wizard.

Clicking the Schedule List icon opens the three step Schedule Test Lists wizard.

1. At the top of the first page of the Wizard, click the Recurring checkbox and then set the schedule you want to use for this test list. This example has the test list run every weekday at 11 pm with no end date set (by default, the wizard sets your test to run five minutes from when you open the wizard).

2. Click the Next button to move to the second page of the wizard where you can select an execution server. At this point, assuming you’ve only set up to run tests on your computer, your computer will be the only one listed (setting up remote execution servers will be covered separately in the next post in this series). You’ll need to check off your computer on this page.

3. Click the Next button to move to the third (and final) page of the Wizard. If you’ve enabled email for TSS, you can use this page both to have an email sent when the test list run finishes (check the “Send result by email” option) and to customize the email. Regardless of what email options you select here, the results of your test list runs are always available on Test Studio’s Results tab.

This example sends an email to peter.vogel@phvis.com, but only when there’s a failure in the test (and, even then, not on transient errors). All the defaults have been taken for the subject line and body content, but the option to have an Excel spreadsheet of the test run’s results be attached to the email has been selected. The dialog gives you a preview on the right of what the email will look like.

Click the Done button and Test Studio automatically displays the Results tab in Timeline mode, showing the schedule of tests for the current week.

Once you’ve made sure you’ve got the schedule you want, you can go home for the day. Test Studio will run your test and upload the results to the storage server for you to review the results. When you come back tomorrow morning, you can return to your project’s Results tab in Test Studio and double-click on the scheduled run’s entry to what happened while you were gone (though you may already have reviewed the results on your phone if you selected the email option). Or, simpler yet, just surf to your Executive Dashboard from anywhere to see what happened in the night.

And then you can get back to your real job.

Appendix: Configuring Testing Services

Just to be clear: You almost certainly won’t need to configure TSS—the defaults set when you install TSS are usually fine. But if Test Studio isn’t letting you schedule a report (and you’ve confirmed that TSS is installed) or you’re getting an error message when working with scheduling, then you may need to change TSS’s default settings.

You can start TSS’s configuration tool by typing “Configure Test Studio Services” into the Windows menu and pressing the Enter key. Use the remote scheduled execution guide Single Machine Setup to review TSS’s settings and diagnose/fix any problems.



You might also find useful information in the Test Studio application logs about any problem you’re having. To get log information, start Test Studio, open a project, and, from the Help dropdown list in the upper right corner of the Test Studio window, select Enable Logging. Perform whatever task is giving you problems and then, back in the Help dropdown list, select View Log to review Test Studio’s logs.



If your problem is related to the Storage Server, the most likely cause is that it has a pre-2018/version 4.0 version of MongoDB installed when you installed TSS. TSS won’t replace existing versions of MongoDB but TSS also won’t work with a version prior to version 4.0. If you do have an earlier version of MongoDB installed, you’ll need to upgrade MongoDB to version 4.0 or later. After you’ve upgraded MongoDB, you’ll need to return to the TSS configuration tool to update the information displayed on the MongoDB and Storage tabs with your new installation settings.



If you have a port conflict with the Executive Dashboard, you can change the port the dashboard uses on the Executive Dashboard tab of the configuration tool (don’t forget to click the Apply button if do make a change). You should also confirm that the Host Name/IP and port for the Storage Service Location on the Executive Dashboard tab matches the settings on Scheduling tab.



Test Studio doesn’t run with Admin privileges and it’s possible that Test Studio isn’t being granted access to the ports it needs. In that case, go the last tab in in the Configuration tool (the Non Admin Port Setup tab) and click the Grant Access button to give all authenticated users access to the ports TSS needs (if you want, you can change the user group specified in this dialog to limit the port access to a smaller group).

 

P.S. I would like to extend my thanks to Elena Tsetkova and Asya Ivanova for their help in completing this blog post.