Telerik blogs
Test StudioT2 Light_1200x303

Scheduling your tests to run on your computer when you’re not around is obviously a good idea and surprisingly easy to do. But the benefits don’t stop there: Scheduling tests lets you run your tests on multiple web browsers and speed up your testing by using headless browser testing.

So you’ve created your first end-to-end (E2E) test and you’ve been running it from Test Studio. In fact, you went on to create more tests—some driven by data in a dataset or using conditional logic to handle multiple scenarios. You’re running all these tests on a regular basis to confirm that your applications continue to work “as expected” as changes are made to them.

Test Studio showing the results of running a successful test

And, quite frankly, it’s getting to be a pain. Your real job is to find problems by doing exploratory testing or to consult with the development team on quality issues or to write new tests or … well, any number of hundreds of other things. Instead, your time (and your computer’s time) are tied up running tests. It would be great to able to have your tests run automatically without your attention. Good news: Test Studio will let you do that.

In addition, having Test Studio schedule your tests to run automatically gives you two additional options that, by now, you probably want.

First: Scheduling tests gives you the ability to run your tests on multiple browsers. Making sure your application works on all the browsers you’re willing to support—cross browser testing—is obviously a good thing … if it can be done automatically.

Second: If you’ve gotten to the point where you’re considering scheduling tests, then you’re probably also at the point where running all your tests is getting to be time-consuming. Scheduling tests also lets you take advantage of headless browser testing, which will dramatically shorten the time your single-browser tests take.

Scheduling tests to run on your computer does require setting up Test Studio Services (TSS). The good news here is that, if you’ve installed Test Studio, you’ve probably installed and configured TSS—you’re already ready to run scheduled tests on your own computer.

Checking That Everything’s Installed

It’s easy to check if you’ve got TSS installed: Open a project in Test Studio (the relevant web project and Test Studio project for this blog post can be downloaded here), select the Project tab in Test Studio’s menu and, in the menu’s Scheduling section, see if the Configuration icon is enabled. If that icon is enabled, you have TSS installed and Test Studio can communicate with it.

The Test Studio menu bar with the Project tab selected. At the right hand end of the graphic, the Configuration icon is circled

If the icon isn’t enabled, then you probably don’t have TSS installed. Fortunately, installing TSS on a computer that already has Test Studio is a snap:

  1. From the Windows menu, go to Add or Remove Programs.
  2. Find the Test Studio entry and pick the Modify option.
  3. When the Test Studio install dialog appears, click the Next button and then the Change button to display the Select features dialog.
  4. In the Select features dialog, add the Storage Service, the Scheduling Server and the Executive Dashboard.

The dialog for customizing your installation of Test Studio showing the components that can be added or removed. The bottom three – Storage Service, Scheduling Server, and Executive Dashboard – are circled. All three of the components are marked as installed on the computer’s hard disk

You can confirm that your installation is working from within Test Studio by going to Project tab on Test Studio’s menu and, in the Scheduling section, clicking the Connect icon. For the purposes of this test, you can treat your computer as if it were a remote scheduling computer by selecting the Run Remotely option and then clicking the Connect button on the dialog.

 The Scheduling Server Settings dialog. The Run Remotely radio button is selected and the Server Name textbox is set to “localhost.” The port is set to the default value of 8009. The Connect button on the dialog is circled. Below the Connect button three icons are displayed indicating that the connection was successful

If Test Studio is able to connect to the Scheduling Server, then the dialog will update to display whether logging is enabled on the Scheduling Server and the View Log/Clear log options for the server. Once you get that feedback, you can leave the setting at Run remotely or reset back to run locally.

To check that the Executive Dashboard is set up to display your test results, start up the web browser of your choice and surf to http://localhost:8085 to see the dashboard’s default page. Later on, you’ll be able to use Test Studio to check that your storage server is set up correctly but, as long as your browser is open, you can do an initial check on the storage server by surfing to http://localhost:27017 (you should get the message: “It looks like you are trying to access MongoDB over HTTP on the native driver port”). Finally, still in your browser, you can check on the scheduling server at http://localhost:8009, which will display an XML document of valid HTTP requests.

If that all works (and it probably will), then, unless you want to get email when your scheduled test runs execute, you don’t need to configure TSS. If one or more of those URLs doesn’t work, see the configuration guide further down in this post.

Configuring for Email Reports

But if, in addition to reviewing your test results in Test Studio’s Results tab, you also want to receive emails about your scheduled test runs, you’ll need to configure Test Studio Services with your email information. To do that:

  1. Start Test Studio.
  2. Open a project.
  3. In Test Studio’s menu, click on the Project tab.
  4. In the Scheduling section of the menu, click on the Configure icon.
  5. When the Configure Test Studio Services dialog appears, pick the Scheduling tab.

The TSS configuration dialog with its third tab -- the Scheduling tab – displayed. The bottom half of the tab window is empty and the arrow with the text “Configure Email (SMTP) Server (optional)” is circled

  1. On the Scheduling tab, expand the Configure Email option and enter your email server (e.g., mymail.com), your send mail port (typically, port 25), the specific email address you want the emails sent to, and the password to be used to sign into your email server with.

The TSS Scheduling tab shown again but with the bottom part expanded to show the entries for configuring the email server to be used by TSS. The entries have been filled with with the server name, port, email address to be used, and password. The ssl checkbox is unchecked

Creating Scheduled Tests

Before scheduling a test run, you have to assemble the tests in a project that you want to run together into a test list (you actually schedule test lists, not individual tests). You can add to your test list either by picking individual tests from your project or by creating a set of rules that will dynamically pick tests from your project at test execution time.

Either way, to create your test list, start Test Studio, open the project with the tests that you want to schedule, and select the Test List tab from the Test Studio menu bar.

The Test Studio menu bar with the Test Lists tab selected showing the options on the tab. The Test List tab itself is circled

If you just want to pick individual tests to run, click on the List button at the left end of the Test List menu to open the Create a New Test List dialog. The dialog has, on the left, the list of the tests in your project and, on the right, the tests you’ve picked to be part of this test list (initially, of course, that list will be empty).

At the top of the dialog, give your test list a name and then use the arrows in between the two lists to move your tests from one list to another. You can use the up and down arrows at the bottom of the list on the right to control the order that your tests are run in.

The Create a New Test List dialog showing the Contoso University project on the left with five tests listed and three tests in the list on the right. In the list of tests on the left all the tests that have been added to the right are grayed out/disabled. The Name textbox at the top of the list has been filled in with “Contoso Edit Tests.” Test Studio is visible behind the dialog with the Test Lists menu tab selected and the first choice on the menu – List – circled to indicate that it is the choice that displayed the dialog

If, instead, you want to have your tests picked at run time, then click the Dynamic List button. That will open the Add Dynamic Test List dialog, which will let you create a set of rules for selecting tests (those rules will be applied when the test list is executed so the tests you see as you set up the list may not be the tests that are run at the scheduled time).

The Add Dynamic Test List dialog. The Test List Name textbox has been filled in with “Run All Bad Data Tests.” A single rule is shown in the Rules section. A rule consist of two dropdown lists and a textbox. The first dropdown list has been set to “Name,” the second dropdown list to “Contains,” and the textbox has been set to the word “Bad.” Below the rule a set of tests that meet the conditions of the rule are listed (e.g., a test called “Update Department Bad Budget”). Behind the dialog the Test Studio menu bar is displayed with the Test Lists tab selected. The second choice from the left – Dynamic List – is circled to indicate that it is the choice that displayed the dialog

You can create rules that select tests based on the test’s name, owner, path to the test file, or priority. You can also select tests based on the three custom properties every test has (CustomProperty1, CustomProperty2 and CustomProperty3).

Designing Projects for Scheduled Tests

As that discussion implies, there are two rules when creating test list:

  1. You can create as many test lists from a project as you want.
  2. All the tests in a test list come from the same project.

So one of the design questions to consider as you divide tests up among projects is: “Do I want to be able to schedule these tests to run together?”

You’ll probably find that you’ve naturally divided tests up among projects in a way that supports the schedules you want. However, you may occasionally find that you have tests in different projects that you want to run together (i.e., your projects are too “fine-grained” for the schedule you want to create). In that case, you’ll need to merge your projects’ tests using Test Studio’s Add Existing Test menu option.

On the other hand, you’ll use your test’s custom properties to let you segment the tests in a project into groups of tests that you want to run together. To set the custom properties on a test, right-click on a test in a project and then select Properties from the popup menu.

The properties window for a test showing CustomProperty1 set to the text “Essential”

Critical Settings

You’re almost ready to start scheduling your test list’s first run. But before you do that, it’s worth looking at the options behind the Edit Settings icon on the Test Lists tab’s Edit section. There are a lot of options here for you but there are a few that you’ll want to consider, even on your first test.

The Test Studio menu bar with the Test List tab selected. In the Edit Section, the Edit Settings icon is circled

For example, with web applications, it’s not unusual for a test to fail because of a transient network error rather than a real error in the application. Rerunning failed tests is one way to check to see if the problem is real or transient (if the test succeeds the second time, the odds are good that the initial failure was transient and there is no underlying bug).

You can use the Edit Settings dialog to control this option. From the Edit Settings dialog’s General tab, set the option to have all failed tests in all your test lists re-run automatically. When you created your test list, you may have noticed a checkbox at the bottom of the dialog that also causes any failed tests in the test list to be rerun automatically. Your choice as to whether you want to set the option as you create the test list or later on in the Test List’s Edit Setting dialog.

The Edit Test List Settings dialog with the first tab – the General tab – selected. In the tab, the Rerun Failed Tests option checked. The option is highlighted and circled

The Web tab of the Edit Settings dialog has two more useful settings.

The Executing Browsers dropdown list lets you select multiple browsers to have your tests run in all of them. One of the choices here is special, though: ChromeHeadless which runs your tests without requiring your web pages to be rendered. Headless browser tests run in far less time than tests that have the browser render its pages.

The Edit Test List Settings dialog with the Web tab selected. In the list of options, the Executing Browser dropdown list is expanded showing a list of browsers: Internet Explorer, Microsoft Edge, Firefox, etc. One of the entries is Chrome Headless (the list of browsers is circled). There is a checkbox beside each browser and two of the browsers – Microsoft Edge and Chrome – are checked off

Headless browser testing doesn’t completely replace testing with multiple browsers. However, you can use it to speed up most of your test runs (e.g., your nightly test runs) and reserve testing with multiple browsers (and have them render their pages) for when you have time—your weekend test runs, for example. You’ll still be able to monitor the progress of your tests even if the browser doesn’t appear—Test Studio opens a command window that it updates with the status of your headless tests as they execute.

And, of course, while you can select multiple browsers as part of building your test list, when your test list runs, TSS will only use the browsers actually installed on the computer where the test executes.

Scheduling Tests

You can now run your test list just by clicking on the Run List button (a good way to check that the tests on your test list don’t interfere with each other). However, if you’ve gone to this much trouble, then you probably want to set up a schedule to run your tests when you’re not around. To set up a schedule for your test list, click on the Schedule List icon to start the scheduling wizard.

The Test Studio menu bar with the Test Lists tab selected. The menu choice at the right end – Schedule List – is circled

Clicking the Schedule List icon opens the three step Schedule Test Lists wizard.

  1. At the top of the first page of the Wizard, click the Recurring checkbox and then set the schedule you want to use for this test list. This example has the test list run every weekday at 11 pm with no end date set (by default, the wizard sets your test to run five minutes from when you open the wizard).

The Schedule Test Lists dialog on step 1 of 3 with the settings specified in the text. The page is titled “Select the Date and Time for TestList(s) Execution”

  1. Click the Next button to move to the second page of the wizard where you can select an execution server. At this point, assuming you’ve only set up to run tests on your computer, your computer will be the only one listed (setting up remote execution servers will be covered separately in the next post in this series). You’ll need to check off your computer on this page.

The second page of the Schedule Test Lists wizard titled “Select Machine(s) to Execute the TestList.” One computer is shown with the name “lp8” and the checkbox to its left checked. The operating system and the icons for the browsers installed on that computer are also shown for the computer.

  1. Click the Next button to move to the third (and final) page of the Wizard. If you’ve enabled email for TSS, you can use this page both to have an email sent when the test list run finishes (check the “Send result by email” option) and to customize the email. Regardless of what email options you select here, the results of your test list runs are always available on Test Studio’s Results tab.

This example sends an email to peter.vogel@phvis.com, but only when there’s a failure in the test (and, even then, not on transient errors). All the defaults have been taken for the subject line and body content, but the option to have an Excel spreadsheet of the test run’s results be attached to the email has been selected. The dialog gives you a preview on the right of what the email will look like.

The third page of the Schedule Test Lists wizard, titled “Select Notification Settings.” The options described in the text have been taken and the preview of the email on the right shows (approximately) what the resulting email will look like.

Click the Done button and Test Studio automatically displays the Results tab in Timeline mode, showing the schedule of tests for the current week.

The Test Studio Results tab showing multiple scheduled test list runs for the current week using the Timeline view. The current day of the week (Friday) is highlighted and has two runs scheduled. In the top left there are options to select Day, Week, and Month views in addition to the Timeline view

Once you’ve made sure you’ve got the schedule you want, you can go home for the day. Test Studio will run your test and upload the results to the storage server for you to review the results. When you come back tomorrow morning, you can return to your project’s Results tab in Test Studio and double-click on the scheduled run’s entry to what happened while you were gone (though you may already have reviewed the results on your phone if you selected the email option). Or, simpler yet, just surf to your Executive Dashboard from anywhere to see what happened in the night.

The Progress Telerik Test Studio Executive Dashboard web page showing the Contoso University Data Drive Test List. The first test list (name Contoso Department Up…) is shown with a large green Pass button, the data and time of the last run, and a bar graph showing that 100% of the last 10 runs have been successful

And then you can get back to your real job.

The Test Studio Results tab in Day view, showing, on the left, a test list run between 8:00 and 9:00am. The test list is selected. On the right the results of the one of the tests are shown. It was a data driven test and was executed three times. All the steps passed.

Appendix: Configuring Testing Services

Just to be clear: You almost certainly won’t need to configure TSS—the defaults set when you install TSS are usually fine. But if Test Studio isn’t letting you schedule a report (and you’ve confirmed that TSS is installed) or you’re getting an error message when working with scheduling, then you may need to change TSS’s default settings.

You can start TSS’s configuration tool by typing “Configure Test Studio Services” into the Windows menu and pressing the Enter key. Use the remote scheduled execution guide Single Machine Setup to review TSS’s settings and diagnose/fix any problems.

The initial tab of the Configure Test Studio Services tool showing the MongoDB tab

You might also find useful information in the Test Studio application logs about any problem you’re having. To get log information, start Test Studio, open a project, and, from the Help dropdown list in the upper right corner of the Test Studio window, select Enable Logging. Perform whatever task is giving you problems and then, back in the Help dropdown list, select View Log to review Test Studio’s logs.

The upper right corner of the Test Studio window showing the Help dropdown list. The list is expanded to show all its choices. The Enable Logging and View Log options in the middle of the menu are selected.

If your problem is related to the Storage Server, the most likely cause is that it has a pre-2018/version 4.0 version of MongoDB installed when you installed TSS. TSS won’t replace existing versions of MongoDB but TSS also won’t work with a version prior to version 4.0. If you do have an earlier version of MongoDB installed, you’ll need to upgrade MongoDB to version 4.0 or later. After you’ve upgraded MongoDB, you’ll need to return to the TSS configuration tool to update the information displayed on the MongoDB and Storage tabs with your new installation settings.

The two MongoDB-related tabs from the TSS Configuration tool (the first two tabs in the tool). The tab on the right is the first tab (labelled MongoDB) and has two textboxes that contain Mongo: The path to the MongoDB data folder and path to the MongoDB executable. The tab on the left, labelled Storage, has a single textbox with MongoDB connection string

If you have a port conflict with the Executive Dashboard, you can change the port the dashboard uses on the Executive Dashboard tab of the configuration tool (don’t forget to click the Apply button if do make a change). You should also confirm that the Host Name/IP and port for the Storage Service Location on the Executive Dashboard tab matches the settings on Scheduling tab.

The Executive Dashboard tab and Scheduling tab from the TSS configuration tool. The Executive Dashboard on the left shows the textboxes holding the Host Name/IP and Port for the scheduling server. Below that is a textbox holding the Port to be used for the Executive Dashboard, a Reset to Default button and an Apply button. The Scheduling tab on the right also shows the Scheduling server’s Host Name/IP and Port (among other information). The Host Name/IP and port textboxes on each tab are circled and joined with a line

Test Studio doesn’t run with Admin privileges and it’s possible that Test Studio isn’t being granted access to the ports it needs. In that case, go the last tab in in the Configuration tool (the Non Admin Port Setup tab) and click the Grant Access button to give all authenticated users access to the ports TSS needs (if you want, you can change the user group specified in this dialog to limit the port access to a smaller group).

The Non Admin Port Setup tab in the TSS configuration tool, showing the ports specified/selected on the other tabs. The User/Group textbox specifies the NT Authority\Authenticated Users group. To the right of the textbox is the Grant Apply button

 


P.S. I would like to extend my thanks to Elena Tsetkova and Asya Ivanova for their help in completing this blog post.


Peter Vogel
About the Author

Peter Vogel

Peter Vogel is a system architect and principal in PH&V Information Services. PH&V provides full-stack consulting from UX design through object modeling to database design. Peter also writes courses and teaches for Learning Tree International.

Related Posts

Comments

Comments are disabled in preview mode.