End-to-end tests are an excellent way to deliver high-quality web applications. Unfortunately, these tests tend to be slower than other tests, creating delays in the development process. Here are five tips you can use to keep your tests running efficiently.
Today’s web applications are more complex than ever. The users of your app expect more than just text and images—they also demand more interactivity and a smooth, seamless experience like they would get from a traditional desktop application. Given the added complexity for building rich, interactive websites, ensuring they work as expected also requires significantly more effort by testing teams worldwide. Thorough testing is necessary to ensure that your web application performs optimally for existing and potential customers.
Many organizations have QA teams that manually go through their sites to test for any defects. However, this practice is inefficient and prone to human error, especially for large sites with tons of functionality. These days, most application development businesses are transitioning toward test automation. By automating a large portion of repetitive testing tasks, these QA teams can focus on other high-value work, such as exploratory testing. Automated tests won’t replace manual testing, but it’s necessary given the demands and expectations of modern software development.
A common form of automated testing for ensuring that your websites function correctly is writing and running end-to-end tests (sometimes called e2e tests). These tests simulate the actions a typical user would perform when visiting your application on a web browser. Instead of having someone go through clicking links, filling out forms and observing the results, an end-to-end test can do that automatically in a fraction of the time.
While end-to-end tests are great for eliminating the need for repetitive and tedious work, they also come with plenty of tradeoffs. Mainly, end-to-end tests are slow compared to other forms of testing, like unit testing and functional testing. Often, they lead to delays in the development and deployment process since the team has to wait for a test suite to finish—sometimes for hours. The lack of quick and reliable feedback is one of the primary reasons organizations abandon their test automation efforts.
If your team isn’t careful, they can find themselves with a suite of slow-running tests clogging up the development pipeline. But it doesn’t always have to be that way. You can take a few steps to make your tests faster—significantly faster, in some cases. This article covers five ways you can improve the speed of your end-to-end tests so you can reap the benefits of test automation without sacrificing development or deployment time.
Modern web applications do lots of background work on a single page, often communicating with multiple services across the world. For example, when a user visits a website, the site will pull in data from third-party services asynchronously to have the information needed to render a given page. Another example is submitting a form on a website, which sends the data to an external database living in another network.
These asynchronous actions provide a smooth experience to the users of a website, only transmitting or fetching the data needed and avoiding full page reloads. But when running end-to-end tests, all of these network connections take up precious testing time. Your tests will have to wait around for multiple network requests to respond. Since you usually won’t have any control over the performance of these calls, your tests can experience severe delays or failures requiring you to re-run your test suite.
Instead of executing these expensive network requests and waiting for their response, most end-to-end testing tools and frameworks allow you to mock any network call made during testing. In the software testing world, a mock is an object that simulates the actual behavior of an existing implementation. In this case, you can simulate the network connection without making the actual request. Mocks help avoid delays from the request, and they also allow you to define its response. They’re handy for ensuring your tests aren’t affected by an external service you can’t manage.
Mocks aren’t a solution to every slow network request, however. While running end-to-end tests, you should always strive to test real behavior whenever possible. But given the potential issues of using third-party services that aren’t under your control, it’s more practical to mock these connections. This way, you control the responses you want to validate in your tests and speed up their execution while avoiding intermittent failures.
Depending on your application, you may need to start or end with a clean slate before running through your end-to-end test suite. Often, that means setting up your application’s database and other data stores with the information needed to execute all tests consistently and cleaning up afterward. That way, you can ensure that your end-to-end tests perform in the same way every time your team or continuous integration systems execute them.
Almost all end-to-end testing tools have special steps to make it easier to handle the initial setup for your tests, along with the actions needed to clean up at the end of your test run. These steps are often known as setup and teardown. The setup and teardown steps help maintain consistency between test runs, but they’re also one of the main problems for slow-running test suites when misused.
Most frameworks will allow different methods for running a setup or teardown step. You can perform your actions only once at the start or end of the test run, or you can have these steps run before and after every test scenario. You can use both in your end-to-end test suite, but knowing which one to use at a given time can have a massive impact on performance. To improve setup and teardown efficiency and avoid long-term maintainability issues, you’ll need to determine the appropriate steps required for your test scenarios.
For instance, if you need to insert thousands of records into a database, you should do that only once at a setup step since it will likely take some time to complete. Likewise, if you need to reset some data after testing, look into putting those as a teardown step for scenarios that need them instead of after every one. Defining the correct amount of setup and teardown functionality takes a bit of planning and work. But when done correctly, it can significantly speed up your tests.
Most end-to-end testing tools run your tests sequentially by default. Your software testing tool goes through each defined test scenario, one by one, running each step in the process. Once the test finishes with either a successful or failed result, it moves on to the next test. If you have dozens of end-to-end tests, each taking a significant amount of time to complete, your entire test run can hold back your team’s work while they wait for the automated test results.
To speed up your test execution, look if your preferred testing tool allows you to run your tests concurrently or in parallel. If available, this mode will enable you to run more than one test at a time. Instead of executing each test one at a time, you can have two or three test scenarios running simultaneously, significantly shortening the time it takes the entire test suite to complete.
Keep in mind that the more tests you run simultaneously, the more resources you’ll need to execute them efficiently. For instance, some organizations use less powerful servers for testing purposes, which might not handle as many requests as your tests need. Also, you might run your tests on a low-powered computer that struggles to run more than a handful of instances of your application without getting bogged down. You’ll need to experiment with what you have available to find out how many concurrent tests work for your situation.
Also, you’ll have to be aware that if your tests require a specific order to run successfully, executing tests concurrently won’t work since you can’t control the order of your tests. As a general rule, your end-to-end tests should run independently from each other without relying on different scenarios to pass. If that isn’t an option for your test suite, you can’t take advantage of running more than one test at a time.
Generally speaking, end-to-end tests are heavy processes that need plenty of computing power. These tests juggle multiple operations at the same time—handling browser instances, rendering your web application, executing steps in your scenarios and more. If you’re a developer, you might also have the entire tech stack for the application running on the same system. If you have a low-end computer running your tests, any tweak won’t have much of a positive effect, and it will still feel painful to wait for your test suite to execute.
The quickest way to solve this issue is to upgrade the systems running your test suite. These days, most computer systems and parts are more powerful for less money. Investing in an upgrade may be a worthwhile investment for the time savings you’ll experience running end-to-end tests or other work tasks. To be an effective automation tester, you need to keep your feedback loops as short as possible, and often better hardware is a great way to achieve that result.
You might not be in a position to upgrade your computers or don’t have the desire to replace a functional system just for running your tests faster. Thankfully, these days you can leverage the power of cloud computing to offload this work when you need it. Cloud services can provide a way to run your automated tests on powerful hardware at a fraction of the price. In some cases, you’ll only need to pay for the computing time you use.
While offloading your tests to a remote server helps keep your system usage at a minimum, it also might not help you experience improvements to your testing times. Because you’re moving your work to an external service on the internet, you’ll have to deal with network latency. Also, most of these cloud services run your tests in virtualized environments, which often aren’t as capable as real hardware. But the flexibility it provides makes the cloud an excellent alternative for your end-to-end tests.
A common mistake that occurs in testing teams that spend time focusing on how to speed up a test suite is neglecting the performance of the application under test. If the application you’re testing doesn’t load or render in an adequate amount of time, no amount of adjustments or modifications will help speed up your end-to-end tests.
Good performance of your web application is a requirement in today’s world. Visitors to your website won’t stick around if it takes more than a few seconds for something to show up on the screen. Sites with delayed interactions or slow animations will appear unreliable and untrustworthy in the eyes of potential customers. Inevitably, your company will lose business and money in the long run because they’re delivering a subpar experience.
Unfortunately, most developers nowadays don’t build software with performance in mind. In a rush to develop and deliver as quickly as possible, organizations often treat performance as an afterthought. It’s only when customers begin complaining and the business isn’t converting as expected that the team starts to pay attention to performance. But by the time they arrive at that point, it may be too late to fix the damage.
If you’re a tester, you may not have the opportunity to fix performance issues directly. However, you do have the ability to ensure performance is a vital part of your testing workflow by adding performance testing in your pipeline. Catching slow actions and inefficient behavior to alert the development team early in the development process will help squash potential problems before your customers do. It will also help with testability, as your end-to-end tests won’t have to struggle with a sluggish application.
In today’s market, it’s crucial to ensure your applications work smoothly and behave as your customers expect. To keep up with the expected pace of delivering high-quality applications, you can leverage the power of test automation and end-to-end testing. End-to-end tests will go through your application just like a typical user would do, making it an excellent choice to ensure your site works as intended.
However, these tests have a few drawbacks, mainly that they’re slow to execute and can create bottlenecks in the development process. (Read more about “How Slow Is Too Slow to Run Your Tests?” here.) You can minimize these delays with a few actions, such as mocking network requests, running multiple tests at the same time and observing how you set up your test suite and each scenario. You can also reduce testing time using more powerful hardware or catching performance degradations in the application as part of your test pipeline.
A slow test suite can become expensive in the long run, and your application’s quality will suffer if you don’t manage your testing times and feedback loops. It’s well worth the effort to speed up your tests and keep them that way.
Editor’s Note: Progress Telerik offers an end-to-end testing tool in Test Studio. It comes with a 30-day free trial so you can explore all Test Studio has to offer.
Dennis Martinez is a freelance automation tester and DevOps engineer living in Osaka, Japan. He has over 19 years of professional experience working at startups in New York City, San Francisco, and Tokyo. Dennis also maintains Dev Tester, writing about automated testing and test automation to help you become a better tester. You can also find him on LinkedIn and his website.
Subscribe to be the first to get our expert-written articles and tutorials for developers!
All fields are required