Hi,
We are experiencing random failures in our current suite of smoke tests that we run against our application. They are usually very stable however, we have recently implemented a service worker on the application and now the tests are failing.
The test step that is failing is a simple click a button and then verify that a div is displayed. We have tested the changes manually and there are no issues with the changes.
We then reverted the build of our application to exclude the service worker change and all tests ran successfully.
Has anyone experienced this issue related to service workers and Telerik failures before?
8 Answers, 1 is accepted
Hi Sheridan,
I am sorry to hear about the random failures in your existing tests. I will be happy to assist you and find out what is happening and haw to fix it. To do that, I need some additional details about the misbehavior. Please elaborate more on the topics below.
- You mentioned that the execution of the test does not fail every time. This might be a synchronization issue, where the new version of the application loads at different rates or is otherwise changed. You can add some execution delay or wait steps whenever the page content is changed. That should improve the stability of the test and could resolve the encountered misbehavior.
- What is the failure error, when the test fails? Please export the full step failure details, that are generated by Test Studio, and attach them to your next reply. I will analyze that information and provide further guidance.
- I assume that other than the service worker, there are other changes in the application under test. You can try to enable SimulateRealClick for the click step that is causing troubles (see simulateRealClick.png). This option will change the simple click to series of actions that simulate the real user's actions. Run the test several times and let me know if that made it more stable.
I hope the above suggestions will be helpful. Of course, if you need further assistance, please let me know and share additional details about the failure.
Regards,
Plamen Mitrev
Progress Telerik
Rank 1
Hi Plamen
Thanks for getting back to me. I've made some of the suggested changes but the tests are still failing.
It's quite strange because the failing step is used in 3 other tests that pass but then on the fourth logout it fails.
Am I able to email you the log file? I'm unable to upload it as it's over the 2MB limit.
Hi Sheridan,
Thank you for testing the suggestions from my previous reply. Since those did not resolve the issue, I need to explore the failure details and the test itself.
Actually, the limit for attachments is 20 MB and you should be able to upload ".zip" files that are smaller than that. However, if you continue to experience troubles with that, please upload them to this private shared folder. You can also open a private support ticket, where we can continue our discussion and you can share more details.
Other than the exported step failure details, please share a sample Test Studio project with a test that reproduces the issue and another test with the same step, which works fine. Although I will not be able to access the application under test, I will explore the steps and configurations you have in place. This might be helpful in the investigation.
Thank you for your cooperation in advance.
Regards,
Plamen Mitrev
Progress Telerik
Rank 1
Hi Plamen
Please find attached the full test logs.
Thank you
Rank 1
Hi Plamen
I have just uploaded our project into the shared folder. The path is Post-deployment smoke tests > SmokeTest.
Hi Sheridan,
I analyzed the execution log and explore the SmokeTest, but it is still not clear to me what is causing the issue. I need you help with some additional information to help in my investigation. Please follow the steps below and attach those resources.
- Clear/Enable the Test Studio application log.
- Execute the SmokeTest locally again.
- Gather the generated application log and attach it as .zip to this thread.
- Open the step failure detail, by double clicking on the red "X" in the failed test step. It should be step 4 in SmokeTest.
- Click the Export button and save the archive on your disk.
- Attach the exported step failure details and send them over in this thread, or in the shared folder.
This issue seems to be specific to the application under test. I am interested to know if it will be possible to provide temporary access to the application under test or a sample application that reproduces the issue. Please discuss this internally and let me know.
Thank you for the continuous cooperation in this discussion.
Regards,
Plamen Mitrev
Progress Telerik
Rank 1
Hi Plamen
I have uploaded the requested application log as well as a data file to access our test site. Hopefully you'll have more luck!
Thanks
Hi Sheridan,
I analyzed the test and the behavior of the application during the execution of "SmokeTest" and I will share my thoughts and suggestions below.
During my exploration of the SmokeTest and its steps, I tried to isolate and test the failing steps. Since those were in step 4 "Execute test 'Service Provider - SmokeTest'", I tried to run only that test. This isolated test run did not reproduce the issue, even when I put it in a loop to repeat it 6 times. This showed that the test is working as expected on its own.
However, when I ran all 4 tests, I encountered failure at step 39 on test "Service Provider - Random In-Progress Job Test". I observed the test execution and noticed that the application under test is stuck after clicking the Notes button successfully. The page is loading for a long time and eventually the test times out. I tried to add some execution delays, wait for exist verification steps and cleared the browser cache, but the result is similar.
In some test runs, the application got stuck during the logout step in the last test and it seems that the behavior is not consistent. Please check what changes were made with the last update and if something could have caused this behavior. Also, observer the test execution and look for the loading indicator in the application and if it loads successfully. I see the grey bar from the screenshot below, which is moving to indicate the page is loading, but it never loads fully.
I noticed that during the test execution, some of the click actions happen at the top left corner of the browser (at 0,0 coordinates). This usually happens when the target element exists in the page, but is not actually visible there. You can enable the annotations during quick execution and observer the test and each actions.
For example, step 16 from test "Random In-Progress Job Test" clicked at 0,0 coordinates. After further inspection of the target element's find logic and the DOM tree of the application, I found that its find expression matches three elements. In this case, Test Studio will interact with the first successful match, which is not visible anywhere in the page but exists in the DOM. The second element is actually the "Charges" button in the left side navigation.
Please analyze the execution in more details and check if some of the steps or element find logic should be improved to identify the correct element. If you have any follow up questions or concerns, let me know and we will continue our discussion and the investigation.
Thank you for your continuous cooperation.
Regards,
Plamen Mitrev
Progress Telerik