Test List Stopped Unexpectedly

[Yan Jun]

Hi All,

I have few Data-Driven test cases that I execute in quick execution and there are no problem with them. But when I create a test list that includes the same test cases, the test list execution failed and stopped unexpectedly.

1. The first case might be caused by the test list setting that I set which is UnexpectedDialogAction : HandleAndFailTest but the other iterations for the same test case don't have this problem. Furthermore, from the video and screenshot of the failed step , I don't see any Dialog box appeared. The failed step is after the browser navigate to a url link. You can refer from the picture "Case 1.png".

2. The second case is the execution stops unexpectedly without any specific error in the middle of the second iteration. The test list result came out as no test steps are executed and there is no iteration completed while actually the first iteration is completed. I can confirm the first iteration is completed because I checked with my AUT and the saved data is there from the first iteration. You can refer from the picture "Case 2.png".

Does anyone experienced this before?

[Plamen Mitrev]

Hi Yan Jun,

It seems to me that there is something in the test list settings that might be causing the difference in behavior. This is why I want to investigate a sample Test Studio project, which simulates the same misbehavior. Is it possible for you to record a sample data driven test in a new project and try to reproduce the issue? If you do so, please attach the zipped project for further investigation.

Something else that can help me find out what is causing the issue, is the application log. Please reproduce each issue and attach the respective application log for more details. I will check for errors and the overall test list execution.

I am interested to know what happens if you configure the test list to HandleAndContinue or DoNotHandle upon encountering an UnexpectedDialogAction. Please monitor the test list execution for any unexpected dialogs and actions during local test list execution and let me know about anything unexpected.

I am looking forward to hearing from you.

Regards,
Plamen Mitrev
Progress Telerik

 

[Yan Jun]

Hi Plamen,

For the sample project you asked before, will it need to be interacting against a public accessible page? Or is it okay for the scripts to interact against the AUT i am testing?  This AUT is not accessible to public.

I tried running the test list in new and current project. The error "Unexpected Dialog:Updating" for HandleAndFailTest happens at different step for each execution although there is no unexpected dialog. For HandleAndContinue or DoNotHandle , the execution was okay and no unexpected dialog came out.. Since Case 1 is not constant, i cannot reproduce Case 2.

Below are the details of the test list execution with attached application log for your reference 

1. HandleAndFailTest - 12.34 pm

2. HandleAndContinue - 12.52 pm

3. DoNotHandle - 02.53 pm

Please note that the failed steps for 2. HandleAndContinue and 3. DoNotHandle cases are expected.

Thank you.

 

[Plamen Mitrev]

Hello Yan Jun,

I analyzed the log and I see that there is an Unexpected dialog detected in several location during the test execution. The first test list execution with HanddleAndFailTest does correctly fail the test after handling the dialog. The next 2 test list execution also encounter an Unexpected dialog and based on the configuration either handle it and continue or take no action against the dialog. Those are all expected scenarios from my point of view.

I might be missing something important about the test scenario and the behavior of the application under test. To help me understand it better, please share more details, like the following.

- 2 short videos of the execution of a new test list, which contains only the test with unexpected dialog. Once with HandleAndFailTest and the next time with HandleAndContinue option.
- A way to reproduce the misbehavior against a publicly accessible application. If you manage to create a project that reproduces the issue, please zip it and attach it to your next reply.
- If you can't reproduce the issue on another application, please share the Test Studio project against the AUT you are testing. I will not be able to run it, but I will still analyze the configurations and get more familiar with the scenario.
- Please attach the zipped step failure details from tests that fail due to unexpected dialog.

I noticed that there are multiple failing tests steps because the test can not find the element or the expected value is not matching. I would advise you to fix those test and I assume that you are aware of those and working towards fixing them.

Thank you for your cooperation in this discussion. I hope I can help you with this issue after reviewing the above resources. If that does not help, we can schedule an online meeting and try to troubleshoot it remotely.

Regards,
Plamen Mitrev
Progress Telerik