Enable "Continue on failure" while using BaseWebAiiTest.ExecuteTest

[Robert]

Hello people at Telerik,

In my test I am using a main test with one coded step that calls other tests using this.ExecuteTest.

This enables a modular setup and implements flow through coded statements like If/Else or Switch.

Currently I am faced with a problem when a test fails:

For example:

However, if a Step within test "A" Fails,  Test "B" will not be executed.

I would like to Execute test B no matter what.

The log shows:

 However, the actual step within test A that fails, is marked "continue on failure" within the UI.

The failing test is a coded step with an assertion.

It seems like the default behaviour of BaseWebAiiTest.ExecuteTest --> "Continue on failure = false".

Is there a way to overload the method, to mark it as Continue on Failure = true?

Thanks in advance.

With friendly regards,

Robert

 

 

 

 

[Elena]

Hi Robert,

Thank you for sharing thorough details of your findings. 

First of all I want to mention that the 'ContinueOnFailure' flag is supposed to be used for debugging purposes while working on the tests and adjusting these.

Then, when we talk about nested tests and that flag it is important to note that it works on single test level. Here is what this means:

• A parent test calls child test A as step 1 and child test B as step 2.
• Test A includes a step which is marked with the 'ContinueOnFailure' flag. 
• The parent test is executed and the marked step in child test A fails.
• The outcome is that the rest of the steps in child test A are executed - because of the set flag. But the overall result of step 1 in the parent test is failure thus the execution of the parent test is terminated and step 2 is never executed. 

If the intention is to continue the execution within the parent test so that also child test B is executed, you need to set the 'ContinueOnFailure' flag to step 1 in the parent test. 

So, considering these notes I have to say that the behavior which you describe for the outcome of the coded step is the expected one. Also, there is no straight forward way to change it - for example with a method overload. Instead you will need to further adjust the code in the coded step. 

A good approach for the described scenario will be to encapsulate the execution of test A within a try..catch block. That way you can ensure that the possible failure of the test will not interrupt the overall execution. 

I hope this information helps you continue working on setting up your automated tests. Let me know in case you have any additional questions.

Regards,
Elena
Progress Telerik

[Robert]

Hello Elena,

Thank you for your detailed explanation regarding the use of Continue on Failure in combination with Test as Steps.

These kind of fundamentals are always useful.

After some experimenting I concluded that "try catch" is a very useful solution to my problem.

I am going to implement this. Thank you very much for bringing this forward. 

FYI: I have implemented a solution based on this article: Skip Iteration - Progress Test Studio. For every Test I have a MainTest with a single coded step that executes SubTests (from a pool of subtests) in a certain order based on information provided in a external data source (Data driven)

Is there another way of handling my 'inappropriate' use of Continue on Failure?

My (data driven) WPF tests generally follow a certain flow for every iteration:

1. Open a Case
2. Change data through functionality provided by the system under test
3. Verify outcomes versus expectations
4. Close the Case

There are two reasons I use Continue on Failure in certain steps within this flow.

A. I want to verify multiple expectations.

For example i, ii and iii.  When verification-step i fails, I still want to verify ii and iii. Marking i as continue on failure ensures ii is also verified and so on.

Is there another way of handling this?

B. I want to Close the Case to setup the application for the next iteration.

When a verification at step 3 fails and step 4 is not executed, the current case will stay active in the application (RadDocumentPane if i'm correct). In the application I am testing, this leads to all kind of problems. Tabs will be open for every case after the failed one. Most of the time the test is not able to recover from this snowball effect.

My solution is to wipe the slate clean at the end of every iteration, so every iteration starts in the same start-up state. The first step of every iteration is restoring the default layout.

 Under normal circumstances only verification - steps fail. Marking them all as Continue on failure ensures Step 4 (closing the case) is executed and the next iteration is able to start clean.

Is there another way of handling this?

Regards,

Robert

 

 

 

 

[Elena]

Hi Robert, 

I am glad to hear that my suggestion turned out to be useful for you. Please, find my comments on the two listed topics. 

A. I want to verify multiple expectations.

From my perspective you have the following options:

• Execute the verification i in code and insert it into a try..catch block. That way you ensure that the test continues despite the outcome of that verification. 
• Another option, which might depend on the overall flow of the scenario, is to use the verification in an if..else statement within the Test Studio UI. 

B. I want to Close the Case to setup the application for the next iteration.

Actually your approach to clear up the application state at the beginning of the test is not bad at all. Because the test might fail earlier than you expect it (where you set the flag 'Continue on failure') and the last clean up step is again not executed. 

And in the situation where it is very essential to have the clear state of the tested application, it is better to have the clear actions at the end and at the beginning - of course executed upon meaningful conditions. That way even if the previous test/iteration did not finish correctly, you can ensure the next one runs as expected. 

I hope that this information is also helpful for you. Let me know if you need additional assistance.

Regards.
Elena

In the scenario when you need to clear up the application for the next iteration

 

[Robert]

Hello Elena,

Thank you for sharing your suggestions. I will consider these going forward.

Implementing try..catch solved multiple issues for me.

• The tests are better at recovery from failures. Especially recovery outside of verifications I marked as "continue on failure"
• Adding Log.WriteLine("....." + exc.Message)  inside the catch block logs the failed teststep at the beginning of an iteration in case of failure. This decreases my debug time significantly. I use a lot of data driven nested tests, which makes the log difficult to read.

So, concluding: I am very happy with the change. Thank you again!

With friendly regards,

Robert

[Elena]

Hi Robert, 

I am really pleased to hear our conversation helped you to improve the testing project you work on. We are always happy to share our knowledge with anyone intrested! 

Thank you also for your cooperation throughout our communication!

Warm regards,
Elena