Test Passes on Exception

[Filip]

I have a Parent test that executes a Child test in a coded step:

[CodedStep(@"Step1", RequiresSilverlight=true)]
public void Step1()
{
    ExecuteTest("Child.aii");
}

The Child step throws an unhandled Exception but this doesn't cause the Parent test to fail and the following steps in Parent are executed (continue on failure is off for each step in Parent). After the entire tests completes it's marked as passed 

I did some debugging and checked the test result in OnAfterTestCompleted() method in both Parent and Child. In Child the status was "Fail" but in Parent it was "Pass". 

Is there any setting that could cause such behavior or is it simply a bug?

PS I just checked and the problem doesn't occur if I execute the Child step as a regular, non coded step (by using Add > Code as step). If I transform this into a coded step again it stops working. This suggests a bug...

[Keaegan]

Hello Filip,

Please try using the following change in your code for running a test as a step:

[CodedStep(@"Step1", RequiresSilverlight=true)]
        public void Step1()
        {

             this.ExecuteTest("Child.aii");

        }

Please let us know if you experience the same issue using the above instead. If this does not resolve the issue, are you able to add a test using the normal "Test As Step" option from the Add menu, then right-click the step and choose Customize Step in Code? We'll look for your reply with the above.

Greetings,
Keaegan
the Telerik team

[Filip]

Hi Telerik

Adding "this." to my code didn't help.

Also, in the last part of my first post I already wrote that I tried adding test as step manually (Add -> Test As Step). Such step works ok (i.e. the entire test fails if the step throws an exception). However, after I transformed it into a coded step (using Customize Step in Code) it stopped working again although I did no modifications to the generated code.

Cheers

Filip

[Cody]

Hi Filip,

I have tried to reproduce this in both our 2010.3.1213 build and our 2010.3.1307 build. Both versions bubble up the exception thrown by the subtest to the parent test which correctly marks the overall result as Failed.

Overall Result: Fail
------------------------------------------------------------
'1/27/2011 4:24:12 PM' - Starting execution....
'1/27/2011 4:24:14 PM' - Detected custom code in test. Locating test assembly: PruebasTestBrickControl.dll.
'1/27/2011 4:24:14 PM' - Assembly Found: C:\Users\gibson\Documents\Support Issues\PruebasTestBrickControl\PruebasTestBrickControl\bin\Debug\PruebasTestBrickControl.dll
'1/27/2011 4:24:14 PM' - Loading code class: 'PruebasTestBrickControl.WebAiiTest2'.
------------------------------------------------------------
'1/27/2011 4:24:15 PM' - 'Pass' : 1. Navigate to : 'http://www.telerik.com/'
'1/27/2011 4:24:17 PM' - 'Fail' : 2. [WebAiiTest2_CodedStep] : @"Execute test 'WebAiiTest1'

>>> Test-as-Step 'WebAiiTest1.aii' log starts:

Overall Result: Fail
'1/27/2011 4:24:17 PM' - 'Fail' : 1. Verify 'TextContent' 'Contains' 'Not Automated Testing' on 'AutomatedLink'
------------------------------------------------------------
Failure Information:
~~~~~~~~~~~~~~~
Content.TextContent of 'AutomatedLink' does not match!

   Match Type: 'Contains'
   Expected Result: 'Not Automated Testing'
   Value at time of failure: 'Automated Testing'
------------------------------------------------------------
'1/27/2011 4:24:17 PM' - Detected a failure. Step is marked 'ContinueOnFailure=False' aborting test execution.
------------------------------------------------------------
'1/27/2011 4:24:17 PM' - Overall Result: Fail
'1/27/2011 4:24:17 PM' - Duration: [0 min: 0 sec: 148 msec]
------------------------------------------------------------

<<< Test-as-Step 'WebAiiTest1.aii' log ends.

------------------------------------------------------------
'1/27/2011 4:24:17 PM' - Detected a failure. Step is marked 'ContinueOnFailure=False' aborting test execution.
------------------------------------------------------------
'1/27/2011 4:24:17 PM' - Overall Result: Fail
'1/27/2011 4:24:17 PM' - Duration: [0 min: 2 sec: 802 msec]
------------------------------------------------------------
'1/27/2011 4:24:17 PM' - Test completed!

What excepting is being thrown by the subtest? Can you send me a test that repros the symptom that I can study?

Kind regards,
Cody

the Telerik team

[Filip]

Because of my company policy I can't send you the failing test. Since this only happens in this particular child test (other children tests work fine) I can't even recrete that with a sample test :\

 I'm just wondering if you can think of any tests elements or settings that could be causing such behaviour?

Cheers
Filip

[Cody]

Hi Filip,

Can you share what exception is being thrown by the subtest? Do you have a stack trace you can share? I can try to replicate that exact same exception and see if it repros the symptom.

No there are no settings or elements that would be able to cause this behavior. I even tried setting "ContineuOnFailure" on the step that calls the subtest. I still could not reproduce this symptom.

Greetings,
Cody

the Telerik team

[Val]

i am seeing same problem where i have a list of this.ExecuteTest("") calls in parent test and i call Assert.IsTrue(false) in one of the child tests in this list and it just continues to run. i dont understand why you guys need to see our tests and files when this is a simple scenario??? Just create 2 simple tests and make one call the other inside of a coded step of the parent one and put "Assert.IsTrue(false);" in a child's test step. The worst thing about this problem is that if i have this parent test databound to a list of 500+ customers under test then this will continue to fail for 500 times until you get the log and go "wt*".

[Cody]

Hello Val,

Thank you for the repro steps. You're right I don't need your tests/files when I am able to reproduce the problem. I asked Filip for his test files so that I can figure out how to repro this problem.

The problem you are running into is that "this.ExecuteTest" doesn't return any results or do anything in anyway to abort the parent test. To overcome this please don't call child tests via code. Use our Test-as-step feature and call your child tests via our standard test step. It's cleaner, requires no code, and the results from the child test will cause the parent test to abort.

Another option is to use our RunHelper class something like this:

ResultSummary results = ArtOfTest.WebAii.Design.Execution.RunHelper.Test(@"c:\TestProjectDir\NoResults.aii", @"c:\TestProjectDir\bin\debug\TestProject.dll");
if (results.Passed == false)
{
    throw new ApplicationException("Child test failed");
}

In the databound parent test scenario you will still be faced with the parent test running 500+ times. We do this deliberately so that the tests will return results for all rows of the table. Most of our customers do not want the test to abort on the first row that failed. They want to know if the rest of the rows pass/fail as well.

Greetings,
Cody
the Telerik team