Hi,
Is there an option in Test Studio to fail the entire test list if it has been executing longer than a certain period of time?
For some reason, a test or two is getting hung up and not failing the test and going on to the next test. When it does that, a simple test list of a few minutes will still be hung after a few hours.
I'm suspecting that the problem with the long running test is it's failing to find elements and not timing out correctly. We are testing Silverlight applications.
Sandy
Is there an option in Test Studio to fail the entire test list if it has been executing longer than a certain period of time?
For some reason, a test or two is getting hung up and not failing the test and going on to the next test. When it does that, a simple test list of a few minutes will still be hung after a few hours.
I'm suspecting that the problem with the long running test is it's failing to find elements and not timing out correctly. We are testing Silverlight applications.
Sandy
7 Answers, 1 is accepted
0
Hello Sandy,
Stoich
the Telerik team
Test Studio is rigged to always fails tests when they take too long. It's a very unusual problem for TestList to be hung. At what point in your TestList execution does it get hung. My guess is that this isn't related to an HTML elements that's missing or something along those lines. The only situation that I can think of is a connection issue if you've got the Silverlight Connect timeout bumped up to an unreasonably high number (screenshot 1).
Currently there's no way to enforce a timeout on a TestList because it's so unusual for it to get hung in the first place.
Stoich
the Telerik team
Vote for Telerik Test Studio at the Annual Automation Honors Voting!
0
Sandy
Top achievements
Rank 1
Rank 1
answered on 17 Aug 2011, 10:32 PM
Stoich,
All the timeouts are set to the default configuration right out of the box, so we haven't messed with those yet. If it's not hanging because the element was not found, then could it be timing issues?
For example, a test that performs these actions:
1. Login to the application (this is a Test as Step that performs some waits as the application loads)
2. Wait for an element to be visible. (the element find logic is using an AutomationId and a XamlTag of grid)
3. Verify text content in the element now that it's visible.
In this test, the application's element in Step 2 is visible for a period of time and then collapses. By watching the test run, I'm guessing that it hasn't completed the wait steps in Step 1 before the element's visibility collapses.
When the test actually gets to Step 2, it should time out and fail the test because the element's visibility is now collapsed and the wait step will never pass. Instead, on some test list runs it just keep waiting for it to be visible beyond the time out period. This is an intermittent issue.
Another example I found was a test that performed these actions:
1. Login to the application
2. Verify an element displays showing the application is busy (this is the application startup that will collapse the busy screen when it's completely loaded)
3. Logout of the application
This test also got hung up for excessively longer than the timeout periods (30 minutes to several hours). Adding an extra step to wait for the busy screen to be collapsed solved that problem. I would expect the test to time out and fail on its own when moving on to Step 3 wasn't possible within the set time out periods.
Sandy
All the timeouts are set to the default configuration right out of the box, so we haven't messed with those yet. If it's not hanging because the element was not found, then could it be timing issues?
For example, a test that performs these actions:
1. Login to the application (this is a Test as Step that performs some waits as the application loads)
2. Wait for an element to be visible. (the element find logic is using an AutomationId and a XamlTag of grid)
3. Verify text content in the element now that it's visible.
In this test, the application's element in Step 2 is visible for a period of time and then collapses. By watching the test run, I'm guessing that it hasn't completed the wait steps in Step 1 before the element's visibility collapses.
When the test actually gets to Step 2, it should time out and fail the test because the element's visibility is now collapsed and the wait step will never pass. Instead, on some test list runs it just keep waiting for it to be visible beyond the time out period. This is an intermittent issue.
Another example I found was a test that performed these actions:
1. Login to the application
2. Verify an element displays showing the application is busy (this is the application startup that will collapse the busy screen when it's completely loaded)
3. Logout of the application
This test also got hung up for excessively longer than the timeout periods (30 minutes to several hours). Adding an extra step to wait for the busy screen to be collapsed solved that problem. I would expect the test to time out and fail on its own when moving on to Step 3 wasn't possible within the set time out periods.
Sandy
0
Hi Sandy,
I was not able to reproduce this issue.
Would it be possible for you to create a test project (TestList included) that runs against a publicly accessible Silverlight app? If you provide such a test project we can reproduce the problem locally and analyze the issue. I would suggest Telerik's Silverlight demo page for this:
http://demos.telerik.com/silverlight/
Alternatively you can send us your actual test project but that would be less useful since we won't be able to actually run it and see the issue first hand.
If you consider any of the data I'm requesting to be too sensitive for a public forum I can provide an e-mail account.
One last thing to check is the setting for the TestList (screenshot 1). Check for any unusually long timeouts.
Kind regards,
Stoich
the Telerik team
I was not able to reproduce this issue.
Would it be possible for you to create a test project (TestList included) that runs against a publicly accessible Silverlight app? If you provide such a test project we can reproduce the problem locally and analyze the issue. I would suggest Telerik's Silverlight demo page for this:
http://demos.telerik.com/silverlight/
Alternatively you can send us your actual test project but that would be less useful since we won't be able to actually run it and see the issue first hand.
If you consider any of the data I'm requesting to be too sensitive for a public forum I can provide an e-mail account.
One last thing to check is the setting for the TestList (screenshot 1). Check for any unusually long timeouts.
Kind regards,
Stoich
the Telerik team
Vote for Telerik Test Studio at the Annual Automation Honors Voting!
0
Sandy
Top achievements
Rank 1
Rank 1
answered on 24 Aug 2011, 02:26 AM
Attached is a sample project using the public Telerik site.
I couldn't reproduce the error by running the test list I created, but it does hang every time for me on Step 3: Wait for Exists 'TelerikEditorTelerikeditorexample' when trying to do a Quick Execution.
If you can see it happening, then I think the concept of tests getting hung up under certain conditions is similar to what we're seeing in our project.
I did expect to see the test list have trouble when I shrunk the browser window so that on the page load, Step 3's TelerikEditorTelerikeditorexample window popped up over the navigation area, covering the BtnNextExampleRadbutton I was trying to click in Step 4. Somehow it clicked right through the window that was in the way and got to the underlying button.
This scenario is typically where I have seen our test project hang during its execution.
I couldn't reproduce the error by running the test list I created, but it does hang every time for me on Step 3: Wait for Exists 'TelerikEditorTelerikeditorexample' when trying to do a Quick Execution.
If you can see it happening, then I think the concept of tests getting hung up under certain conditions is similar to what we're seeing in our project.
I did expect to see the test list have trouble when I shrunk the browser window so that on the page load, Step 3's TelerikEditorTelerikeditorexample window popped up over the navigation area, covering the BtnNextExampleRadbutton I was trying to click in Step 4. Somehow it clicked right through the window that was in the way and got to the underlying button.
This scenario is typically where I have seen our test project hang during its execution.
0
Hello Sandy,
thank you for providing us with a reliable way to reproduce this issue.
I've logged this as a bug, you can track its progress here:
http://www.telerik.com/support/pits.aspx#/public/test-studio/7600
I've went ahead and added 800 Telerik points to your account.
The issue seems to be related specifically with the RadRichTextBox control. Test recorder against other controls don't seem to exhibit the same problematic behavior.
Regards,
Stoich
the Telerik team
thank you for providing us with a reliable way to reproduce this issue.
I've logged this as a bug, you can track its progress here:
http://www.telerik.com/support/pits.aspx#/public/test-studio/7600
I've went ahead and added 800 Telerik points to your account.
The issue seems to be related specifically with the RadRichTextBox control. Test recorder against other controls don't seem to exhibit the same problematic behavior.
Regards,
Stoich
the Telerik team
Vote for Telerik Test Studio at the Annual Automation Honors Voting!
0
Sandy
Top achievements
Rank 1
Rank 1
answered on 26 Aug 2011, 08:10 PM
Stoich,
Thanks for the point goodies. Do you think fixing this bug for the RadRichTextBox will affect other controls as well?
We aren't using Telerik's Radcontrols in our project, but intermittently experience the issue of Test Studio/Internet Explorer stops responding. It's been seen in Quick Execution, but it impacts us more when it happens during our automated Test List executions.
I originally posted the question in the forums first in case there was a quick fix to keep the Test Lists from running long. So far, we've been able to identify problem areas and add extra steps inside our tests as a work around. We have a subscription to support, if needed, to continue debugging.
Sandy
Thanks for the point goodies. Do you think fixing this bug for the RadRichTextBox will affect other controls as well?
We aren't using Telerik's Radcontrols in our project, but intermittently experience the issue of Test Studio/Internet Explorer stops responding. It's been seen in Quick Execution, but it impacts us more when it happens during our automated Test List executions.
I originally posted the question in the forums first in case there was a quick fix to keep the Test Lists from running long. So far, we've been able to identify problem areas and add extra steps inside our tests as a work around. We have a subscription to support, if needed, to continue debugging.
Sandy
0
Hi Sandy,
a fix for the the RadRichTextBox issue is unlikely to resolve the other issues you're describing. However, we can't be 100% certain of that since our engineers are still working on the issue.
Please go ahead and give us more details on the other issues you're experiencing so that we analyze and eventually resolve them.
All the best,
Stoich
the Telerik team
a fix for the the RadRichTextBox issue is unlikely to resolve the other issues you're describing. However, we can't be 100% certain of that since our engineers are still working on the issue.
Please go ahead and give us more details on the other issues you're experiencing so that we analyze and eventually resolve them.
All the best,
Stoich
the Telerik team
Vote for Telerik Test Studio at the Annual Automation Honors Voting!