Testloop is unstable - browser closes unexpectely?

[Annette]

We are currently evaluating Telerik Test Studio, and I have a problem trying to get my tests to run steadily in a loop. 

To evaluate, I have written a simple test, which 1) logs the user in to our website, 2) checks that the user is properly logged in (information concerning the user is validated) 3) logs the user out.

This test runs smoothly, but when I put a loop around the test and make it run e.g. 100 iterations (in order to evaluate whether the automated test is stable), it always fails sooner or later. Sometimes on the 3.rd iteration, sometime on the 84.th etc…

When it fails, I get an error in IE 9, saying that the browser has stopped working. When I try to run the same test In Firefox, the browser also closes unexpectedly  sooner or later – but without any errors from the browser. (It is significantly more unstable in Firefox than in IE.) Not surprisingly, the error I get in Telerik Test Studio says that it cant locate the elements it is looking for - since the browser has been closed.  ("is not connected to remote to be able to process command")

Any ideas?

Best regards
Annette

[Ivaylo]

Hello Annette,

From your explanation I assume you are using data binding in your test. The issue you are experiencing is most probably because you are running out of memory at certain point and that is the reason for the inconsistent behavior you are describing (failing on different iterations). However we have quite useful property called ReuseAppWindow defining the number of iterations to reuse the application window in. You can set this value to 10, 15 or 20 (default value = 0) and give it a try. 
In order to proceed select the test and navigate to it's properties pane on the left, refer to the screenshot attached for more information.

Regards,
Ivaylo
the Telerik team

[Annette]

Unfortunately this does not solve the problem. I have tried adjusting the ReuseAppWindow-parameter to 20, 10 and 1 - but the result is the same. The browser (IE 9 or Firefox 13) still closes sooner or later - usually sooner. Typically somewhere between the 3rd and the 20th loop.
I am not using Data Binding - I have tried to make the test at simple as possible. (I was experiencing unstability in another test and set up this test in order to pinpoint the cause of the unstability.)

/Annette

PS: I have mailed one of your colleages concerning this problem a couple of weeks ago (2/8/2012) - and again last week, but I still have not recieved an answer. (I enclosed the test and the FailLogs).

PPS: The error I get:

ExecuteCommand failed!
InError set by the client. Client Error:
mozCommandProcessor: ProcessInformationCommand(): Document is not ready to retrieve its markup.
BrowserCommand (Type:'Information',Info:'DocumentMarkup',Action:'NotSet',Target:'null',Data:'',ClientId:'1ede603e-ebd2-418a-9467-3c1e59d39f8c',HasFrames:'False',FramesInfo:'',TargetFrameIndex:'-1',InError:'True',Response:'mozCommandProcessor: ProcessInformationCommand(): Document is not ready to retrieve its markup.')
InnerException: none.

[Annette]

I have asked a colleague to run the test on his PC. He gets the same error - IE crashes after 30 loops.
I have now submitted a support ticket with the zipped test.

/Annette

[Ivaylo]

Hello Annette,

Please note that it is bad practice to run a loop so many times. For such purposes we strongly recommend to use data binding. Actually the ReuseAppWindow property works for data bind tests. Usually when testing login functionality it is better to test it with multiple users instead with only one user, in such cases data binding is also quite helpful. More about data binding you can read in this article.

Kind regards,
Ivaylo
the Telerik team

[Annette]

Hallo again

I tried switching to Data Binding (using a set of data, which contained 500 rows with the same userID + password). This didn't solve the problem. I also tried different settings of the ReuseAppWindow-parameter in combination with the Data Binding (setting it to 20, 10, 1) - but still no result. (The browser still closed unexpectedly sooner or later).  
However, I did get the suggestion in the support ticket I issued, that I should try to convert some of the lines in my test to "Test as step". Since the full test was only 35 lines, I didn't find this necessary when I recorded the test.
Now I have moved some of the lines to 6 separate steps, and it looks like it has solved the problem. The test can now run the 500 cycles without failing. Initially when I tried the "Test as step"-version I had the ReuseAppWindow set to 1, but when I switch it to 20, it still runs smoothly. So to me it looks like it was the "Test as step" which made the difference. 

Best regards
Annette Hørning

[Ivaylo]

Hello Annette,

I am glad that your issue has been fixed, and I am really surprised that the test as step fixed it. 
Thank you for uploading your project in the ticket opened so we can test and see the issue.
I hope adding a test as step is working for you fine.

If anything comes up please update the forum post or the the ticket.

Regards,
Ivaylo
the Telerik team

[Annette]

I am surprised that it worked too - that is why it was the last thing I tried.
As mentioned earlier I initially tried introducing the Data Driven test and using the ReuseAppWindow. This made the test significantly more stable (now failing roughly between iteration 50 and 200.) 
However, as mentioned I couldn't get it to run eg. 500 iterations withou failing before I introduced the "Test as step".
Just for fun (well some sort of fun :-)) I tried switching back to the Data Driven test without steps tonight, and it failed after iteration 172. 

I have no explanation why the introduction of steps should make a difference - and it could be a coincidence. (Since 500 iterations takes almost 4 hours, I havn't any statistical evidence prooving that it will always work with the test steps and never without.) 

But I would like to add, that I noticed that the test always failed immidiately after trying to navigate to a new page. (In the test I navigate between 6 different pages.) I broke the test down into test steps, and made a new step each time the test sends the user to a new page. I am not a developer, so obviously I don't know what kind of magic is going on. But it could seem as if the interaction between the browser that is busy navigating to a new window - and Telerik Test Studio - is improved when the steps are introduced. But I'm just guessing.

Just writing this as a possible help for others. I consider my own problem solved :-)

Best regards
Annette

[Ivaylo]

Hello Annette,

Thank you for your update.
I am happy to hear that you are considering your problem is solved.
I will keep the written in mind and will do some further testing. If anything comes up I will update this forum post.

Kind regards,
Ivaylo
the Telerik team