[Solved] WebUI (Crash)Test Studio

[Espen Harlinn]

Once more I attempt to create a test containing more than 80 steps, and WebUI Test Studio crashes at exactly step 80. Needless to say this doesn't exactly make me swoon with delight...

Sending you the project isn't an option, but believe me: what I'm trying to do here isn't exactly rocket science either.

Most of the complexity of the pages comes from DNN 4.9.4 - and I gather that support for DNN is one of Teleriks key selling points.

So now I'm wondering how many of the people who actually bought this product uses it...

To sum things up:
WebUI Test Studio crashes when a test reaches about 80 steps
WebUI Test Studio has problems recording the text I enter into fields
Exceptions when executing tests - once this occurs I have to restart Visual Studio 2008
WebUI Test Studio seems to have some trouble with the integration into Visual Studio 2008 - as in setting SelectDropDownType to ‘ByText’ and still having to set SelectionIndex for a SelectDropDownActionDescriptor

Regards
Espen Harlinn

[Cody]

Hello Espen Harlinn,

We are very sorry you are having such difficulties with our product. We'd like to dig deeper into the crashing issues you are experiencing. Is it pretty easy for you to recreate this crash? Can you send us a stack trace of the crash? Would it be possible to schedule a Live Meeting with you where you can show us the crash happening and how you are able to make it happen?

I believe we understand the other issues you are running into:

• Recording of typed text may not capture the right text. This is an old issue we are aware of and are working on improving this with our 2.0 version currently under development.
• You must still enter an Index number even though you are selecting a drop down by text. We can address this in our 2.0 release by changing this property to become readonly.
• Tool acts very slow with a large number of steps. We suspect and the crashing may be related. My co-worker is working on trying to reproduce this today.

Kind regards,
Cody
the Telerik team

Instantly find answers to your questions on the new Telerik Support Portal.
Watch a video on how to optimize your support resource searches and check out more tips on the blogs.

[Espen Harlinn]

Is it pretty easy for you to recreate this crash?

When a test is at about eighty steps Visual Studio 2008 sometimes crashes. So I might work for several hours and not experience a crash…since I'm trying to work within the limits I've experienced for the tool I try to keep the tests as short as possible, and I really, really need to get this work finished. WebUI was chosen because it seemed, at the time, like a timesaving tool compared to coding directly against the IE interfaces. Given the choice I probably would have spent some more time evaluating the various alternatives available before starting the project. That time wasn't available, and I obviously have even less time available now - as in things are getting somewhat “tight”.

Can you send us a stack trace of the crash?

All I see at that point is the Visual Studio 2008 “crash” dialog – offering to restart Visual Studio 2008

Would it be possible to schedule a Live Meeting with you where you can show us the crash happening and how you are able to make it happen?

I really don’t know – this is something I will have to take up with the customer I’m working for at the moment.

Tool acts very slow with a large number of steps. We suspect and the crashing may be related.

Eighty steps should be well within what should be possible to handle for a “test” tool – apart from that I think you are right, since I usually experience slow performance before the tool crashes.

Regards

Espen Harlinn

[Konstantin Petkov]

Hi Espen,

Thanks for getting back to us providing more information.

I'm following this thread closely so please let me comment on a note you've posted.

Eighty steps should be well within what should be possible to handle for a “test” tool

This is of course correct. Based on my experience working on test automation however, I wouldn't recommend creating such large tests. It's all about maintenance that would make me avoid using such tests. Imagine that test prepared and running, then after some time a change in the application causing the test to fail at verification step '75' for example. That could be a nightmare to replicate and found what is going on.

WebUI Test Studio exposes various options/tools to maintain the tests easier like the property grid for each test step and the Visual StoryBoard. Still, I believe one should really consider splitting the tests into smaller parts as a best practice. Apart from maintenance that could help in tests reuse, although you may have probably already found the "test as step" option, let me point you to this video that worth's following.

I hope this helps. Please let us know a Live Meeting can be scheduled. Thank you!

Best wishes,
Konstantin Petkov
the Telerik team

Instantly find answers to your questions on the new Telerik Support Portal.
Watch a video on how to optimize your support resource searches and check out more tips on the blogs.

[Espen Harlinn]

This is of course correct. Based on my experience working on test automation however, I wouldn't recommend creating such large tests. It's all about maintenance that would make me avoid using such tests. Imagine that test prepared and running, then after some time a change in the application causing the test to fail at verification step '75' for example. That could be a nightmare to replicate and found what is going on.

If I had any say it is more than a bit likely that WebUI Test Studio wouldn’t be my initial choice. I could possibly have chosen the WebAii Testing Framework, but it’s likely I would have chosen the Selenium toolchain (see: http://seleniumhq.org/) - not because I think its superior - but having the source code for this kind of tool is often quite useful. If there is a limitation in the tool you can "fix" it and move on with your work.

For the simplest test I would have chosen “recording” tools – for the rest: I would have written them in c#, java, ruby, python, or whatever programming language the tool supported.

You absolutely have a point when you mention the maintenance problems regarding WebUI Test Studio tests – and while some of them could be worked around using Add Test as Step, the lack of “gouping” and “iteration” functionality seems a severe limitation.

The editing functionality of WebUI Test Studio isn't much to write home about either.

Initially I estimated about 500 hours for the test project I’m working on – and I got about 100 hours and a tool, that I had no part in choosing, which behaves erratically to complete the job. So I’m a bit miffed.

Taking the price tag of “WebUI Test Studio” into consideration I certainly feel that this tool should be more stable than what I’ve experienced. The customer has as far as I know decided that WebUI Test Studio is a product he is going to buy – I’m just the guy who is stuck with getting the thing to work…

you may have probably already found the "test as step" option

I most certainly would have organized the project differently had I known the tool beforehand. As for the video – I wish people would stop doing this, and write instead. I’m not an illiterate, and I find well written documentation a superior “learning source” compared to videos.

So basically I'm between a rock and a hard place.

Regards

Espen Harlinn

[Espen Harlinn]

Here is one stack trace I'm tired of:
ArtOfTest.WebAii.Design.Exceptions.ExecutionException: Unhandled exception thrown during automation step execution ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   ved System.Text.RegularExpressions.Match..ctor(Regex regex, Int32 capcount, String text, Int32 begpos, Int32 len, Int32 startpos)
   ved System.Text.RegularExpressions.RegexRunner.InitMatch()
   ved System.Text.RegularExpressions.RegexRunner.Scan(Regex regex, String text, Int32 textbeg, Int32 textend, Int32 textstart, Int32 prevlen, Boolean quick)
   ved System.Text.RegularExpressions.Regex.Run(Boolean quick, Int32 prevlen, String input, Int32 beginning, Int32 length, Int32 startat)
   ved System.Text.RegularExpressions.Regex.Match(String input, Int32 startat)
   ved ArtOfTest.WebAii.Core.Parser.GetNextTag(Int32 startPosition, Int32& tagLength, String& tagContent, String& tagName, String& tagRemainder)
   ved ArtOfTest.WebAii.Core.Parser.Parse()
   ved ArtOfTest.WebAii.Core.Parser.Parse(String text)
   ved ArtOfTest.WebAii.Core.TreeBuilder.BuildDomTree(String text, Boolean trimWhiteSpace)
   ved ArtOfTest.WebAii.Core.Browser.UpdateDomTree(String domString)
   ved ArtOfTest.WebAii.Core.Browser.RefreshDomTree()
   ved ArtOfTest.WebAii.Core.Browser.ExecuteCommand(BrowserCommand request, Boolean performDomRefresh, Boolean waitUntilReady)
   ved ArtOfTest.WebAii.Core.Browser.ExecuteCommand(BrowserCommand request)
   ved ArtOfTest.WebAii.Core.Actions.Click(Element targetElement)
   ved ArtOfTest.WebAii.Controls.HtmlControls.HtmlControl.Click()
   ved ArtOfTest.WebAii.Controls.HtmlControls.HtmlControl.Click(Boolean isClosing)
   ved ArtOfTest.WebAii.Design.IntrinsicTranslators.Descriptors.ClickActionDescriptor.Execute(Browser browser)
   ved ArtOfTest.WebAii.Design.Execution.ExecutionEngine.ExecuteStep()
   --- End of inner exception stack trace ---
   ved ArtOfTest.WebAii.Design.Execution.ExecutionEngine.ExecuteStep()

[Cody]

Hi Espen Harlinn,

Thank you very much for the stack trace. We are studying it and trying to figure out what could cause this error at that point in the tool.

We are wondering:

• What is the physical size of the .aii file that you were using when this crashed?
• What is the physical size of the matching .resx file for the .aii file?
• Do you have a code behind file included in this test?
• Can you tell us how much memory is the devenv.exe process using just before you launch a test like this one?
• Is the website you're testing in English or some other language (Norweigan perhaps?)? We're beginning to wonder if it's possible we're dealing with a globalization/localization issue. Same question for the OS you're using... is it the English version of Windows... and the .NET framework you have installed... is it English version or have you installed a language pack such as is offered here?

Regards,
Cody
the Telerik team

Instantly find answers to your questions on the new Telerik Support Portal.
Watch a video on how to optimize your support resource searches and check out more tips on the blogs.

[Espen Harlinn]

What is the physical size of the .aii file that you were using when this crashed?

Files and File sizes as reported by the dir command:

·         AddAssetsToOrder.aii 160 488

·         AddAssetsToOrder4-5-6.aii  148 089

·         AddAssetsToOrder7-8-9.aii    2 340

·         CreateAssetClass.aii  61 445

·         CreateAssetTF111.aii 101 298

·         CreateAssetTF112.aii 101 298

·         CreateAssetTF113.aii 101 298

·         CreateAssetTF114.aii 101 304

·         CreateAssetTF115.aii 101 304

·         CreateAssetTF116.aii 101 298

·         CreateAssetTF117.aii 101 298

·         CreateAssetType.aii 61 142

·         CreateCustomer.aii 48 608

·         CreateDynamicFieldsADC.aii 278 821

·         CreateDynamicFieldsADP.aii 278 823

·         CreateDynamicFieldsAHEA.aii 278 830

·         CreateEmployeeForCustomer.aii 29 007

·         CreateOrder1.aii 54 970

·         CreateOrder2.aii 54 944

·         CreateOrder3.aii 54 944

·         CreateOrder4.aii 54 944

·         CreateOrder5.aii 54 944

·         CreateOrder6.aii 54 944

·         CreateOrder7.aii 54 944

·         CreateOrder8.aii 54 944

·         CreateOrder9.aii 54 944

·         LogonTestUser.aii 16 132

·         ValidateAndEditAssetClass.aii 19 374

·         ValidateAndEditAssetType.aii 190 916

·         ValidateAndEditCustomer.aii 74 284

·         ValidateAssets.aii 10 413

·         ValidateAssetTF111.aii 111 703

·         ValidateAssetTF112.aii 111 703

·         ValidateAssetTF113.aii 111 703

·         ValidateAssetTF114.aii 111 709

·         ValidateAssetTF115.aii 111 709

·         ValidateAssetTF116.aii 111 703

·         ValidateAssetTF117.aii 111 703

·         ValidateDynamicFieldsADC.aii 138 513

·         ValidateDynamicFieldsADP.aii 138 515

·         ValidateDynamicFieldsAHEA.aii 138 530

The stack trace happened during the execution of a step in AddAssetsToOrder4-5-6.aii – this isn’t consistent though.

What is the physical size of the matching .resx file for the .aii file?

The corresponding resx file has the size: 7 996KB

Do you have a code behind file included in this test?

No

Can you tell us how much memory is the devenv.exe process using just before you launch a test like this one?

Devenv.exe uses around 420 MB when I’m working with WebUI Test Studio, at this point I've done no design work and executed no tests - just opened the test project. I don't think resx was the best choice for storing images - why didn't you go for straight png files instead? Thumbnails? There really isn't a good reason why everything should be loaded into memory.

On another computer I've a far more complex Visual Studio 2008 installation, but no WebUI Test Studio - and when I open a solution containing the Winforms user interface for a database containing about 50 tables and nearly 700 columns, a web service providing CRUD for those tables, opens the designer on 10 of the more complex forms, and start a debug session - Visual Studio 2008 uses 415 MB of memory.

Is the website you're testing in English or some other language (Norweigan perhaps?)? We're beginning to wonder if it's possible we're dealing with a globalization/localization issue. Same question for the OS you're using... is it the English version of Windows... and the .NET framework you have installed... is it English version or have you installed a language pack such as is offered here?

I’m running a Norwegian version of Windows XP and the OS is up to date. Normally I would have installed an English version of the OS, but in this case the computer belongs to the customer, who bought it preconfigured with the Norwegian version of the OS.

.Net was installed from the English redist, and there is no “local” version of Visual Studio 2008.

The web site is primarily in English, but some Norwegian words are used here and there. Surprisingly enough none of them contain any of the special Scandinavian characters, but exactly how this should cause your software to fail seems rather strange.

DevEnv and IE uses COM, COM uses BSTR, BSTR is Unicode, and the odd codepoints in Unicode isn’t something we usually have to think about in our part of the world.

I guess WebUI Test Studio is built using .Net – so strings are Unicode. Since I’m not privy to the code, I’m unable to tell you anything more about your code, but I would be more than a bit surprised if this turns out to have anything to do with localization/internationalization - you would almost have to have made an conscious effort to achieve this..

Apart from the horrible memory consumption, my guess would be that somehow/somewhere a .Net exception is not caught before passing control from the .Net runtime to COM and DevEnv.exe.

My advice would be to use something like log4net and wrap each of your implementation of methods belonging to the DevEnv COM interfaces with logging functionality – that way you would have something to work with in case of support scenarios for future versions of WebUI Test Studio.

A test tool without logging… well – I’ll rest my case.

Regards

      Espen Harlinn

[Cody]

Hello Espen Harlinn,

Thank you very much for those details. We looked over the files and their sizes, while they are on the large size, we don't think those sizes are causing any problems.

Regarding localization issues, we had a problem in the past where the locale setting changed the numbering format which caused us to crash. In that instance a string is used to convey rectangle coordinates and we were setup to parse it using English rules but what we got was a non-English format over the COM interface. This is how different languages/locales can wreak havoc on programs not setup to handle them.

We used to have a problem with simulated typing of non-ASCII characters into text boxes. We fixed that by adding a true UNICODE function to handle them.

Another remote possibility is when we go to parse the HTML from the web page badly formatted HTML code or some odd ball character could conceivably cause our HTML parser to misbehave. We've gone to great lengths to make it bullet proof, but it's very difficult to make it 100% bullet proof. At this time we have no evidence to suggest it is the HTML parser that is the culprit.

Regarding our using a resx file, it's been our experience that using this single file works just as well as a bunch of separate .png's plus most of our customers, when it comes to source control,  very much appreciate dealing with just a few files per test instead of one (or more) files per test step.

I do have some encouraging news. Yesterday I created a 114 step test. It runs great, until I abort the test while running it via Quick Execute and try to edit the elements. Then, just the one time (haven't had time to repeat the experiment), I saw memory consumption go through the roof. I'll be spending the day investigating this scenario.

Kind regards,
Cody
the Telerik team

Instantly find answers to your questions on the new Telerik Support Portal.
Watch a video on how to optimize your support resource searches and check out more tips on the blogs.

[Cody]

Hello Espen Harlinn,

I have been successful at uncovering a rather nasty memory leak. It happens when:

1. You have a large test like you have.
2. You edit the properties of an element in Element Explorer that happens to be used/shared by multiple tests.
3. Or instead of 2, you edit the properties of a web page in Element Explorer that happens to have elements underneath it that are used/shared by multiple tests.

We have dedicated one engineer to work on this problem specifically. Thank you for the report and your patience. I'll update you as we make progress.

Kind regards,
Cody
the Telerik team

Instantly find answers to your questions on the new Telerik Support Portal.
Watch a video on how to optimize your support resource searches and check out more tips on the blogs.

[Espen Harlinn]

Thank you for your efforts so far.

At this point my trial of WebUI Test Studio is not longer valid, and while we find that WebUI Test Studio is an interesting and promising product, we have been unable to reach a decision about buying the product or not.

Since WebUI Test Studio comes with a rather hefty price tag, we feel that we should be able to experience a somewhat more problem free period of use. So far our experiences have been both painful and costly.

If WebUI Test Studio could live up to its promise we wouldn’t hesitate to buy the product.

As it stands now we wonder if we could reach some sort of accommodation with you that enabled us to test the product until we have a version of the product that doesn’t force us to spend considerable more time “debugging” WebUI Test Studio and it’s idiosyncrasies than actually creating tests. 

At that time we would like to buy licenses for the product.

Regards

Espen Harlinn

[Konstantin Petkov]

Hi Espen,

We decided to extend your trial so that you can continue with your evaluation. Please let us know if you have more feedback.

Best wishes,
Konstantin Petkov
the Telerik team

Instantly find answers to your questions on the new Telerik Support Portal.
Watch a video on how to optimize your support resource searches and check out more tips on the blogs.

[Konstantin Petkov]

Hi Espen,

We have uploaded a build of WebUI Test Studio v1.1 fixing the memory leak Cody talked about before in this thread. You can find it uploaded here (this is a temporary location).

Please give it a go and let us know how your tests perform. Based on your feedback we are about to publish this build as unofficial update part of the Latest Internal Build group of the product Q2 SP2 release.

The release notes follow:
•    Memory use optimizations.
•    Fixed case where user was always notified that a code behind file is missing after properly deleting it from the project.
•    Updated test template to reference the correct assembly version number.

Best wishes,
Konstantin Petkov
the Telerik team

Instantly find answers to your questions on the new Telerik Support Portal.
Watch a video on how to optimize your support resource searches and check out more tips on the blogs.

[Konstantin Petkov]

Hello,

This is just to update this thread we uploaded the latest internal build of WebUI Test Studio v1.1 today fixing the memory leak discussed here. The build is accessible for everyone who has downloaded the latest official release corresponding to the 2009 Q2 SP2. Here is a direct link to the internal build:

WebUI Test Studio v2009.2.1013

Best wishes,
Konstantin Petkov
the Telerik team

Instantly find answers to your questions on the new Telerik Support Portal.
Watch a video on how to optimize your support resource searches and check out more tips on the blogs.