This is a migrated thread and some comments may be shown as answers.

"Out of Memory" errors when running full suite of UI tests in Test Manager environment

36 Answers 1308 Views
General Discussions
This is a migrated thread and some comments may be shown as answers.
This question is locked. New answers and comments are not allowed.
Shashi
Top achievements
Rank 1
Shashi asked on 13 Mar 2012, 01:22 AM

Hello,

I was wondering if you are aware of - or working on - any memory issues related to Test Studio.  
I am asking for the following reason:

 


We have a suite of 70 UI tests (written with WebUI 2010 and Test Studio 2011.1) that run every night against one of our QA environments.  We also run this suite on demand against other environments.  The tests run in a TFS 2010/Microsoft Test Manager 2010 environment - test runner is the Visual Studio Test Agent (QTAgent32.exe) which is controlled by the Visual Studio Test Controller service (QTController32.exe).

 

For the last month to month and a half, almost every run of the suite has been constantly failing - with the last 10-15 tests of the suite either failing or not being executed.  If you rerun the failed or not executed tests, they pass without any issues. 

 


Troubleshooting has revealed that the failures are due to Out of Memory exceptions being thrown by the Visual Studio Test Agent (QTAgent32.exe).  I know this by looking in Event Viewer on the test server.  Sometimes the QTAgent process terminates which is when the last few tests are not executed.

 

When I view the QTAgent32.exe process in Task Manager after the exception is thrown, the memory it is consuming is high - anywhere from 850 MB to 1.2 GB.  Restarting the controller and the agent sets the memory consumption back to its baseline value of about 15K.  However, doing this before the test run does not help - the next run of the suite almost always fails due to the same issues - so the memory is building up during the run.

 


I also observed the QTAgent32 process in Task Manager while a test is running. It looks like memory consumed by QTAgent32 rapidly increases when a test runs - but this memory is never released when the test terminates.  This is seen both on the remote server and with the test running locally (using Test View in Visual Studio).  Also, this is observed both with a simple Login test using our application or an even simpler test running against one of Microsoft’s sample Silverlight apps.

 


Please let us know if this issue has already been reported (I thought I saw an issue related to virtual memory in PITS - not sure if this is related).  If there have been fixes in this regard recently, please point me to the relevant release/build number (I am using a slightly older version of Test Studio - 2011.1.829.0).

 


Thanks,

Shashi 

36 Answers, 1 is accepted

Sort by
0
Shashi
Top achievements
Rank 1
answered on 14 Mar 2012, 06:51 PM

Can someone respond to this?

 

Thanks,

Shashi

0
Plamen
Telerik team
answered on 15 Mar 2012, 11:49 AM
Hi Shashi,

I'm sorry you are experiencing this problem. It may be worth your while to upgrade to our latest internal build 2011.2.1506. You can download it from your Telerik account here: Public URL.  It includes a LOT of bug fixes and optimizations(more than 500 bugs found and fixed) since our 2011.1.829 release. One of them may address this problem.

You can also try to close and reopen the browser between each test within the list by setting the RecycleBrowser in Test List Setting to False. This assumes that the start of test two is not dependent on the browser state at the end of test one.

Please let me know if the issue persist after upgrading!

Kind regards,
Plamen
the Telerik team
Quickly become an expert in Test Studio, check out our new training sessions!
Test Studio Trainings
0
Shashi
Top achievements
Rank 1
answered on 15 Mar 2012, 08:59 PM
Hi Plamen,

Thanks for your response.  To respond to your suggestions:

a) I do not reuse the browser between tests - so the browser does shut down at the end of the test and a new browser instance is created by the next test.

b) I upgraded to a trial version of the latest official release (2011.2.1413.0) and am still seeing the problem when running the tests in local.   

Here are the memory consumptions of QTAgent32.exe that I observed :

Login test for our Silverlight application:
Before run -  15 MB (not 15K as I stated in my original post - my mistake)
After run -  127 MB

I also ran the first two steps of the SimpleGoogleSearch sample test (other two steps were disabled as the third step failed).  I got the following numbers:

Before run - 15 MB
After run - 58 MB

In both cases:
- Test was executed in Test View in Visual Studio
- QTAgent32.exe was shut down in Task Manager before running the test (it automatically restarts).

I also dumped the memory of the QTAgent32.exe process (using the adplus.bat utility that comes with Microsoft's Debugging Tools for Windows) before and after the run and then loaded them for analysis into a .NET memory profiler (SciTech Software's .NET Memory Profiler 4 evaluation version).  Results seem to indicate that Test Studio objects are one of the major contributors to the increased memory consumption of QTAgent32.exe after the run - it appears that these objects are not released (and therefore are not garbage-collected) at the end of the test run.  NOTE:  QTAgent32.exe does not stop running after the test run and therefore we cannot rely on that to release the objects - the application that instantiates the objects (in this case Test Studio) needs to be responsible for releasing the objects to GC. 

Given this, I have the following question:

I am aware that I can implement event handlers CleanUp and/or OnAfterTestCompleted in every test.  Is there something I need to do (or can do) in one or both of these methods to force all Test Studio objects to be released/deleted?  Note that I have already tried forcing the GC to run in one of these methods (even though this is strongly discouraged by MS) but it didn't seem to have any effect.

I  look forward to your responses to the above.  Please let me know if you have any other suggestions.

Thanks,
Shashi   

0
Plamen
Telerik team
answered on 19 Mar 2012, 01:34 PM
Hello Shashi,

Thank you for the detailed description of the problem. I was able to reproduce it and find a simple solution. In Visual Studio, go to Tools-->Options... . Expand the Test Tools node and select Test Execution. Under Performance, uncheck the "Keep test execution engine running between test runs" checkbox. Here's a video demonstrating the process. This should resolve the problem you are experiencing.

Greetings,
Plamen
the Telerik team
Quickly become an expert in Test Studio, check out our new training sessions!
Test Studio Trainings
0
Shashi
Top achievements
Rank 1
answered on 19 Mar 2012, 02:55 PM
Plamen,

Thanks for your response.  I will try your suggestion out.  However, I have a question regarding your solution:

Assuming this solution works, do you know of a way to configure this in Microsoft Test Manager?  After I got your response, I searched Microsoft's forums - I found that this question has been asked multiple times but no answer has been forthcoming from MS.  As I described below, the Out of Memory issue occurs when the full suite runs in Test Manager - and for something to be considered a solution (or more precisely a workaround), it needs to work in that scenario.  I also looked in the test settings associated with our test plan and didn[t see any options that looked like it would accomplish the same thing.

Also, the above solution would only be a workaround for the problem - not a resolution of the root cause.  Were you able to reproduce my memory profiling results?  If so, is Telerik looking at the root cause of this issue i.e. Test Studio objects not being released to garbage collection?  Please do let me know if there is anything I can do in my code to force this - or if there is any other way I can help you find and fix the issue.

Thanks,
Shashi
0
Shashi
Top achievements
Rank 1
answered on 20 Mar 2012, 01:11 AM
Plamen,

I tried the setting and it does terminate the QTAgent32 after a test run. 

However, the option controls termination of QTAgent32 after a *test run* - not after a single test. This means that, if I run a suite of multiple tests (for example, select multiple tests in Test List and run) and the option is unchecked, QTAgent32 does not terminate until the last test finishes executing. 

Extending that to the original Test Manager scenario (assuming that there was a corresponding option there), the agent would not terminate until all 70 tests finished execution.  As I have mentioned earlier, the Out of Memory issue occurs even if I restart QTAgent32 and the QTController *just before* the test run. 

Taking the above facts together, it follows that the option is not a solution for the original issue - even if there was a corresponding setting in Test Manager.  But thanks for the suggestion anyway - it was worth a try.  Please let me know if you or your colleagues have other ideas.  Also, please let me know if you were able to repro the issues in memory profiling and the next steps if any.

This is a pretty serious issue for us - and so your help would be greatly appreciated.  Again, let me know if there is anything I can do to help in finding a solution.

To other Test Studio users who are reading this:  If any of you are running Test Studio tests with Visual Studio/TFS/Test Manager 2010 and have encountered and found a way to resolve this issue, it would be great if you could share what you did.

Thanks,
Shashi
0
Plamen
Telerik team
answered on 21 Mar 2012, 01:26 PM
Hello Shashi,

Yes, we were able to reproduce the issue. The QTAgent memory usage is increasing after each test execution and if you have a large amount of tests within the test plan you will run out of memory at some point. Currently we are not completely sure whether this is a bug in our tool or in the Microsoft Test Manager, so we need some more time testing the issue. We'll try to replicate the same scenario (without using Test Studio) using only Microsoft Coded UI tests and see what will happen. I'll get back to you with the results.   

Greetings,
Plamen
the Telerik team
Quickly become an expert in Test Studio, check out our new training sessions!
Test Studio Trainings
0
Shashi
Top achievements
Rank 1
answered on 21 Mar 2012, 08:49 PM
Plamen,

Thanks for your response.  I have logged Support issue 525489 for this issue and have uploaded a screen shot of the .NET memory profiler results for a run of a test that logs in to our application and then quits.  I look forward to your feedback - please let me know if you have further questions and/or need anything else from me.

Thanks,
Shashi 
0
Dino
Top achievements
Rank 1
answered on 15 May 2012, 12:36 AM
I am seeing the same behaviour, where when running multiple tests QTAgent is consuming more memory with each test, ultimately resulting in an Out of Memory exception.

If I disable the statement in our code which grabs a screenshot of the desktop...

testManager.Log.CaptureDesktop(logFile)

...then I do not encounter any memory problems - QTAgent memory usage remains low and constant. I am taking multiple screenshots per test method. Commenting out this statement is the only change I am making.

I am using Telerik.Testing.Framework.2011.2.1305

Any comments or help would be appreciated.

Dino
0
Cody
Telerik team
answered on 15 May 2012, 10:49 PM
Hello Dino,

We do cache the last 5 images (for no apparent good reason that I can identify). The memory consumption should stop growing rapidly after the first 5 captures. Can you watch for this?

We expect memory consumption to slowly grow as the test runs and the results for each step is recorded in memory. These results are not flushed to disk until the test has finished executing.

All the best,
Cody
the Telerik team
Quickly become an expert in Test Studio, check out our new training sessions!
Test Studio Trainings
0
Dino
Top achievements
Rank 1
answered on 17 May 2012, 06:25 PM
Hi Cody

Memory consumption grows beyond 5 captures. In one sample run I am taking 100 captures and I am observing a +/- 25 MB increase per CaptureDesktop call. I am disposing of all WebAii objects in my code at the end of each test. Forcing GC.Collect() in my code has no effect.
  • Can you confirm that within your code you are explicitly disposing of the Bitmap/Image object as soon as it has been written to disk? Looking at other peoples implementations of Bitmap objects, I see that the Bitmap class is a wrapper around code in gdiplus.dll (unmanaged code):

Unlike the majority of the objects in the .NET Framework, GDI handles are not automatically disposed when your application no longer holds a reference to them. This, as you may have already guessed, can lead to memory allocations that are never released. Over time (application up-time), this could lead to large consumption of memory usage and ultimately an "Out of Memory" exception.
- http://www.codeproject.com/Articles/165443/Disposing-GDI-Objects-in-C-NET

Thanks
Dino


0
Shashi
Top achievements
Rank 1
answered on 17 May 2012, 06:55 PM
Hi Cody,

I have been following your discussion with Dino with great interest. 

I have some questions regarding the following statement:

"We do cache the last 5 images (for no apparent good reason that I can identify). "

What do you capture images of?
When exactly is the image captured? (or what triggers the image capture)?
What factors would impact (increase) the size of the images? (Number of steps?  Number of tests?  Content of application page? Other?)
It seems to me you are saying these images are captured by default - is that correct?  If so, is there a way to turn image capture on/off and/or control the number of image captures? (for tests developed with the Visual Studio plugin).

Your help is greatly appreciated.

Thanks,
Shashi
0
Cody
Telerik team
answered on 18 May 2012, 10:51 PM
Hello Shashi,

I'm going to answer your questions a little out of sequence simply because it all will make better sense that way:

It seems to me you are saying these images are captured by default - is that correct?

No. Only when you explicitly add a Capture Browser/Capture Desktop command.

What do you capture images of?

The browser window or the entire desktop.

When exactly is the image captured?

Answered already above.

What factors would impact (increase) the size of the images?

The size of the browser window or the resolution of the monitor.

If so, is there a way to turn image capture on/off...

Probably obvious by now... simply don't add a Capture Browser/Window test step.

and/or control the number of image captures?

There's no way to control how many images we're going to cache internally. It's fixed at 5. But even then it's only the last 5 Capture Browser/Window test steps. So if you test does not use this, we won't cache anything.

All the best,
Cody
the Telerik team
Quickly become an expert in Test Studio, check out our new training sessions!
Test Studio Trainings
0
Dino
Top achievements
Rank 1
answered on 31 May 2012, 06:32 PM
Hello

Any update on my previous post? Can you confirm correct and active disposal of image objects (and not passively waiting for GC).

Dino
0
Cody
Telerik team
answered on 06 Jun 2012, 09:51 PM
Hi Dino,

I apologize for the delay getting back to you.

To verify our handling of browser image captures I created a test that takes 100 browser captures. As you can see in this video memory use stabilized once memory use reached about 150MB. I am unable to reproduce any excessive memory use problem. I attached the test I used to study this. You're free to run the test yourself.

Kind regards,
Cody
the Telerik team
Quickly become an expert in Test Studio, check out our new training sessions!
Test Studio Trainings
0
Cody
Telerik team
answered on 13 Jun 2012, 09:34 PM
Hello Dino,

Just following up with you. Did you try running the attached project? How did memory behave for you with that project?

If you can help point us to the specific steps needed to get into excessive memory use by Test Studio we'd be delighted to pursue it. So far we've not been able to come up with any.

All the best,
Cody
the Telerik team
Quickly become an expert in Test Studio, check out our new training sessions!
Test Studio Trainings
0
Max
Top achievements
Rank 1
answered on 25 Mar 2013, 02:18 PM
Did a solution/fix ever come out of this Shashi's initial issue?
0
Cody
Telerik team
answered on 25 Mar 2013, 07:46 PM
Hello Max,

By "Shashi's initial issue" are you referring to her memory consumption problem she was observing? No we never heard back from her so we never really got to the bottom of what she was experiencing.

Since then we did uncover some memory leak problems in the product. We have fixed some of them and are actively working on fixing more. Is there something specific you are observing or some other problem you are running into you want assistance with?

Regards,
Cody
the Telerik team
Quickly become an expert in Test Studio, check out our new training sessions!
Test Studio Trainings
0
Jem
Top achievements
Rank 1
answered on 04 Apr 2013, 07:38 AM
Further to my colleague Max's post I have been investigating this issue and can reproduce the problem with a minimal console application which starts a simple WPF window application and then closes it again. It does not exhibit this behavior simply opening and closing the application. It does not exhibit the behavior when finding and clicking a WPF button. However it does exhibit it when finding a 3rd party control (but taking no further action on that control). The 3rd party control we are using is  an Actipro docking control - but I think that this is immaterial.

The application does not use any QTAgent or run under mstest; it is simply a stand alone application which links to the art-of-test libraries.

It takes 16,000 + iterations to fail with an out of memory exception.

The version of ArtOfTest  we are using is:

// Type: ArtOfTest.WebAii.Silverlight.VisualFind
// Assembly: ArtOfTest.WebAii, Version=2012.2.1420.0, Culture=neutral, PublicKeyToken=4fd5f65be123776c
// Assembly location: C:\Program Files (x86)\Telerik\Test Studio\Bin\ArtOfTest.WebAii.dll

The output window (in Visual Studio 2012 running the application in Debug mode) contains the following:

The thread 'Pipe.ArtOfTest.WebAii.BrowserProvisioner' (0x3306c) has exited with code 0 (0x0).
The thread 'UAI Dispatcher Thread' (0x316c8) has exited with code 0 (0x0).
The thread '<No Name>' (0x332a8) has exited with code 0 (0x0).
The thread '<No Name>' (0x33394) has exited with code 0 (0x0).
The thread '<No Name>' (0x3c0c) has exited with code 0 (0x0).
The thread 'Remoted async command listener' (0x3305c) has exited with code 0 (0x0).
The thread 'Pipe.ArtOfTest.WebAii.BrowserProvisioner' (0x3283c) has exited with code 0 (0x0).
The thread 'UAI Dispatcher Thread' (0x332fc) has exited with code 0 (0x0).
The thread '<No Name>' (0x33238) has exited with code 0 (0x0).
A first chance exception of type 'System.OutOfMemoryException' occurred in System.dll

(call stack is minimal)

The failing code is:on the LaunchNewApplication line.

using ArtOfTest.WebAii.Core;
using ArtOfTest.WebAii.Wpf;
using TestOfArtTest;

namespace TestOfArtOfTest
{
    public static class AppWrap
    {
   
        public static WpfApplication Application { get; private set; }
        public static MainWindow MainWindow { get; private set; }
        public static Manager Manager { get; private set; }
        public const string MainWindowCaption = "~TestWindow";
  
        /// <summary>
        ///     Stops this instance.
        /// </summary>
        public static void Close()
        {
            if (Application != null &&
                Manager != null)
            {
                Application.Quit();
                Manager.Dispose();
                Application = null;
                Manager = null;
            }
        }

        /// <summary>
        ///     Starts the test runner.
        /// </summary>
        public static void Start(string exePath, string logPath)
        {
            StartUpManager(exePath, logPath);

            Application = Manager.LaunchNewApplication(Manager.Settings.Wpf.DefaultApplicationPath); // OOM after 16000+ cycles

            ConnectToMainWindow(MainWindowCaption.ToUpperInvariant());
        }

     
    

        private static void ConnectToMainWindow(string windowName)
        {
            var window = Application.WaitForWindow(windowName, TimePeriod.TwoMinutes);
            MainWindow = new MainWindow(window);
        }

        private static void StartUpManager(string ExePath, string logPath)
        {
            var settings = new Settings
            {
                ClientReadyTimeout = TimePeriod.OneMinute,
                ExecuteCommandTimeout = TimePeriod.ThirtySeconds,
                ExecutionDelay = TimePeriod.HalfASecond,
                UnexpectedDialogAction = UnexpectedDialogAction.HandleAndFailTest,
                LogLocation = logPath,
                DisableDialogMonitoring = true,
                Wpf = new Settings.WpfSettings
                {
                    DefaultApplicationPath = ExePath,
                },
                SimulatedMouseMoveSpeed = 0.5f
            };

            Manager = new Manager(settings);
            Manager.Start();
        }
    }

}





The console app is as follows:
namespace TestOfArtTest
{
    class Program
    {
        private const string testExePath = @"C:\Projects\TestOfArtApp\TestOfArtApp\bin\Debug\TestOfArtApp.exe";
        static void Main(string[] args)
        {
             
            Console.WriteLine("Press <Return> to start");
            Console.ReadLine();
            for (int i = 0; i < 1000000; i++)
            {
                Console.WriteLine("Iteration: " + (i+1));
                AppWrap.Start(testExePath,"");
                AppWrap.MainWindow.ButtonToClick.User.Click();
                AppWrap.MainWindow.Dock.PinWindow("TestToolWindow"); // <-- without this line all is OK
                AppWrap.Close();
                GC.Collect();
            }
            Console.WriteLine("Press <Return> to exit");
            Console.ReadLine();
     
        }
    }
}

using ArtOfTest.WebAii.Controls.Xaml.Wpf;
using ArtOfTest.WebAii.Wpf;
 
namespace TestOfArtTest
{
    public static class TimePeriod
    {
        public const int FiveMilliSeconds = 5;
        public const int HalfASecond = 500;
        public const int OneSecond = 1000;
        public const int FiveSeconds = 5000;
        public const int TenSeconds = 10000;
        public const int ThirtySeconds = 30000;
        public const int OneMinute = 60000;
        public const int TwoMinutes = 120000;
        public const int FiveMinutes = 300000;
        public const int TenMinutes = 600000;
    }
    public class MainWindow : Window
    {
             public WpfWindow WpfWindow { get; private set; }
             private DockSite dockingRoot;
 
             /// <summary>
        /// Initialises a new instance of the <see cref="MainWindow"/> class.
        /// </summary>
        /// <param name="wpfWindow">
        /// The wpf window.
        /// </param>
        public MainWindow(WpfWindow wpfWindow)
        {
            this.WpfWindow = wpfWindow;
            wpfWindow.Find.WaitOnElementsTimeout = TimePeriod.OneMinute;
            this.Find.WaitOnElementsTimeout = TimePeriod.OneMinute;
            this.Wait.Timeout = TimePeriod.OneMinute;
        }
 
        private Button ClickButton;
 
        public Button ButtonToClick
        {
            get { return this.ClickButton ?? (this.ClickButton = this.WpfWindow.Find.ByType<Button>()); }
 
        }
        public DockSite Dock
        {
            get { return this.dockingRoot ?? (this.dockingRoot = this.WpfWindow.Find.ByType<DockSite>()); }
        }
   
 
 
    }
}

The essence of the line which causes failure is:

this.WpfWindow.Find.ByType<
DockSite>

which is looking for the Actipro control.
(it takes no further action).

The working set of the test application was at 1.5 GB at the time of the exception.

I could send the full solution if required - but it would probably require an Actipro license. I think you should be able to reproduce it with any user control, but I don't have enough time left in my time-box.

0
Plamen
Telerik team
answered on 04 Apr 2013, 03:57 PM
Hello Jem,

Thanks for providing the code snippets.

Looking your code I'm not sure I understand your usecase completely. It looks like you're using our Testing Framework in a WPF application and you are also using the framework in a Console application to test your WPF application. If this is correct, it is a really complicated scenario and Test Studio is not designed to work in that way. You can use our framework to create tests against WPF applications, but we don't recommend to use it in the WPF app itself. 

Regarding the line which causes the failure:
this.WpfWindow.Find.ByType<DockSite>();
This code is incorrect in the context of our framework. ByType<T>() is a generic method that can only be used with type arguments that implement IFrameworkElement.
public T ByType<T>() where T : IFrameworkElement
This leads me to believe that you probably created a control wrapper for the DockSite control in order to use it, but it seems you forgot to provide the code for that. I think this could also be the root cause of the issue. 

Also it will be very difficult for us to replicate the problem you're experiencing from the provided code snippets alone. In order to replicate the issue we need a sample WPF app and a copy of your test that we can run against that app. If these files are too big to attach to a support ticket, just let me know and I'll share a private Dropbox folder for you to upload the files to. Once we have a local repro, we'll try to find the root cause and determine how Test Studio is involved in this and fix it(in the event there's a bug).     

Kind regards,
Plamen
the Telerik team
Quickly become an expert in Test Studio, check out our new training sessions!
Test Studio Trainings
0
Jem
Top achievements
Rank 1
answered on 05 Apr 2013, 07:00 AM
Hi Riamen,

I am not sure what you mean by a using the test framework in a wpf application. I am not (as far as I know). I am using it in a .NET 4.0 console application. This then 'tests' a wpf app - which is the correct use, - no?

I have attached the full project - I haven't had time to clean it up completely - some of the stuff in it is probably irrelevant.
As you will appreciate the product we are actually testing is extremely large and fairly complex and is related to formula 1 telemetry.
I have tried to cut out everything I could to get to a minimal exhibitor of the problem. You will appreciate that our real tests on our full application  result in out of memory after about 50 tests.

DockSite is indeed a wrapper class for the Actipro control (Actipro is a wpf docking framework and so comprises a set of FrameworkElements):
  public class DockSite : FrameworkElement
    {...}
     which we find in the application using:

   public DockSite Dock
        {
            get { return this.dockingRoot ?? (this.dockingRoot = this.WpfWindow.Find.ByType<DockSite>()); }
        }

this is defined by Actipro thus:
namespace ActiproSoftware.Windows.Controls.Docking
{
  [ContentProperty("Content")]
  public class DockSite : Control, IDockTarget
  {
....
}

A  'Control' is a FrameworkElement:

namespace System.Windows.Controls
{
  /// <summary>
  /// Represents the base class for user interface (UI) elements that use a <see cref="T:System.Windows.Controls.ControlTemplate"/> to define their appearance.
  /// </summary>
  public class Control : FrameworkElement
  {


I appreciate that we may indeed be using the test framework incorrectly - but as the documentation is a bit sparse we have to proceed by trial and error. Any pointers would be appreciated...

Even so, even if we are using the calls incorrectly - disconnecting the application, closing and disposing of our Manager should not cause any memory leakage should it?

Hope this helps,
Regards
Jem
 






0
Plamen
Telerik team
answered on 10 Apr 2013, 08:41 AM
Hello Jem,

Thank you for providing the project. I have installed a trial version of the Actipro controls for WPF and I was able to reproduce the memory leak. I filed a bug on this behavior and provided the project to our developers for further investigation. You can track its progress here: Public URL. I hope they'll be able to find the root cause and fix it soon.

I also updated your Telerik points accordingly.

Thanks again for your cooperation.

All the best,
Plamen
the Telerik team
Quickly become an expert in Test Studio, check out our new training sessions!
Test Studio Trainings
0
CCG
Top achievements
Rank 2
answered on 02 Sep 2013, 01:14 PM
Hi

My name is Jeremy and we are having the same problem with out of memory in a UI test. In one single TestMethod we are visiting more then 80 sites in our web based system using the BaseTest.ActiveBrowser.NavigateTo(pageUri) method. When the test is visiting the 75'th page we always get a "Out of memory" - exception regardless of what page is at the 75'th place in line to be navigated to.

Here are the exception we get: 
-------------

ArtOfTest.WebAii.Exceptions.ExecuteCommandException: ExecuteCommand failed!
InError set by the client. Client Error:
System.OutOfMemoryException: Could not get the outerHTML property. Not enough storage is available to complete this operation.

   at mshtml.HTMLHtmlElementClass.IHTMLElement_get_outerHTML()
   at ArtOfTest.InternetExplorer.IECommandProcessor.SetCurrentDocumentMarkup(BrowserCommand& response)
   at ArtOfTest.InternetExplorer.IECommandProcessor.ProcessInformationCommands(BrowserCommand request)
   at ArtOfTest.InternetExplorer.IECommandProcessor.ProcessCommand(WebBrowserClass ieInstance, BrowserCommand request, IHTMLDocument2 document)
BrowserCommand (Type:'Information',Info:'DocumentMarkup',Action:'NotSet',Target:'null',Data:'',ClientId:'Client_ce9ca102-a49f-4f6d-ac1f-7b4036d36eaf',HasFrames:'False',FramesInfo:'',TargetFrameIndex:'-1',InError:'True',Response:'System.OutOfMemoryException: Could not get the outerHTML property. Not enough storage is available to complete this operation.

   at mshtml.HTMLHtmlElementClass.IHTMLElement_get_outerHTML()
   at ArtOfTest.InternetExplorer.IECommandProcessor.SetCurrentDocumentMarkup(BrowserCommand& response)
   at ArtOfTest.InternetExplorer.IECommandProcessor.ProcessInformationCommands(BrowserCommand request)
   at ArtOfTest.InternetExplorer.IECommandProcessor.ProcessCommand(WebBrowserClass ieInstance, BrowserCommand request, IHTMLDocument2 document)')
InnerException: none.


   at ArtOfTest.WebAii.Core.Browser.ExecuteCommandInternal(BrowserCommand request)
   at ArtOfTest.WebAii.Core.Browser.ExecuteCommand(BrowserCommand request, Boolean performDomRefresh, Boolean waitUntilReady)
   at ArtOfTest.WebAii.Core.Browser.RefreshDomTree()
   at ArtOfTest.WebAii.Core.Browser.ExecuteCommand(BrowserCommand request, Boolean performDomRefresh, Boolean waitUntilReady)
   at ArtOfTest.WebAii.Core.Browser.ExecuteCommand(BrowserCommand request)
   at ArtOfTest.WebAii.Core.Browser.InternalNavigateTo(Uri uri, Boolean useDecodedUrl)
   at ArtOfTest.WebAii.Core.Browser.NavigateTo(Uri uri, Boolean useDecodedUrl)

-------------

If we split the test to multiple TestMethods each taking a chunk of 50 pages to navigate to it all works fine.

We are using Windows 7 64bit Enterprise and Windows Server 2008 R2 64bit Enterprise along with Internet Explorer 9 (version 9.0.8112.16421) all computers have lots of memory and diskspace.

Please get back to us so if there is a better way to handle this or if this is a known issue please post a link to the issue so we can track it.

Besides this, thanks for an awesome framework!

Best regards
Jeremy
0
Ivaylo
Telerik team
answered on 05 Sep 2013, 01:17 PM
Hello Jeremy,

I am sorry to hear you are experiencing this issue.
Are you experiencing this issue on all the machines in your environment?
Unfortunately I cannot provide you a link to a known issue since we will need first to reproduce this behavior and then log it as a known issue. If you can provide us with your test/project we will be glad to take a look and investigate further.
I am also happy you have been able to find a workaround for this problem by dividing your test to multiple test methods.

Regards,
Ivaylo
Telerik
Quickly become an expert in Test Studio, check out our new training sessions!
Test Studio Trainings
0
Jeff
Top achievements
Rank 1
answered on 11 Sep 2013, 08:23 PM
I'm having a similar issue running automated web tests through MSBuild.  The test agent continues to build up memory allocation for QTAgent32_40.exe until around the 75th test which then stops executing.  I'm using a mix of CodedUI and Test Studio scripts with this test run.  The error I see in MTM when the agent stops executing the tests is:

"The test adapter 'WebAiiTestAdapter' threw an exception while running test 'TCxxxx'. Exception of type 'System.OutOfMemoryException' was thrown."

QTAgent32_40.exe had around 1.4 gigs of memory allocated to it at this point. 

Video of agent and processes at the point of failure: http://screencast.com/t/KQJusAyvQ3ob

I just ran only the codedUI tests (around 65 of them) and the memory usage did not go past 400MB. Does anyone have any suggestions or a resolution that worked for them that I could try?  BTW - There are no screen capture steps in my scripts.

Thanks,

Jeff
0
Shashi
Top achievements
Rank 1
answered on 11 Sep 2013, 09:31 PM
Hi Jeff,

We have implemented the following measures to work around the issue:

a) Split the tests into multiple smaller suites - which are then scheduled one after the other in a full test run (each suite can also be run individually).
b) Kill the QTAgent process and then wait for it to automatically restart before scheduling the test run of each suite (this is done in a proprietary Windows workflow app that initiates, monitors and reports results of a test run).
c) Scheduled tasks to restart the test controller on the test servers regularly.

While these measures have certainly helped (a lot), they don't completely eliminate the problem.  Out of Memory errors still occur sporadically - but incidence is nowhere near as we were seeing when I first reported this issue.  There are also a few downsides to some of the measures.  It would certainly be ideal if Telerik looked further and resolved the memory issues (which, based on your report, apparently still exist).  It would be great to get an update on where things stand. 

Hope that helps,
Shashi    
0
Jeff
Top achievements
Rank 1
answered on 12 Sep 2013, 12:53 PM
Hi Shashi,

Thanks for the response. I will attempt to mirror what you have done to see if that helps any over the next few days (early next week). I will most certainly keep you posted. 

Have you upgraded your Test Studio and Execution server editions to the latest release? If so, did you notice a difference after doing so or are you still having the same problem? 
 
Thanks,

Jeff
0
Shashi
Top achievements
Rank 1
answered on 12 Sep 2013, 07:27 PM
Hi Jeff,

No, we are not on the latest release of Test Studio - but the last official 2012 release (2012.2.1420.0).  We use the VS plugin - not standalone. 

Also, we do not use the Execution server to run our tests.  Instead, we run the tests using Microsoft's tcm.exe (command line version of MTM) which works with Visual Studio 2010 Test Controller and Test Agent installed on the test servers.

Shashi
0
Ganga
Top achievements
Rank 1
answered on 16 Sep 2013, 04:56 AM

Hi,

I am also getting frequent Out Memory exception for my test. My environment settings are:

Visual Studio Ultimate 2012

Target Platform : .Net Framework 4.5

Telerik Testing Framework Product version: 131.8.6.0

O/S: Windows 7 Enterprise

Test Execution: Keep test execution engine running between test runs (unchecked)


Here are the exception I get: 

System.OutOfMemoryException: Could not get the outerHTML property. Not enough storage is available to complete this operation.

   at mshtml.HTMLHtmlElementClass.IHTMLElement_get_outerHTML()

   at ArtOfTest.InternetExplorer.IECommandProcessor.SetCurrentDocumentMarkup(BrowserCommand& response)

   at ArtOfTest.InternetExplorer.IECommandProcessor.ProcessInformationCommands(BrowserCommand request)

   at ArtOfTest.InternetExplorer.IECommandProcessor.ProcessCommand(WebBrowserClass ieInstance, BrowserCommand request, IHTMLDocument2 document)

BrowserCommand (Type:'Information',Info:'DocumentMarkup',Action:'NotSet',Target:'null',Data:'',ClientId:'Client_e0094032-05c4-4123-bb08-88f4fcc56722',HasFrames:'False',FramesInfo:'',TargetFrameIndex:'-1',InError:'True',Response:'System.OutOfMemoryException: Could not get the outerHTML property. Not enough storage is available to complete this operation.

   at mshtml.HTMLHtmlElementClass.IHTMLElement_get_outerHTML()

   at ArtOfTest.InternetExplorer.IECommandProcessor.SetCurrentDocumentMarkup(BrowserCommand& response)

   at ArtOfTest.InternetExplorer.IECommandProcessor.ProcessInformationCommands(BrowserCommand request)

   at ArtOfTest.InternetExplorer.IECommandProcessor.ProcessCommand(WebBrowserClass ieInstance, BrowserCommand request, IHTMLDocument2 document)')

InnerException: none.


thx
Ganga S
0
Jeff
Top achievements
Rank 1
answered on 16 Sep 2013, 04:59 PM
Hi Ganga,

Are you using MSBuild to run your tests?  I'd be interested in discussing your set up and compare notes with what we're trying to do on our end to share successes and roadblocks.  Telerik stated in a support ticket that their knowledge with MSBuild is quite limited so I'm tring to hear from other users on what they have been able to accomplish with Visual Stuido/TFS 2012.  Most of the documentation I have read out there has been for the 2010 version.

Thanks,

Jeff
0
Ganga
Top achievements
Rank 1
answered on 16 Sep 2013, 09:47 PM

Hi Jeff,

We have created small window application to run our tests.  I am running my tests on Debug mode all the time using Visual Studio Ultimate 2012.


I ran the test yesterday on Chrome web browser instead IE 9.0. All the tests were executed without “Out Memory issue”. I will run the test again on Chrome web browser and let you know the status.


PS: My test only take a screenshot (browser window view port) for failed scenario. The last run has lots failures (total number of screenshots has been taken = 63)


I hope this help!! I am happy to help, If you need any info.

Cheers,
Ganga S

0
Jeff
Top achievements
Rank 1
answered on 17 Sep 2013, 12:57 PM
Hi Ganga,

Thanks for the reply!  I would be interested in learning more about the window application you created to run your tests.  Are you using Microsoft Test Manager to store you test cases and metrics?  You can private message me since we may be getting off topic here at:  jpagano@softerware.com.

Please let me know if you are still getting the "Out Of Memory" issue with the chrome runs.  As of right now, I am unable to run any of my tests with the newest version of Test Studio with MSBuild.  I have opend a support ticket and awaiting their response.  The build actually completes fine, but no tests are executed on the test agent.  It was my hope that upgrading to the newest version of Test Studio would help with the "Out of Memory" issues, but I can verify that until my tests actually run. 

I will keep you posted as well as new information comes to me.

Thanks,

Jeff

0
Ganga
Top achievements
Rank 1
answered on 18 Sep 2013, 10:50 PM

Hi,

So far I observed followings:


Test Run on IE:
After Out of Memory exception, the IE web browser is not usable. I have to kill the browser in order to run the test again.


Test Start at : 9.56am

Out of Memory Exception at : 10.36am

Total Memory: 12 GB

Physical Memory usage before: 4.50 GB

Physical Memory usage during the exception: 6.00 GB


Test Run on Chrome & FireFox:
didn't get the Out of Memory exception and all the test were executed…

All the tests ran less than 30-35 minutes.

0
Ganga
Top achievements
Rank 1
answered on 09 Oct 2013, 10:37 PM

Any update for this issue?

Really appreciate your support….

Cheers,
Ganga S

0
Ivaylo
Telerik team
answered on 14 Oct 2013, 02:03 PM
Hello Ganga,

Please note that the out of memory exceptions don't refer to the physical memory at all. This is a Windows limitation, and it refers to the OS memory management rather than Test Studio. Please check this article for more information.
Furthermore I've checked the crash reports we receive in such cases and I couldn't find something related to your account for the last few months. 

Regards,
Ivaylo
Telerik
The New Release of Telerik Test Studio Is Here! Download, install,
and send us your feedback!
0
Ganga
Top achievements
Rank 1
answered on 14 Oct 2013, 09:13 PM

Hi Ivaylo,

Thanks for your response. 

I have posted my crash report on 15th Sep 2013.


http://www.telerik.com/automated-testing-tools/community/forums/test-studio-express/general-discussions/out-of-memory-errors-when-running-full-suite-of-ui-tests-in-test-manager-environment.aspx#2787848

I discovered additional information about the out of memory exception.


In our case, IE 9 process is not releasing memory. IE 9 process is crashing, if I navigate to many pages continuously for 30 minutes. (I found this article about IE 9 limitation)


In our case, it is not a Testing Framework issue.


I can able to replicate same issue, if run manually on IE 9.0. 



Thanks
Ganga S 

Tags
General Discussions
Asked by
Shashi
Top achievements
Rank 1
Answers by
Shashi
Top achievements
Rank 1
Plamen
Telerik team
Dino
Top achievements
Rank 1
Cody
Telerik team
Max
Top achievements
Rank 1
Jem
Top achievements
Rank 1
CCG
Top achievements
Rank 2
Ivaylo
Telerik team
Jeff
Top achievements
Rank 1
Ganga
Top achievements
Rank 1
Share this question
or