Rank 1
I’m getting “Wait for condition has timed out” exceptions when calling LaunchNewBrowser. The strange thing is that this problem doesn't happen all the time (except in one case, see below).
Stacktrace:
at ArtOfTest.Common.WaitSync.CheckResult(WaitSync wait, String extraExceptionInfo)
at ArtOfTest.Common.WaitSync.For[T](Predicate`1 predicate, T target, Boolean invertCondition, Int32 timeout)
at ArtOfTest.WebAii.Core.Manager.WaitForBrowserToConnect(Int32 browserIndexToWaitFor)
at ArtOfTest.WebAii.Core.Manager.LaunchNewBrowser(BrowserType browserToLaunch, Boolean waitForBrowserToConnect, ProcessWindowStyle windowStyle, String arguments)
at ArtOfTest.WebAii.Core.Manager.LaunchNewBrowser(BrowserType browserToLaunch, Boolean waitForBrowserToConnect)
Things to note about our setup:
1. We are using Telerik Testing Framework 2011.1 (2011.1.609). However, this problem also happened when we were running version 2010.1.
2. On most of our servers (a mix of Server 2003, 2008, 32 bit, 64 bit, IE8, IE9) we’re getting this message regularly, but not all the time. Roughly, 80% of the time there is no problem, 20% is showing this exception (or similar problems). Usually, it helps to just retry calling .LaunchNewBrowser, but not always.
3. When this happens, a new browser is started and visible without any delays, but Telerik Testing Framework is not able to connect to it (see below).
4. As of today, we have one new Server 2003 which always displays this behavior. This server has never had an earlier version of Telerik Testing Framework, but when I install the version we used previously (2010.1), we’re seeing normal behavior (80% OK, 20% error).
5. I’m including a snippet of our application’s trace log.
Framework: InternetExplorerActions.LaunchNewBrowserInstance() : Process launched (ID=5460, Path="\Device\HarddiskVolume1\Program Files\Internet Explorer\IEXPLORE.EXE", Arguments="-nomerge about:blank").
Framework: InternetExplorerActions.LaunchNewBrowserInstance() : Attempting to attach on IE frame (HWND=197334)...
Framework: InternetExplorerActions.LaunchNewBrowserInstance() : Process "URLTester" (ID=2728) is assumed to host the controller.
Uncategorized: Connector.InjectCode() : ------- Function Start 'Connector.InjectCode' -------
Uncategorized: Connector.InjectCode() : Connection string: C:\WINDOWS\assembly\GAC_MSIL\ArtOfTest.InternetExplorer\2011.1.609.0__5339893a7cefe4d6\ArtOfTest.InternetExplorer.dll?ArtOfTest.InternetExplorer.ArtOfTestPluginEntryPoint?2728
Framework: InternetExplorerActions.WaitForDocument() : Attempting to get IHTMLDocument2 from Internet Explorer HWND 197334 on another thread...
Uncategorized: <>c__DisplayClasse.<WaitForDocument>b__7() : Attempting to get document from window handle
Framework: InternetExplorerActions.WaitForDocument() : IHTMLDocument2 successfully retrieved.
Framework: Connector.InjectCode() : Sending WM_COPYDATA to HWND 197334 (lpData = "C:\WINDOWS\assembly\GAC_MSIL\ArtOfTest.InternetExplorer\2011.1.609.0__5339893a7cefe4d6\ArtOfTest.InternetExplorer.dll?ArtOfTest.InternetExplorer.ArtOfTestPluginEntryPoint?2728")...
Framework: Connector.InjectCode() : Returned from sending WM_COPYDATA (LRESULT=0, GetLastError()=DS_S_SUCCESS).
Uncategorized: Connector.InjectCode() : ------- Function End 'Connector.InjectCode' -------
<The exception occurs after this point>
6. If I compare this with the log of regular (correct) cases, I'm missing this part:
Framework: BrowserProvisioner.ServiceThread() : A new client has connected to the provisioning pipe.
Framework: BrowserRemoted.AsyncListenerThreadEntry() : Not connected; this will now wait indefinitely for a pipe client connection...
Framework: BrowserRemoted.AsyncListenerThreadEntry() : A new client has connected to the pipe.
Framework: BrowserRemoted.AsyncListenerThreadEntry() : Now waiting indefinitely for a command from the pipe client...
Framework: Manager.AddNewBrowser() : Added client ID "Client_81b0ef79-6ece-4f38-97e1-1332561eab77" (now there is/are 1 total).
Framework: Connector.InjectCode() : Returned from sending WM_COPYDATA (LRESULT=0, GetLastError()=DS_S_SUCCESS).
Uncategorized: Connector.InjectCode() : ------- Function End 'Connector.InjectCode' -------
7. Therefore, either the client is not connecting through the pipe, or BrowserProvisioner.ServiceThread() isn't running. I looked at the internal ArtOfTest.WebAii code using JustDecompile, in order to try and understand what is happening, but I haven't been able to figure out which code is calling BrowserProvisioner.ServiceThread.
8. I have tried disabling the Named Pipes restriction in Local Security Policies, but this doesn't seem to make any difference, and it wouldn't explain why the problem occurs *some* of the time.
9. Obviously, our servers and browsers are configured as per Telerik's documentation.
Apologies for this long message. We would really like to get some more control over this problem, as it would greatly improve the stability of our software. Please let us know if you need more information. Thanks very much in advance.
Kind regards,
JP
23 Answers, 1 is accepted
Rank 1
You are the first person to report this type of intermittent connect problem. None of our other customers have an intermittent connect problem (it either works 100% or doesn't connect at all). I am suspicious that an antivirus product may be interfering with our connect process. We do have one other customer that is running a McAfee Enterprise product and we have confirmed it is blocking our connection to the browser. We're still trying to get it resolved for that customer.
If you have an anti-virus installed, can you disable or uninstall it and see if the problem persists?
Cody
the Telerik team
Have you looked at the new Online User Guide for Telerik Test Studio?
Rank 1
Thanks for your insight, I had a look at your suggestion.
1. Our server that always shows this problem (100%) does not have any anti-virus software installed.
2. We have one server that runs fine for about 24 hours (no problems whatsoever) and then suddenly produces the problem 100% of the time. I can only fix this by rebooting the server; restarting my own app doesn't help.
This server did have a McAfee product (installed by our provider) but after disabling it, I don't seen any change in behavior.
Although it was good to verify this, I don't think that this is the cause of our particular problem. I'm guessing that it's a problem with the communication through named pipes. Is it possible for you to elaborate a bit on that? Is there something I can do to "reset" any named pipes related resources that may have been locked after a certain time? (since part of our problem is solved after resetting the server from item #2 above).
And, to take a slightly different approach, which code is responsible for calling BrowserProvisioner.ServiceThread()? I noticed from looking at Telerik's trace output that the output from that method is missing when our problem occurs. Is it possible that the browser does try to connect, but the BrowserProvisioner somehow doesn't get round to calling namedPipeServerStream.WaitForConnection()?
Thanks for any insight, I appreciate it.
Rank 1
1. This time I witnessed the moment where it goes from working correctly 100% of the time, to not working at all. Our software performs browser tests all the time (one test per minute on average), and the problem started from one moment to the next.
2. I used ProcessExplorer to look at named pipes being created.
2a. During 'problem circumstances', I can still see that a named pipe is being created (e.g. \Device\NamedPipe\Pipe.ArtOfTest.WebAii.BrowserProvisioner_-258246580).
2b. During 'working circumstances', I see two additional named pipes (.AsyncPipe and .CommandPipe). Those two are not created when the problem occurs. That's probably obvious, but I thought I'd mention it.
3. Previously you mentioned that 3rd party software, such as anti-virus software, might be interfering with the process of connecting to the browser. Is it possible that other software starts to do something similar at some point? I noticed that our provider installed a number of uptime- and performance monitoring tools. I'm not sure what those tools are doing exactly, apart from watching performance counters, but perhaps there is an overlap with anti-virus software. Any thoughts?
Thank you again for your insight.
In "C:\Program Files (x86)\Telerik\Test Studio 2011.1\Logs" on the failing machine should be a "Telerik Test Studio Trace Log.txt" file. Can you send us that for analysis please (place it into a .zip file first).
Kind regards,Cody
the Telerik team
Rank 1
We don't have Test Studio installed, just Telerik Testing Framework. As such, we only have a C:\Program Files\Telerik\Testing Framework 2011.1 folder, and it doesn't contain a trace log file.
I'll be happy to generate trace logs though, if I know what you need. I noticed the ArtOfTest.WebAii.Core.Settings.CreateLogFile property; we have this set to false. If I set it to true, will that generate the log file you need? I'll see what it does.
Regards,
JP
Sorry about that. I forgot for moment you were using just the framework.
There is a registry key to set instead. Add the registry keys as shown in the attached screenshot (use Wow6432 if it's a 64-bit machine). Set TraceLoceLogEnabled to 1. Set the path to whatever works for you.
Cody
the Telerik team
Rank 1
Got it, thanks. The trace file is being generated now. I'll leave it running until the problem pops up again (probably at some point during the night) and I'll send it to you tomorrow.
Regards,
JP
Rank 1
I've sent the trace file via e-mail to support@telerik.com. I hope it will be helpful for investigating the problem. Thank you.
Kind regards,
JP
Thanks for that! We have received it and are analyzing it (yes it is large). I'll get back to you very soon with our findings.
Kind regards,Cody
the Telerik team
Thanks again for supplying us the log file. We spent considerable time studying it, but we weren't able to identify what is causing this problem. Basically all it shows is that all of a sudden, with no explanation we're unable to connect to IE when expected.
We have a few questions about your test environment:
1) What are you using to run your tests, MSTest, NUnit, TFS, ArtOfTest.Runner.exe, etc.
2) Which version of IE?
3) Which version of Windows? 32-bit or 64-bit?
4) Just to confirm, you're using our Testing Framework version 2011.1.712?
Here's an experiment to try please - the next time it breaks, before rebooting the machine, open Task Manager. See if there's a left over IE process hanging around as shown in the attached screen shot. If so, kill it/them and try running a test again and see if it starts working. Hopefully this narrows down the problem.
Cody
the Telerik team
Rank 1
Thank you for your time and analyzing the log file. To answer your questions:
1) We're not using a test framework, but our own (32-bit) proprietary application.
2) The server in question has IE8.
3) Windows Server 2003 SP2 32-bit.
4) Actually, I was using Testing Framework version 2011.1.609. Yesterday I upgraded the server and our application to 2011.1.712, but the server has run into the same problem since.
The only IE process instances I can find hanging around after it breaks, are those created by the Testing Framework *after* the breaking point. When I kill them (I do this in my code after a while in fact), nothing changes. The server needs an actual reboot.
Obviously I wish there was a way to clean something up using code.
A few notes from my side:
- We are using settings.XMultiMgr = true because we need to run multiple tests simultaneously. I'm guessing that this connects to the browser in a particular way?
- Although we had similar problems with Testing Framework 2010.3.1213, it was never as bad as this. Apparently the differences between these versions are quite fundamental, for our situation anyway. I know that there have been quite a few versions since. I'm willing to try each of them in order to narrow down which version introduced this problem. Do you think this might be helpful?
- Previously you mentioned that 3rd party software, such as anti-virus software, might be interfering with the process of connecting to the browser. Is it possible that other software starts to do something similar at some point? I noticed that our provider installed a number of uptime- and performance monitoring tools. I'm not sure what those tools are doing exactly, apart from watching performance counters, but perhaps there is an overlap with anti-virus software. Any thoughts?
Thanks guys.
Thank you for the additional information. You are one of the very few people using XMultiMgr. It only slightly modifies how we connect to the browser. The difference is almost insignificant. We doubt it has any bearing on this problem. I am going to bet the problem got worse when we shipped our first 2011.1.xxx build. The way we connect to the browser changed radically between that version and the older 2010.3.1213. If you really feel like testing it, here's a link to that older version.
"Is it possible that other software starts to do something similar at some point?" While almost anything is possible at this point, I tend to doubt other software is interfering. The one documented case is McAfee Enterprise is blocking our Windows Global Hook API call that we started making in 2011.1.502. The symptom is immediate, i.e. we can't connect to the browser at all under any condition. We're still working on that particular issue.
After discussing it with my co-worker, we thing the best approach is if you can help us reproduce the problem. If we can repro it then we can study it in our source code debugging environment and determine the root cause. Since you are using your own own (32-bit) proprietary application to run tests, can you send me that application? I shouldn't need the tests you are running as I should be able to create my own and plug them into your home grown framework. If this is doable I'll email you instructions to access a private dropbox folder for sending the files to me privately and securely.
Cody
the Telerik team
Rank 1
I needed a couple of days to do some further testing. Thanks for your thoughts regarding XMultiMgr and interference of third party software.
Your comment about the radical changes in the first 2011.1.xxx build prompted me to do some version testing. I went ahead and tried every version you have available for download at the moment, which is why it took me a number of days to complete the test results.
I tried 2011.1.712, 2011.1.609, 2011.1.512, 2010.3.1421 and 2010.3.1213, but all of them are having this problem. Originally we were using 2010.1.518 which doesn't have the breaking problem.
This implies that it wasn't the radical change in 2011.1.512, but some version after 2010.1.518 that introduced this problem for us. I'm guessing that there were at least two other versions in 2010. If you still have those available, I'll be happy to test them, so we can narrow down as much as possible.
Hopefully this will point us to the change that is giving us trouble. Thanks for your insight.
Best regards,
Jean-Pascal
You were using 2010.1.518 and it does not exhibit this problem? That' is a pretty old version (over a year). Yes we have had many releases between that one and the most current. Here are links to them (and one before):
http://dl.dropbox.com/u/12886581/Telerik.WebAii.Framework.2010.1.412.msi
http://dl.dropbox.com/u/12886581/Telerik.WebAii.Framework.2010.2.830.msi
http://dl.dropbox.com/u/12886581/Telerik.WebAii.Framework.2010.3.1208.msi
http://dl.dropbox.com/u/12886581/Telerik.WebAii.Framework.2010.3.1213.msi
http://dl.dropbox.com/u/12886581/Telerik.WebAii.Framework.2010.3.1307.msi
http://dl.dropbox.com/u/12886581/Telerik.WebAii.Framework.2010.3.1418.msi
Cody
the Telerik team
Rank 1
I did several tests with different versions.
- 2010.2.830 and earlier versions work okay (occasional problems, just like 2010.1.518)
- 2010.3.1208 and later versions break after a while.
Apparently, the changes in 2010.3.1208 are somehow causing a problem in our setup. Hopefully this will help to narrow it down. Thanks very much for looking into it.
Best regards,
JP
Rank 1
I forgot to say that 2010.3.1208 logs the following info when it breaks
[11:00 AM] - Exception Details:
[11:00 AM] - ------------------
[11:00 AM] - System.Runtime.InteropServices.COMException (0x80004005): Error HRESULT E_FAIL has been returned from a call to a COM component. at ArtOfTest.WebAii.BrowserSpecialized.InternetExplorer.NativeMethods.ObjectFromLresult(UIntPtr lResult, Guid refiid, IntPtr wParam) at ArtOfTest.WebAii.BrowserSpecialized.InternetExplorer.InternetExplorerActions.GetDocumentFromHwnd(IntPtr hwnd)
at ArtOfTest.WebAii.BrowserSpecialized.InternetExplorer.InternetExplorerActions.<>c__DisplayClass8.<InjectCode>b__7(Object o)
[11:00 AM] - Attempting to get document from window handle
[11:00 AM] - Internet Explorer Window handle (139657874) has not initialized its document
Hope this helps.
Regards, JP
Thank you for the additional data points on this problem. We are reviewing it and discussing our next step. Unfortunately this is likely to slide into next week. I am out of the office Friday and Monday. I expect we'll get back to you on Tuesday next week with our course of action.
All the best,Cody
the Telerik team
Rank 1
Hope you are doing well. I'm just writing to inquire whether you have an update on this issue, or if you want me to do additional tests. Thank you.
Regards,
Jean-Pascal
I am sorry, but no real update. We've reviewed the changes between 2010.2.xxx and 2010.3.xxx and don't see anything suspicious. I'm going to try and replicate this on a Server 2003 VM. Can you assist with this? Are you using some sort of test execution framework in which your tests plug in? Can you share that with me?
Best wishes,Cody
the Telerik team
Check out Telerik Trainer, the state of the art learning tool for Telerik products.
Rank 1
Thanks.
For all versions of Windows please go to (or create) the Key:
HKEY_CURRENT_USER\Software\Wow6432Node\Telerik\Test Studio
Create a DWORD
TraceLogEnabled
and set it to 1.
I've attached a screenshot showing the correct settings that work on 32 bit Windows XP. Keep in mind use of these keys for creating the log file was added about May of 2011. If you're using a version older than that it will ignore these keys.
Cody
the Telerik team
Check out Telerik Trainer, the state of the art learning tool for Telerik products.
Rank 1