Hi,
we are facing a problem with the ArtOfTest.Runner, after completing the tests the program never quits. The longest duration of the test we ran was two days, which we had to cancel. The error occurs very often. We are using the ArtOfTest.Runner from VSTS, but the failures also occure on the machine itself as seen in the console. The machine with the Test Studio is a VM in Azure that connects on startup via VPN to the right webserver.
The call is "C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe" list="C:\Users\XXX\Documents\XXX.TelerikTestStudio\TestLists\XYZ.aiilist" persistOnEachStep=true junit settings="C:\Users\XXX\Documents\XXX.TelerikTestStudio\Environment\test.config" also with staging.config which is basically same, but has another URL.
Nothing shows up in the logs.
Note: We are also facing a problem with the jUnit parameter xyz.junit.xml not beeing in right format. I opened another thread for this.
Thanks for helping in advance,
Greg
7 Answers, 1 is accepted
Thanks for reporting that.
It is really strange that you observe that misbehavior and it will be also quite difficult to be investigated due to the setup in use. Though I would like to share with you we have implemented in our first release this year R1 2018 (v.2018.1.130) a test list setting which addresses such misbehavior.
This is called RunnerResponseTimeout and if enabled will kill the runner if this hangs for some reason and will store the generated results. The only thing to watch out for this setting is to set such amount of timeout which will exceed the test list duration as otherwise the runner will be killed before the test list has finished.
Could you please give it a try and let me know if the issue could be handled with its help?
Thanks in advance for your cooperation!
Regards,
Elena Tsvetkova
Progress Telerik
Rank 1
Hi Elena,
i think its not necessary to configure such a complex environment as we have, because the failure is clearly replicable locally within the console apllication. If it may help, the last log entry is:
[06/19 15:38:27,ArtOfTest.WebAii.HttpProxy.exe(4188:1),Execution] Program.Main() : Received Stop command from control pipe.
[06/19 15:38:27,ArtOfTest.WebAii.HttpProxy.exe(4188:1),Execution] Program.Main() : Terminating proxy.
then nothing happens and we have to cancel the ArtOfTest.Runner from within the console. I have tried it with proxy-settings on and off -> same behavior.
We also tried before to add the RunnerResponseTimeout, but the ArtOfTest.Runner could not load the *.config file when we added this parameter and used the default settings from the Test Studio (without a warning in console, it says it only in the trace log). Can you provide the right way how to add this to our configs?
Thank you very much for helping,
Greg
Rank 1
Hi again,
i really think it could be a settings-parameter problem from ArtOfTest.Runner, we use the default config from your website https://docs.telerik.com/teststudio/features/test-runners/artoftest-runner with different values for timeouts and the baseUrl, have added "recycleBrowser" and without the wpf-settings:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<configSections>
<section type="ArtOfTest.WebAii.Core.SettingsConfigSectionHandler,ArtOfTest.WebAii" name="WebAii.Settings"/>
<section type="ArtOfTest.WebAii.Core.WebSettingsConfigSectionHandler,ArtOfTest.WebAii" name="WebAii.Settings.Web"/>
</configSections>
<WebAii.Settings elementWaitTimeout="500000" xMultiMgr="false" unexpectedDialogAction="HandleAndFailTest" createLogFile="true" waitCheckInterval="500000" logAnnotations="false" annotationMode="All" annotateExecution="true" executionDelay="0" queryEventLogErrorsOnExit="false" clientReadyTimeout="20000" executionTimeout="500000" logLocation="C:\Log\"></WebAii.Settings>
<WebAii.Settings.Web recycleBrowser="true" verboseHttpProxy="false" enableSilverlight="true" useHttpProxy="false" enableUILessRequestViewing="false" aspNetDevServerPort="-1" webAppPhysicalPath="" localWebServer="None" silverlightApplicationPath="" silverlightConnectTimeout="500000" enableScriptLogging="false" defaultBrowser="Chrome" killBrowserProcessOnClose="false" baseUrl="http://abc.xyz.io/"> </WebAii.Settings.Web>
</configuration>
It would be nice if you could take a look at the website https://docs.telerik.com/teststudio/features/test-runners/artoftest-runner how the config looks there, it seems to me the default config there with all parameters is not correct / completed anymore.
We also have tried to only use the baseUrl-parameter in different ways, but errors occured in the trace log -> could not load config.
Thanks,
Greg
Rank 1
first we created a new config file. we also figured out which parameters are not "supported". From the AiiList we copied every existing setting parameter and made it camelCase. Then we started it few times till no bug are showing up in the trace log. The ArtOfTest.Runner cannot parse the following settings:
executeInDevelopmentTests="false"
executeCommandTimeout="20000"
runnerResponseTimeout="0.0"
annotationMode="0"
isProfilingExecution="false"
executingBrowsers=""
useMultiBrowserExecution="false"
enableUiLessRequestViewing="false"
autoCalibrateBrowsers="false"
As we would like to use some of these settings, it would be great if you can help us implement them.
FYI: When we passed only one parameter it seemed that the ArtOfTest.Runner used the very default values and not as expected those from the testlist. It would be great if we could have an implementation to change this behavior.
Our new config-file looks like this, but at least one parameter - probably in the core-settings - is blocking the ArtOfTest.Runner from finishing:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<configSections>
<section type="ArtOfTest.WebAii.Core.SettingsConfigSectionHandler,ArtOfTest.WebAii" name="WebAii.Settings"/>
<section type="ArtOfTest.WebAii.Core.WebSettingsConfigSectionHandler,ArtOfTest.WebAii" name="WebAii.Settings.Web"/>
<section type="ArtOfTest.WebAii.Core.WpfSettingsConfigSectionHandler,ArtOfTest.WebAii" name="WebAii.Settings.Wpf"/>
</configSections>
<WebAii.Settings createLogFile="true" logLocation="C:\\WebAiiLog\\" queryEventLogErrorsOnExit="false" logAnnotations="true" simulatedMouseMoveSpeed="0.3" waitCheckInterval="500" elementWaitTimeout="15000" executionDelay="0" unexpectedDialogAction="HandleAndFailTest" annotateExecution="true" xMultiMgr="true" clientReadyTimeout="60000" disableDialogMonitoring="false"></WebAii.Settings>
<WebAii.Settings.Web recycleBrowser="true" aspNetDevServerPort="-1" localWebServer="None" webAppPhysicalPath="" defaultBrowser="Chrome" enableScriptLogging="false" baseUrl="http://abc.xyz.io/" killBrowserProcessOnClose="false" useHttpProxy="false" enableSilverlight="false" verboseHttpProxy="false" silverlightConnectTimeout="60000" silverlightApplicationPath=""></WebAii.Settings.Web>
<WebAii.Settings.Wpf defaultApplicationPath="" />
</configuration>
Further important information: As we changed the config-file the example-test ran without problems, but in our normal tests the ArtOfTest.Runner very often doesen't stop. Is there maybe some "old" method that should not be used anymore?
In the trace log shows up:
...
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:1),Framework] BrowserProvisioner.EndService() : Provisioning pipe connected.
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:9)] First trace message from background thread "Pipe.ArtOfTest.WebAii.BrowserProvisioner" (managed ID = 9, native ID = 2500).
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:9),Framework] BrowserProvisioner.ServiceThread() : A new client has connected to the provisioning pipe but this will be treated as a termination cue.
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:1),Framework] BrowserProvisioner.EndService() : Provisioning thread terminated.
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:1),DialogPlayback] DialogMonitor.AddDialog() : Added {OnBeforeUnloadDialog(Dismiss=OK)} for monitoring.
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:1),Uncategorized] Log.WriteLineInternal() : Closing Window
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:1),Framework] Manager.RemoveBrowser() : Removed client ID "the guid" (0 remaining).
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:1),Framework] Manager.SetActiveBrowser() : Active browser is now null.
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:1),Warning] Manager.RemoveBrowser() : Attempted to remove a client ID "present guid" that was not in the list.
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:15)] First trace message from foreground thread "AnnotatorUIThread" (managed ID = 15, native ID = 8820).
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:15),Uncategorized] AdornerWindow.ShutDown() : AdornerWnd::ShutDown::Start
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:15),Uncategorized] AdornerWindow.ShutDown() : AdornerWnd::LocationListener::Disposed
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:15),Uncategorized] AdornerWindow.ShutDown() : AdornerWnd::DetachWrappedWindow::Start
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:15),Uncategorized] AdornerWindow.ShutDown() : AdornerWnd::DetachWrappedWindow::End
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:15),Uncategorized] AdornerWindow.ShutDown() : AdornerWnd::DispatcherBeginInvokeShutdown::Start
[06/20 13:58:47,ArtOfTest.Runner.exe(3580:15),Uncategorized] AdornerWindow.ShutDown() : AdornerWnd::Close::Start
[06/20 13:58:48,ArtOfTest.Runner.exe(3580:1),DialogPlayback] DialogMonitor.Stop() : Dialog monitoring is being halted.
[06/20 13:58:48,ArtOfTest.Runner.exe(3580:1),UIAutomation] AutomationExtensions.AddAutomationEventHandler() : Added a(n) WindowPatternIdentifiers.WindowOpenedProperty handler on element System.Windows.Automation.AutomationElement scope Descendants.
As a workaround we deleted the core-settings attributes, but we are not sure which config the ArtOfTest.Runner uses in this case. It would be nice if we can set them in the config-file or it is ensured that it uses the testlist settings. So one of the core-settings is as said before the blocker.
We hope this helps if you have further questions let us know,
Greg
Thank you for the additional notes.
My initial suggestion for RunnerResponseTIme setting will actually not be applicable for the run you perform - it is available for scheduled jobs only. Please excuse me for the misunderstanding.
Though I tested on my end with the provided Settings file and I would like you to remove annotateExecution="true" argument from the settings file and then try the execution again.
Next question will be if you need all of the settings you have listed in the config file and mainly the recycleBrowser and killBrowserProcessOnClose. If you have any doubts what these are meant for you could find details in the test list settings article.
Please let me know if I could assist further and thanks for your cooperation!
Regards,
Elena Tsvetkova
Progress Telerik
Rank 1
Hi Elena,
removing the annotateExecution="true" argument from the settings file did the trick: Now the ArtOfTest.Runner does his job correctly with our defined arguments and finishes.
We have set the killBrowserProcessOnClose to false, we don't need it necessarily and recycleBrowser is set to true, which is a nice feature for our tests but also not absolutely needed. Do you suggest to remove them, so the application runs more smoothely and stable?
So far thanks for helping and we are really intressted in improvements to our test-setup,
Greg
Thank you for the feedback!
I am glad to know that removing the annotations helped on your end as well. The rest of the settings could be beneficial depending on the scenarios covered by the automation tests you have. And if these are useful for you I wouldn't advise to remove these. You could try few executions with these settings enabled or disabled in the possible pairs to combine them and double check which is more convenient for you. Though, as you said recycleBrowser is beneficial it is better to leave it enabled.
Usually each setup is quite specific as it involves different environments and application under test so each solution is specific as well. I do understand this is not a straight forward answer but Test Studio allows you to customize the automation as per different requirements.
I hope the test execution will run smoothly as of now. Though in case of any further queries please do not hesitate to contact me back. Thanks!
Regards,
Elena Tsvetkova
Progress Telerik