Issue running WPF UI tests from TFS 2017

[Logan]

Hello!

 

I have been running into trouble running tests from TFS 2017 builds for several weeks now. I will do my best to sum up the situation and point out important details.

 

I have a c# project containing my UI tests that use ArtOfTest.WebAii.dll. The general structure of the tests is: 

 

01.try
02.{
03.    log.Info("Starting ConnectAndDisposeTest");
04.
05.    var manager = new Manager(true);
06.    Process proc = Process.GetProcessesByName("WPF").FirstOrDefault();
07.    if (proc == null)
08.    {
09.         log.Info("Launching app");
10.         manager.LaunchNewApplication(@"C:\repo\WPF.exe");
11.         log.Info("Launched app");
12.    }
13.    else
14.    {
15.    log.Info("Connecting to app");
16.    manager.ConnectToApplication(proc);
17.    log.Info("Connected to app");
18.    }
19.
20.    log.Info("Disposing");
21.
22.     manager.Dispose();
23.
24.    log.Info("Disposed");
25.    await Task.Delay(1.Seconds());
26.    }
27.    catch (AppDomainUnloadedException ex)
28.    {
29.    log.Fatal(ex);
30.    }
31.    catch (Exception ex)
32.    {
33.log.Error(ex);
34.    }
35.}

 

When I run this test with  `vstest.console.exe .\WPFTest.dll /TestCaseFilter:FullyQualifiedName~ConnectAndDispose /InIsolation`, I see the following error in the console:

Warning: System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain. This can happen if the test(s) started a thread but did not stop it. Make sure that all the threads started by the test(s) are stopped before completion.

 

In addition to this, when TFS runs these tests as part of the `Run Functional Test` task (https://github.com/Microsoft/vsts-tasks/blob/master/Tasks/RunDistributedTests/README.md), the test runner intermittently stops responding, stops sending results to TFS, and causes the test runner for that machine to abort and fail those tests.

 

I've done nearly everything I can to try to identify what might be causing the hanging/unresponsive behavior, but I've reached the end of my expertise. What can I do to diagnose the problem and/or work around it to prevent the test runner from hanging? Is it possible that this `System.AppDomainUnloadedException` is causing this unresponsive behavior? 

 

Please let me know if you need additional details. Thank you!

[Elena]

Hi Logan,

Thank you for sharing details on your observations. 

I reviewed the shared information and also searched the web for the warning you get. It seems to be a common warning which is mostly related to the way the coded solution is structured and if all exceptions are correctly caught. I would like to kindly ask you to research for possible solutions about the warning in question since you are familiar with your whole project and any advice on my end might not be correct. 

There is a suggestion I would like to place - you could remove the code related to the Telerik Testing Framework (LaunchNewApplication, ConnectToApplication, Dispose) and double check if the warning appear again. This will exclude the relation of the issue to the Telerik Framework. 

Please let me know if I could assist you with anything else. Thanks!

Regards,
Elena Tsvetkova
Telerik by Progress

 

 

[Logan]

Hello Elena. Thanks for getting back to me!

Unfortunately, when I removed those (LaunchNewApplication, ConnectToApplication, Dispose) calls, the error no longer appeared, so it is related to the Telerik Framework.

 

My test class is inheriting from "BaseWpfTest", which gives me access to the "Initialize" function. 

I have run the test both with and without the "Initialize()" call, in both the ReSharper test runner as well as the Visual Studio test runner (vstest).

Here are my results:

-- NOT Calling Initialize() =>

[TestClass]

public class AppDomainTest : BaseWpfTest

{

    [TestMethod]

    public void ConnectAndDisposeTest()

    {

        //Initialize();

    }

}

 

Console output from vstest:

------ Discover test started ------
========== Discover test finished: 54 found (0:00:00.1559993) ==========
------ Run test started ------
========== Run test finished: 1 run (0:00:00.6764968) ==========

 

(No console output from ReSharper)

 

TRACE Output (from ReSharper):

2017.04.19 16:10:40.636    INFO BuildStage started

2017.04.19 16:10:40.636    INFO Build policy: Automatic

2017.04.19 16:10:43.645    INFO 1 projects built

2017.04.19 16:10:43.645 VERBOSE     WPF.Test.UI

2017.04.19 16:10:43.645    INFO BuildStage completed

2017.04.19 16:10:43.648    INFO RefreshProjectPropertiesStage started

2017.04.19 16:10:43.648    INFO RefreshProjectPropertiesStage completed

2017.04.19 16:10:43.648    INFO DiscoveryStage started

2017.04.19 16:10:43.649    INFO Test exploration required for 1 projects

2017.04.19 16:10:43.649 VERBOSE     WPF.Test.UI

2017.04.19 16:10:43.862    INFO DiscoveryStage completed

2017.04.19 16:10:43.862    INFO BuildPipelineStage started

2017.04.19 16:10:43.862    INFO BuildPipelineStage completed

2017.04.19 16:10:43.862    INFO ApplyRulesStage started

2017.04.19 16:10:43.863    INFO Got 1 elements (1 explicit) to run

2017.04.19 16:10:43.863    INFO Marking 1 elements as Pending

2017.04.19 16:10:43.864    INFO Marking elements as Pending completed

2017.04.19 16:10:43.864    INFO Distributing 1 elements into runs

2017.04.19 16:10:43.864    INFO Host Controller initialization started

2017.04.19 16:10:43.864 VERBOSE Host Controller: ProcessTaskRunnerHostController

2017.04.19 16:10:43.864    INFO Host Controller initialization completed

2017.04.19 16:10:43.865    INFO Distributing elements completed

2017.04.19 16:10:43.865    INFO Created 1 unit-test runs

2017.04.19 16:10:43.865    INFO ApplyRulesStage completed

2017.04.19 16:10:43.865    INFO RunTestsStage started

2017.04.19 16:10:43.865 VERBOSE Running up to 4 unit-test runs in parallel

2017.04.19 16:10:43.865    INFO Run: 4b7d9f25-d2be-44ab-9f77-26bb79f84324 - Started

2017.04.19 16:10:43.865 VERBOSE     Provider: MSTest

    Project: WPF.Test.UI

    Target Framework:

    Strategy: JetBrains.ReSharper.UnitTestProvider.MSTest11.MsTestInProcessStrategy

    Runtime Enviroment:

        Platform Type: x86

        Platform Version: v4_5

2017.04.19 16:10:44.138    INFO Run: 4b7d9f25-d2be-44ab-9f77-26bb79f84324 - Completed

2017.04.19 16:10:44.138    INFO RunTestsStage completed

2017.04.19 16:10:44.139    INFO Finishing run

2017.04.19 16:10:44.139    INFO Run is finished

 

-- WITH Calling Initialize() =>

[TestClass]
public class AppDomainTest : BaseWpfTest
{
    [TestMethod]
    public void ConnectAndDisposeTest()
    {
        Initialize();
    }
}

Console output from vstest:

Discover test started ------
========== Discover test finished: 54 found (0:00:00.1615246) ==========
------ Run test started ------
System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain. This can happen if the test(s) started a thread but did not stop it. Make sure that all the threads started by the test(s) are stopped before completion.
========== Run test finished: 1 run (0:00:01.1459267) ==========

Console output from ReSharper:
Debug Trace:
Framework: BrowserProvisioner.EndService() : Connecting to the provisioning pipe "Pipe.ArtOfTest.WebAii.BrowserProvisioner" in order to unblock and eventually terminate it...
Framework: BrowserProvisioner.EndService() : Provisioning pipe connected.
Framework: BrowserProvisioner.ServiceThread() : A new client has connected to the provisioning pipe but this will be treated as a termination cue.
Framework: BrowserProvisioner.EndService() : Provisioning thread terminated.
DialogPlayback: DialogMonitor.Stop() : Dialog monitoring is being halted.
UIAutomation: AutomationExtensions.AddAutomationEventHandler() : Added a(n) WindowPatternIdentifiers.WindowOpenedProperty handler on element System.Windows.Automation.AutomationElement scope Descendants.


TRACE Output (from ReSharper):
2017.04.19 16:22:07.364    INFO BuildStage started
2017.04.19 16:22:07.364    INFO Build policy: Automatic
2017.04.19 16:22:09.141    INFO 1 projects built
2017.04.19 16:22:09.141 VERBOSE     WPF.Test.UI
2017.04.19 16:22:09.141    INFO BuildStage completed
2017.04.19 16:22:09.144    INFO RefreshProjectPropertiesStage started
2017.04.19 16:22:09.144    INFO RefreshProjectPropertiesStage completed
2017.04.19 16:22:09.144    INFO DiscoveryStage started
2017.04.19 16:22:09.145    INFO Test exploration required for 1 projects
2017.04.19 16:22:09.145 VERBOSE     WPF.Test.UI
2017.04.19 16:22:09.825    INFO DiscoveryStage completed
2017.04.19 16:22:09.825    INFO BuildPipelineStage started
2017.04.19 16:22:09.825    INFO BuildPipelineStage completed
2017.04.19 16:22:09.825    INFO ApplyRulesStage started
2017.04.19 16:22:09.826    INFO Got 1 elements (1 explicit) to run
2017.04.19 16:22:09.826    INFO Marking 1 elements as Pending
2017.04.19 16:22:09.827    INFO Marking elements as Pending completed
2017.04.19 16:22:09.827    INFO Distributing 1 elements into runs
2017.04.19 16:22:09.828    INFO Host Controller initialization started
2017.04.19 16:22:09.828 VERBOSE Host Controller: ProcessTaskRunnerHostController
2017.04.19 16:22:09.828    INFO Host Controller initialization completed
2017.04.19 16:22:09.828    INFO Distributing elements completed
2017.04.19 16:22:09.828    INFO Created 1 unit-test runs
2017.04.19 16:22:09.828    INFO ApplyRulesStage completed
2017.04.19 16:22:09.828    INFO RunTestsStage started
2017.04.19 16:22:09.829 VERBOSE Running up to 4 unit-test runs in parallel
2017.04.19 16:22:09.829    INFO Run: 25f50617-364d-4c60-a4f1-eca45cf0aa10 - Started
2017.04.19 16:22:09.829 VERBOSE     Provider: MSTest
    Project: WPF.Test.UI
    Target Framework:
    Strategy: JetBrains.ReSharper.UnitTestProvider.MSTest11.MsTestInProcessStrategy
    Runtime Enviroment:
        Platform Type: x86
        Platform Version: v4_5
2017.04.19 16:22:11.406    WARN System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain. This can happen if the test(s) started a thread but did not stop it. Make sure that all the threads started by the test(s) are stopped before completion.
2017.04.19 16:22:11.426    INFO Run: 25f50617-364d-4c60-a4f1-eca45cf0aa10 - Completed
2017.04.19 16:22:11.426    INFO RunTestsStage completed
2017.04.19 16:22:11.426    INFO Finishing run
2017.04.19 16:22:11.427    INFO Run is finished

 

As you can see, even when the only call to the Telerik Testing Framework is "Initialize()", it still results in an AppDomainUnloaded exception when the test is disposed. 

[Elena]

Hello Logan,

Thanks for providing further details. I am sorry to hear you are still facing the error. 

Would it be possible to provide me your project for further research? I would like to inspect it and double check how it is build to try to reproduce the issue on my end. 

Thanks in advance for your time and cooperation! 

Regards,
Elena Tsvetkova
Telerik by Progress

 

 

[Logan]

 It's very simple: just add the ArtOfTest.WebAii.dll as a reference, subclass the BaseWpfTest, and call Initialize in the test. 

 

Did you try doing that yourself? Because that's all it takes, and that's all I've uploaded...

[Elena]

Hello Logan,

Thanks for getting back to me. 

Since I tried to reproduce the same behavior on few different machines and it was unsuccessful I would like to have few more details about the setup you have in use. Please let me know more about the exact setup you have.

Please let me know which are the Visual Studio and Telerik Framework versions on your local machine where the project is maintained and please confirm if the test runs when executed locally.

Additionally let me know which version of Visual Studio you have on the TFS build machine and how is the build configured (xaml build or new TFS build steps). Having this information I will try again to reproduce the environment and the same behavior. 

Thank you once again for your time and cooperation! 

Regards,
Elena Tsvetkova
Telerik by Progress

 

 

[Logan]

I am using Visual Studio 2017. I have tested this under both Testing_Framework_2017_1_0207_2_Free and Testing_Framework_2017_1_0328_3_Free. It's quite inconsistent, and sometimes causes the VSTest runner to abort with the message

The active Test Run was aborted because the execution process exited unexpectedly. The test execution process crashed while running the tests. To investigate further, open file:///C:/Users/username/AppData/Local/CrashDumps/vstest.executionengine.x86.exe(1).39616.dmp file in Visual Studio and choose "Debug in mixed mode".

I've opened the dump files several times, but it's been very difficult to diagnose. All the dump file indicates is that some thread was being aborted. I'm confident it's not in any code I've written because I can get the error to occur using just the "Initialize()" call from "BaseWpfTest".

 

I'm also using TFS 2017 Update 1. I am using the "Visual Studio Test Agent Deployment" and "Run Functional Tests" build tasks. This problem has occurred in both the 2015 and 2017 Functional Test Runners supplied by TFS.

I don't believe it's related to the version of the runner, as it happens both when the tests are executed by the Test Agent from TFS as well as when I run them locally in Visual Studio.

 

[Logan]

I'm using Visual Studio 2017. I have seen this error occur in both the Testing_Framework_2017_1_0207_2_Free and Testing_Framework_2017_1_0328_3_Free versions of the Telerik Testing Framework.

I will also randomly get the following error (both from TFS and from local Visual Studio test runs):

The active Test Run was aborted because the execution process exited unexpectedly. The test execution process crashed while running the tests. To investigate further, open file:///C:/Users/username/AppData/Local/CrashDumps/vstest.executionengine.x86.exe(1).39616.dmp file in Visual Studio and choose "Debug in mixed mode".

When I open the dump file, I see that the exception was a "Thread was aborted" error. However, I'm confident it's nothing in my code as I'm only calling the "Initialize()" method on the "BaseWpfTest" class. 

I am using the "Visual Studio Test Agent Deployment" and "Run Functional Tests" build tasks on a TFS 2017 Update 1 server. I have seen this error occur using both the 2015 and 2017 versions of the Visual Studio Test Agent from TFS.

Again, the error seems to come from some thread or resource not being properly disposed by the "BaseWpfTest" class. I have seen the AppDomainUnloadedException warning as well as the "Test Run was aborted" error in both TFS Test Agent runs as well as locally in Visual Studio. 

[Elena]

Hi Logan,

Thanks for all the provided details. 

I would like to note that Visual Studio and TFS 2017 are not yet supported from our side. This might be the reason for the behavior you are facing. 

Meanwhile you could try to create a Telerik test in a Visual studio test project (right click the project name -> Add New Item... ->Test and then choose a Telerik wpf test) and give it a try. Would that make any difference in the behavior? 

We manged to test on our end and run a Telerik test with TFS 2017. However some further time will be required to continue the research. I will refer to the additional information you have shared and will contact you in case of further assistance required. 

Thanks for your understanding! 

Regards,
Elena Tsvetkova
Telerik by Progress

 

 

[Logan]

Please be aware that I do not have Test Studio installed and therefore I'm not using the Test Studio Visual Studio plugin. I'm using the free version of the Testing Framework, which doesn't require installation of Test Studio.

I also just ran my test under Visual Studio 2015, and the exact same behavior occurred. This may be an issue with how MSTest and VSTest are handling multithreading errors, but it is clear to me that your framework (ArtOfTest.WebAii.dll) is causing these errors, regardless of Visual Studio version or TFS version.

If anyone at Telerik (QA, developers, etc) knows of a way to fix or work around this, I would very much like to know what that might be.

[Elena]

Hi Logan,

Thanks for getting back to me. 

Please note that all tests on our end were applied on an environment based on the details you have shared. Thus the Telerik Testing Framework is installed on the test machine along with Visual Studio 2015 and TFS 2017. 

I will kindly ask you again to create a unit test using the Telerik WPF test template as I described in my previous reply. The template includes all required references on top of these you have manually added to the provided project. 

Please find attached a sample project created as per the above instructions along with some screenshots demonstrating the successful TFS build and the test execution. Please note that there are plenty of features in the TFS tasks but the sample project ran successfully. The different tasks might evoke behavior that is not expected. 

I hope this will be helpful to you to find out what causes the observed warning on your end. Thanks in advance for your cooperation! 

Regards,
Elena Tsvetkova
Telerik by Progress

 

 

[Lorenz]

Hi,

We are having similar issue here. Logan, were you able to solve this issue? Or did you find a workaround?

 

Here is our case:

We have a set of acceptance tests using Telerik Test Framework which are executed on TeamCity with NUnit 2: no problem.

Now we are trying to migrate to NUnit3 but we are facing the AppDomainUnloadException issue. All tests are executed successfully but at the end it seems there is a timeout when nunit agent tries to unload the AppDomain. And then no Test Results file is generated:

[Exec] Server stack trace:
[Exec]    at NUnit.Engine.Services.DomainManager.DomainUnloader.Unload()
[Exec]    at NUnit.Engine.Services.DomainManager.Unload(AppDomain domain)
[Exec]    at NUnit.Engine.Runners.TestDomainRunner.UnloadPackage()
[Exec]    at NUnit.Engine.Runners.AbstractTestRunner.Unload()
[Exec]    at NUnit.Engine.Runners.AbstractTestRunner.Dispose(Boolean disposing)
[Exec]    at NUnit.Engine.Runners.AbstractTestRunner.Dispose()
[Exec]    at NUnit.Engine.Runners.AggregatingTestRunner.RunTestsSequentially(ITestEventListener listener, TestFilter filter, List`1 results, Boolean disposeRunners)
[Exec]    at NUnit.Engine.Runners.AggregatingTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
[Exec]    at NUnit.Engine.Runners.AbstractTestRunner.Run(ITestEventListener listener, TestFilter filter)
[Exec]    at NUnit.Engine.Agents.RemoteTestAgent.Run(ITestEventListener listener, TestFilter filter)
[Exec]    at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
[Exec]    at System.Runtime.Remoting.Messaging.StackBuilderSink.PrivateProcessMessage(RuntimeMethodHandle md, Object[] args, Object server, Object[]& outArgs)
[Exec]    at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)
[Exec]
[Exec] Exception rethrown at [0]:
[Exec]    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
[Exec]    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
[Exec]    at NUnit.Engine.ITestEngineRunner.Run(ITestEventListener listener, TestFilter filter)
[Exec]    at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)
[Exec]
[Exec] Test Run Summary
[Exec]   Overall result: Failed
[Exec]   Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
[Exec]   Start time: 2017-09-07 06:25:11Z
[Exec]     End time: 2017-09-07 06:30:51Z
[Exec]     Duration: 339.754 seconds

We have 4 Test Scenario written with SpecFlow and 2 of them are Scenario Outline (equivalent to NUnit  TestCase attribute). Here are the flow of these tests:

- Inheriting from BaseWpfTest

- In BeforeScenario (Equivalent to NUnit SetUp) method: call of Intitialize method:

1.[BeforeScenario]
2.public void BeforeScenario()
3.{
4.    BuildLog.Info("BeforeScenario: Initialize ArtOfTest.WebAii.Core.Manager for UI Automation");
5.    this.Initialize(Console.Out.WriteLine);
6.}

 

-  Start the WPF Application:

1.var processInfo = new ProcessStartInfo(ExecutablePath) { Arguments = string.Join(" ", arguments), UseShellExecute = false };
2.WpfApplication app = this.Manager.LaunchNewApplication(processInfo);

 

- Search some Visual Elements and make some assertions

- In AfterScenario (Equivalent to NUnit TearDown): call of CleanUp method

1.BuildLog.Info("AfterScenario: CleanUp ArtOfTest.WebAii.Core.Manager after UI Automation");
2.this.CleanUp();

 

- In AfterTestRun (Equivalent to NUnit OneTimeTearDown): Call to ShutDown method

1.[AfterTestRun]
2.public static void AfterTestRun()
3.{
4.    BuildLog.Info("AfterTestRun: ShutDown ArtOfTest.WebAii.Core.Manager");
5.    BaseWpfTest.ShutDown();
6.}

 

Thanks,

Benoît

[Logan]

I'm sorry Benoit, but I've long since abandoned that project. I went so far as to try to create a proxy for the Telerik framework as a separate process to avoid crashing the test runner process, but it proved to be too difficult and our priorities shifted away from UI testing. 

 

That's the best workaround I could come up with: instead of referencing the Telerik UI testing dll directly, write some sort of proxy program that references the Telerik dll in a completely separate process. 

[Elena]

Hello,

@Logan: Thank you for getting in the conversation! I appreciate it! 

@Benoit: Please let me know if you would need further assistance on the topic! 

Thank you! 

Regards,
Elena Tsvetkova
Progress Telerik

 

 

[Lorenz]

Hi Elena,

We would appreciate some assistance on this topic.

For additional info: if I enable only one test among the 10 using the manager (other ones are mark as Ignored), sometimes nunit was able to execute it an unload the app domain, but most of the case it fails.

When at least 2 tests are enabled, test execution systematically fails at the end when unloading app domain.

 

Thanks,

Kind regards,

Benoît

[Elena]

Hello Benoît,

Thank you for the additional details. 

I would like to ask for your assistance so that we could investigate the issue on our end. I would like to kindly ask you to prepare a sample project which you could reproduce the issue with. 

Next very important details would be the exact NUnit version you are using to build the failing tests. 

I would appreciate if you could also record a short video demonstrating your actions to face the exception. 

Please let me know if you could provide these. Thank you in advance for your cooperation!

Regards,
Elena Tsvetkova
Progress Telerik

 

 

[Lorenz]

Hello Elena,

Last week I was preparing a sample project to attach to this topic. And I noticed it was working well :/

I'm not 100& sure yet but it seems that the AppDomain unload exception is thrown only when our tests are executed with JetBrains dotCover. When I run them directly with the NUnit3 console, it seems to work... I will investigate that a bit deeper.

 

By the way, we are using the following versions of NUnit components (from NuGet):

- NUnit v3.7.1

- NUnit.Console v3.7.0

- NUnit.ConsoleRunner 3.7.0

 

Kind regards,

Benoît

[Elena]

Hi Benoît,

Thank you very much for your cooperation! 

I am glad to know you have identified some pattern of working and non-working setups. In case there is anything I could be helpful with please do not hesitate to get back to this thread again. 

Thanks! 

Regards,
Elena Tsvetkova
Progress Telerik