I have been running into trouble running tests from TFS 2017 builds for several weeks now. I will do my best to sum up the situation and point out important details.
I have a c# project containing my UI tests that use ArtOfTest.WebAii.dll. The general structure of the tests is:
When I run this test with `vstest.console.exe .\WPFTest.dll /TestCaseFilter:FullyQualifiedName~ConnectAndDispose /InIsolation`, I see the following error in the console:
In addition to this, when TFS runs these tests as part of the `Run Functional Test` task (https://github.com/Microsoft/vsts-tasks/blob/master/Tasks/RunDistributedTests/README.md), the test runner intermittently stops responding, stops sending results to TFS, and causes the test runner for that machine to abort and fail those tests.
I've done nearly everything I can to try to identify what might be causing the hanging/unresponsive behavior, but I've reached the end of my expertise. What can I do to diagnose the problem and/or work around it to prevent the test runner from hanging? Is it possible that this `System.AppDomainUnloadedException` is causing this unresponsive behavior?
Please let me know if you need additional details. Thank you!
17 Answers, 1 is accepted
Thank you for sharing details on your observations.
I reviewed the shared information and also searched the web for the warning you get. It seems to be a common warning which is mostly related to the way the coded solution is structured and if all exceptions are correctly caught. I would like to kindly ask you to research for possible solutions about the warning in question since you are familiar with your whole project and any advice on my end might not be correct.
There is a suggestion I would like to place - you could remove the code related to the Telerik Testing Framework (LaunchNewApplication, ConnectToApplication, Dispose) and double check if the warning appear again. This will exclude the relation of the issue to the Telerik Framework.
Please let me know if I could assist you with anything else. Thanks!
Telerik by Progress
Hello Elena. Thanks for getting back to me!
Unfortunately, when I removed those (LaunchNewApplication, ConnectToApplication, Dispose) calls, the error no longer appeared, so it is related to the Telerik Framework.
My test class is inheriting from "BaseWpfTest", which gives me access to the "Initialize" function.
I have run the test both with and without the "Initialize()" call, in both the ReSharper test runner as well as the Visual Studio test runner (vstest).
Here are my results:
-- NOT Calling Initialize() =>
-- WITH Calling Initialize() =>
As you can see, even when the only call to the Telerik Testing Framework is "Initialize()", it still results in an AppDomainUnloaded exception when the test is disposed.
Thanks for providing further details. I am sorry to hear you are still facing the error.
Would it be possible to provide me your project for further research? I would like to inspect it and double check how it is build to try to reproduce the issue on my end.
Thanks in advance for your time and cooperation!
Telerik by Progress
It's very simple: just add the ArtOfTest.WebAii.dll as a reference, subclass the BaseWpfTest, and call Initialize in the test.
Did you try doing that yourself? Because that's all it takes, and that's all I've uploaded...
Thanks for getting back to me.
Since I tried to reproduce the same behavior on few different machines and it was unsuccessful I would like to have few more details about the setup you have in use. Please let me know more about the exact setup you have.
Please let me know which are the Visual Studio and Telerik Framework versions on your local machine where the project is maintained and please confirm if the test runs when executed locally.
Additionally let me know which version of Visual Studio you have on the TFS build machine and how is the build configured (xaml build or new TFS build steps). Having this information I will try again to reproduce the environment and the same behavior.
Thank you once again for your time and cooperation!
Telerik by Progress
I am using Visual Studio 2017. I have tested this under both Testing_Framework_2017_1_0207_2_Free and Testing_Framework_2017_1_0328_3_Free. It's quite inconsistent, and sometimes causes the VSTest runner to abort with the message
The active Test Run was aborted because the execution process exited unexpectedly. The test execution process crashed while running the tests. To investigate further, open file:///C:/Users/username/AppData/Local/CrashDumps/vstest.executionengine.x86.exe(1).39616.dmp file in Visual Studio and choose "Debug in mixed mode".
I've opened the dump files several times, but it's been very difficult to diagnose. All the dump file indicates is that some thread was being aborted. I'm confident it's not in any code I've written because I can get the error to occur using just the "Initialize()" call from "BaseWpfTest".
I'm also using TFS 2017 Update 1. I am using the "Visual Studio Test Agent Deployment" and "Run Functional Tests" build tasks. This problem has occurred in both the 2015 and 2017 Functional Test Runners supplied by TFS.
I don't believe it's related to the version of the runner, as it happens both when the tests are executed by the Test Agent from TFS as well as when I run them locally in Visual Studio.
I will also randomly get the following error (both from TFS and from local Visual Studio test runs):
When I open the dump file, I see that the exception was a "Thread was aborted" error. However, I'm confident it's nothing in my code as I'm only calling the "Initialize()" method on the "BaseWpfTest" class.
I am using the "Visual Studio Test Agent Deployment" and "Run Functional Tests" build tasks on a TFS 2017 Update 1 server. I have seen this error occur using both the 2015 and 2017 versions of the Visual Studio Test Agent from TFS.
Again, the error seems to come from some thread or resource not being properly disposed by the "BaseWpfTest" class. I have seen the AppDomainUnloadedException warning as well as the "Test Run was aborted" error in both TFS Test Agent runs as well as locally in Visual Studio.
Thanks for all the provided details.
I would like to note that Visual Studio and TFS 2017 are not yet supported from our side. This might be the reason for the behavior you are facing.
Meanwhile you could try to create a Telerik test in a Visual studio test project (right click the project name -> Add New Item... ->Test and then choose a Telerik wpf test) and give it a try. Would that make any difference in the behavior?
We manged to test on our end and run a Telerik test with TFS 2017. However some further time will be required to continue the research. I will refer to the additional information you have shared and will contact you in case of further assistance required.
Thanks for your understanding!
Telerik by Progress
Please be aware that I do not have Test Studio installed and therefore I'm not using the Test Studio Visual Studio plugin. I'm using the free version of the Testing Framework, which doesn't require installation of Test Studio.
I also just ran my test under Visual Studio 2015, and the exact same behavior occurred. This may be an issue with how MSTest and VSTest are handling multithreading errors, but it is clear to me that your framework (ArtOfTest.WebAii.dll) is causing these errors, regardless of Visual Studio version or TFS version.
If anyone at Telerik (QA, developers, etc) knows of a way to fix or work around this, I would very much like to know what that might be.
Thanks for getting back to me.
Please note that all tests on our end were applied on an environment based on the details you have shared. Thus the Telerik Testing Framework is installed on the test machine along with Visual Studio 2015 and TFS 2017.
I will kindly ask you again to create a unit test using the Telerik WPF test template as I described in my previous reply. The template includes all required references on top of these you have manually added to the provided project.
Please find attached a sample project created as per the above instructions along with some screenshots demonstrating the successful TFS build and the test execution. Please note that there are plenty of features in the TFS tasks but the sample project ran successfully. The different tasks might evoke behavior that is not expected.
I hope this will be helpful to you to find out what causes the observed warning on your end. Thanks in advance for your cooperation!
Telerik by Progress
We are having similar issue here. Logan, were you able to solve this issue? Or did you find a workaround?
Here is our case:
We have a set of acceptance tests using Telerik Test Framework which are executed on TeamCity with NUnit 2: no problem.
Now we are trying to migrate to NUnit3 but we are facing the AppDomainUnloadException issue. All tests are executed successfully but at the end it seems there is a timeout when nunit agent tries to unload the AppDomain. And then no Test Results file is generated:
We have 4 Test Scenario written with SpecFlow and 2 of them are Scenario Outline (equivalent to NUnit TestCase attribute). Here are the flow of these tests:
- Inheriting from BaseWpfTest
- In BeforeScenario (Equivalent to NUnit SetUp) method: call of Intitialize method:
- Start the WPF Application:
- Search some Visual Elements and make some assertions
- In AfterScenario (Equivalent to NUnit TearDown): call of CleanUp method
- In AfterTestRun (Equivalent to NUnit OneTimeTearDown): Call to ShutDown method
I'm sorry Benoit, but I've long since abandoned that project. I went so far as to try to create a proxy for the Telerik framework as a separate process to avoid crashing the test runner process, but it proved to be too difficult and our priorities shifted away from UI testing.
That's the best workaround I could come up with: instead of referencing the Telerik UI testing dll directly, write some sort of proxy program that references the Telerik dll in a completely separate process.
@Logan: Thank you for getting in the conversation! I appreciate it!
@Benoit: Please let me know if you would need further assistance on the topic!
We would appreciate some assistance on this topic.
For additional info: if I enable only one test among the 10 using the manager (other ones are mark as Ignored), sometimes nunit was able to execute it an unload the app domain, but most of the case it fails.
When at least 2 tests are enabled, test execution systematically fails at the end when unloading app domain.
Thank you for the additional details.
I would like to ask for your assistance so that we could investigate the issue on our end. I would like to kindly ask you to prepare a sample project which you could reproduce the issue with.
Next very important details would be the exact NUnit version you are using to build the failing tests.
I would appreciate if you could also record a short video demonstrating your actions to face the exception.
Please let me know if you could provide these. Thank you in advance for your cooperation!
Last week I was preparing a sample project to attach to this topic. And I noticed it was working well :/
I'm not 100& sure yet but it seems that the AppDomain unload exception is thrown only when our tests are executed with JetBrains dotCover. When I run them directly with the NUnit3 console, it seems to work... I will investigate that a bit deeper.
By the way, we are using the following versions of NUnit components (from NuGet):
- NUnit v3.7.1
- NUnit.Console v3.7.0
- NUnit.ConsoleRunner 3.7.0
Thank you very much for your cooperation!
I am glad to know you have identified some pattern of working and non-working setups. In case there is anything I could be helpful with please do not hesitate to get back to this thread again.