I have enabled the code activity workflow in a TFS build that runs unit tests.  My understanding is that JustMockTestRunner uses MSTest.exe to execute them.  Is it possible to use VSTest.Console.exe instead?

Hi Sven,

Unfortunately, at this stage this is not possible out of the box. However, could you please elaborate a bit more on the matter, as we might be able to suggest some possible solutions. For example, could you share the following information with us:
 - What is the TFS version used?
 - Why do you need to use vstest instead of mstest?

I am looking forward to your reply.

Regards,
Kaloyan
Telerik

TFS 2012.

We don't necessarily need vstest.exe but I was interested in possible usage.  This may be necessary when running tests developed using runsettings over testsettings file.  Also, it is supposed to be faster.  We just updated our JustMock library to latest version from one a year older.  Our previous run times were around 20 minutes.  They are now reaching 50 minutes. 

Can you tell me which version of MSTest.exe is being called by JustMockTestRunner.exe on a machine with both VS 2010 and VS 2012 installed? 

Hello Sven,

In principle, to enable the JustMock profiler (regardless of use case) you just need to set 3 environment variables before the test run starts and unset them afterwards. The JustMock code activities make this process straightforward, but they're not absolutely necessary. The built-in TFS code activities are sufficient to set those variables, albeit not as straightforward.

JustMock also provides the JustMockRunner executable which can wrap any process start so that JustMock is enabled for that process. If you use this executable, then you don't have to muck with environment variables, but you will have to provide the full command line that starts VSTest and you won't be able to use the built-in code activity that already does that.

The code activity for TFS 2010 uses MSTest from Visual Studio 2010, and the code activity fort TFS 2012 uses MSTest from Visual Studio 2012 by default. You can override the MSTest executable for either code activity by setting the MsTestExecutable property of the code activity to the full path the the MSTest executable (MSTest.exe) on the build machine.

As for the experienced slow-down, I have a hunch that the slow-down is caused by our change of the default behavior for mocks from Behavior.Loose to Behavior.RecursiveLoose. My suggestion is that you profile your tests to determine the cause of the slow execution. It is likely caused by a few mocked types throughout your test cases. Changing the Create call from Mock.Create<T>() to Mock.Create<T>(Behavior.Loose) will speed-up the usage of that mock. Naturally, this is only an informed guess and the information from the profiling session should be used to determine the true cause.

If you use JustCode and the JustCode Unit Test Runner, then you can profile your tests right away.
If you prefer to use only JustTrace, then you can use its unit test run integration.
Do not forget to link the JustTrace and JustMock profilers before profiling.

If you have any additional questions, please don't hesitate to ask.

Regards,
Stefan
Telerik

I have been having issues getting code coverage to work with JustMockTestRunner.  I have implemented the TFS build activity as described in the help document.  Everything was working fine prior to my recent upgrade of the JustMock library so I am sure that it is set up correctly. 

For the latest release of JustMock, have there been fundamental changes to the activity?  Do I need to rebuild that part of the workflow?

Secondly, I noticed that there is a tool that will allow you to select which profilers are enabled to work with JustMock.  Do I need to set the active profilers on the build machine or can I do that through the build workflow itself?  Does the enabling of the profilers require restart of windows to work?

Thanks in advance Stefan.  You have been most helpful.

Hello Sven,

Yes, you need to run the configuration tool on the build machine and select the profilers to link. You don't have to restart Windows. The profiler linking takes effect for all processes started after the linking. Currently, there is no way to configure profiler linking in a way other than through the configuration GUI.

I don't think there were any changes to the activity recently. What sort of problems are you having? Can you be more specific?

Regards,
Stefan
Telerik

Essentially we are not getting code coverage since our update.  When I run mstest manually, I can get everything to work except, of course, the JustMock tests which require the JustMock profiler.  When I run JustMockTestRunner manually, the tests run but I get a code coverage error:

Code coverage collection error: Collection engine threw exception:
Error VSP10041 : True.

I have tried every possible combination of profiler settings when running JustMockTestRunner and nothing makes a difference.  I am currently trying to narrow down exactly where the problem is occurring but it is going slow.  Any ideas?

Hi Sven,

We are currently investigating what could be causing such issue on our side. Unfortunately, we will require some more time for this.

Meanwhile, can you check if removing the JustMockTestRunner activities completely from your workflow and linking the profilers from the JustMock Configuration tool will solve the matter? I ask this, as normally, when you link our profiler with a code coverage tool, we expect the other tool to initialize us in the test process. In other words, the JustMockTestRunner code activity should be unnecessary in such cases.

We will get back to you with results from our investigation as soon as possible. Thank you for the understanding in advance.

Regards,
Kaloyan
Telerik

Hi Sven,

I configured a new TFS 2012 and using Visual Studio 2012 I found two possible approaches for running JustMock test cases with enabled Code Coverage. Here they are:

Using Visual Studio Test Runner

This is simple. The only things you need to run elevated JustMock tests with enabled code coverage, using vstest are:
 1. As a starting point, you need to link the JustMock's profiler with the Visual Studio 2012 Code Coverage/IntelliTrace. This is done, from the JustMock Configuration tool.
 2. Then, you will need to add the following environment variable on system level: "JUSTMOCK_INSTANCE" with value, "1".
 3. Finally, you have to set the Test Runner in the build definition to Visual Studio Test Runner and from its options, to Enable Code Coverage.

After the above steps, the elevated JustMock tests should pass upon build and code coverage should be produced.

Note: This works with the default build template and workflow. In other words, you don't need to add any JustMockTestRunner activities or something else.
Note: This will work, only when the Code Coverage is enabled for your test runs!

Using MSTest Runner

In order to enable code coverage with MSTest, you need to use a *.testsettings file for your project and follow the JustMock's Code Activity Workflow. You can further check this article, that explains how to enable Code Coverage in the *.testsettings file. Note that, as MSTest uses the VS 2010 Code Coverage, you don't need to link any profilers.

I hope this helps. Please, let me know if you require further assistance.

Regards,

Kaloyan
Telerik

Thanks for all the information.

I have already implemented the workflow changes so I am familiar with how to set up code coverage already.  In fact, everything was working perfectly prior to the update of the JustMock version. 

I redesigned the build definition/workflow and am still using MSTest to run everything.  We like using the testsettings file rather than runsettings (which by default does not produce .trx and .coverage files).  I also disables the workflow just mock custom activities and am just using the environment variable settings on the build agent.  This seems to be working now.  Code coverage is being generated normally.

However, on occasion, the build trips up and not all the assemblies are instrumented.  I can't help but think that this has something to do with the JustMock profiler as this was never a problem before the update.  I will keep investigating and if I discover something new or have additional questions, I'll ask.  Thanks again.