Mocks failing to work correctly in large test runs?

[Cameron]

I am using JustMock in ~140 tests, and am experiencing some strange issues with a group of test failing when the entire set of test is run, but passing when run independently.  Currently it takes 3 test runs for all test to pass, with no changes to the code between runs.

They seem to be failing because mocks are not correctly intercepting method/property calls.

Are there any known issue with the mocks interfering with each other when used in multiple tests?  Due to legacy library use there is a lot of Final and Static mocking in the tests, could this have something to do with it?

[Ricky]

Hi Cameron,

Thank you for sending the issue. The issue could happen if a method / property is invoked directly before the acutal test method where it is failing. Let's consider the following example:

We have our first test method that looks like:

[TestMethod]
public void Test1()
{
  Foo.StaticCall();
}

Next, we have another method that looks like:

[TestMethod]
public void Test2()
{
 bool called = false;
 Mock.Arrange(() => Foo.StaticCall())
   .DoInstead(() => called = true);
 Assert.IsTrue(called);
}

Now if the first test executes before the second. Test2 is going to fail. As Foo.StaticCall is already compiled and  we can't intercept it during OnJITCompilationStarted in Mock.Arrange.

Because JustMock intercepts or setups concrete methods (Ex. Static methods) on demand thus avoiding any attribute centric approach, one way to get around this issue is to initialize the call during test  class initialization. The attribute for it is ClassInitailize(MSTest) / TestFixutreSetup(NUnit) , other test frameworks also have similar initialization constructs.

Therefore in our example, we can do the following:

[ClassInitailize]
public static void Initialize(TestContext context)
{
   Mock.Initialize<Foo>();

}

Similarly, we can setup the same for a particular method:

[ClassInitailize]
public static void Initialize(TestContext context)
{
   Mock.Partial<Foo>().For(() => Foo.StaticCall());

}

This technique will ensure that all the target calls are initialized properly thus interfering tests wont fail.


Additionally, another way is to ensure the order in which the tests are executing. Here, if Test2 executes before Test1 then both will pass.

Please do write back for any further questions.

Kind regards,
Ricky
the Telerik team

[Stefen]

Alternatively if you explicitly mock the return value on every test that access your value/method then your tests will pass as expected. I'm haven't tested if this works for methods, but it solved the issue for a boolean property.  

[Ricky]

Hi Cameron,

Stefan is right. In that regard, you have to arrange each expected member which either does CallOriginal or goes through some specific expectation set by you.

Kind Regards
Mehfuz
the Telerik team

[Suneco]

I'm having thesame problem, but I'm not using static classes/methods, so the solution to the problem Ricky posted doen't work.

My "Test1" tests a method, let's say Foo.Fill()      <-- (not a static method)

And "Test2" tries to mock the Foo.Fill(), but in stead of mocking the method, it executes the real method.

[Christian]

Hi,
The Example provided in this thread also works for non-static classes.
You have Initialize the complete class (Mock.Initialize<class>()) before the fist instance is created.

HTH
Christian

[Ricky]

Hi Jeroen,
Thanks again for bringing up the question.

Christian is correct about the fact that Mock.Initialize also works for instance methods. In addition, I would suggest you to take a look at this post for further reading:

http://www.telerik.com/help/justmock/basic-usage-mock-initialize.html

Kind Regards
Ricky
the Telerik team

[Pizza]

I have tried using the initializers and am still seeing failures when running it with other tests in my solution.

I've create a new test (mine being the only test that exist so there are no other calls being made that may intercept).

Here is the following scenario I'm facing:

I have a static class Foo
I have a static method FooMethod
I have a static property FooProperty -> This is the unit test I've created

public static string FooMethod(string s)
{
    return string;
}
public static boolean FooProperty
{
    var v = FooMethod(somestring);
    return Convert.ToBoolean(v);
}

In my test, I am trying to mock the FooMethod that returns a string value
public void UnitTest
{
        Mock.Create(typeof(Foo));   //I'm noticing that I do not need to create anything and my tests will still pass.

        Mock.Arrange( () => Foo.FooMethod(Arg.IsAny<String>())).IgnoreInstance().Returns("string");

        bool b = Foo.FooProperty;
       
}

Any suggestions?
Thanks

[Kaloyan]

Hello Lisa Le,

Thank you for contacting Telerik support central.

To assist you with the issue, I would recommend you to use "Mock.SetupStatic" in order to prepare your classes for static mocking. Please check the following example:

[TestMethod]
public void ShouldTestFooPropertyGetWhenTrue()
{
    Mock.SetupStatic(typeof(Foo),Behavior.CallOriginal);

    Mock.Arrange(() => Foo.FooMethod(Arg.IsAny<String>())).IgnoreInstance().Returns(System.Boolean.TrueString);

    bool b = Foo.FooProperty;

    Assert.IsTrue(b);
}

Here, I updated your test with the Mock.SetupStatic and also, asserted that "b" should be true at the end. This test method is behaving as expected. For further guidance, you could check this article about static mocking with JustMock.

However, if this does not help, I would need you to send me the exact exception report you are experiencing in your large test runs with the exact tests that are failing and the code being tested.

I hope this helps. Please notify us if you need further help.

Kind regards,
Kaloyan
the Telerik team

[Pizza]

Hi Kaloyon,

I have tried your suggested solution below and am still facing the same problem. Please have a look at my scenario below that I'm noticing:

I added 2 test methods in the same class:
When I run them individually - they pass
When I run all the test methods in this class - they pass
When I run it with other test methods outside the class - they fail.

If I'm mocking them per test method, I don't see how this could be intercepted from anywhere else.


public class FooTests
{

 

[

 

TestMethod()]

 

 

 

public void FooPropertyStringValueReturnsTrue()

 

{

 

 

    Mock.SetupStatic(typeof(Foo), Behavior.CallOriginal);

 

 

 

    Mock.Arrange(() => Foo.FooMethod(Arg.IsAny<String>())).IgnoreInstance().Returns("1");

 

 

 

    bool actual = Foo.FooProperty;

 

 

 

    Assert.IsTrue(actual);

 

}

 

 

 

[TestMethod()]

 

 

 

public void FooPropertyStringValueReturnsFalse()

 

{

 

 

    Mock.SetupStatic(typeof(Foo), Behavior.CallOriginal);

 

 

 

    Mock.Arrange(() => Foo.FooMethod(Arg.IsAny<String>())).IgnoreInstance().Returns("0");

 

 

 

    bool actual = Foo.FooProperty;

 

 

 

    Assert.IsTrue(actual);

 

}
}

Any other ideas of what may be causing the failure?

[Kaloyan]

Hi again Lisa,

Thank you for the information provided. I must note that, you are correct that mocking is per test method and the different mocks should not be intercepted in other tests, than theirs. However, the reason of your failing tests could be that, not the mocks of "Foo", but the "Foo" class itself has already been JIT compiled before the test methods were able to intercept it.

I managed to reproduce the issue by creating the simple test method:

[TestMethod]
public void ATestMethod1()
{
    string s = "test";
    var initializer = Foo.FooMethod(s);
}

When I ran all the test in my solution, "ATestMethod1" did execute first. This lead to the failure of the other tests, trying to intercept "Foo.FooMethod()".

However, to avoid this, I would suggest you the using of Mock.Initialize on more global level. Another solution could be to split your test methods in two projects, one for elevated testing (using the profiler) and another one for simple testing.

I hope this helps. Please, contact us further if you need more assistance.

Kind regards,
Kaloyan
the Telerik team

[Pizza]

Kaloyan,

I tried placing this in my testsetup and it didn' twork. I also tried placing this in the class setup and it also didn't work:

Mock.Partial(typeof(Foo)).For(() => Foo.FooMethod(Arg.IsAny<String>)));

This is as simple as my unit test goes to separate them into separate projects. I have also looked into using the Sequential Mocking and this too fails when i do it this way. I get teh same kind of result: 1 - passes when running  2 - fails when you run with OTHER tests outside the class. Can this be further investigated or any other suggestions?

 

 

Mock.Arrange(() => Foo.FooMethod(Arg.IsAny<String>())).IgnoreInstance().Returns("1").InSequence();

 

 

 

Mock.Arrange(() => Foo.FooMethod(Arg.IsAny<String>())).IgnoreInstance().Returns("0").InSequence();

 

 

 

bool actualOne = Foo.FooProperty;

 

 

 

bool actualTwo = Foo.FooProperty;

 

 

 

Assert.IsTrue(actualOne);

 

 

 

Assert.IsFalse(actualTwo);


Thanks

 

[Kaloyan]

Hello Lisa,

It is unfortunate to hear that the issue still remains.

In order to investigate further, I would be needing your whole test project(you should be able to attach it to the post as a .ZIP), including the other tests that are causing the issue. I need this, so that I could check what in JustMock could be causing such an unexpected behavior.

Please notify me if this is doable for you.

Thank you for the collaboration and the understanding in advance.

All the best,
Kaloyan
the Telerik team

[Pizza]

Hi Kaloyan,
I've discussed this same issue with a colleague of mine and for soem reason, it works on his machine when he uses all code we've both specified. It's possible that there's some kind of environment issue? I got some assistance and  was able to create the unit test without using any JustMocks altogether. If you do encounter any issues similar to the above again, please share. Thanks for all the help.

[Kaloyan]

Hi again Lisa,

It is good to know that you have found a solution to the issue.

However, if there is anything that matches your case, I will be glad to share it with you and help you further on the matter.

Greetings,
Kaloyan
the Telerik team