Mocks failing to work correctly in large test runs?

15 posts, 0 answers
  1. Cameron
    Cameron avatar
    2 posts
    Member since:
    Dec 2010

    Posted 21 Dec 2010 Link to this post

    I am using JustMock in ~140 tests, and am experiencing some strange issues with a group of test failing when the entire set of test is run, but passing when run independently.  Currently it takes 3 test runs for all test to pass, with no changes to the code between runs.

    They seem to be failing because mocks are not correctly intercepting method/property calls.

    Are there any known issue with the mocks interfering with each other when used in multiple tests?  Due to legacy library use there is a lot of Final and Static mocking in the tests, could this have something to do with it?
  2. Ricky
    Admin
    Ricky avatar
    467 posts

    Posted 23 Dec 2010 Link to this post

    Hi Cameron,

    Thank you for sending the issue. The issue could happen if a method / property is invoked directly before the acutal test method where it is failing. Let's consider the following example:

    We have our first test method that looks like:

    [TestMethod]
    public void Test1()
    {
      Foo.StaticCall();
    }

    Next, we have another method that looks like:

    [TestMethod]
    public void Test2()
    {
     bool called = false;
     Mock.Arrange(() => Foo.StaticCall())
       .DoInstead(() => called = true);
     Assert.IsTrue(called);
    }

    Now if the first test executes before the second. Test2 is going to fail. As Foo.StaticCall is already compiled and  we can't intercept it during OnJITCompilationStarted in Mock.Arrange.

    Because JustMock intercepts or setups concrete methods (Ex. Static methods) on demand thus avoiding any attribute centric approach, one way to get around this issue is to initialize the call during test  class initialization. The attribute for it is ClassInitailize(MSTest) / TestFixutreSetup(NUnit) , other test frameworks also have similar initialization constructs.

    Therefore in our example, we can do the following:

    [ClassInitailize]
    public static void Initialize(TestContext context)
    {
       Mock.Initialize<Foo>();
        
    }

    Similarly, we can setup the same for a particular method:

    [ClassInitailize] 
    public static void Initialize(TestContext context) 
       Mock.Partial<Foo>().For(() => Foo.StaticCall());
          
    }

    This technique will ensure that all the target calls are initialized properly thus interfering tests wont fail.


    Additionally, another way is to ensure the order in which the tests are executing. Here, if Test2 executes before Test1 then both will pass.

    Please do write back for any further questions.

    Kind regards,
    Ricky
    the Telerik team

    Browse the videos here>> to help you get started with JustMock
  3. DevCraft R3 2016 release webinar banner
  4. Stefen
    Stefen avatar
    18 posts
    Member since:
    Oct 2010

    Posted 13 Mar 2012 Link to this post

    Alternatively if you explicitly mock the return value on every test that access your value/method then your tests will pass as expected. I'm haven't tested if this works for methods, but it solved the issue for a boolean property.  
  5. Ricky
    Admin
    Ricky avatar
    467 posts

    Posted 16 Mar 2012 Link to this post

    Hi Cameron,

    Stefan is right. In that regard, you have to arrange each expected member which either does CallOriginal or goes through some specific expectation set by you.


    Kind Regards
    Mehfuz
    the Telerik team
    Sharpen your .NET Ninja skills! Attend Q1 webinar week and get a chance to win a license! Book your seat now >>
  6. Jeroen Speldekamp
    Jeroen Speldekamp avatar
    30 posts
    Member since:
    Mar 2005

    Posted 25 Sep 2012 Link to this post

    I'm having thesame problem, but I'm not using static classes/methods, so the solution to the problem Ricky posted doen't work.

    My "Test1" tests a method, let's say Foo.Fill()      <-- (not a static method)

    And "Test2" tries to mock the Foo.Fill(), but in stead of mocking the method, it executes the real method.
  7. Christian
    Christian avatar
    3 posts
    Member since:
    Sep 2012

    Posted 25 Sep 2012 Link to this post

    Hi,
    The Example provided in this thread also works for non-static classes.
    You have Initialize the complete class (Mock.Initialize<class>()) before the fist instance is created.

    HTH
    Christian
  8. Ricky
    Admin
    Ricky avatar
    467 posts

    Posted 27 Sep 2012 Link to this post

    Hi Jeroen,
    Thanks again for bringing up the question.

    Christian is correct about the fact that Mock.Initialize also works for instance methods. In addition, I would suggest you to take a look at this post for further reading:

    http://www.telerik.com/help/justmock/basic-usage-mock-initialize.html


    Kind Regards
    Ricky
    the Telerik team

    Time to cast your vote for Telerik! Tell DevPro Connections and Windows IT Pro why Telerik is your choice. Telerik is nominated in a total of 25 categories.

  9. Azzip
    Azzip avatar
    5 posts
    Member since:
    Feb 2013

    Posted 21 Feb 2013 Link to this post

    I have tried using the initializers and am still seeing failures when running it with other tests in my solution.

    I've create a new test (mine being the only test that exist so there are no other calls being made that may intercept).

    Here is the following scenario I'm facing:

    I have a static class Foo
    I have a static method FooMethod
    I have a static property FooProperty -> This is the unit test I've created

    public static string FooMethod(string s)
    {
        return string;
    }
    public static boolean FooProperty
    {
        var v = FooMethod(somestring);
        return Convert.ToBoolean(v);
    }

    In my test, I am trying to mock the FooMethod that returns a string value
    public void UnitTest
    {
            Mock.Create(typeof(Foo));   //I'm noticing that I do not need to create anything and my tests will still pass.

            Mock.Arrange( () => Foo.FooMethod(Arg.IsAny<String>())).IgnoreInstance().Returns("string");

            bool b = Foo.FooProperty;
           
    }

    Any suggestions?
    Thanks

  10. Kaloyan
    Admin
    Kaloyan avatar
    872 posts

    Posted 22 Feb 2013 Link to this post

    Hello Lisa Le,

    Thank you for contacting Telerik support central.

    To assist you with the issue, I would recommend you to use "Mock.SetupStatic" in order to prepare your classes for static mocking. Please check the following example:
    [TestMethod]
    public void ShouldTestFooPropertyGetWhenTrue()
    {
        Mock.SetupStatic(typeof(Foo),Behavior.CallOriginal);
     
        Mock.Arrange(() => Foo.FooMethod(Arg.IsAny<String>())).IgnoreInstance().Returns(System.Boolean.TrueString);
     
        bool b = Foo.FooProperty;
     
        Assert.IsTrue(b);
    }
    Here, I updated your test with the Mock.SetupStatic and also, asserted that "b" should be true at the end. This test method is behaving as expected. For further guidance, you could check this article about static mocking with JustMock.

    However, if this does not help, I would need you to send me the exact exception report you are experiencing in your large test runs with the exact tests that are failing and the code being tested.

    I hope this helps. Please notify us if you need further help.

    Kind regards,
    Kaloyan
    the Telerik team
    Share what you think about JustTrace & JustMock with us, so we can become even better! You can use the built-in feedback tool inside JustTrace, our forums, or our JustTrace or JustMock portals.
  11. Azzip
    Azzip avatar
    5 posts
    Member since:
    Feb 2013

    Posted 22 Feb 2013 Link to this post

    Hi Kaloyon,

    I have tried your suggested solution below and am still facing the same problem. Please have a look at my scenario below that I'm noticing:

    I added 2 test methods in the same class:
    When I run them individually - they pass
    When I run all the test methods in this class - they pass
    When I run it with other test methods outside the class - they fail.

    If I'm mocking them per test method, I don't see how this could be intercepted from anywhere else.


    public class FooTests
    {

     

    [

     

    TestMethod()]

     

     

     

    public void FooPropertyStringValueReturnsTrue()

     

    {

     

     

        Mock.SetupStatic(typeof(Foo), Behavior.CallOriginal);

     

     

     

        Mock.Arrange(() => Foo.FooMethod(Arg.IsAny<String>())).IgnoreInstance().Returns("1");

     

     

     

        bool actual = Foo.FooProperty;

     

     

     

        Assert.IsTrue(actual);

     

    }

     

     

     

    [TestMethod()]

     

     

     

    public void FooPropertyStringValueReturnsFalse()

     

    {

     

     

        Mock.SetupStatic(typeof(Foo), Behavior.CallOriginal);

     

     

     

        Mock.Arrange(() => Foo.FooMethod(Arg.IsAny<String>())).IgnoreInstance().Returns("0");

     

     

     

        bool actual = Foo.FooProperty;

     

     

     

        Assert.IsTrue(actual);

     

    }
    }

    Any other ideas of what may be causing the failure?

  12. Kaloyan
    Admin
    Kaloyan avatar
    872 posts

    Posted 25 Feb 2013 Link to this post

    Hi again Lisa,

    Thank you for the information provided. I must note that, you are correct that mocking is per test method and the different mocks should not be intercepted in other tests, than theirs. However, the reason of your failing tests could be that, not the mocks of "Foo", but the "Foo" class itself has already been JIT compiled before the test methods were able to intercept it.

    I managed to reproduce the issue by creating the simple test method:
    [TestMethod]
    public void ATestMethod1()
    {
        string s = "test";
        var initializer = Foo.FooMethod(s);
    }

    When I ran all the test in my solution, "ATestMethod1" did execute first. This lead to the failure of the other tests, trying to intercept "Foo.FooMethod()".

    However, to avoid this, I would suggest you the using of Mock.Initialize on more global level. Another solution could be to split your test methods in two projects, one for elevated testing (using the profiler) and another one for simple testing.

    I hope this helps. Please, contact us further if you need more assistance.

    Kind regards,
    Kaloyan
    the Telerik team
    Share what you think about JustTrace & JustMock with us, so we can become even better! You can use the built-in feedback tool inside JustTrace, our forums, or our JustTrace or JustMock portals.
  13. Azzip
    Azzip avatar
    5 posts
    Member since:
    Feb 2013

    Posted 26 Feb 2013 Link to this post

    Kaloyan,

    I tried placing this in my testsetup and it didn' twork. I also tried placing this in the class setup and it also didn't work:

    Mock.Partial(typeof(Foo)).For(() => Foo.FooMethod(Arg.IsAny<String>)));

    This is as simple as my unit test goes to separate them into separate projects. I have also looked into using the Sequential Mocking and this too fails when i do it this way. I get teh same kind of result: 1 - passes when running  2 - fails when you run with OTHER tests outside the class. Can this be further investigated or any other suggestions?

     

     

    Mock.Arrange(() => Foo.FooMethod(Arg.IsAny<String>())).IgnoreInstance().Returns("1").InSequence();

     

     

     

    Mock.Arrange(() => Foo.FooMethod(Arg.IsAny<String>())).IgnoreInstance().Returns("0").InSequence();

     

     

     

    bool actualOne = Foo.FooProperty;

     

     

     

    bool actualTwo = Foo.FooProperty;

     

     

     

    Assert.IsTrue(actualOne);

     

     

     

    Assert.IsFalse(actualTwo);


    Thanks

     

  14. Kaloyan
    Admin
    Kaloyan avatar
    872 posts

    Posted 27 Feb 2013 Link to this post

    Hello Lisa,

    It is unfortunate to hear that the issue still remains.

    In order to investigate further, I would be needing your whole test project(you should be able to attach it to the post as a .ZIP), including the other tests that are causing the issue. I need this, so that I could check what in JustMock could be causing such an unexpected behavior.

    Please notify me if this is doable for you.

    Thank you for the collaboration and the understanding in advance.

    All the best,
    Kaloyan
    the Telerik team
    Share what you think about JustTrace & JustMock with us, so we can become even better! You can use the built-in feedback tool inside JustTrace, our forums, or our JustTrace or JustMock portals.
  15. Azzip
    Azzip avatar
    5 posts
    Member since:
    Feb 2013

    Posted 07 Mar 2013 Link to this post

    Hi Kaloyan,
    I've discussed this same issue with a colleague of mine and for soem reason, it works on his machine when he uses all code we've both specified. It's possible that there's some kind of environment issue? I got some assistance and  was able to create the unit test without using any JustMocks altogether. If you do encounter any issues similar to the above again, please share. Thanks for all the help.
  16. Kaloyan
    Admin
    Kaloyan avatar
    872 posts

    Posted 08 Mar 2013 Link to this post

    Hi again Lisa,

    It is good to know that you have found a solution to the issue.

    However, if there is anything that matches your case, I will be glad to share it with you and help you further on the matter.

    Greetings,
    Kaloyan
    the Telerik team
    Share what you think about JustTrace & JustMock with us, so we can become even better! You can use the built-in feedback tool inside JustTrace, our forums, or our JustTrace or JustMock portals.
Back to Top
DevCraft R3 2016 release webinar banner