Tests fail in large runs but pass individually

[Dominic Faraday]

Hi,

I am having a problem with JustMock where tests pass when run individually but fail when all tests are run. I am not using any static or final mocking.

In the following code, the GetSectionsForAppByAppId_WithAppIdForSingleApp_ReturnsGetSectionsForAppResponse test fails.

   [TestClass]
   public class MobileAppSectionServiceTests
   {
        private const int AppId = 1;
        private const int LocalCultureId = 2;
 
        private IEnumerable<IMobileAppSection> sections;
        private GetSectionsForAppRequest request;
        private MobileAppSection section;
 
        [TestInitialize]
        public void TestInit()
        {
            section = new MobileAppSection();
            section.Id = AppId;
            section.Name = "SectionName";
            section.Summary = "Section Summary";
 
            sections = new List<IMobileAppSection> { section };
            request = new GetSectionsForAppRequest { MobileAppId = AppId, LocalCultureId = LocalCultureId};
        }
 
        [TestMethod]
        public void GetSectionsForAppByAppId_WithAppIdForSingleApp_ReturnsGetSectionsForAppResponse()
        {
            MobileAppSectionService service = new MobileAppSectionService(null);
            Mock.Arrange<IEnumerable<IMobileAppSection>>(() => service.GetSectionDomainEntitiesForAppByCulture(AppId, LocalCultureId)).Returns(sections);
            GetSectionsForAppResponse response = service.GetSectionsForAppByCulture(request);
            
            Assert.IsNotNull(response, "Response is not null");
        }
 
        [TestMethod]
        public void GetSectionsForAppByAppId_WithAppIdForSingleApp_ReturnsResponseWithSections()
        {
            IMobileAppSectionMapper _mobileAppSectionMapper = Mock.Create<IMobileAppSectionMapper>();
            MobileAppSectionService service = new MobileAppSectionService(_mobileAppSectionMapper);
            Mock.Arrange<IEnumerable<IMobileAppSection>>(() => _mobileAppSectionMapper.GetSectionsForMobileApp(AppId, LocalCultureId)).Returns(sections);
            
            GetSectionsForAppResponse response = service.GetSectionsForAppByCulture(request);
            
            Assert.IsTrue(response.MobileAppSections.Any());
        }
    }

The error message is:

Test method MobileAppDelivery.Service.Tests.Unit.MobileAppSectionServiceTests.GetSectionsForAppByAppId_WithAppIdForSingleApp_ReturnsGetSectionsForAppResponse threw exception: 
System.NullReferenceException: Object reference not set to an instance of an object.

The NullReferenceException is thrown the following method of the MobileAppSectionService class.

    public List<IMobileAppSection> GetSectionDomainEntitiesForAppByCulture(int appId, int cultureId)
    {
       List<IMobileAppSection> sections = _mobileAppSectionMapper.GetSectionsForMobileApp(appId, cultureId); 
       return sections;
    }

where _mobileAppSectionMapper is null. Rewriting the test method as the following fixes the problem

        [TestMethod]
        public void GetSectionsForAppByAppId_WithAppIdForSingleApp_ReturnsGetSectionsForAppResponse()
        {
            IMobileAppSectionMapper mapper = Mock.Create<IMobileAppSectionMapper>();
 
            MobileAppSectionService service = new MobileAppSectionService(mapper);
            Mock.Arrange<IEnumerable<IMobileAppSection>>(() => service.GetSectionDomainEntitiesForAppByCulture(AppId, LocalCultureId)).DoNothing().Returns(sections);
            GetSectionsForAppResponse response = service.GetSectionsForAppByCulture(request);
            
            Assert.IsNotNull(response, "Response is not null");
        }

So why does the original test only pass when run individually? 
My instincts tell me that the original test was not written correctly but it still passed. 

Thanks!

[Ricky]

Hi Dominic,

Thanks for reporting the problem. However, I can see that the following is a final method and in your first test, you were partially mocking it but for some reason it is calling the main implementation.

public List<IMobileAppSection> GetSectionDomainEntitiesForAppByCulture(int appId, int cultureId)
        {
            List<IMobileAppSection> sections = _mobileAppSectionMapper.GetSectionsForMobileApp(appId, cultureId);

            return sections;
        }

In order to pinpoint the issue, I would request you to send me a sample project where it throws the exception. I will use it to further debug and investigate the problem for you.

Kind Regards,
Mehfuz
the Telerik team

[Dominic Faraday]

Hi,

I have had this exact issue occur two more times and my colleagues experience it too.

I am not able to send you an example project which fails because it requires many tests for this to happen. I would have to pretty much send you the entire code base.

In this new example the test fails because according to the report the method marked as MustBeCalled is not called.

        [TestMethod]
        public void App_logs_are_persisted()
        {
            LogMobileAppActionMapper logAppMapper = Mock.Create<LogMobileAppActionMapper>();
            ContentUpdater updater = new ContentUpdater(null, null, logAppMapper, null);
            Mock.Arrange(() => logAppMapper.SaveCollection(Arg.IsAny<List<MobileAppLoggableActionBase>>())).MustBeCalled();
            updater.PersistAppLogEntries();
 
            Mock.Assert(logAppMapper);
        }

    public class ContentUpdater : IContentUpdater
    {        
        public List<MobileAppLoggableActionBase> MobileAppLogEntries { get; set; }
        public ContentUpdater
        (
            IMobileAppMapper mobileAppMapper, 
            ISectionContentMapper sectionContentMapper, 
            ILogMobileAppActionMapper logMobileAppActionMapper, 
            ILogSectionContentActionMapper logSectionContentActionMapper)
        {
            _mobileAppMapper = mobileAppMapper;
            _sectionContentMapper = sectionContentMapper;
            _logSectionContentActionMapper = logSectionContentActionMapper;
            _logMobileAppActionMapper = logMobileAppActionMapper;
 
            UploadedMobileAppDataCollection = new List<MobileAppData>();
            DomainContentsForSection = new List<ISectionContent>();
 
            SectionLogEntries = new List<SectionContentLoggableActionBase>();
            MobileAppLogEntries = new List<MobileAppLoggableActionBase>();
        }


        public void PersistAppLogEntries()
        {
            _logMobileAppActionMapper.SaveCollection(MobileAppLogEntries);
        }
}

    public class LogMobileAppActionMapper : DataMapperBase, ILogMobileAppActionMapper

       {

        public void SaveCollection(List<MobileAppLoggableActionBase> mobileAppLogEntries)
        {
            foreach (MobileAppLoggableActionBase mobileAppLoggableAction in mobileAppLogEntries)
            {
                Save(mobileAppLoggableAction);
            }
        }
 
        public void Save(MobileAppLoggableActionBase logEntry)
        {
            MobileAppLogEntity entity = ConvertDomainEntityToLlblGenEntity(logEntry) as MobileAppLogEntity;
            SaveLogEntityToDb(entity);
        }
 
        public override CommonEntityBase ConvertDomainEntityToLlblGenEntity(IDomainEntity domainEntityToConvert)
        {
            MobileAppLoggableActionBase logActionBase = domainEntityToConvert as MobileAppLoggableActionBase;
            MobileAppLogEntity entity = new MobileAppLogEntity();
 
            entity.MobileAppLogId = logActionBase.Id;
            entity.MobileAppId = logActionBase.MobileAppId;
            entity.UserName = logActionBase.UserName;
            entity.ActionDescription = logActionBase.ActionToString();
            entity.ActionDateTime = logActionBase.ActionDateTime;
            entity.CultureId = logActionBase.CultureId;
            return entity;
        }

      }

    public abstract class MobileAppLoggableActionBase 
    {
        protected MobileAppLoggableActionBase()
        {
            ActionDateTime = DateTime.UtcNow;
            IsNew = true;
            UserName = "user name";
        }
 
        public bool IsNew { get; set; }
        public DateTime ActionDateTime { get; set; }
        public string ActionDescription { get; set; }
        public string UserName { get; set; }
        public int Id { get; set; }
        public abstract string ActionToString();
    }
 
    public class LogMobileAppUnapprovalByUpdate : MobileAppLoggableActionBase
    {
        public override string ActionToString()
        {
            return "Unapproved because of update";
        }
    }

[Larry]

Mehfuz,

I am having the same problem.  I can send the code, as I am only writing it to evaluate the product and to do a training session for the other developers here.  I also have another problem that I am seeing, though I am not 100% sure if I am doing it correctly.

-Larry

[Ricky]

Hi Larry,
Thanks for keeping us informed about the problem. Also, thanks in advance for sharing your samples. This will let me investigate the problem and send you a possible solution.

However, this can happen due to the ordering of the same method used in different tests; since by design .net profiler can intercept a member only once during OnJITCompilationStarted and if it is invoked manually first then prevents JustMock from intercepting it. However even if you have such tests, it is possible to pre-initialize the member using Mock.Initialize{T} to resolve this side effect.

In order to make sure that Mock.Initalize{T} is called before any other tests are being executed. You can put it inside a static block that is invoked by .net framework before any other members in the class.

For example:

static TestClass()
{
    Mock.Partial<MyClass>().For((x) => x.Delete());
}

Once after the above block, the following two consecutive tests will work as expected; despite the first one calls the Delete method directly:

[TestMethod]
public void  Test1()
{
   new MyClass().Delete();
}

[TestMethod]
public void  Test2()

{

   var myClass = Mock.Create<MyClass>();
   Mock.Arrange(() => myClass.Delete()).DoNothing().MustBeCalled();

   myClass.Delete();

   Mock.Assert(myClass);
}

Hope this helps to solve the issue.


Kind regards,
Mehfuz
the Telerik team

[Dominic Faraday]

Using Mock.Initialize<> did not solve the issue

        [TestMethod]
        public void App_logs_are_persisted()
        {
            LogMobileAppActionMapper logAppMapper = Mock.Create<LogMobileAppActionMapper>();
            JsonUpdater updater = new JsonUpdater(null, null, logAppMapper, null);
            Mock.Arrange(() => logAppMapper.SaveCollection(Arg.IsAny<List<MobileAppLoggableActionBase>>())).MustBeCalled();
            updater.PersistAppLogEntries();
 
            Mock.Assert(logAppMapper);
        }
        
        public class JsonUpdater : IJsonUpdater
        {
           ...
 
           public void PersistAppLogEntries()
           {
               _logMobileAppActionMapper.SaveCollection(MobileAppLogEntries);
           }
        }


The above test fails in large test runs stating the method which must be called isn't. This is the only unit test which calls
logAppMapper.SaveCollection and updater.PersistAppLogEntries so I cannot see how other tests could break this. Also I have had this occur
with a test class that had only one test method. 

[Ricky]

Hi Dominic,

Thanks for contacting us regarding the matter. Since you are also having this problem with a test class that has only one test method, I would request you to send me a sample project with the same so that I can further investigate the problem. I need to debug through the issue to identify the actual cause and will send you a possible solution.

Kind Regards,
Mehfuz
the Telerik team

[Dominic Faraday]

Hi Mehfuz,

I cannot supply a code sample where this error occurs. The error occurs regularly only with large test runs. It would require me to send you the entire solution which is 500mb in size. I have tried extracting a snippet from it but the error no longer occurs. I don't think there is much value in sending you a test project which does not produce this bug.

After a discussion with my manager we will be happy to let you remotely connect to one of our development pcs and investigate the error.

[Ricky]

Hi Dominic,
Thanks again for contacting us.  If you want me to investigate the problem by connecting remotely, you can always reach me at  mehfuz.hossain att telerik dot com (making the email spam safe).  However, please also try the latest version and see whether it solves your issue. 


Kind Regards,
Mehfuz
the Telerik team