Tests fail in large runs but pass individually

9 posts, 0 answers
  1. Dominic Faraday
    Dominic Faraday avatar
    4 posts
    Member since:
    Jan 2010

    Posted 07 Feb 2012 Link to this post

    Hi,

    I am having a problem with JustMock where tests pass when run individually but fail when all tests are run. I am not using any static or final mocking.

    In the following code, the GetSectionsForAppByAppId_WithAppIdForSingleApp_ReturnsGetSectionsForAppResponse test fails.

       [TestClass]
       public class MobileAppSectionServiceTests
       {
            private const int AppId = 1;
            private const int LocalCultureId = 2;
     
            private IEnumerable<IMobileAppSection> sections;
            private GetSectionsForAppRequest request;
            private MobileAppSection section;
     
            [TestInitialize]
            public void TestInit()
            {
                section = new MobileAppSection();
                section.Id = AppId;
                section.Name = "SectionName";
                section.Summary = "Section Summary";
     
                sections = new List<IMobileAppSection> { section };
                request = new GetSectionsForAppRequest { MobileAppId = AppId, LocalCultureId = LocalCultureId};
            }
     
            [TestMethod]
            public void GetSectionsForAppByAppId_WithAppIdForSingleApp_ReturnsGetSectionsForAppResponse()
            {
                MobileAppSectionService service = new MobileAppSectionService(null);
                Mock.Arrange<IEnumerable<IMobileAppSection>>(() => service.GetSectionDomainEntitiesForAppByCulture(AppId, LocalCultureId)).Returns(sections);
                GetSectionsForAppResponse response = service.GetSectionsForAppByCulture(request);
                
                Assert.IsNotNull(response, "Response is not null");
            }
     
            [TestMethod]
            public void GetSectionsForAppByAppId_WithAppIdForSingleApp_ReturnsResponseWithSections()
            {
                IMobileAppSectionMapper _mobileAppSectionMapper = Mock.Create<IMobileAppSectionMapper>();
                MobileAppSectionService service = new MobileAppSectionService(_mobileAppSectionMapper);
                Mock.Arrange<IEnumerable<IMobileAppSection>>(() => _mobileAppSectionMapper.GetSectionsForMobileApp(AppId, LocalCultureId)).Returns(sections);
                
                GetSectionsForAppResponse response = service.GetSectionsForAppByCulture(request);
                
                Assert.IsTrue(response.MobileAppSections.Any());
            }
        }
    
    The error message is:
    
    Test method MobileAppDelivery.Service.Tests.Unit.MobileAppSectionServiceTests.GetSectionsForAppByAppId_WithAppIdForSingleApp_ReturnsGetSectionsForAppResponse threw exception: 
    System.NullReferenceException: Object reference not set to an instance of an object. The NullReferenceException is thrown the following method of the MobileAppSectionService class.
        public List<IMobileAppSection> GetSectionDomainEntitiesForAppByCulture(int appId, int cultureId)
        {
           List<IMobileAppSection> sections = _mobileAppSectionMapper.GetSectionsForMobileApp(appId, cultureId); 
           return sections;
        }
    
    where _mobileAppSectionMapper is null. Rewriting the test method as the following fixes the problem
    
    
            [TestMethod]
            public void GetSectionsForAppByAppId_WithAppIdForSingleApp_ReturnsGetSectionsForAppResponse()
            {
                IMobileAppSectionMapper mapper = Mock.Create<IMobileAppSectionMapper>();
     
                MobileAppSectionService service = new MobileAppSectionService(mapper);
                Mock.Arrange<IEnumerable<IMobileAppSection>>(() => service.GetSectionDomainEntitiesForAppByCulture(AppId, LocalCultureId)).DoNothing().Returns(sections);
                GetSectionsForAppResponse response = service.GetSectionsForAppByCulture(request);
                
                Assert.IsNotNull(response, "Response is not null");
            }
    
    So why does the original test only pass when run individually? 
    My instincts tell me that the original test was not written correctly but it still passed. 
    
    Thanks!
    
  2. Ricky
    Admin
    Ricky avatar
    467 posts

    Posted 10 Feb 2012 Link to this post

    Hi Dominic,

    Thanks for reporting the problem. However, I can see that the following is a final method and in your first test, you were partially mocking it but for some reason it is calling the main implementation.
    public List<IMobileAppSection> GetSectionDomainEntitiesForAppByCulture(int appId, int cultureId)
            {
                List<IMobileAppSection> sections = _mobileAppSectionMapper.GetSectionsForMobileApp(appId, cultureId);
      
                return sections;
            }


    In order to pinpoint the issue, I would request you to send me a sample project where it throws the exception. I will use it to further debug and investigate the problem for you.

    Kind Regards,
    Mehfuz
    the Telerik team
    Sharpen your .NET Ninja skills! Attend Q1 webinar week and get a chance to win a license! Book your seat now >>
  3. DevCraft R3 2016 release webinar banner
  4. Dominic Faraday
    Dominic Faraday avatar
    4 posts
    Member since:
    Jan 2010

    Posted 13 Feb 2012 Link to this post

    Hi,

    I have had this exact issue occur two more times and my colleagues experience it too.

    I am not able to send you an example project which fails because it requires many tests for this to happen. I would have to pretty much send you the entire code base.

    In this new example the test fails because according to the report the method marked as MustBeCalled is not called.

            [TestMethod]
            public void App_logs_are_persisted()
            {
                LogMobileAppActionMapper logAppMapper = Mock.Create<LogMobileAppActionMapper>();
                ContentUpdater updater = new ContentUpdater(nullnull, logAppMapper, null);
                Mock.Arrange(() => logAppMapper.SaveCollection(Arg.IsAny<List<MobileAppLoggableActionBase>>())).MustBeCalled();
                updater.PersistAppLogEntries();
     
                Mock.Assert(logAppMapper);
            }
    
    
        public class ContentUpdater : IContentUpdater
        {        
            public List<MobileAppLoggableActionBase> MobileAppLogEntries { getset; }
            public ContentUpdater
            (
                IMobileAppMapper mobileAppMapper, 
                ISectionContentMapper sectionContentMapper, 
                ILogMobileAppActionMapper logMobileAppActionMapper, 
                ILogSectionContentActionMapper logSectionContentActionMapper)
            {
                _mobileAppMapper = mobileAppMapper;
                _sectionContentMapper = sectionContentMapper;
                _logSectionContentActionMapper = logSectionContentActionMapper;
                _logMobileAppActionMapper = logMobileAppActionMapper;
     
                UploadedMobileAppDataCollection = new List<MobileAppData>();
                DomainContentsForSection = new List<ISectionContent>();
     
                SectionLogEntries = new List<SectionContentLoggableActionBase>();
                MobileAppLogEntries = new List<MobileAppLoggableActionBase>();
            }
    
    
    public void PersistAppLogEntries()         {             _logMobileAppActionMapper.SaveCollection(MobileAppLogEntries);         } }


        public class LogMobileAppActionMapper : DataMapperBaseILogMobileAppActionMapper
    
           {
            public void SaveCollection(List<MobileAppLoggableActionBase> mobileAppLogEntries)
            {
                foreach (MobileAppLoggableActionBase mobileAppLoggableAction in mobileAppLogEntries)
                {
                    Save(mobileAppLoggableAction);
                }
            }
     
            public void Save(MobileAppLoggableActionBase logEntry)
            {
                MobileAppLogEntity entity = ConvertDomainEntityToLlblGenEntity(logEntry) as MobileAppLogEntity;
                SaveLogEntityToDb(entity);
            }
     
            public override CommonEntityBase ConvertDomainEntityToLlblGenEntity(IDomainEntity domainEntityToConvert)
            {
                MobileAppLoggableActionBase logActionBase = domainEntityToConvert as MobileAppLoggableActionBase;
                MobileAppLogEntity entity = new MobileAppLogEntity();
     
                entity.MobileAppLogId = logActionBase.Id;
                entity.MobileAppId = logActionBase.MobileAppId;
                entity.UserName = logActionBase.UserName;
                entity.ActionDescription = logActionBase.ActionToString();
                entity.ActionDateTime = logActionBase.ActionDateTime;
                entity.CultureId = logActionBase.CultureId;
                return entity;
            }
          }

        public abstract class MobileAppLoggableActionBase 
        {
            protected MobileAppLoggableActionBase()
            {
                ActionDateTime = DateTime.UtcNow;
                IsNew = true;
                UserName = "user name";
            }
     
            public bool IsNew { getset; }
            public DateTime ActionDateTime { getset; }
            public string ActionDescription { getset; }
            public string UserName { getset; }
            public int Id { getset; }
            public abstract string ActionToString();
        }
     
        public class LogMobileAppUnapprovalByUpdate : MobileAppLoggableActionBase
        {
            public override string ActionToString()
            {
                return "Unapproved because of update";
            }
        }

  5. Larry
    Larry avatar
    1 posts
    Member since:
    Jan 2012

    Posted 13 Feb 2012 Link to this post

    Mehfuz,

    I am having the same problem.  I can send the code, as I am only writing it to evaluate the product and to do a training session for the other developers here.  I also have another problem that I am seeing, though I am not 100% sure if I am doing it correctly.

    -Larry
  6. Ricky
    Admin
    Ricky avatar
    467 posts

    Posted 15 Feb 2012 Link to this post

    Hi Larry,
    Thanks for keeping us informed about the problem. Also, thanks in advance for sharing your samples. This will let me investigate the problem and send you a possible solution.

    However, this can happen due to the ordering of the same method used in different tests; since by design .net profiler can intercept a member only once during OnJITCompilationStarted and if it is invoked manually first then prevents JustMock from intercepting it. However even if you have such tests, it is possible to pre-initialize the member using Mock.Initialize{T} to resolve this side effect.

    In order to make sure that Mock.Initalize{T} is called before any other tests are being executed. You can put it inside a static block that is invoked by .net framework before any other members in the class.

    For example:

    static TestClass()
    {
        Mock.Partial<MyClass>().For((x) => x.Delete());
    }


    Once after the above block, the following two consecutive tests will work as expected; despite the first one calls the Delete method directly:

    [TestMethod]
    public void  Test1()
    {
       new MyClass().Delete();
    }
      
    [TestMethod]
    public void  Test2()
      
    {
      
       var myClass = Mock.Create<MyClass>();
       Mock.Arrange(() => myClass.Delete()).DoNothing().MustBeCalled();
     
       myClass.Delete();
       
       Mock.Assert(myClass);  
    }



    Hope this helps to solve the issue.


    Kind regards,
    Mehfuz
    the Telerik team
    Sharpen your .NET Ninja skills! Attend Q1 webinar week and get a chance to win a license! Book your seat now >>
  7. Dominic Faraday
    Dominic Faraday avatar
    4 posts
    Member since:
    Jan 2010

    Posted 17 Feb 2012 Link to this post

    Using Mock.Initialize<> did not solve the issue

            [TestMethod]
            public void App_logs_are_persisted()
            {
                LogMobileAppActionMapper logAppMapper = Mock.Create<LogMobileAppActionMapper>();
                JsonUpdater updater = new JsonUpdater(nullnull, logAppMapper, null);
                Mock.Arrange(() => logAppMapper.SaveCollection(Arg.IsAny<List<MobileAppLoggableActionBase>>())).MustBeCalled();
                updater.PersistAppLogEntries();
     
                Mock.Assert(logAppMapper);
            }
    
            
    public
     class JsonUpdater : IJsonUpdater   { ...  
               public void PersistAppLogEntries()
               {
                   _logMobileAppActionMapper.SaveCollection(MobileAppLogEntries);
               }
            }
    
    
    The above test fails in large test runs stating the method which must be called isn't. This is the only unit test which calls
    logAppMapper.SaveCollection and updater.PersistAppLogEntries so I cannot see how other tests could break this. Also I have had this occur
    with a test class that had only one test method. 
    
  8. Ricky
    Admin
    Ricky avatar
    467 posts

    Posted 20 Feb 2012 Link to this post

    Hi Dominic,

    Thanks for contacting us regarding the matter. Since you are also having this problem with a test class that has only one test method, I would request you to send me a sample project with the same so that I can further investigate the problem. I need to debug through the issue to identify the actual cause and will send you a possible solution.


    Kind Regards,
    Mehfuz
    the Telerik team
    Sharpen your .NET Ninja skills! Attend Q1 webinar week and get a chance to win a license! Book your seat now >>
  9. Dominic Faraday
    Dominic Faraday avatar
    4 posts
    Member since:
    Jan 2010

    Posted 21 Feb 2012 Link to this post

    Hi Mehfuz,

    I cannot supply a code sample where this error occurs. The error occurs regularly only with large test runs. It would require me to send you the entire solution which is 500mb in size. I have tried extracting a snippet from it but the error no longer occurs. I don't think there is much value in sending you a test project which does not produce this bug.

    After a discussion with my manager we will be happy to let you remotely connect to one of our development pcs and investigate the error.
  10. Ricky
    Admin
    Ricky avatar
    467 posts

    Posted 24 Feb 2012 Link to this post

    Hi Dominic,
    Thanks again for contacting us.  If you want me to investigate the problem by connecting remotely, you can always reach me at  mehfuz.hossain att telerik dot com (making the email spam safe).  However, please also try the latest version and see whether it solves your issue. 


    Kind Regards,
    Mehfuz
    the Telerik team
    Sharpen your .NET Ninja skills! Attend Q1 webinar week and get a chance to win a license! Book your seat now >>
Back to Top
DevCraft R3 2016 release webinar banner