Mock.Create performance issue with a large interface

[J]

Hello,

I want to mock a large interface (~630 methods). The following line takes several minutes and several GBytes of memory:

Dim mJob As IJobInterface = Mock.Create(Of IJobInterface)()

Am I doing something wrong or is there another way to mock that object. I need this mock for every unit test so I need a faster way to do this.

Cheers

[Stefan]

Hello J,

I tried mocking an interface with 1000 methods, each with 11 parameters, and it took only 3 seconds for the mock to be created.

public interface IHugeInterface
{
    void Do0(object x, int a0, int a1, int a2, int a3, int a4, int a5, int a6, int a7, int a8, int a9);
    void Do1(object x, int a0, int a1, int a2, int a3, int a4, int a5, int a6, int a7, int a8, int a9);
    void Do2(object x, int a0, int a1, int a2, int a3, int a4, int a5, int a6, int a7, int a8, int a9);

....

Could you share more details about that IJobInterface that takes so much time? If it's not a problem, you could just send us its source and the source of all dependencies.

Regards,
Stefan
Telerik

[J]

Hello Stefan,

sorry I can't give you the source code but i have the interop.dll for you. IJobInterface is an Interface of a C++ COM class. Add it as a reference and import the e3 namespace.

ftp://e3-delivery:vK5Bj7sPSLCe9yj@194.39.140.20/20131028-0e1d92da-a6ff-4448-aed5-ae212227d687/e3.Interop.dll

There are a lot more interfaces inside just ignore them.

Greetings

[Stefan]

Hi J,

I referenced your interop assembly and tried the following code:

[TestMethod]
public void TestPerformance()
{
    var t = DateTime.UtcNow;
    var mock = Mock.Create<IJobInterface>();
    Mock.Arrange(() => mock.GetSignalCount()).Returns(123);
    Assert.Equal(123, mock.GetSignalCount());
    var t2 = DateTime.UtcNow - t;
}

On my machine, the test passes in about 1.5 seconds. I tried both with and without the profiler enabled. Could I ask you to profile the unit test and see what takes so much time?

Regards,
Stefan
Telerik

[J]

Hi Stefan,

after some test runs I've figured out the following behaivior for your TestPerformance method:

1. Run unit test in VS08 - duration 20 seconds, almost no memory consumption
2. Debug unit test in VS08: duration 20 minutes then OutOfMemoryException

The run with the ANTS Performance profiler leads to the same result as 1. (20s, no memory consumption). But it still takes much longer than 1.5s on your machine..
Also I figured out that high memory consumption cames from the VS08 process (devenv.exe) neither VSTestHost.exe nor MSTest.exe.

I couldn't profile the high memory consumption issue if you know a way to do this please tell me. I've attached a profiler screenshot of the 20s run.

Cheers

PS: Sorry the first attachment was the wrong screenshot but it's quite the same.. 20sRun-New.png is the right one.

[Stefan]

Hi J,

I was able to reproduce the 20s test run time. It appears to happen only when running in the .NET 2.0/3.5 runtime. On .NET 4.0 there is no such problem.

The good news is that it happens only for the first mock created for a given type. There was almost no difference in run time between creating one mock of IJobInterface and a hundred. In other words, the 20s performance penalty is constant, regardless of the number of unit tests run.

I'm still investigating the root cause of this issue. At this point I cannot promise that the performance of Mock.Create<T> for interfaces with a large number of members running on .NET 3.5 will be improved.

If the high memory consumption is coming from devenv itself, then a profiler will probably be of little use. Your test is probably hitting some hot path in the debugger or some VS add-in. I suggest you turn to general troubleshooting techniques to resolve this issue, as it appears to be unrelated to JustMock itself. I successfully debugged the test running on .NET 3.5 with the VS2013 debugger. So it seems that it might be a bug in VS2008.

I hope that the above helps you move forward. If there's anything else that I can help you with, don't hesitate to write us back again.

Regards,
Stefan
Telerik

[J]

Hello Stefan,

thanks for your help. I've converted the solution to .NET 4.0 (VS2013) - the behaivior is more weird now :D

A test run fills the memory and takes about 1 minute now but at the end the debugger seem to crash. Everything in VS looks ok.. instead of CPU is doing nothing and the memory usage is cleared again from this point and I have to stop the test run manually.

Performance Profiler run leads to the same result as before but it runs now about 1 minute
I'm completely confused now..

Best Regards

[Stefan]

Hello J,

Upon further investigation it appears that the VS2013 debugger also runs out of memory, so it's not a VS version issue. Also, the difference in behavior between .NET 3.5 and .NET 4.0 was caused by the way interop types get embedded. If I run the test on .NET 4.0 without embedding the interop types, then the behavior is the same - a lot of waiting ensues. The same happens with a synthetic interface.

The main problem here is that IJobInterface doesn't have 600 members - it has 6000. It implements about 15 versions of the interface, each with about 600 members and the grand total of members to mock goes to about 6000. When I tried to mock a synthetically generated interface with 6000 members, the runtime went from a second to over a minute (compared to the same interface with 1000 members).

I will continue investigating both the debugger issue and the performance issue.

Regards,
Stefan
Telerik

[J]

Hello Stefan,

ok didn't know that all methods are implemented while mocking this object. It would actually be enough if only the newest version of a method is implemented here at least in this case...

Meanwhile I tried some other mocking frameworks - the job object is always the problem except for RhinoMocks.
With the DynamicMockWithRemoting class IJobInterface is mocked in no time. Although I have to ask myself what this mock actually doesn't have compared to the others.

Greetings

[Kaloyan]

Hello J,

Thank you for your help.

Unfortunately, we will require some additional time in order to investigate the issue. We will notify you as soon as we can provide further information.

Thank you for the understanding.

Regards,
Kaloyan
Telerik

[Stefan]

Hello J,

In the next JustMock release (due this week) you will be able to filter which methods should be intercepted. Here's a sample how to do it:

var mock = Mock.Create<e3.IJobInterface>(conf => {
    conf.SetInterceptorFilter(mi => mi.DeclaringType == typeof(e3.IJobInterface));
});

In the above code I specified that I only want to intercept methods that are declared by the IJobInterface itself, and not by any of the old version interfaces that it inherits from. The above code drastically reduces the time to create the mock type as well as the memory usage in the VS debugger when stepping over.

I sincerely hope that this helps your scenario.

Regards,
Stefan
Telerik

[J]

Hi Stefan,

ok sounds good. I will test it.

Best regards

[J]

Hello Stefan,

is it already released? Where can I find it?

Regards

[Todor]

Hi J,

During the QA stage we discovered a blocking bug and we had to postpone the release. The good news is that the fix is verified and we are releasing next Monday.

Regards,
Todor
Telerik

[J]

Hello

I could test it now but still get an exception:

Test method JustMockTestProject.JustMockTestClass.GetFlagnoteIdsWithJustMock threw exception:  System.ArgumentException: Do not supply contructor arguments when mocking an interface or delegate..

   at Telerik.JustMock.Core.MocksRepository.Create(Type type, MockCreationSettings settings)
   at Telerik.JustMock.MockBuilder.Create(MocksRepository repository, Type type, Object[] constructorArgs, Nullable`1 behavior, Type[] additionalMockedInterfaces, Nullable`1 mockConstructorCall, IEnumerable`1 additionalProxyTypeAttributes, List`1 supplementaryBehaviors, List`1 fallbackBehaviors, List`1 mixins, Expression`1 interceptorFilter)
   at Telerik.JustMock.Mock.<>c__DisplayClass69`1.<Create>b__68()
   at Telerik.JustMock.Core.ProfilerInterceptor.GuardInternal[T](Func`1 guardedAction)
   at Telerik.JustMock.Mock.Create[T](Object[] args)
   at JustMockTestProject.JustMockTestClass.GetFlagnoteIdsWithJustMock() in JustMockTestClass.vb:line 56

I tried the line Stefan mentioned...

var mock = Mock.Create<e3.IJobInterface>(conf => {
    conf.SetInterceptorFilter(mi => mi.DeclaringType == typeof(e3.IJobInterface));
});

..and converted it to VB:

mJob = Mock.Create(Of IJobInterface)(Function(conf) conf.SetInterceptorFilter(Function(mi) mi.DeclaringType Is GetType(e3.IJobInterface)))

Is that correct?

Cheers

[Kaloyan]

Hello J,

The VB syntax should use Sub instead of Function. Like this:

mJob = Mock.Create(Of IJobInterface)(Sub(conf) conf.SetInterceptorFilter(Function(mi) mi.DeclaringType Is GetType(e3.IJobInterface)))

Please, let me know if this solves the issue.

Regards,
Kaloyan
Telerik

[J]

Hello Kaloyan,

unfortunately it doesn't. .NET 3.5 VB syntax does not allow 'Subs' in lambda expressions...

Best regards

[Kaloyan]

Hello J,

I just tried the following test inside a .NET 3.5 VB test project, referring the e3 assembly:

<TestMethod()>
Public Sub TestMethod1()
    Dim mJob = Mock.Create(Of IJobInterface)(Sub(conf) conf.SetInterceptorFilter(Function(mi) mi.DeclaringType Is GetType(e3.IJobInterface)))
End Sub

It compiled and passed inside Visual Studio 2012.

I assume the compiler of VS2008 does not allow the Sub usage inside .NET 3.5 VB lambda expressions. Can you please check if this is passing on your side in VS2010 or later?

Thanks.

Regards,
Kaloyan
Telerik

[J]

Hello Kaloyan,

i've finally managed to test it :D
It works with VS2013 (.NET 4.0) .. still takes approximately 30 seconds in debug mode but it's much better then 20 minutes.

Thank you all for help.
Best regards.

[Stefan]

Hello J,

I'm glad that the problem is now resolved to your satisfaction. If there's anything else you need help with, don't hesitate to contact us again.

Regards,
Stefan
Telerik