JustMock Profiler on Linux - Profiler not enabled

[Sandro]

Hi there

We are trying to move our current build agents from Windows to Linux (Ubuntu 20.04) for all our dotnet6.0 applications and trying to integrate the JustMock Profiler on this machine because we depend on it in several tests.

Therefore, we followed your integration article "JustMock Integration on Linux".

We have copied the Profiler binaries to the build agent and referencing it through the environment variable:

root@8d248b92b005:/azp/Backend# printenv | grep 'CORE\|JUST'
CORECLR_PROFILER_PATH=/opt/TelerikJustMock/JustMock_2022_1_223_1_Dev/Libraries/CodeWeaver/64/libTelerik.CodeWeaver.Profiler.so
CORECLR_PROFILER={B7ABE522-A68F-44F2-925B-81E7488E9EC0}
JUSTMOCK_INSTANCE=1
CORECLR_ENABLE_PROFILING=1

The csproj definition for referencing the JustMock library looks (at the moment) like the following:

...
<ItemGroup>
    <Reference Include="Telerik.JustMock">
        <HintPath Condition="Exists('C:\Program Files (x86)\Progress\Telerik JustMock\Libraries\netcoreapp2.0\Telerik.JustMock.dll')">C:\Program Files (x86)\Progress\Telerik JustMock\Libraries\netcoreapp2.0\Telerik.JustMock.dll</HintPath>
        <HintPath Condition="Exists('/opt/TelerikJustMock/JustMock_2022_1_223_1_Dev/Libraries/netcoreapp2.0/Telerik.JustMock.dll')">/opt/TelerikJustMock/JustMock_2022_1_223_1_Dev/Libraries/netcoreapp2.0/Telerik.JustMock.dll</HintPath>
        <Private>True</Private>
    </Reference>
</ItemGroup>
...

In the tests we are calling the following method to check for the Profiler to be enabled:

public class Test_JustMockProfiler_Enabled
{
	[Fact]
	public void ProfilerMustBeEnabled()
	{
		Mock.IsProfilerEnabled.Should().BeTrue();
	}
}

Then running the tests with the 'dotnet test' command:

dotnet test /path/to/Tests.dll --no-build --no-restore -l:trx --results-directory ./TestResults/

But still we get the following error:

Error Message:
   Expected Mock.IsProfilerEnabled to be true, but found False.

Installed dotnet:

root@8d248b92b005:/azp/Backend# dotnet --info
.NET SDK (reflecting any global.json):
 Version:   6.0.402
 Commit:    6862418796

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  20.04
 OS Platform: Linux
 RID:         ubuntu.20.04-x64
 Base Path:   /usr/share/dotnet/sdk/6.0.402/

global.json file:
  Not found

Host:
  Version:      6.0.10
  Architecture: x64
  Commit:       5a400c212a

.NET SDKs installed:
  6.0.402 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.10 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.10 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

 

Is there anything we are missing or doing wrong? 
How do we properly enable the Profile on Linux to run in 'dotnet test'?

Thanks for your help in advance, cheers.

[Ivo]

Hello Sandro,

Everything seems to be correct, but let us check the profiler shared library dependencies using ldd. Please provide the output from the command line.

Regards,
Ivo
Progress Telerik

[Sandro]

Hi Ivo,

Here is the output:

root@79d902e9c4ea:/azp# ls -la /opt/TelerikJustMock/JustMock_2022_3_912_3_Dev/Libraries/CodeWeaver/64/
total 6540
drwxr-xr-x 2 root root    4096 Oct 21 13:25 .
drwxr-xr-x 3 root root    4096 Oct 21 13:25 ..
-rwxr-xr-x 1 root root       0 Oct 21 13:17 libTelerik.CodeWeaver.Profiler.so
-rwxr-xr-x 1 root root       0 Oct 21 13:17 libTelerik.CodeWeaver.Profiler.so.2022
-rwxr-xr-x 1 root root 6688512 Sep 12 15:12 libTelerik.CodeWeaver.Profiler.so.2022.3.912.3
root@79d902e9c4ea:/azp# ldd /opt/TelerikJustMock/JustMock_2022_3_912_3_Dev/Libraries/CodeWeaver/64/libTelerik.CodeWeaver.Profiler.so
        not a dynamic executable
root@79d902e9c4ea:/azp# ldd /opt/TelerikJustMock/JustMock_2022_3_912_3_Dev/Libraries/CodeWeaver/64/libTelerik.CodeWeaver.Profiler.so.2022
        not a dynamic executable
root@79d902e9c4ea:/azp# ldd /opt/TelerikJustMock/JustMock_2022_3_912_3_Dev/Libraries/CodeWeaver/64/libTelerik.CodeWeaver.Profiler.so.2022.3.912.3
        linux-vdso.so.1 (0x00007ffd5bd21000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f39a3ef8000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f39a3ef2000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f39a3d10000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f39a3bc1000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f39a3ba6000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f39a39b4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f39a421a000)

Anything wrong with that?

I also checked if these shared libraries are available on the system and it seems we are missing some of them, respectively one is found at a different path:

• linux-vdso.so.1
• /lib64/ld-linux-x86-64.so.2

root@79d902e9c4ea:/azp# ldconfig -p | grep 
root@79d902e9c4ea:/azp# ldconfig -p | grep libpthread.so.0
        libpthread.so.0 (libc6,x86-64, OS ABI: Linux 3.2.0) => /lib/x86_64-linux-gnu/libpthread.so.0
root@79d902e9c4ea:/azp# ldconfig -p | grep libdl.so.2
        libdl.so.2 (libc6,x86-64, OS ABI: Linux 3.2.0) => /lib/x86_64-linux-gnu/libdl.so.2
root@79d902e9c4ea:/azp# ldconfig -p | grep libstdc++.so.6
        libstdc++.so.6 (libc6,x86-64) => /lib/x86_64-linux-gnu/libstdc++.so.6
root@79d902e9c4ea:/azp# ldconfig -p | grep libm.so.6
        libm.so.6 (libc6,x86-64, OS ABI: Linux 3.2.0) => /lib/x86_64-linux-gnu/libm.so.6
root@79d902e9c4ea:/azp# ldconfig -p | grep libgcc_s.so.1
        libgcc_s.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libgcc_s.so.1
root@79d902e9c4ea:/azp# ldconfig -p | grep libc.so.6
        libc.so.6 (libc6,x86-64, OS ABI: Linux 3.2.0) => /lib/x86_64-linux-gnu/libc.so.6
root@79d902e9c4ea:/azp# ldconfig -p | grep /lib64/ld-linux-x86-64.so.2
root@79d902e9c4ea:/azp# ldconfig -p | grep ld-linux-x86-64.so.2
        ld-linux-x86-64.so.2 (libc6,x86-64) => /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2

I hope this helps to give us the right advise to get it working. :)

Am I right, that the libTelerik.CodeWeaver.Profiler.so and libTelerik.CodeWeaver.Profiler.so.2022 are just symlinks to the effective binary libTelerik.CodeWeaver.Profiler.so.2022.3.912.3?

Thanks for your help. 

[Ivo]

Yes, exactly. Upon extraction, they have lost their targets and that is the reason for failure. As far as I know, there is a targz option to preserve the symlinks. Alternatively, you can just recreate them manually in the following way:

libTelerik.CodeWeaver.Profiler.so -> libTelerik.CodeWeaver.Profiler.so.2022

libTelerik.CodeWeaver.Profiler.so.2022 -> libTelerik.CodeWeaver.Profiler.so.2022.3.912.3

 

[Sandro]

Many thanks, the missing symlinks solved the issue.

I used now the following command to extract the tarball:

tar -h --overwrite -xvf ./JustMock_2022_1_223_1_Dev.tar.gz

Now, my test is passing.

Cheers

[Ivo]

That is great, I am happy to hear that the issue is finally resolved.

[Sandro]

Hi Ivo

We are running now the tasks in a Azure DevOps pipeline and face now the issue that we have an undefined error after the pipeline is done (tests are performed successfully!):

Our setup is like we discussed above (Linux Docker Container), so we have the profiler installed at: 

/opt/TelerikJustMock/JustMock_2022_3_11009_1_Dev/Libraries/CodeWeaver/64/libTelerik.CodeWeaver.Profiler.so

And we have the following environment variables set:

CORECLR_PROFILER={B7ABE522-A68F-44F2-925B-81E7488E9EC0}
CORECLR_PROFILER_PATH=/opt/TelerikJustMock/JustMock_2022_3_11009_1_Dev/Libraries/CodeWeaver/64/libTelerik.CodeWeaver.Profiler.so
CORECLR_ENABLE_PROFILING=1
JUSTMOCK_INSTANCE=1

This error pops up no matter if the profiles is used within a test task in a pipline run. Still I paste the test task in here for completeness:

- task: DotNetCoreCLI@2
  displayName: 'dotnet test'
  inputs:
    command: 'test'
    configuration: 'Release'
    projects: |
      **/*.Tests.dll
      !**/*TestAdapter.dll
      !**/obj/**
      !**/bin/**/ref/**
    nobuild: true
    publishTestResults: true
    arguments: '/Framework:.NETCoreApp,Version=v6.0'          
  enabled: 'true'

We also checked the Agent and Worker logs in the _diag folder of the Azure DevOps agent but could not find any error message, even in diagnostics mode and debugging enabled.

Is there anything wrong with our setup?

What is meant with the documentation stating "[extracted-files-path]runtimeslinux-x64native" for the path:

CORECLR_PROFILER_PATH = [extracted-files-path]runtimeslinux-x64nativelibTelerik.CodeWeaver.Profiler.so

Thanks for helping.

 

[Ivo]

It looks really strange indeed, it is hard to be found anything wrong since the setup looks fine to me at a first glance. It might be a shot in the dark, but try to disable the steps starting from the end of the pipeline until the build succeeds, this way you could isolate the failing task and continue digging further into details.

[Sandro]

Hi Ivo

Thanks for your answer.

Yes I agree, it is very strange. And I already worked through your approach with disabling task by task but we always get the error. What I found out was that, as soon as I do not start the container with the four needed environment variables (CORECLR_PROFILER, CORECLR_PROFILER_PATH, CORECLR_ENABLE_PROFILING and JUSTMOCK_INSTANCE) to be set with the corresponding values, the error is not popping up during the pipeline run. Sure, the tests using the JustMock profiles will then fail, but at least I have no undefined error for the pipeline run.

Is there any dependency which I may miss for the JustMock profiler to work?

Running the tests with the explicit command on the agent directly will also not produce any error:

/usr/bin/dotnet test /azp/_work/1/s/tests/JustMockTest.Tests/bin/Release/net6.0/JustMockTest.Tests.dll --logger trx --results-directory /azp/_work/_temp /Framework:.NETCoreApp,Version=v6.0

It would be great if you could investigate and see if you find any bug for this. If you need any logs, I could provide those.

Thanks.

 

[Ivo]

Probably the exact moment of setting up the profiler environment is important. If the variables are set for the whole container it could be a source of issues since each process hosting the CLR will be instrumented. My recommendation is to limit the scope of profiler variables just for the test runner, one possible option is to use runsettings, take a look at this article.

[Sandro]

Thanks for this recommendation, solved the problem. Superb support!

[Ivo]

Glad to hear that, happy mocking!