This is a migrated thread and some comments may be shown as answers.

Error converting dump files

4 Answers 122 Views
General Discussions
This is a migrated thread and some comments may be shown as answers.
This question is locked. New answers and comments are not allowed.
Pierre-Andre van Leeuwen
Top achievements
Rank 1
Pierre-Andre van Leeuwen asked on 24 Mar 2014, 08:52 PM
Hi
I have a few dump files (.jtdump) that were created on our productions server - Windows 2008 R2. The process is a 64 bit .NET service. The dump files range from 500MB to 3GB in size. I have the Debugging Tools for Windows installed. On my workstation, when I try to convert the dump file to a snapshot in JustTrace, I get the following error message: The following files were not converted: xxxxx. Reason: Input string was not in a correct format. I looks like it can be opened on the server, but unfortunately it doesn't have enough memory available to complete the process. I also opened the dump files on my workstation using windbg.exe and that worked, so they seem to be intact.

Environment:
OS: Windows 8.1 (X64)
JustTrace: Version Q1 2014 Build 2014.1.1425.4
.NET: 4.5.1
Windbg.exe: 6.3.9600.16384 AMD 64

Any suggestions?

4 Answers, 1 is accepted

Sort by
0
Pierre-Andre van Leeuwen
Top achievements
Rank 1
answered on 25 Mar 2014, 08:10 AM
I see from the log file that there are a number of symbol load errors. I can also see that the symbol path passed to cdb is not the one I configured in the JustTrace options.

Here is the content of the log file:

2014-03-25 10:03:10: <Environment>: arch=x64; memory (MB)=16308; running in VM=False; OS=Microsoft Windows NT 6.3.9600.0;
2014-03-25 10:03:11: <Page>: id='Backstage.StartHere', referrer='', ui='Backstage', profilee='', profiler=''
2014-03-25 10:03:14: <Page>: id='Backstage.Options', referrer='Backstage.StartHere', ui='Backstage', profilee='', profiler=''
2014-03-25 10:03:20: <Page>: id='Backstage.OpenSnapshots', referrer='Backstage.Options', ui='Backstage', profilee='', profiler=''
2014-03-25 10:03:20: <Event>: category='Open Snapshots', id='Import Snapshots', value='', page='Backstage.OpenSnapshots', ui='Toolbar', profilee='', profiler=''
2014-03-25 10:03:30: CDB Output:
2014-03-25 10:03:30: Microsoft (R) Windows Debugger Version 6.3.9600.16384 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Temp\Gendac.Sefeko.V1.Server 2014-03-24 18-02-44 5364.jtdump]
User Mini Dump File with Full Memory: Only application data is available
Dir entry 2, FunctionTableStream stream has too many elements (0x1469 > 0x3e8)
Comment: 'This dump is done with Telerik JustTrace'
************* Symbol Path validation summary **************
Response                         Time (ms)     Location
OK                                             d:\PvLUserData\Temp\Telerik\JustTrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}
Symbol search path is: d:\PvLUserData\Temp\Telerik\JustTrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Machine Name:
Debug session time: Mon Mar 24 20:03:55.000 2014 (UTC + 2:00)
System Uptime: 5 days 11:04:18.601
Process Uptime: 0 days 4:48:22.000
................................................................
................................................................
.........................
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll -
************* Symbol Loading Error Summary **************
Module name            Error
ntdll                  PDB not found : d:\pvluserdata\temp\telerik\justtrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}\symbols\dll\ntdll.pdb
You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for KERNELBASE.dll -
ntdll!NtWaitForSingleObject+0xa:
00000000`777b12fa c3              ret
0:000> 0:000> *** ERROR: Symbol file could not be found.  Defaulted to export symbols for clr.dll -
************* Symbol Loading Error Summary **************
Module name            Error
clr                    PDB not found : d:\pvluserdata\temp\telerik\justtrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}\symbols\dll\clr.pdb
You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
PDB symbol for clr.dll not loaded
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is
                in the version directory or on the symbol path
            3) or, if you are debugging a dump file, verify that the file
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on supported cross platform architecture as
                the dump file. For example, an ARM dump file must be debugged
                on an X86 or an ARM machine; an AMD64 dump file must be
                debugged on an AMD64 machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.
0:000> #JTStart DumpDomain
0:000> The version of SOS does not match the version of CLR you are debugging.  Please
load the matching version of SOS for the version of CLR you are debugging.
CLR Version: 4.0.30319.18444
SOS Version: 4.0.30319.34011
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is
                in the version directory or on the symbol path
            3) or, if you are debugging a dump file, verify that the file
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on supported cross platform architecture as
                the dump file. For example, an ARM dump file must be debugged
                on an X86 or an ARM machine; an AMD64 dump file must be
                debugged on an AMD64 machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.
0:000> #JTEnd DumpDomain
0:000> #JTStart GcHandles
0:000> The version of SOS does not match the version of CLR you are debugging.  Please
load the matching version of SOS for the version of CLR you are debugging.
CLR Version: 4.0.30319.18444
SOS Version: 4.0.30319.34011
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is
                in the version directory or on the symbol path
            3) or, if you are debugging a dump file, verify that the file
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on supported cross platform architecture as
                the dump file. For example, an ARM dump file must be debugged
                on an X86 or an ARM machine; an AMD64 dump file must be
                debugged on an AMD64 machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.
0:000> #JTEnd GcHandles
0:000> #JTStart DumpHeap
0:000> The version of SOS does not match the version of CLR you are debugging.  Please
load the matching version of SOS for the version of CLR you are debugging.
CLR Version: 4.0.30319.18444
SOS Version: 4.0.30319.34011
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is
                in the version directory or on the symbol path
            3) or, if you are debugging a dump file, verify that the file
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on supported cross platform architecture as
                the dump file. For example, an ARM dump file must be debugged
                on an X86 or an ARM machine; an AMD64 dump file must be
                debugged on an AMD64 machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.
0:000> #JTEnd DumpHeap
Microsoft (R) Windows Debugger Version 6.3.9600.16384 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Temp\Gendac.Sefeko.V1.Server 2014-03-24 18-02-44 5364.jtdump]
User Mini Dump File with Full Memory: Only application data is available
Dir entry 2, FunctionTableStream stream has too many elements (0x1469 > 0x3e8)
Comment: 'This dump is done with Telerik JustTrace'
************* Symbol Path validation summary **************
Response                         Time (ms)     Location
OK                                             d:\PvLUserData\Temp\Telerik\JustTrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}
Symbol search path is: d:\PvLUserData\Temp\Telerik\JustTrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Machine Name:
Debug session time: Mon Mar 24 20:03:55.000 2014 (UTC + 2:00)
System Uptime: 5 days 11:04:18.601
Process Uptime: 0 days 4:48:22.000
................................................................
................................................................
.........................
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll -
************* Symbol Loading Error Summary **************
Module name            Error
ntdll                  PDB not found : d:\pvluserdata\temp\telerik\justtrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}\symbols\dll\ntdll.pdb
You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for KERNELBASE.dll -
ntdll!NtWaitForSingleObject+0xa:
00000000`777b12fa c3              ret
0:000> 0:000> *** ERROR: Symbol file could not be found.  Defaulted to export symbols for clr.dll -
************* Symbol Loading Error Summary **************
Module name            Error
clr                    PDB not found : d:\pvluserdata\temp\telerik\justtrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}\symbols\dll\clr.pdb
You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
PDB symbol for clr.dll not loaded
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is
                in the version directory or on the symbol path
            3) or, if you are debugging a dump file, verify that the file
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on supported cross platform architecture as
                the dump file. For example, an ARM dump file must be debugged
                on an X86 or an ARM machine; an AMD64 dump file must be
                debugged on an AMD64 machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.
0:000> #JTStart TraverseHeap
0:000> The version of SOS does not match the version of CLR you are debugging.  Please
load the matching version of SOS for the version of CLR you are debugging.
CLR Version: 4.0.30319.18444
SOS Version: 4.0.30319.34011
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is
                in the version directory or on the symbol path
            3) or, if you are debugging a dump file, verify that the file
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on supported cross platform architecture as
                the dump file. For example, an ARM dump file must be debugged
                on an X86 or an ARM machine; an AMD64 dump file must be
                debugged on an AMD64 machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.
0:000> #JTEnd TraverseHeap2014-03-25 10:03:30: System.FormatException: Input string was not in a correct format.
   at System.Number.ParseUInt64(String value, NumberStyles options, NumberFormatInfo numfmt)
   at Telerik.JustTrace.Api.Data.Memory.ClrProfilerLogDataLoader`1.ProcessRootRefsMessage(Int64 offset) in c:\J\jobs\JT_CI\workspace\JustTrace\JustTrace\Development\Telerik.JustTrace.Api\Data\Memory\ClrProfilerLogDataLoader.cs:line 270
   at Telerik.JustTrace.Api.Data.Memory.ClrProfilerLogFileProcessor.ReadTypeStats(StreamReader reader) in c:\J\jobs\JT_CI\workspace\JustTrace\JustTrace\Development\Telerik.JustTrace.Api\Data\Memory\ClrProfilerLogFileProcessor.cs:line 119
   at Telerik.JustTrace.Api.Data.Memory.ClrProfilerLogFileProcessor.ProcessLog() in c:\J\jobs\JT_CI\workspace\JustTrace\JustTrace\Development\Telerik.JustTrace.Api\Data\Memory\ClrProfilerLogFileProcessor.cs:line 46
   at Telerik.JustTrace.Api.Session.Profilers.DumpReader.ProcessClrProfilerTraverseHeapFile(String clrProfilerLogfile, DateTime timestamp, Boolean is64Bit) in c:\J\jobs\JT_CI\workspace\JustTrace\JustTrace\Development\Telerik.JustTrace.Api\Session\Profilers\DumpReader.cs:line 764
   at Telerik.JustTrace.Api.Session.Profilers.DumpReader.GetSnapshotFromDump(String dumpfile, DateTime timestamp, TimeSpan timeout, String debuggingToolsForWindowsFolder) in c:\J\jobs\JT_CI\workspace\JustTrace\JustTrace\Development\Telerik.JustTrace.Api\Session\Profilers\DumpReader.cs:line 273
   at Telerik.JustTrace.SnapshotsStartPagePresenter.#8ld(String #9ld, String #7ld, DateTime #Ze, Int32 #bo) in c:\J\jobs\JT_CI\workspace\JustTrace\JustTrace\Development\JustTrace\Modules\Snapshots\StartPageWidget\SnapshotsStartPagePresenter.cs:line 472
   at Telerik.JustTrace.SnapshotsStartPagePresenter.#CZ.#Rpd() in c:\J\jobs\JT_CI\workspace\JustTrace\JustTrace\Development\JustTrace\Modules\Snapshots\StartPageWidget\SnapshotsStartPagePresenter.cs:line 343
2014-03-25 10:03:33: <Page>: id='Inactive', referrer='Backstage.OpenSnapshots', ui='Unspecified', profilee='', profiler=''

0
Martin
Telerik team
answered on 27 Mar 2014, 11:28 AM
Hello Pierre,

It seems that there's a bug in JustTrace. Opening a dump fails when the machine from which the dump is taken and the machine on which it is being opened have different CLR versions.

We will fix it in one of our next public releases. In the meantime you can work around it by installing JustTrace on a machine with the same OS and CLR versions. Note that some Windows Updates modify the CLR version, so having the same updates installed might also be necessary. 

I am sorry for the inconvenience that this issue is causing you. To thank you for reporting it I have added some Telerik points to your account.

Kind regards,
Martin
Telerik
 

Build cross-platform mobile apps using Visual Studio and .NET. Register for the online webinar on 03/27/2014, 11:00AM US ET.. Seats are limited.

0
Pierre-Andre van Leeuwen
Top achievements
Rank 1
answered on 27 Mar 2014, 12:47 PM
Thanks for the response. I decompiled JustTrace using JustDecompile and saw that cdb.exe was passed the command: .loadby sos clr. Perhaps adding and option to JustTrace where one can specify an alternative location for sos.dll and mscordacwks.dll would solve the problem.

I finally managed to simulate the memory leak on a test machine and managed to find the leak using JustTrace, so all is well thank you!
0
Martin
Telerik team
answered on 27 Mar 2014, 12:54 PM
Hi again,

I'm glad that you've managed to resolve the issue.

Thank you for the suggestion. We actually embed the mscordacwks.dll and sos.dll files inside the dump file and do not need such an option. We just need to fix our code to load them from there.

Regards,
Martin
Telerik
 

Build cross-platform mobile apps using Visual Studio and .NET. Register for the online webinar on 03/27/2014, 11:00AM US ET.. Seats are limited.

 
Tags
General Discussions
Asked by
Pierre-Andre van Leeuwen
Top achievements
Rank 1
Answers by
Pierre-Andre van Leeuwen
Top achievements
Rank 1
Martin
Telerik team
Share this question
or