Error converting dump files

5 posts, 0 answers
  1. Pierre-Andre van Leeuwen
    Pierre-Andre van Leeuwen avatar
    12 posts
    Member since:
    Feb 2010

    Posted 24 Mar 2014 Link to this post

    Hi
    I have a few dump files (.jtdump) that were created on our productions server - Windows 2008 R2. The process is a 64 bit .NET service. The dump files range from 500MB to 3GB in size. I have the Debugging Tools for Windows installed. On my workstation, when I try to convert the dump file to a snapshot in JustTrace, I get the following error message: The following files were not converted: xxxxx. Reason: Input string was not in a correct format. I looks like it can be opened on the server, but unfortunately it doesn't have enough memory available to complete the process. I also opened the dump files on my workstation using windbg.exe and that worked, so they seem to be intact.

    Environment:
    OS: Windows 8.1 (X64)
    JustTrace: Version Q1 2014 Build 2014.1.1425.4
    .NET: 4.5.1
    Windbg.exe: 6.3.9600.16384 AMD 64

    Any suggestions?
  2. Pierre-Andre van Leeuwen
    Pierre-Andre van Leeuwen avatar
    12 posts
    Member since:
    Feb 2010

    Posted 25 Mar 2014 Link to this post

    I see from the log file that there are a number of symbol load errors. I can also see that the symbol path passed to cdb is not the one I configured in the JustTrace options.

    Here is the content of the log file:

    2014-03-25 10:03:10: <Environment>: arch=x64; memory (MB)=16308; running in VM=False; OS=Microsoft Windows NT 6.3.9600.0;
    2014-03-25 10:03:11: <Page>: id='Backstage.StartHere', referrer='', ui='Backstage', profilee='', profiler=''
    2014-03-25 10:03:14: <Page>: id='Backstage.Options', referrer='Backstage.StartHere', ui='Backstage', profilee='', profiler=''
    2014-03-25 10:03:20: <Page>: id='Backstage.OpenSnapshots', referrer='Backstage.Options', ui='Backstage', profilee='', profiler=''
    2014-03-25 10:03:20: <Event>: category='Open Snapshots', id='Import Snapshots', value='', page='Backstage.OpenSnapshots', ui='Toolbar', profilee='', profiler=''
    2014-03-25 10:03:30: CDB Output:
    2014-03-25 10:03:30: Microsoft (R) Windows Debugger Version 6.3.9600.16384 AMD64
    Copyright (c) Microsoft Corporation. All rights reserved.
    Loading Dump File [C:\Temp\Gendac.Sefeko.V1.Server 2014-03-24 18-02-44 5364.jtdump]
    User Mini Dump File with Full Memory: Only application data is available
    Dir entry 2, FunctionTableStream stream has too many elements (0x1469 > 0x3e8)
    Comment: 'This dump is done with Telerik JustTrace'
    ************* Symbol Path validation summary **************
    Response                         Time (ms)     Location
    OK                                             d:\PvLUserData\Temp\Telerik\JustTrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}
    Symbol search path is: d:\PvLUserData\Temp\Telerik\JustTrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}
    Executable search path is:
    Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
    Product: Server, suite: Enterprise TerminalServer SingleUserTS
    Machine Name:
    Debug session time: Mon Mar 24 20:03:55.000 2014 (UTC + 2:00)
    System Uptime: 5 days 11:04:18.601
    Process Uptime: 0 days 4:48:22.000
    ................................................................
    ................................................................
    .........................
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll -
    ************* Symbol Loading Error Summary **************
    Module name            Error
    ntdll                  PDB not found : d:\pvluserdata\temp\telerik\justtrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}\symbols\dll\ntdll.pdb
    You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
    You should also verify that your symbol search path (.sympath) is correct.
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for KERNELBASE.dll -
    ntdll!NtWaitForSingleObject+0xa:
    00000000`777b12fa c3              ret
    0:000> 0:000> *** ERROR: Symbol file could not be found.  Defaulted to export symbols for clr.dll -
    ************* Symbol Loading Error Summary **************
    Module name            Error
    clr                    PDB not found : d:\pvluserdata\temp\telerik\justtrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}\symbols\dll\clr.pdb
    You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
    You should also verify that your symbol search path (.sympath) is correct.
    PDB symbol for clr.dll not loaded
    Failed to load data access DLL, 0x80004005
    Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
                2) the file mscordacwks.dll that matches your version of clr.dll is
                    in the version directory or on the symbol path
                3) or, if you are debugging a dump file, verify that the file
                    mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
                4) you are debugging on supported cross platform architecture as
                    the dump file. For example, an ARM dump file must be debugged
                    on an X86 or an ARM machine; an AMD64 dump file must be
                    debugged on an AMD64 machine.
    You can also run the debugger command .cordll to control the debugger's
    load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
    If that succeeds, the SOS command should work on retry.
    If you are debugging a minidump, you need to make sure that your executable
    path is pointing to clr.dll as well.
    0:000> #JTStart DumpDomain
    0:000> The version of SOS does not match the version of CLR you are debugging.  Please
    load the matching version of SOS for the version of CLR you are debugging.
    CLR Version: 4.0.30319.18444
    SOS Version: 4.0.30319.34011
    Failed to load data access DLL, 0x80004005
    Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
                2) the file mscordacwks.dll that matches your version of clr.dll is
                    in the version directory or on the symbol path
                3) or, if you are debugging a dump file, verify that the file
                    mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
                4) you are debugging on supported cross platform architecture as
                    the dump file. For example, an ARM dump file must be debugged
                    on an X86 or an ARM machine; an AMD64 dump file must be
                    debugged on an AMD64 machine.
    You can also run the debugger command .cordll to control the debugger's
    load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
    If that succeeds, the SOS command should work on retry.
    If you are debugging a minidump, you need to make sure that your executable
    path is pointing to clr.dll as well.
    0:000> #JTEnd DumpDomain
    0:000> #JTStart GcHandles
    0:000> The version of SOS does not match the version of CLR you are debugging.  Please
    load the matching version of SOS for the version of CLR you are debugging.
    CLR Version: 4.0.30319.18444
    SOS Version: 4.0.30319.34011
    Failed to load data access DLL, 0x80004005
    Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
                2) the file mscordacwks.dll that matches your version of clr.dll is
                    in the version directory or on the symbol path
                3) or, if you are debugging a dump file, verify that the file
                    mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
                4) you are debugging on supported cross platform architecture as
                    the dump file. For example, an ARM dump file must be debugged
                    on an X86 or an ARM machine; an AMD64 dump file must be
                    debugged on an AMD64 machine.
    You can also run the debugger command .cordll to control the debugger's
    load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
    If that succeeds, the SOS command should work on retry.
    If you are debugging a minidump, you need to make sure that your executable
    path is pointing to clr.dll as well.
    0:000> #JTEnd GcHandles
    0:000> #JTStart DumpHeap
    0:000> The version of SOS does not match the version of CLR you are debugging.  Please
    load the matching version of SOS for the version of CLR you are debugging.
    CLR Version: 4.0.30319.18444
    SOS Version: 4.0.30319.34011
    Failed to load data access DLL, 0x80004005
    Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
                2) the file mscordacwks.dll that matches your version of clr.dll is
                    in the version directory or on the symbol path
                3) or, if you are debugging a dump file, verify that the file
                    mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
                4) you are debugging on supported cross platform architecture as
                    the dump file. For example, an ARM dump file must be debugged
                    on an X86 or an ARM machine; an AMD64 dump file must be
                    debugged on an AMD64 machine.
    You can also run the debugger command .cordll to control the debugger's
    load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
    If that succeeds, the SOS command should work on retry.
    If you are debugging a minidump, you need to make sure that your executable
    path is pointing to clr.dll as well.
    0:000> #JTEnd DumpHeap
    Microsoft (R) Windows Debugger Version 6.3.9600.16384 AMD64
    Copyright (c) Microsoft Corporation. All rights reserved.
    Loading Dump File [C:\Temp\Gendac.Sefeko.V1.Server 2014-03-24 18-02-44 5364.jtdump]
    User Mini Dump File with Full Memory: Only application data is available
    Dir entry 2, FunctionTableStream stream has too many elements (0x1469 > 0x3e8)
    Comment: 'This dump is done with Telerik JustTrace'
    ************* Symbol Path validation summary **************
    Response                         Time (ms)     Location
    OK                                             d:\PvLUserData\Temp\Telerik\JustTrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}
    Symbol search path is: d:\PvLUserData\Temp\Telerik\JustTrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}
    Executable search path is:
    Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
    Product: Server, suite: Enterprise TerminalServer SingleUserTS
    Machine Name:
    Debug session time: Mon Mar 24 20:03:55.000 2014 (UTC + 2:00)
    System Uptime: 5 days 11:04:18.601
    Process Uptime: 0 days 4:48:22.000
    ................................................................
    ................................................................
    .........................
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll -
    ************* Symbol Loading Error Summary **************
    Module name            Error
    ntdll                  PDB not found : d:\pvluserdata\temp\telerik\justtrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}\symbols\dll\ntdll.pdb
    You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
    You should also verify that your symbol search path (.sympath) is correct.
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for KERNELBASE.dll -
    ntdll!NtWaitForSingleObject+0xa:
    00000000`777b12fa c3              ret
    0:000> 0:000> *** ERROR: Symbol file could not be found.  Defaulted to export symbols for clr.dll -
    ************* Symbol Loading Error Summary **************
    Module name            Error
    clr                    PDB not found : d:\pvluserdata\temp\telerik\justtrace\{4c3177a6-4f32-4a2b-bdc4-882b45110fc0}\symbols\dll\clr.pdb
    You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
    You should also verify that your symbol search path (.sympath) is correct.
    PDB symbol for clr.dll not loaded
    Failed to load data access DLL, 0x80004005
    Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
                2) the file mscordacwks.dll that matches your version of clr.dll is
                    in the version directory or on the symbol path
                3) or, if you are debugging a dump file, verify that the file
                    mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
                4) you are debugging on supported cross platform architecture as
                    the dump file. For example, an ARM dump file must be debugged
                    on an X86 or an ARM machine; an AMD64 dump file must be
                    debugged on an AMD64 machine.
    You can also run the debugger command .cordll to control the debugger's
    load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
    If that succeeds, the SOS command should work on retry.
    If you are debugging a minidump, you need to make sure that your executable
    path is pointing to clr.dll as well.
    0:000> #JTStart TraverseHeap
    0:000> The version of SOS does not match the version of CLR you are debugging.  Please
    load the matching version of SOS for the version of CLR you are debugging.
    CLR Version: 4.0.30319.18444
    SOS Version: 4.0.30319.34011
    Failed to load data access DLL, 0x80004005
    Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
                2) the file mscordacwks.dll that matches your version of clr.dll is
                    in the version directory or on the symbol path
                3) or, if you are debugging a dump file, verify that the file
                    mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
                4) you are debugging on supported cross platform architecture as
                    the dump file. For example, an ARM dump file must be debugged
                    on an X86 or an ARM machine; an AMD64 dump file must be
                    debugged on an AMD64 machine.
    You can also run the debugger command .cordll to control the debugger's
    load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
    If that succeeds, the SOS command should work on retry.
    If you are debugging a minidump, you need to make sure that your executable
    path is pointing to clr.dll as well.
    0:000> #JTEnd TraverseHeap2014-03-25 10:03:30: System.FormatException: Input string was not in a correct format.
       at System.Number.ParseUInt64(String value, NumberStyles options, NumberFormatInfo numfmt)
       at Telerik.JustTrace.Api.Data.Memory.ClrProfilerLogDataLoader`1.ProcessRootRefsMessage(Int64 offset) in c:\J\jobs\JT_CI\workspace\JustTrace\JustTrace\Development\Telerik.JustTrace.Api\Data\Memory\ClrProfilerLogDataLoader.cs:line 270
       at Telerik.JustTrace.Api.Data.Memory.ClrProfilerLogFileProcessor.ReadTypeStats(StreamReader reader) in c:\J\jobs\JT_CI\workspace\JustTrace\JustTrace\Development\Telerik.JustTrace.Api\Data\Memory\ClrProfilerLogFileProcessor.cs:line 119
       at Telerik.JustTrace.Api.Data.Memory.ClrProfilerLogFileProcessor.ProcessLog() in c:\J\jobs\JT_CI\workspace\JustTrace\JustTrace\Development\Telerik.JustTrace.Api\Data\Memory\ClrProfilerLogFileProcessor.cs:line 46
       at Telerik.JustTrace.Api.Session.Profilers.DumpReader.ProcessClrProfilerTraverseHeapFile(String clrProfilerLogfile, DateTime timestamp, Boolean is64Bit) in c:\J\jobs\JT_CI\workspace\JustTrace\JustTrace\Development\Telerik.JustTrace.Api\Session\Profilers\DumpReader.cs:line 764
       at Telerik.JustTrace.Api.Session.Profilers.DumpReader.GetSnapshotFromDump(String dumpfile, DateTime timestamp, TimeSpan timeout, String debuggingToolsForWindowsFolder) in c:\J\jobs\JT_CI\workspace\JustTrace\JustTrace\Development\Telerik.JustTrace.Api\Session\Profilers\DumpReader.cs:line 273
       at Telerik.JustTrace.SnapshotsStartPagePresenter.#8ld(String #9ld, String #7ld, DateTime #Ze, Int32 #bo) in c:\J\jobs\JT_CI\workspace\JustTrace\JustTrace\Development\JustTrace\Modules\Snapshots\StartPageWidget\SnapshotsStartPagePresenter.cs:line 472
       at Telerik.JustTrace.SnapshotsStartPagePresenter.#CZ.#Rpd() in c:\J\jobs\JT_CI\workspace\JustTrace\JustTrace\Development\JustTrace\Modules\Snapshots\StartPageWidget\SnapshotsStartPagePresenter.cs:line 343
    2014-03-25 10:03:33: <Page>: id='Inactive', referrer='Backstage.OpenSnapshots', ui='Unspecified', profilee='', profiler=''

  3. DevCraft banner
  4. Martin
    Admin
    Martin avatar
    29 posts

    Posted 27 Mar 2014 Link to this post

    Hello Pierre,

    It seems that there's a bug in JustTrace. Opening a dump fails when the machine from which the dump is taken and the machine on which it is being opened have different CLR versions.

    We will fix it in one of our next public releases. In the meantime you can work around it by installing JustTrace on a machine with the same OS and CLR versions. Note that some Windows Updates modify the CLR version, so having the same updates installed might also be necessary. 

    I am sorry for the inconvenience that this issue is causing you. To thank you for reporting it I have added some Telerik points to your account.

    Kind regards,
    Martin
    Telerik
     

    Build cross-platform mobile apps using Visual Studio and .NET. Register for the online webinar on 03/27/2014, 11:00AM US ET.. Seats are limited.

  5. Pierre-Andre van Leeuwen
    Pierre-Andre van Leeuwen avatar
    12 posts
    Member since:
    Feb 2010

    Posted 27 Mar 2014 in reply to Martin Link to this post

    Thanks for the response. I decompiled JustTrace using JustDecompile and saw that cdb.exe was passed the command: .loadby sos clr. Perhaps adding and option to JustTrace where one can specify an alternative location for sos.dll and mscordacwks.dll would solve the problem.

    I finally managed to simulate the memory leak on a test machine and managed to find the leak using JustTrace, so all is well thank you!
  6. Martin
    Admin
    Martin avatar
    29 posts

    Posted 27 Mar 2014 Link to this post

    Hi again,

    I'm glad that you've managed to resolve the issue.

    Thank you for the suggestion. We actually embed the mscordacwks.dll and sos.dll files inside the dump file and do not need such an option. We just need to fix our code to load them from there.

    Regards,
    Martin
    Telerik
     

    Build cross-platform mobile apps using Visual Studio and .NET. Register for the online webinar on 03/27/2014, 11:00AM US ET.. Seats are limited.

     
Back to Top
DevCraft banner