Why do I get weird function names on my stack? (a discussion on symbols)

Symbols can contain information about global variables, local variables, function names, parameters, structures and source line numbers.

There are 3 types of symbols

• export symbols
• pdb symbols (public symbols) and
• private pdb symbols (private symbols)

The export symbols are part of the dll itself. For example ntdll.dll and kernel32.dll expose a big part of their functions as export symbols so that they can be called as APIs, but most dlls that you find in a process have a very small set of exported symbols. Generally export symbols don’t contain parameter info for the functions and since very few functions are exposed this way you can’t really relay on the validity of the stacks when you have only export symbols.

Public symbols contain some basic symbols such as function names and global variables, but again, not all function names are exposed in public symbols. The developer of the dll chooses what to expose as public symbols and thus he/she can hide anything that they feel would give away too much information about the implementation.

Private symbols contain pretty much everything listed in the first paragraph

When debugging, symbols are matched up with the respective dlls or exe by a GUID linking the dll/exe to a symbol file (thanks Pavan for the correction). This means that if you have multiple ntdll.pdb in your symbol search path, the debugger will know which one corresponds to your particular version of ntdll.dll. The search path is given by .sympath, and apart from what is listed in your sympath the debugger will also look in the directory where the dll is loaded from as well as anything in the paths given in the environment variable _NT_SYMBOL_PATH.

So… what happens if you have the wrong symbols

Let’s look at this stack with public symbols for

mscorsvr.dll: (sorry about the formatting)

  54  Id: 62c.1590 Suspend: 1 Teb: 7ffa2000 Unfrozen
ChildEBP RetAddr  Args to Child
1212ef44 7c59a030 00000090 00000000 1212ef64 ntdll!NtWaitForSingleObject+0xb [i386\usrstubs.asm @ 2004]
1212ef6c 7c57b3db 00000090 00009c40 00000000 kernel32!WaitForSingleObjectEx+0x71 [D:\nt\private\windows\base\client\synch.c @ 1309]
1212ef7c 791b578b 00000090 00009c40 00000000 kernel32!WaitForSingleObject+0xf [D:\nt\private\windows\base\client\synch.c @ 1217]
1212efa0 791dbe6e 00000000 00000000 00000000 mscorsvr!ThreadpoolMgr::WorkerThreadStart+0x3a
1212ffb4 7c57b388 0d406838 00000002 00000000 mscorsvr!ThreadpoolMgr::intermediateThreadProc+0x44
1212ffec 00000000 791dbe2d 0d406838 00000000 kernel32!BaseThreadStart+0x52 [D:\nt\private\windows\base\client\support.c @ 460]

We are creating a worker thread and then sit and wait for work

Let’s look at the same stack with export symbols for mscorsvr:

0:054> kb
ChildEBP RetAddr  Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
1212ef6c 7c57b3db 00000090 00009c40 00000000 ntdll!NtWaitForSingleObject+0xb
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for mscorsvr.dll -
1212efa0 791dbe6e 00000000 00000000 00000000 kernel32!WaitForSingleObject+0xf
1212ffb4 7c57b388 0d406838 00000002 00000000 mscorsvr!GetCompileInfo+0x8e99
1212ffec 00000000 791dbe2d 0d406838 00000000 kernel32!lstrcmpiW+0xb7

The debugger is nice and tells us that we couldn’t find the symbol file for mscorsvr, but yet it gives us a function name (from the export symbols) so it appears that we’re calling some function called GetCompileInfo, hmm… very strange, why did it pick that name?

If we were to list the symbols (export symbols for mscorsvr) we would get a list that looks like this

0:054> x mscorsvr!*
791b0000 mscorsvr!Ordinal73 =
791b0000 mscorsvr!Ordinal76 =
791b0000 mscorsvr!Ordinal77 =
791b0000 mscorsvr!Ordinal75 =
791b0000 mscorsvr!Ordinal78 =
791b0000 mscorsvr!Ordinal71 =
791b0000 mscorsvr!Ordinal79 =
791b0000 mscorsvr!Ordinal74 =
791b0000 mscorsvr!Ordinal72 =
791d2fd5 mscorsvr!GetCompileInfo =
791e0920 mscorsvr!GetAssemblyMDImport ()
…

So we are executing something at address 791dbe6e (791d2fd5+0x8e99), if we take a peak at the symbols with public symbols loaded, we would see that that address landed inside the intermediateThreadProc function (+0x44 to be exact)

791bbb78 mscorsvr!SimpleRWLock::LeaveRead =
793eaf50 mscorsvr!ExecutionManager::m_dwReaderCount =
791dbe2d mscorsvr!ThreadpoolMgr::intermediateThreadProc =
793035c5 mscorsvr!VtBoolMarshaler::UnmarshalComToNativeIn =
791f8e0d mscorsvr!LayoutClassPtrMarshaler::ClearNativeTemp =

So in essence, the reason it picked the name GetCompileInfo was that that was the last symbol it could find, preceding the address that we were executing at.

You can see how this can easily get confusing. Oh and btw, not only did it give us a completely wrong function name, but if you take a closer look at the two stacks, we lost a whole function call on the stack with the export symbols (The call to WorkerThreadStart). The reason for this is that the function we were supposed to look at was at address 791b578b and this address is located before our first exported function so it can’t even resolve it to something fake.

How do you know if your symbols are good

For starters, just by looking at the stacks we can see that the function names seem pretty weird. It doesn’t make sense that a thread would start with lstrcmpiW, they usually start with BaseThreadStart or something to that effect. The second thing that gives it away just by looking at the stack is that we were supposed to be 0x8e99 instructions in to the function GetCompileInfo, that is 36505 instructions, wow, that would be a mighty long function.

The more definite way of knowing would be to run lmv on the executable. (notice the extra m before mscorsvr for match pattern)

0:054> lmv mmscorsvr
start    end        module name
791b0000 79418000   mscorsvr   (export symbols)       mscorsvr.dll
    Loaded symbol image file: mscorsvr.dll
    Image path: C:\WINNT\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll
    Image name: mscorsvr.dll
    Timestamp:        Thu Jul 15 09:26:32 2004 (40F631A8)
    CheckSum:         002664D4
    ImageSize:        00268000
    File version:     1.1.4322.2032
    Product version:  1.1.4322.2032
    File flags:       20 (Mask 3F) Special
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04e4
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft .NET Framework
    InternalName:     MSCORSVR.DLL
    OriginalFilename: mscorsvr.dll
    ProductVersion:   1.1.4322.2032
    FileVersion:      1.1.4322.2032
    FileDescription:  Microsoft .NET Runtime Common Language Runtime - Server
    LegalCopyright:   Copyright © Microsoft Corporation 1998-2002. All rights reserved.
    LegalTrademarks:  Microsoft® is a registered trademark of Microsoft Corporation. Windows(TM) is a trademark of Microsoft Corporation
    Comments:         Microsoft .NET Runtime Common Language Runtime - Server

lmv will give you a lot of good information about the dll, but in this particular case what we are interested is what is printed after the module name, i.e export symbols in this case. (You can also get this for all modules by running lm)

One last comment on this: As you may have noticed, going to the public Microsoft symbols server you sometimes get public symbols and sometimes end up with export symbols for mscorsvr.dll for example. This is because some cases like certain hotfixes and private builds have not been uploaded to the public symbol server.

Lazy symbol loading and downloading symbols for offline debugging

If you run lm you will notice that a lot of the dll and exe will have their symbols listed as deferred. This is because the debugger doesn’t actually load all the symbols when the debugger starts (as this would take a looooooooong time especially if you have a remote symbol store). Symbols are loaded on an as needed bases, i.e. when you run kb (to list the stacks) or run x to examine the symbols for a particular executable, or any other command that requires symbol lookup.

Sometimes though, it is useful to download all the symbols for a process, a common example would be if your application is running on a machine with no internet access and you want to be able to live debug it.

In this case what you can do is this:

1. Take a dump of the process when all modules are loaded and open it on a machine with internet access.
2. Set the sympathy to srv*c:\myappsymbols*http://msdl.microsoft.com/download/symbols;
3. run .reload /i /f to force load all the symbols for the dump

This will download all the symbols available on Microsoft’s public symbol server that match your dlls into the c:\myappsymbols and then you can simply copy this folder over to the machine without internet access and set the symbol path there to srv*c:\myappsymbols*c:\myappsymbols;c:\folderwithanyadditionalsymbols.

But… some of the functions on my stack only show up as addresses, what is up with that

This really doesn’t have anything to do with symbols but I thought it was worth mentioning anyways. In the call stack below we can see one function call only listed by it’s address… 0xd44334a, the reason for this is that it is a managed function which doesn’t have a native translation, and after all windbg is merely a native debugger. You can run !clrstack or !ip2md on the address to get the .net function name.

0:045> kb
ChildEBP RetAddr  Args to Child
1182ecd0 7c59c5b7 7920d6a7 00000010 0f39c800 ntdll!NtYieldExecution+0xb [i386\usrstubs.asm @ 2108]
1182ecd4 7920d6a7 00000010 0f39c800 00000010 kernel32!SwitchToThread+0x6 [D:\nt\private\windows\base\client\thread.c @ 2611]
1182ed00 791c0283 0f39c800 00000010 00000000 mscorsvr!gc_heap::allocate_more_space+0xe3
1182ef28 791b6930 0f39c800 00000010 00000000 mscorsvr!GCHeap::Alloc+0x7b
1182ef3c 791c0359 00000010 00000000 00000000 mscorsvr!Alloc+0x3a
1182ef5c 791c03a7 0d421928 1182efec 062eff1c mscorsvr!FastAllocateObject+0x25
1182efc8 0d44334a 00000000 1182f01c 1182f040 mscorsvr!JIT_NewFast+0x2c
WARNING: Frame IP not in any known module. Following frames may be wrong.
1182effc 791b33ec 1182f114 791b3c96 1182f050 0xd44334a
1182f004 791b3c96 1182f050 00000000 1182f028 mscorsvr!CallDescrWorker+0x30
1182f114 791b5612 00bf1bdb 79b7a000 00000004 mscorsvr!MethodDesc::CallDescr+0x1b8
1182f1d0 791b56c6 79bf1bdb 79b7a000 79bd1d16 mscorsvr!MethodDesc::CallDescr+0x4f
1182f1f8 791b5c40 1182f210 0ba57968 0f39c7c8 mscorsvr!MethodDesc::Call+0x97
1182f2ac 791b5b97 0f3a6480 00000001 00000000 mscorsvr!AddTimerCallbackEx+0x165
1182f2c0 791b4de8 1182f374 791b3cf4 ffffffff mscorsvr!AddTimerCallback_Wrapper+0x13
1182f308 791b5ae5 0ba57968 791b5b84 1182f374 mscorsvr!Thread::DoADCallBack+0x5c
1182f3c4 791b59ce 0f3a6480 00000001 00000001 mscorsvr!AddTimerCallbackEx+0x165
1182f3d8 791b59a5 0f3a6480 00000001 791b5996 mscorsvr!AddTimerCallback+0x10
1182f3ec 791b5802 0f3a2230 0f3b6bf0 793ea988 mscorsvr!ThreadpoolMgr::AsyncTimerCallbackCompletion+0xf
1182f400 791b57c1 0f3b6bf0 00000000 791b5751 mscorsvr!ThreadpoolMgr::ExecuteWorkRequest+0x19
1182f420 791dbe6e 00000000 00000000 00000000 mscorsvr!ThreadpoolMgr::WorkerThreadStart+0x129

0:045> !clrstack
Loaded Son of Strike data table version 5 from "C:\WINNT\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll"
Thread 45
ESP         EIP
0x1182efa4  0x77f88fe3 [FRAME: HelperMethodFrame]
0x1182efd0  0x0d44334a [DEFAULT] [hasThis] Void System.Timers.Timer.MyTimerCallback(Object)
0x1182f2cc  0x791b33ec [FRAME: ContextTransitionFrame]

How about symbols for managed dlls? Do I need them

Technically for debugging with windbg the answer is that you would need them very seldom or never. If you do have managed symbols and the dlls are compiled in debug mode you can get line numbers and source files. You can also get some parameter info when using !clrstack –p. But all in all, the extra “stuff” you get from managed symbols is very little when using a native debugger like windbg.

Over and out…