10 minute read

I have written earlier about how to track down exceptions using configuration scripts for adplus.

Most of the time the methods are short enough that just knowing what function you got an exception in, is enough for you to track down the why, but we all know that we don’t live in that perfect world where we write completely modular applications and everything is nicely lined up

Let’s say you have found this exception on the heap…

0:025> !dumpobj 02b7191c
Name: System.NullReferenceException
MethodTable: 7915ec4c
EEClass: 791ea18c
Size: 72(0x48) bytes (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)

Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fa3e0  40000b5        4        System.String  0 instance 00000000 _className
79109208  40000b6        8 ...ection.MethodBase  0 instance 00000000 _exceptionMethod
790fa3e0  40000b7        c        System.String  0 instance 00000000 _exceptionMethodString
790fa3e0  40000b8       10        System.String  0 instance 02b719bc _message
79113dfc  40000b9       14 ...tions.IDictionary  0 instance 00000000 _data
790fa9e8  40000ba       18     System.Exception  0 instance 00000000 _innerException
790fa3e0  40000bb       1c        System.String  0 instance 00000000 _helpURL
790f9c18  40000bc       20        System.Object  0 instance 02b71a38 _stackTrace
790fa3e0  40000bd       24        System.String  0 instance 00000000 _stackTraceString
790fa3e0  40000be       28        System.String  0 instance 00000000 _remoteStackTraceString
790fed1c  40000bf       34         System.Int32  0 instance        0 _remoteStackIndex
790f9c18  40000c0       2c        System.Object  0 instance 00000000 _dynamicMethods
790fed1c  40000c1       38         System.Int32  0 instance -2147467261 _HResult
790fa3e0  40000c2       30        System.String  0 instance 00000000 _source
790fe160  40000c3       3c        System.IntPtr  0 instance 34270984 _xptrs
790fed1c  40000c4       40         System.Int32  0 instance -1073741819 _xcode
0:025> !printexception 02b7191c
Exception object: 02b7191c
Exception type: System.NullReferenceException
Message: Object reference not set to an instance of an object.
InnerException: <none>
StackTrace (generated):
    SP       IP       Function
    020AF378 029C3269 DisplayUserInfo.Page_Load(System.Object, System.EventArgs)

StackTraceString: <none>

We know from the stack that the NullReferenceException occurred in the DisplayUserInfo.Page_Load function, but how do you know exactly where inside that function? and how do you know what caused it?

The first thing I usually do is look at the code if I have it. And if the code is not readily available I extract the dll from the dump using !savemodule or !saveallmodules (!sam) in sos.dll, which will give me an exact copy of the dll loaded in memory when the dump was taken. (Slight caveat… the !sam functionality does not exist in the 2.0 version of sos.dll yet so for 2.0 you will have to use !savemodule)

So for this exception I know that we are faulting at IP (instruction pointer) 0x029C3269, and I can use this to first get the method descriptor for the method (using !ip2md - Instruction Pointer to Method Descriptor)…

0:025> !ip2md 0x029C3269
MethodDesc: 0ee335b8
Method Name: DisplayUserInfo.Page_Load(System.Object, System.EventArgs)
Class: 0297a5b8
MethodTable: 0ee335ec
mdToken: 06000013
Module: 0ee329c4
IsJitted: yes
m_CodeOrIL: 029c3110

..and then dump out the method table to find out what dll this code is compiled to.

0:025> !dumpmt 0ee335ec
EEClass: 0297a5b8
Module: 0ee329c4
Name: DisplayUserInfo
mdToken: 02000004  (C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\debuggersamples\e7443224\5232f845\App_Web_dmjhn1yn.dll)
BaseSize: 0x180
ComponentSize: 0x0
Number of IFaces in IFaceMap: c
Slots in VTable: 130

Once we know the dll, we can find the loading address by running lmv m<assemblyname>

0:025> lmv mApp_Web_dmjhn1yn
start    end        module name
0f280000 0f288000   App_Web_dmjhn1yn   (deferred)
    Image path: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\debuggersamples\e7443224\5232f845\App_Web_dmjhn1yn.dll
    Image name: App_Web_dmjhn1yn.dll
    Using CLR debugging support for all symbols
    Has CLR image header, track-debug-data flag not set
    Timestamp:        Thu May 18 13:24:36 2006 (446C5974)
    CheckSum:         00000000
    ImageSize:        00008000
    File version:     0.0.0.0
    Product version:  0.0.0.0
    File flags:       0 (Mask 3F)
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0000.04b0
    InternalName:     App_Web_dmjhn1yn.dll
    OriginalFilename: App_Web_dmjhn1yn.dll
    ProductVersion:   0.0.0.0
    FileVersion:      0.0.0.0
    FileDescription:
    LegalCopyright:

And given this we can now extract this assembly (App_Web_dmjhn1yn.dll) from the memory dump

0:025> !savemodule 0f280000 f:\App_Web_dmjhn1yn.dll
3 sections in file
section 0 - VA=2000, VASize=1af4, FileAddr=200, FileSize=1c00
section 1 - VA=4000, VASize=2c8, FileAddr=1e00, FileSize=400
section 2 - VA=6000, VASize=c, FileAddr=2200, FileSize=200

Ok… so now we have the dll… but how do we get to the actual code?

Well… we could open it with ildasm.exe and browse to the DisplayUserInfo.Page_Load function which would give us somewhat readable IL code. It’s actually not half-bad, but it doesn’t give you as clear of a picture as the code would.

The tool I like to use is Lutz Roeder’s reflector which gives me the following output if I browse to the DisplayUserInfo.Page_Load function

protected void Page_Load(object sender, EventArgs e)
{
      try
      {
            this.LblWelcomeMsg.Text = "Welcome " + this.Session["username"];

            if (this.Session["role"].ToString() == "Administrator")
            {
                  this.btnEditRoll.Enabled = true;
            }
            else
            {
                  this.btnEditRoll.Enabled = false;
            }

            TableHeaderRow row1 = new TableHeaderRow();
            TableHeaderCell cell1 = new TableHeaderCell();
            cell1.Text = "Blogs";
            row1.Cells.Add(cell1);
            this.tblBlogRoll.Rows.Add(row1);
            ArrayList list1 = (ArrayList) this.Session["BlogRoll"];

            for (int num1 = 0; num1 < list1.Count; num1++)
            {
                  TableCell cell2 = new TableCell();
                  cell2.Text = list1[num1].ToString();
                  TableRow row2 = new TableRow();
                  row2.Cells.Add(cell2);
                  this.tblBlogRoll.Rows.Add(row2);
            }
      }
      catch (Exception)
      {
            base.Response.Write("An exception occurred");
      }
}

Now how cool is that, that is pretty much an exact replica of the original code…

but… that alone doesn’t really tell us where the exception occurred, so let’s go back to the instruction pointer and un-assemble the function in the dump using !u, and then we can search for the instruction closest to our current instruction pointer to see exactly where we are at…

0:025> !u 029C3269
Normal JIT generated code
DisplayUserInfo.Page_Load(System.Object, System.EventArgs)
Begin 029c3110, size 214
029c3110 55              push    ebp
029c3111 8bec            mov     ebp,esp
029c3113 57              push    edi
029c3114 56              push    esi
029c3115 53              push    ebx
029c3116 83ec20          sub     esp,20h
029c3119 33c0            xor     eax,eax
029c311b 8945e8          mov     dword ptr [ebp-18h],eax
029c311e 894ddc          mov     dword ptr [ebp-24h],ecx
029c3121 8b45dc          mov     eax,dword ptr [ebp-24h]
029c3124 8945d4          mov     dword ptr [ebp-2Ch],eax
029c3127 8bb86c010000    mov     edi,dword ptr [eax+16Ch]
029c312d 8b354444a30a    mov     esi,dword ptr ds:[0AA34444h]
029c3133 8bc8            mov     ecx,eax
029c3135 8b01            mov     eax,dword ptr [ecx]
029c3137 ff90a8010000    call    dword ptr [eax+1A8h]
029c313d 8bc8            mov     ecx,eax
029c313f 8b151844a30a    mov     edx,dword ptr ds:[0AA34418h]
029c3145 3909            cmp     dword ptr [ecx],ecx
029c3147 e8c4b7de65      call    System_Web_ni!System.Web.SessionState.HttpSessionState.get_Item(System.String) (687ae910)
029c314c 8bd0            mov     edx,eax
029c314e 8bce            mov     ecx,esi
029c3150 e863e79a76      call    USERENV!ProcessGPORegistryPolicy+0xdf (769ae763) (USERENV!ProcessGPORegistryPolicy)
029c3155 8bd0            mov     edx,eax
029c3157 8bcf            mov     ecx,edi
029c3159 8b01            mov     eax,dword ptr [ecx]
029c315b ff90fc010000    call    dword ptr [eax+1FCh]
029c3161 8b4dd4          mov     ecx,dword ptr [ebp-2Ch]
029c3164 8b01            mov     eax,dword ptr [ecx]
029c3166 ff90a8010000    call    dword ptr [eax+1A8h]
029c316c 8bc8            mov     ecx,eax
029c316e 8b152044a30a    mov     edx,dword ptr ds:[0AA34420h]
029c3174 3909            cmp     dword ptr [ecx],ecx
029c3176 e895b7de65      call    System_Web_ni!System.Web.SessionState.HttpSessionState.get_Item(System.String) (687ae910)
029c317b 8bc8            mov     ecx,eax
029c317d 8b01            mov     eax,dword ptr [ecx]
029c317f ff5028          call    dword ptr [eax+28h]
029c3182 8b152444a30a    mov     edx,dword ptr ds:[0AA34424h]
029c3188 8bc8            mov     ecx,eax
029c318a e861879876      call    USERENV!MyRegLoadKeyEx+0x21a (76988761) (USERENV!MyRegLoadKeyEx)
029c318f 25ff000000      and     eax,0FFh
029c3194 7418            je      App_Web_dmjhn1yn!DisplayUserInfo.Page_Load(System.Object, System.EventArgs)+0x9e (029c31ae)
029c3196 8b45d4          mov     eax,dword ptr [ebp-2Ch]
029c3199 8b8874010000    mov     ecx,dword ptr [eax+174h]
029c319f ba01000000      mov     edx,1
029c31a4 8b01            mov     eax,dword ptr [ecx]
029c31a6 ff9098010000    call    dword ptr [eax+198h]
029c31ac eb13            jmp     App_Web_dmjhn1yn!DisplayUserInfo.Page_Load(System.Object, System.EventArgs)+0xb1 (029c31c1)
029c31ae 8b45d4          mov     eax,dword ptr [ebp-2Ch]
029c31b1 8b8874010000    mov     ecx,dword ptr [eax+174h]
029c31b7 33d2            xor     edx,edx
029c31b9 8b01            mov     eax,dword ptr [ecx]
029c31bb ff9098010000    call    dword ptr [eax+198h]
029c31c1 b9c86ca568      mov     ecx,offset System_Web_ni+0x496cc8 (68a56cc8)
029c31c6 e851ee78ff      call    0215201c (JitHelp: CORINFO_HELP_NEWSFAST)
029c31cb 8bf0            mov     esi,eax
029c31cd 8bce            mov     ecx,esi
029c31cf e82448ef65      call    System_Web_ni!System.Web.UI.WebControls.TableHeaderRow..ctor() (688b79f8)
029c31d4 b9a08ca668      mov     ecx,offset System_Web_ni+0x4a8ca0 (68a68ca0)
029c31d9 e83eee78ff      call    0215201c (JitHelp: CORINFO_HELP_NEWSFAST)
029c31de 8bf8            mov     edi,eax
029c31e0 8bcf            mov     ecx,edi
029c31e2 e86144ef65      call    System_Web_ni!System.Web.UI.WebControls.TableHeaderCell..ctor() (688b7648)
029c31e7 8b156844a30a    mov     edx,dword ptr ds:[0AA34468h]
029c31ed 8bcf            mov     ecx,edi
029c31ef 8b01            mov     eax,dword ptr [ecx]
029c31f1 ff9014020000    call    dword ptr [eax+214h]
029c31f7 8bce            mov     ecx,esi
029c31f9 8b01            mov     eax,dword ptr [ecx]
029c31fb ff90f0010000    call    dword ptr [eax+1F0h]
029c3201 8bc8            mov     ecx,eax
029c3203 8bd7            mov     edx,edi
029c3205 3909            cmp     dword ptr [ecx],ecx
029c3207 e85c3fef65      call    System_Web_ni!System.Web.UI.WebControls.TableCellCollection.Add(System.Web.UI.WebControls.TableCell) (688b7168)
029c320c 8b45d4          mov     eax,dword ptr [ebp-2Ch]
029c320f 8b8870010000    mov     ecx,dword ptr [eax+170h]
029c3215 8b01            mov     eax,dword ptr [ecx]
029c3217 ff9028020000    call    dword ptr [eax+228h]
029c321d 8bc8            mov     ecx,eax
029c321f 8bd6            mov     edx,esi
029c3221 3909            cmp     dword ptr [ecx],ecx
029c3223 e8a048ef65      call    System_Web_ni!System.Web.UI.WebControls.TableRowCollection.Add(System.Web.UI.WebControls.TableRow) (688b7ac8)
029c3228 8b4dd4          mov     ecx,dword ptr [ebp-2Ch]
029c322b 8b01            mov     eax,dword ptr [ecx]
029c322d ff90a8010000    call    dword ptr [eax+1A8h]
029c3233 8bc8            mov     ecx,eax
029c3235 8b156c44a30a    mov     edx,dword ptr ds:[0AA3446Ch]
029c323b 3909            cmp     dword ptr [ecx],ecx
029c323d e8ceb6de65      call    System_Web_ni!System.Web.SessionState.HttpSessionState.get_Item(System.String) (687ae910)
029c3242 8bf0            mov     esi,eax
029c3244 85f6            test    esi,esi
029c3246 7418            je      App_Web_dmjhn1yn!DisplayUserInfo.Page_Load(System.Object, System.EventArgs)+0x150 (029c3260)
029c3248 813eb0361079    cmp     dword ptr [esi],offset mscorlib_ni+0x436b0 (791036b0)
029c324e 7502            jne     App_Web_dmjhn1yn!DisplayUserInfo.Page_Load(System.Object, System.EventArgs)+0x142 (029c3252)
029c3250 eb0e            jmp     App_Web_dmjhn1yn!DisplayUserInfo.Page_Load(System.Object, System.EventArgs)+0x150 (029c3260)
029c3252 8bd6            mov     edx,esi
029c3254 b9b0361079      mov     ecx,offset mscorlib_ni+0x436b0 (791036b0)
029c3259 e84e895277      call    mscorwks!JIT_ChkCastClassSpecial (79eebbac)
029c325e 8bf0            mov     esi,eax
029c3260 8975d8          mov     dword ptr [ebp-28h],esi
029c3263 33db            xor     ebx,ebx
029c3265 8b4dd8          mov     ecx,dword ptr [ebp-28h]
029c3268 8b01            mov     eax,dword ptr [ecx]
029c326a ff5040          call    dword ptr [eax+40h]
029c326d 85c0            test    eax,eax
029c326f 0f8e87000000    jle     App_Web_dmjhn1yn!DisplayUserInfo.Page_Load(System.Object, System.EventArgs)+0x1ec (029c32fc)
029c3275 b9cc00a568      mov     ecx,offset System_Web_ni+0x4900cc (68a500cc)
029c327a e89ded78ff      call    0215201c (JitHelp: CORINFO_HELP_NEWSFAST)
029c327f 8bf8            mov     edi,eax
029c3281 8bcf            mov     ecx,edi
029c3283 e89897e765      call    System_Web_ni!System.Web.UI.WebControls.TableCell..ctor() (6883ca20)
029c3288 8bd3            mov     edx,ebx
029c328a 8b4dd8          mov     ecx,dword ptr [ebp-28h]
029c328d 8b01            mov     eax,dword ptr [ecx]
029c328f ff5054          call    dword ptr [eax+54h]
029c3292 8bc8            mov     ecx,eax
029c3294 8b01            mov     eax,dword ptr [ecx]
029c3296 ff5028          call    dword ptr [eax+28h]
029c3299 8bd0            mov     edx,eax
029c329b 8bcf            mov     ecx,edi
029c329d 8b01            mov     eax,dword ptr [ecx]
029c329f ff9014020000    call    dword ptr [eax+214h]
029c32a5 b9843da668      mov     ecx,offset System_Web_ni+0x4a3d84 (68a63d84)
029c32aa e86ded78ff      call    0215201c (JitHelp: CORINFO_HELP_NEWSFAST)
029c32af 8bf0            mov     esi,eax
029c32b1 8bce            mov     ecx,esi
029c32b3 e8b057e665      call    System_Web_ni!System.Web.UI.WebControls.TableRow..ctor() (68828a68)
029c32b8 8bce            mov     ecx,esi
029c32ba 8b01            mov     eax,dword ptr [ecx]
029c32bc ff90f0010000    call    dword ptr [eax+1F0h]
029c32c2 8bc8            mov     ecx,eax
029c32c4 8bd7            mov     edx,edi
029c32c6 3909            cmp     dword ptr [ecx],ecx
029c32c8 e89b3eef65      call    System_Web_ni!System.Web.UI.WebControls.TableCellCollection.Add(System.Web.UI.WebControls.TableCell) (688b7168)
029c32cd 8b45d4          mov     eax,dword ptr [ebp-2Ch]
029c32d0 8b8870010000    mov     ecx,dword ptr [eax+170h]
029c32d6 8b01            mov     eax,dword ptr [ecx]
029c32d8 ff9028020000    call    dword ptr [eax+228h]
029c32de 8bc8            mov     ecx,eax
029c32e0 8bd6            mov     edx,esi
029c32e2 3909            cmp     dword ptr [ecx],ecx
029c32e4 e8df47ef65      call    System_Web_ni!System.Web.UI.WebControls.TableRowCollection.Add(System.Web.UI.WebControls.TableRow) (688b7ac8)
029c32e9 83c301          add     ebx,1
029c32ec 8b4dd8          mov     ecx,dword ptr [ebp-28h]
029c32ef 8b01            mov     eax,dword ptr [ecx]
029c32f1 ff5040          call    dword ptr [eax+40h]
029c32f4 3bc3            cmp     eax,ebx
029c32f6 0f8f79ffffff    jg      App_Web_dmjhn1yn!DisplayUserInfo.Page_Load(System.Object, System.EventArgs)+0x165 (029c3275)
029c32fe 8b4ddc          mov     ecx,dword ptr [ebp-24h]
029c3301 e84a2be165      call    System_Web_ni!System.Web.UI.Page.get_Response() (687d5e50)
029c3306 8bc8            mov     ecx,eax
029c3308 8b155c44a30a    mov     edx,dword ptr ds:[0AA3445Ch]
029c330e 3909            cmp     dword ptr [ecx],ecx
029c3310 e8e3b4cd65      call    System_Web_ni!System.Web.HttpResponse.Write(System.String) (6869e7f8)
029c3315 e8df2f5d77      call    mscorwks!JIT_EndCatch (79f962f9)
029c331a 8d65f4          lea     esp,[ebp-0Ch]
029c331d 5b              pop     ebx
029c331e 5e              pop     esi
029c331f 5f              pop     edi
029c3320 5d              pop     ebp
029c3321 c20400          ret     4

The bolded line…

029c3268 8b01            mov     eax,dword ptr [ecx]

…is the line right prior to our current IP, which means that it was the line that caused the NullReferenceException.

I have marked a few lines around it in gray that will help us compare the disassembly to code.

Disassembly:

029c3223 e8a048ef65      call    System_Web_ni!System.Web.UI.WebControls.TableRowCollection.Add(System.Web.UI.WebControls.TableRow) (688b7ac8)
029c3228 8b4dd4          mov     ecx,dword ptr [ebp-2Ch]
029c322b 8b01            mov     eax,dword ptr [ecx]
029c322d ff90a8010000    call    dword ptr [eax+1A8h]
029c3233 8bc8            mov     ecx,eax
029c3235 8b156c44a30a    mov     edx,dword ptr ds:[0AA3446Ch]
029c323b 3909            cmp     dword ptr [ecx],ecx
029c323d e8ceb6de65      call    System_Web_ni!System.Web.SessionState.HttpSessionState.get_Item(System.String) (687ae910)
029c3242 8bf0            mov     esi,eax
029c3244 85f6            test    esi,esi
029c3246 7418            je      App_Web_dmjhn1yn!DisplayUserInfo.Page_Load(System.Object, System.EventArgs)+0x150 (029c3260)
029c3248 813eb0361079    cmp     dword ptr [esi],offset mscorlib_ni+0x436b0 (791036b0)
029c324e 7502            jne     App_Web_dmjhn1yn!DisplayUserInfo.Page_Load(System.Object, System.EventArgs)+0x142 (029c3252)
029c3250 eb0e            jmp     App_Web_dmjhn1yn!DisplayUserInfo.Page_Load(System.Object, System.EventArgs)+0x150 (029c3260)
029c3252 8bd6            mov     edx,esi
029c3254 b9b0361079      mov     ecx,offset mscorlib_ni+0x436b0 (791036b0)
029c3259 e84e895277      call    mscorwks!JIT_ChkCastClassSpecial (79eebbac)
029c325e 8bf0            mov     esi,eax
029c3260 8975d8          mov     dword ptr [ebp-28h],esi
029c3263 33db            xor     ebx,ebx
029c3265 8b4dd8          mov     ecx,dword ptr [ebp-28h]
029c3268 8b01            mov     eax,dword ptr [ecx]
029c326a ff5040          call    dword ptr [eax+40h]
029c326d 85c0            test    eax,eax
029c326f 0f8e87000000    jle     App_Web_dmjhn1yn!DisplayUserInfo.Page_Load(System.Object, System.EventArgs)+0x1ec (029c32fc)
029c3275 b9cc00a568      mov     ecx,offset System_Web_ni+0x4900cc (68a500cc)
029c327a e89ded78ff      call    0215201c (JitHelp: CORINFO_HELP_NEWSFAST)
029c327f 8bf8            mov     edi,eax
029c3281 8bcf            mov     ecx,edi
029c3283 e89897e765      call    System_Web_ni!System.Web.UI.WebControls.TableCell..ctor() (6883ca20)

Code from reflector:

this.tblBlogRoll.Rows.Add(row1);
ArrayList list1 = (ArrayList) this.Session["BlogRoll"];
for (int num1 = 0; num1 < list1.Count; num1++)
{
      TableCell cell2 = new TableCell();
      cell2.Text = list1[num1].ToString();
      TableRow row2 = new TableRow();

We can see the call to this.tblBlogRoll.Rows.Add (System_Web_ni!System.Web.UI.WebControls.TableRowCollection.Add), and the call to this.Session[“BlogRoll”] (System_Web_ni!System.Web.SessionState.HttpSessionState.get_Item), followed by the cast to ArrayList (mscorwks!JIT_ChkCastClassSpecial)

After our bolded line we can see the call to new TableCell (System_Web_ni!System.Web.UI.WebControls.TableCell..ctor), which means that the bolded line must be one of the instructions on the for line

for (int num1 = 0; num1 < list1.Count; num1++)

More specifically it is the list1.Count that is causing the NullReferenceException, in other words list1 is null because this.Session[“BlogRoll”] was empty, and we NullReference when we try to get to the Count property, so to avoid this we need to do a null check on the Session[“BlogRoll”] before assigning it to the ArrayList.

Btw, this method doesn’t only apply to exceptions. You can use it to figure out where exactly you take a lock and other similar things as well, but you’ll probably need it mostly for exceptions.

Til next time…