Quick Debugging Tip: Disable apppool recycling while getting crash dumps

I have probably mentioned this before, but I think it is worth mentioning again because of the frequency of this issue.

When debugging a crash, we usually get customers to use a crash rule in debug diag or to use adplus -crash to get memory dumps at the point of the crash. Both of these attach to the process and gathers dumps when the process is shutting down, which is exactly what we want.

The problem is that very often we will get what I call “false-positive” dumps. I.e. dumps when the process is shutting down, but not dumps of an actual crash, or at least not the shutdown we are trying to troubleshoot.

A typical characteristic of such a memory dump is that thread 0 is in the middle of a “normal shutdown”

0:000> kL
ChildEBP RetAddr
0006fc4c 79f96d9b kernel32!ExitProcess
0006fe74 79f96dc7 mscorwks!SafeExitProcess+0x11a
0006fe80 79f0c4a4 mscorwks!HandleExitProcessHelper+0x25
0006ff10 7901145b mscorwks!CorExitProcess+0x242
0006ff20 77bcaddb mscoree!CorExitProcess+0x46
0006ff2c 77bcaefb msvcrt!__crtExitProcess+0x29
0006ff5c 77bcaf52 msvcrt!doexit+0x81
0006ff70 01001a38 msvcrt!exit+0x11
0006ffc0 77e523cd w3wp!wmainCRTStartup+0x140
0006fff0 00000000 kernel32!BaseProcessStart+0x23

By normal, I mean that the process is shutting down because the process is either being IISReset, or it is shutting recycled due to health monitoring settings.

If it was an actual crash, you would typically see some 2nd chance exceptions occurring before the crash, generating 2nd chance exception dumps, and the process shutdown dump would only contain one thread (the last thread standing when the process is shut down). Btw, this “last thread standing” is usually not thread 0 (the main thread), but rather some unsuspecting “bystander” thread that just happened to be the last thread hanging around at the crime scene, some thread that usually has nothing to do with the actual crash.

Another typical trait of a “false-positive” is that the process has been up for roundabout 20 minutes

0:000> .time
Debug session time: Thu Sep 11 17:52:34.000 2008 (GMT+2)
System Uptime: 1 days 9:59:44.203
Process Uptime: 0 days 0:22:03.000
  Kernel time: 0 days 0:00:00.000
  User time: 0 days 0:00:00.000

Now, observant readers have probably figured out what is going on here. We get memory dumps when the process shuts down because of the idle timeout

By default, the worker process will shutdown after 1740 mins of processing (29 hrs), and/or when the process has been idle for 20 minutes.

So if either the process has been idle for 20 minutes, or the processing time is 29 hrs, the process will recycle and you will get a “crash” dump if you have a debugger attached.

In summary, if you are trying to get dumps of a real crash, you need to disable these two settings in order to stop getting “false-positive” dumps, and get dumps on the real crash.

Laters, Tess