.NET Debugging Demos Lab 7: Memory Leak

This is the last debugging lab in the .NET Debugging Labs series. By now you should have the basics down for troubleshooting hangs, perf issues, memory leaks and crashes in .net applications. Hope you have enjoyed your debugging sessions.

The last one in the series is a managed memory leak caused by holding on to resources in an unexpected way. Since it is the last one I have tried to make the questions a little bit less leading than the previous ones :)

Previous labs and setup instructions

If you are new to the debugging labs, here you can find information on how to set up the labs as well as links to the previous labs in the series.

• Information and setup instructions
• Lab 1: Hang
• Lab 2: Crash
• Lab 3: Memory
• Lab 4: High CPU hang
• Lab 5: Crash
• Lab 6: Memory

Problem description

The problem description is very similar to Lab 6.

We have started getting out of memory exceptions on the Buggy Bits site and we have been able to determine a scenario in which we think we are leaking memory but we can’t seem to figure out where the memory is going.

The leak seems to be occurring on our News page for example and we can reproduce it by stress testing.

It seems like it is leaking just a small bit every time but since it is something that customers look at a lot and over time the process will crash with an out of memory exception.

Reproduce the issue and gather data

1. Start the Buggy Bits application and browse to the News site
2. Start monitoring performance counters with

3. Stress the application

    .\tinyget.ps1 -url https://localhost:44350/News -numTimes 1000
   

4. After tinyget has finished, capture a memory dump

    dotnet-dump collect -n iisexpress
   

   or with procdump

    procdump64.exe -ma iisexpress.exe
   

Review the performance counters to figure out what we are leaking

1. Check Working Set, GC Heap Size and Number of Assemblies Loaded

   • Can you tell if the issue we are facing is a virtual bytes leak, a native leak or a .NET leak?

Debug the memory dump

1. Open the memory dump in WinDbg, load up the symbols and load sos.dll (see information and setup instructions for more info)

   • What is the size of the memory dump (on disk)

2. Run !eeheap -gc and !dumpheap -stat

   • What is the size of the .NET heap according to !eeheap -gc?
   • Is most of the memory stored on the regular heaps or on the large object heap?

   We saw from the performance counters that we appeared to be leaking .net memory, so the next step is to determine what the memory is used for.

3. Run !dumpheap -stat

   • What types of objects seem to use up most of the memory?
   • Looking at the 10-20 bottommost object types in !dumpheap -stat, can you see any patterns among the objects? I.e. can they be grouped in any way?

   Normally you will be able to see patterns such as many data related objects, many ui related objects, many xml related objects etc. like in this memory investigation

   • Looking at the 10-20 bottom most object types in !dumpheap -stat, do the quantities of each of them seem normal or does anything seem out of the ordinary?

4. Dump the large objects using !dumpheap -min 0n85000

   • What type of objects are stored on the Large Object Heap (LOH)?

5. Take any of the objects on the LOH and run !gcroot <object address> to find out where they are rooted

   • What types of roots would you normally see when running !gcroot and what do they mean?

   An example of a root chain would look like this (if we are looking to see why the string at address 02ebf628 is still around)

    !gcroot 02ebf628
    ...
    DOMAIN(001CCA88):HANDLE(Strong):2c41188:Root: 02ec05c0(MyNamespace.Person)->
    02ec0578(MyNamespace.Address)->
    02ebf628(System.String)
   

   In this case we have a strongly rooted (static) object of type MyNamespace.Person at 02ec05c0. Person has a member variable of type Address at 02ec0578 and Address has a member variable of type string at 02ebf628, so the reason that the string is still around is because it is linked to an Address that is linked to a Person and that Person is stored as a static variable. Unless the chain is broken the string will not be available for collection until the application domain is recycled since static objects never go out of scope.

   Look at the root chain of the object that you did !gcroot on.

   • Why is it sticking around? How could the root chain be broken?

   Note: If you walk it from the bottom up and reach an object type that you don’t recognize, look it up on the docs to get more info about it in order to understand why there is a link.

Putting it all together and determining the cause of the memory leak

1. Look at the code for the News controller and use the knowledge you gathered from the debugging session to figure out how the leak was generated.

   • How long will the objects stick around?
   • What can you do to avoid this type of event handler leak?

Resolve the issue and rerun the test to verify the solution

1. Search this site or the Microsoft docs to find a resolution to the problem and resolve the issue
2. Rerun the test to verify that the assembly “leak” no longer exists.

Have fun,

Tess