ASP.NET Memory Leak: Byte arrays rooted in System.Security.Policy.Evidence

Today I got a question from a reader (Chris) about a memory leak they are seeing in their application

When we do a '!dumpheap -min 85000 -type Byte[]' we can see 100s of byte array objects using up ~545MB of memory. A majority of the objects are the same size (either 4.5 or 9MB in size). Looking at the memory addresses, they all appear to be different copies of our assemblies. And when we do !GcRoot on those addresses they all either have no results returned, or show a rooted System.Security.Policy.Evidence object

DOMAIN(000FDC18):HANDLE(Strong):-:Root:-(System.Security.Policy.Evidence)->
(System.Collections.ArrayList+SyncArrayList)->
(System.Collections.ArrayList)->
(System.Object[])->
(System.Security.Policy.Hash)->
(System.Byte[])

Through PerfMon we can see our Large Object Heap size grow and grow, and through !dumpheap it looks like it is getting fragmented.

We have not figured out how/why so many copies of our main assembly’s would be loaded by different rooted System.Security.Policy.Evidence object over and over again and never released. Our code never creates and new AppDomains and never manually loads any assembly. It receives a lot of web service requests, makes some database queries and/or makes its own web service requests and then returns the results.

Chris did a lot of ground work here and did everything exactly right

1. Looked at the .net memory (!eeheap -gc) to find out that that is where his memory issue is, since most of the memory is managed (.net)
2. Looked at the large objects (!dumpheap -min 85000) and identified that the byte[]s on the LOH were taking up most of the .net memory
3. Compared the byte[]s to find a pattern (they were all 4,5 MB byte arrays and 9 MB byte arrays)
4. Looked at the contents of the byte arrays (dc <addr of byte array>) and noticed that the byte arrays contained text that resembled some of his assemblies, and not only that, he noticed that they were the same assemblies so it looks like the same data is loaded over and over
5. Looked at the roots for the byte[]s and found that they were rooted in System.Security.Policy.Evidence

From here it gets a bit tricky to figure out the cause of the issue. But it turns out that when you load an assembly that reference another assembly, a security.policy.hash is generated. The hash it generates is about the same size as the referenced dlls, so that is where the 4.5 and 9 MB come from. This is then cached in case another dll is referencing the same dll so the hash wont be regenerated.

That is all well and good, the question is why are we then generating so many of these policy hashes?

There was an issue in .net 2.0 RTM where, if memory pressure is high (i.e. if system memory is low) we would flush some items from cache which meant that we need to regenerate these policy.hashes. Of course this in turn may cause more memory pressure so we flush again and create again etc. etc.

Solution

This issue is fixed in http://support.microsoft.com/default.aspx/kb/929963 FIX: The ASP.NET cache may flush some assemblies if the system memory is low, which is included in SP1 for .Net framework 2.0, so preferably one should install SP1 to fix this issue.

Have a good one, Tess