ASP.NET Memory - How much are you caching? + an example of .foreach
I was writing a post on debugging a high memory usage problem caused by storing too much in session scope, but I realized I got in to tangent discussions all the time so I decided to create a separate post on caching first.
Caching is by far the most common cause of high memory problems. It’s an understatement to say that the cache is often overused.
As with everything else nothing comes for free, so the performance you gain by storing items in cache has to be compared to the performance you loose by having to scavenge this memory. Be careful with the design and set expiration times etc. appropriately, and make sure that what you cache will actually be accessed enough times to make it worth while.
Sos.dll has some nice features for examining your cache through the !dumpaspnetcache command; however before we get in to that, let’s look at some commands to determine if you have a cache problem.
Where is the cache
Each HttpRuntime (running application) has a reference to a System.Web.Caching.CacheMultiple object (the _cache field), where the cached objects for that application is stored. In the particular version I am running (1.1.4322.2300), the _cache field is located at offset 0xc from the address of the HttpRuntime object.
0:023> !do 0x061b47d4
Name: System.Web.HttpRuntime
MethodTable 0x01b29dfc
EEClass 0x01ee8314
Size 116(0x74) bytes
GC Generation: 2
mdToken: 0x02000079 (c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll)
FieldDesc*: 0x01b29554
MT Field Offset Type Attr Value Name
0x01b29dfc 0x4000683 0x4 CLASS instance 0x00000000 _namedPermissionSet
0x01b29dfc 0x4000684 0x8 CLASS instance 0x061b4950 _fcm
0x01b29dfc 0x4000685 0xc CLASS instance 0x061b4ca0 _cache
0x01b29dfc 0x4000686 0x54 System.Boolean instance 0 _isOnUNCShare
0x01b29dfc 0x4000687 0x10 CLASS instance 0x061b6e18 _profiler
0x01b29dfc 0x4000688 0x14 CLASS instance 0x061b6e34 _timeoutManager
0x01b29dfc 0x4000689 0x18 CLASS instance 0x021b368c _requestQueue
0x01b29dfc 0x400068a 0x55 System.Boolean instance 0 _apartmentThreading
0x01b29dfc 0x400068b 0x56 System.Boolean instance 0 _beforeFirstRequest
0x01b29dfc 0x400068c 0x60 VALUETYPE instance start at 0x061b4834 _firstRequestStartTime
0x01b29dfc 0x400068d 0x57 System.Boolean instance 1 _firstRequestCompleted
0x01b29dfc 0x400068e 0x58 System.Boolean instance 0 _userForcedShutdown
0x01b29dfc 0x400068f 0x59 System.Boolean instance 1 _configInited
0x01b29dfc 0x4000690 0x50 System.Int32 instance 1 _activeRequestCount
0x01b29dfc 0x4000691 0x5a System.Boolean instance 0 _someBatchCompilationStarted
0x01b29dfc 0x4000692 0x5b System.Boolean instance 0 _shutdownInProgress
0x01b29dfc 0x4000693 0x1c CLASS instance 0x00000000 _shutDownStack
0x01b29dfc 0x4000694 0x20 CLASS instance 0x00000000 _shutDownMessage
0x01b29dfc 0x4000695 0x68 VALUETYPE instance start at 0x061b483c _lastShutdownAttemptTime
0x01b29dfc 0x4000696 0x5c System.Boolean instance 1 _enableHeaderChecking
0x01b29dfc 0x4000697 0x24 CLASS instance 0x061b7014 _handlerCompletionCallback
0x01b29dfc 0x4000698 0x28 CLASS instance 0x061b7030 _asyncEndOfSendCallback
0x01b29dfc 0x4000699 0x2c CLASS instance 0x061b704c _appDomainUnloadallback
0x01b29dfc 0x400069a 0x30 CLASS instance 0x00000000 _initializationError
0x01b29dfc 0x400069b 0x34 CLASS instance 0x00000000 _appDomainShutdownTimer
0x01b29dfc 0x400069c 0x38 CLASS instance 0x061d4b24 _codegenDir
0x01b29dfc 0x400069d 0x3c CLASS instance 0x02188f1c _appDomainAppId
0x01b29dfc 0x400069e 0x40 CLASS instance 0x02188f68 _appDomainAppPath
0x01b29dfc 0x400069f 0x44 CLASS instance 0x0218a434 _appDomainAppVPath
…
I’m not going to go in too much in the nitty-gritty but basically this CacheMultiple has a _caches Object[] member variable, which contains some CacheSingles and they each have an _entries member variable which contains the actual CacheEntries, if you want to delve in to them manually. Luckily for us, !dumpaspnetcache automates this as you will see later.
Getting the size of the cache
The reason that I still mention where the cache is stored, is because with this knowledge we can now compose a command that goes through all our HttpRuntimes and dumps out the name of the app and how much it stores in cache.
The pseudo-code for doing this would be:
For each (HttpRuntime in (all the runtimes on the heap))
Print “*******************” //to divide the entries
Dump out the AppDomainID
Print “Cache Size: “
Dump out the size of the cache
End for each
Translated to windbg terms:
To find all the runtimes on the heap, first find the methodtable for System.Web.HttpRuntime and then run !dumpheap –mt.
First run !dumpheap –type HttpRuntime which gives us all objects whose name contains HttpRuntime and their method tables (MT).
0:023> !dumpheap -type HttpRuntime
Using our cache to search the heap.
Address MT Size Gen
0x061d4dbc 0x020c0010 12 -1 System.Web.Configuration.HttpRuntimeConfigurationHandler
0x021b321c 0x0209ff40 52 -1 System.Web.Configuration.HttpRuntimeConfig
0x061b47d4 0x01b29dfc 116 -1 System.Web.HttpRuntime
Statistics:
MT Count TotalSize Class Name
0x020c0010 1 12 System.Web.Configuration.HttpRuntimeConfigurationHandler
0x0209ff40 1 52 System.Web.Configuration.HttpRuntimeConfig
0x01b29dfc 1 116 System.Web.HttpRuntime
Total 3 objects, Total size: 180
And then do !dumpheap –mt 0x01b29dfc –short
0:023> !dumpheap -mt 0x01b29dfc -short
0x61b47d4
The -short switch gives us only the address of the objects so we can use it more easily with .foreach. In this case I had only one HttpRuntime because I have only one vdir that I have accessed since the start of the process.
The AppDomainId (IIS name for the app) is stored in the _appDomainAppId member variable for HttpRuntime at offset 0x3c…
0x01b29dfc 0x400069d 0x3c CLASS instance 0x02188f1c _appDomainAppId
…so to dump this out given a HttpRuntime address you would run
0:023> !dumpobj poi(0x61b47d4+0x3c)
String: /LM/w3svc/1/root/MemoryIssues
The poi operator gives us a dword stored at the address we are giving it, i.e. in this case it will give us what is stored at 0x61b47d4+0x3c, where 0x61b47d4 is the address of the HttpRuntime, so it will give us 0x02188f1c, which is the address of _appDomainAppId from above.
The reason we use this technique is so we can automate it to run on all the HttpRuntime objects on the heap, since with .foreach where we will only know the address of each HttpRuntime object.
Much the same way we get the size of the cache by running !objsize on the _cache member variable
0x01b29dfc 0x4000685 0xc CLASS instance 0x061b4ca0 _cache
0:023> !objsize poi(0x61b47d4+0xc)
sizeof(0x61b4ca0) = 1,336,709,120 (0x4fac9000) bytes (System.Web.Caching.CacheMultiple)
Now we have all the components to run a command that will give us the name and the cache size for all HttpRuntime objects.
A short comment on .foreach before we continue
In its simplest form the .foreach command looks like this:
.foreach ( Variable { InCommands } ) {
OutCommands
}
Where InCommands is the commands that will give you the array/collection to loop through, and the OutCommands is a semicolon separated list of what commands you want to run each iteration.
Our command will look like this:
.foreach (runtime {!dumpheap -mt 0x01b29dfc -short}){.echo ******************;!dumpobj poi(${runtime}+0x3c);.echo cache size:;!objsize poi(${runtime}+0xc) }
We have to enclose runtime (our foreach variable) in ${} so that it doesn’t get interpreted as a symbol.
And the output looks like this:
******************
String: /LM/w3svc/1/root/MemoryIssues
cache size:
sizeof(0x61b4ca0) = 1,336,709,120 (0x4fac9000) bytes (System.Web.Caching.CacheMultiple)
If I would have had more than one application it would have listed all of them one after another, separating them with a line of ***
So here we can see that my one application MemoryIssues, has a cache that contains 1,336,709,120 bytes. WOW!!! that’s a lot!!! We definitely seem to have some sort of cache problem.
What is in my cache? (Using !dumpaspnetcache)
!dumpaspnetcache –stat gives you a nice overview over what types of objects you are storing in cache. (check the post on !dumpheap –stat to understand TotalSize).
0:023> !dumpaspnetcache -stat
Going to dump the ASP.NET Cache.
MT Count TotalSize Class Name
0x0211cc9c 1 20 System.Web.Security.FileSecurityDescriptorWrapper
0x020c242c 2 56 System.Web.UI.ParserCacheItem
0x0206c66c 5 260 System.Web.Configuration.HttpConfigurationRecord
0x0c2e7014 1 316 System.Web.Mobile.MobileCapabilities
0x79b94638 4 376 System.String
0x0c2eaeb4 151 7,248 System.Web.SessionState.InProcSessionState
Total 164 objects, Total size: 8,276
!dumpaspnetcache –s gives you a summary of your objects with priority (removable or NotRemovable etc.) along with create time, expires and last updated which can be useful to see if you are caching something for too long.
The ASP.NET 1.1 cache uses an LRU (Least Recently Used) model for removing objects from cache if memory is needed, for the objects that are not marked NotRemovable.
In this example we can see for example some **InProcSessionState** items (sessions) that started around 15:31:36 and expire on 15:51:36 (after 20 min) and during that time they can not be removed (NotRemovable) by someone trying to clean up the cache to lower memory usage.
0:023> !dumpaspnetcache -s
Going to dump the ASP.NET Cache.
Address MT Priority Create Time Expires Last Updated Key Class Name
0x061d2658 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:35 System.Web.HttpConfig: System.Web.Configuration.HttpConfigurationRecord
0x021b1acc 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:35 System.Web.HttpConfig:/MEMORYISSUES System.Web.Configuration.HttpConfigurationRecord
0x061b3f7c 0x79b94638 Normal 01/25/2006 15:31:35 01/25/2006 15:41:35 01/25/2006 15:31:35 ISAPIWorkerRequest.MapPath:/MemoryIssues/StoringStuffInSessions.aspx System.String
0x021b503c 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.HttpConfig:/MEMORYISSUES/STORINGSTUFFINSESSIONS.ASPX System.Web.Configuration.HttpConfigurationRecord
0x021b80b4 0x0211cc9c Normal 01/25/2006 15:31:36 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.Security.FileSecurityDescriptorWrapper:c:\inetpub\wwwroot\MemoryIssues\StoringStuffInSessions.aspx System.Web.Security.FileSecurityDescriptorWrapper
0x021bc74c 0x020c242c NotRemovable 01/25/2006 15:31:36 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.UI.TemplateParser:/MemoryIssues/StoringStuffInSessions.aspx System.Web.UI.ParserCacheItem
0x022028fc 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:x4ud0tb53l2wfo554aw0yny4 System.Web.SessionState.InProcSessionState
0x02202a10 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:iw0thd55js0d44qjm2cwao45 System.Web.SessionState.InProcSessionState
0x02202cfc 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:2r1dtbzh00tanc451pejqqrl System.Web.SessionState.InProcSessionState
…
And finally, if you want to drill down to what cache objects were largest you can use the .foreach again
0:023> .foreach (cachedObject {!dumpaspnetcache -short}) {!objsize ${cachedObject}}
sizeof(0x626b1b8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6268284) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6265318) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62623ac) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x625f440) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x625c4d4) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6259568) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62565fc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6253690) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6250724) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x624d7b8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x624a84c) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62485e8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6248334) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6248080) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6247dcc) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62479a8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62476bc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
…
So in this particular case, most of our cache is used up by large sessions (~12 and 8 MB each), given this knowledge we would of course drill down into the sessions and find out what is in them, but I’ll leave that for my next post.
Phew! :) At last I have the prerequisites for the post I’m writing on sessions
Until then…
