Tess Ferrandez

Principal Software Engineer at Microsoft. Solving the problems of the world with a debugger, a sprinkle of ML and a bag of chips

3 minute read

In August I wrote about how you could cause a nasty high memory situation by using CacheItemRemovedCallbacks improperly, today I got a case where improper use of CacheItemRemovedCallback caused a crash, so I figured I’ll use it to show you how you can use them to bring down your website in seconds…

Problem description

The scenario is a web site that constantly crashes within minutes or seconds of startup, and a dump has been taken on process shutdown (adplus -crash -pn w3wp.exe)

Debugging the issue

If we open the memory dump with windbg and take a quick peak at the call stack that calls Kernel32!TerminateProcess (with kb) we see:

0:026> kb
ChildEBP RetAddr Args to Child
03744980 7a059b67 ffffffff 800703e9 9fa3661f kernel32!TerminateProcess
037449e4 7a059c79 0374498c 00166268 00004000 mscorwks!EEPolicy::HandleFatalStackOverflow+0xd8
03744a08 79fce2f0 00000001 00000000 00000000 mscorwks!EEPolicy::HandleStackOverflow+0x157
03744a24 7c82eeb2 03744b08 0376f27c 03744b1c mscorwks!COMPlusFrameHandler+0xd8
03744a48 7c82ee84 03744b08 0376f27c 03744b1c ntdll!ExecuteHandler2+0x26
03744af0 7c82ecc6 03743000 03744b1c 03744b08 ntdll!ExecuteHandler+0x24
03744af0 77e55e02 03743000 03744b1c 03744b08 ntdll!KiUserExceptionDispatcher+0xe
03744e3c 79feef3c e053534f 00000000 00000000 kernel32!RaiseException+0x53
03744e54 7a0900a3 00000001 79124228 03744ec8 mscorwks!ReportStackOverflow+0x7e
03744e64 79e88774 00000094 00000000 00080000 mscorwks!Alloc+0x3b
03744f30 79e7e877 653dbde2 03744fe0 00000001 mscorwks!AllocateArrayEx+0x1d1
03744ff4 652c012b 00000000 1cd07984 1cd07a0c mscorwks!JIT_NewArr1+0x167

So the cause for the process termination was a fatal StackOverflowException, which is typically caused by a recursive loop.

The next step is to take a look at the .net call stack (with !clrstack from sos.dll), and doing so will reveal a long call stack with the following contents

<stack cut to save space>
03761b14 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
03761b4c 05d54422 MyNameSpace.MyClass.RefreshCache(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
03761b8c 6601e39c System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
03761bc8 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
03761c04 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
03761cd4 65fa82ea System.Web.Caching.CacheMultiple.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
03761cf8 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
03761d54 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
03761d8c 05d54422 MyNameSpace.MyClass.RefreshCache(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
03761dcc 6601e39c System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
03761e08 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
03761e44 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
03761f14 65fa82ea System.Web.Caching.CacheMultiple.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
03761f38 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
03761f94 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
03761fcc 05d54422 MyNameSpace.MyClass.RefreshCache(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
0376200c 6601e39c System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
03762048 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
03762084 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
03762154 65fa82ea System.Web.Caching.CacheMultiple.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
03762178 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
037621d4 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
0376220c 05d54422 MyNameSpace.MyClass.RefreshCache(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
0376224c 6601e39c System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
03762288 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
037622c4 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
03762394 65fa82ea System.Web.Caching.CacheMultiple.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
037623b8 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
03762414 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
0376244c 05d54422 MyNameSpace.MyClass.RefreshCache(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
0376248c 6601e39c System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
037624c8 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
<stack cut to save space>

So what do we have here?

Well, we can see that MyNameSpace.MyClass.RefreshCache inserts something in cache, which in turn updates the cache and for some reason closes out a cache item, and calls the RefreshCache function again.

If we take a peak at the MyNameSpace.MyClass.RefreshCache function it looks like this:

protected void RefreshCache(string key, object item, CacheItemRemovedReason reason)
{
    ...
    ContextHandler.Cache.Insert(
        "MyCacheItem",
        CacheValue,
        null,
        DateTime.Now.AddHours(9),
        TimeSpan.Zero,
        CacheItemPriority.NotRemovable,
        new CacheItemRemovedCallback(this.RefreshCache));
}

So from first look, it looks like a very logical way to perform what needs to be performed. What we want to do is insert a new item, and in the case where it is removed we want to reinsert it using the same function. In fact, from looking at it, it is very hard to see how this could ever cause a problem.

But, dear Watson, there is a major problem with this code….

When an item is inserted the first time into cache everything will be cool, but what will happen the next time you go through this function?

Answer: The insert will insert a new cache item “MyCacheItem”, and at the same time it will remove the previous copy of this item, causing the CacheItemRemovedCallback handler to be called (RefreshCache), this will in turn insert a new copy, remove the old one and so on and so on… and we have ourselves a never ending recursive loop that causes a StackOverflowException and a crashing website.

Final thoughts

The morale of the story is, don’t use the CacheItemRemovedCallback to add a new copy of the same cache item. Instead use sliding expiration, and instead add the cached item when you try to access the cached item and this item is null.

Til next time, Tess

You May Also Enjoy

Naming tests

7 minute read

I’m a sucker for clean, minimalistic code, that is easy to read, understand and maintain.

Organizing ASP.NET Core Minimal APIs

4 minute read

I often hear (on social media or direct comments) people saying you can’t use minimal APIs for “real production” apps.