Tess Ferrandez

Principal Software Engineer at Microsoft. Solving the problems of the world with a debugger, a sprinkle of ML and a bag of chips

6 minute read

When you use very complex row filters or expressions on datasets or data tables you may end up getting a StackoverflowException.

Eber was running into this and comment on another post. Since it is something we see from time to time and it was a bit to long to answer in the comments, here is the why and the how.

Problem description

When browsing an ASP.NET site, intermittently we get “Internet Explorer cannot display the webpage” accompanied by the following event in the event log

Event Type:       Error
Event Source:     .NET Runtime 2.0 Error Reporting
Event Category:   None
Event ID:         5000
Date:             2008-03-31
Time:             09:59:27
User:             N/A
Computer:         MYMACHINE
Description:
EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.3959, P3 45d6968e, P4 system.data, P5 2.0.0.0, P6 471ebf27, P7 1840, P8 0, P9 system.stackoverflowexception, P10 NIL.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Generating dumps

Following the instructions in the StackOverflow lab we end up with a log file where the last event/exception before the crash is the c00000fd StackOverflow exception

Mon Mar 31 10:14:11.557 2008 (GMT+2): (2994.2848): Stack overflow - code c00000fd (first chance)
---
--- 1st chance Stackoverflow exception ----
---------------------------------------------------------------

The next step is to get a memory dump on the actual exception so we can set up a config file similar to the unknown.cfg that we used in the lab, but in this case we will be dumping on the c00000fd exception instead (and save it as SOF.cfg in the debuggers directory)

<adplus>
    <settings>
        <runmode> CRASH </runmode>
    </settings>

    <!-- defining and configuring exceptions -->
    <exceptions>
        <!-- First we redefine all exceptions -->
        <config>
            <code>AllExceptions</code>
            <actions1>Log</actions1>
            <actions2>MiniDump;Log;EventLog</actions2>
        </config>

        <newexception>
            <code> c00000fd </code>
            <name> Stackoverflow </name>
        </newexception>
        <!-- Configuring the custom exception -->
        <config>
            <code> c00000fd </code>
            <actions1>FullDump;Log;EventLog</actions1>
            <actions2>FullDump;Log;EventLog</actions2>
        </config>
    </exceptions>
</adplus>

Finally we run this with adplus -pn w3wp.exe -p SOF.cfg which generates a memory dump on the first chance Stackoverflow Exception when we reproduce the issue.

Debugging the issue

If we open up the dump file in windbg and run kb 2000 and !clrstack to see the native and .net stacks we see the following:

0:016> kb
ChildEBP RetAddr  Args to Child
024f37d4 652857b0 00000000 00000200 06f37854 System_Data_ni!_load_config_used+0x114f8b
024f4230 652857b0 00000000 00000200 06f37854 System_Data_ni!_load_config_used+0x1140e0
024f4c8c 652857b0 00000000 00000200 06f37854 System_Data_ni!_load_config_used+0x1140e0
024f56e8 652857b0 00000000 00000200 06f37854 System_Data_ni!_load_config_used+0x1140e0
024f6144 652857b0 00000000 00000200 06f37854 System_Data_ni!_load_config_used+0x1140e0
...

0:016> .loadby sos mscorwks
0:016> !clrstack
OS Thread Id: 0x2848 (16)
ESP       EIP
024f2dac 6528665b System.Data.BinaryNode.EvalBinaryOp(Int32, System.Data.ExpressionNode, System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[])

So in the native stack (kb) there is clear evidence of recursion (a method calling itself in a recursive loop) but the symbols are not matching up since this is .net methods.

The !clrstack output on the other hand only shows one frame System.Data.BinaryNode.EvalBinaryOp, so the recursion isn’t showing up in there. Sometimes we can get a weird .net stack like this with !clrstack if we are in the middle of handling an exception which is exactly what is happening here.

I have mentioned the !dumpstack command before. This command dumps the raw stack without checking it and doing stack walks to see if it is valid, so you should use it with caution but in some cases like this it may prove very useful in order to see more of the stack.

0:016> !dumpstack
OS Thread Id: 0x2848 (16)
Current frame: (MethodDesc 0x654910c8 +0x1b System.Data.BinaryNode.EvalBinaryOp(Int32, System.Data.ExpressionNode, System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
ChildEBP RetAddr  Caller,Callee
024f37d4 652857b0 (MethodDesc 0x65491070 +0x1c System.Data.BinaryNode.Eval(System.Data.DataRow, System.Data.DataRowVersion)), calling (MethodDesc 0x654910c8 +0 System.Data.BinaryNode.EvalBinaryOp(Int32, System.Data.ExpressionNode, System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
024f37f0 65285a79 (MethodDesc 0x654910b8 +0x11 System.Data.BinaryNode.Eval(System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
024f37f8 6528b5b0 (MethodDesc 0x654910c8 +0x4f70 System.Data.BinaryNode.EvalBinaryOp(Int32, System.Data.ExpressionNode, System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[])), calling (MethodDesc 0x654910b8 +0 System.Data.BinaryNode.Eval(System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
024f4230 652857b0 (MethodDesc 0x65491070 +0x1c System.Data.BinaryNode.Eval(System.Data.DataRow, System.Data.DataRowVersion)), calling (MethodDesc 0x654910c8 +0 System.Data.BinaryNode.EvalBinaryOp(Int32, System.Data.ExpressionNode, System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
024f424c 65285a79 (MethodDesc 0x654910b8 +0x11 System.Data.BinaryNode.Eval(System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
024f4254 6528b5b0 (MethodDesc 0x654910c8 +0x4f70 System.Data.BinaryNode.EvalBinaryOp(Int32, System.Data.ExpressionNode, System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[])), calling (MethodDesc 0x654910b8 +0 System.Data.BinaryNode.Eval(System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
024f4c8c 652857b0 (MethodDesc 0x65491070 +0x1c System.Data.BinaryNode.Eval(System.Data.DataRow, System.Data.DataRowVersion)), calling (MethodDesc 0x654910c8 +0 System.Data.BinaryNode.EvalBinaryOp(Int32, System.Data.ExpressionNode, System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
024f4ca8 65285a79 (MethodDesc 0x654910b8 +0x11 System.Data.BinaryNode.Eval(System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
024f4cb0 6528b5b0 (MethodDesc 0x654910c8 +0x4f70 System.Data.BinaryNode.EvalBinaryOp(Int32, System.Data.ExpressionNode, System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[])), calling (MethodDesc 0x654910b8 +0 System.Data.BinaryNode.Eval(System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
024f56e8 652857b0 (MethodDesc 0x65491070 +0x1c System.Data.BinaryNode.Eval(System.Data.DataRow, System.Data.DataRowVersion)), calling (MethodDesc 0x654910c8 +0 System.Data.BinaryNode.EvalBinaryOp(Int32, System.Data.ExpressionNode, System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
024f5704 65285a79 (MethodDesc 0x654910b8 +0x11 System.Data.BinaryNode.Eval(System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
024f570c 6528b5b0 (MethodDesc 0x654910c8 +0x4f70 System.Data.BinaryNode.EvalBinaryOp(Int32, System.Data.ExpressionNode, System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[])), calling (MethodDesc 0x654910b8 +0 System.Data.BinaryNode.Eval(System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
024f6144 652857b0 (MethodDesc 0x65491070 +0x1c System.Data.BinaryNode.Eval(System.Data.DataRow, System.Data.DataRowVersion)), calling (MethodDesc 0x654910c8 +0 System.Data.BinaryNode.EvalBinaryOp(Int32, System.Data.ExpressionNode, System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
024f6160 65285a79 (MethodDesc 0x654910b8 +0x11 System.Data.BinaryNode.Eval(System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
024f6168 6528b5b0 (MethodDesc 0x654910c8 +0x4f70 System.Data.BinaryNode.EvalBinaryOp(Int32, System.Data.ExpressionNode, System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[])), calling (MethodDesc 0x654910b8 +0 System.Data.BinaryNode.Eval(System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[]))
...
0252f26c 6528d658 (MethodDesc 0x654841f8 +0x88 System.Data.DataExpression.Bind(System.Data.DataTable)), calling mscorwks!JIT_Writeable_Thunks_Buf+0x11f
0252f284 6520dbd2 (MethodDesc 0x65482ee0 +0x12 System.Data.DataView.SetIndex(System.String, System.Data.DataViewRowState, System.Data.IFilter)), calling (MethodDesc 0x65482ee8 +0 System.Data.DataView.SetIndex2(System.String, System.Data.DataViewRowState, System.Data.IFilter, Boolean))
0252f298 6520b8e7 (MethodDesc 0x65482d08 +0xb3 System.Data.DataView.set_RowFilter(System.String))
0252f2b0 0f0207e3 (MethodDesc 0x2c38dc0 +0x143 DataSetPage.Page_Load(System.Object, System.EventArgs))
0252f2cc 66f12980 (MethodDesc 0x66f1bcd0 +0x10 System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs))
0252f2d8 6628efd2 (MethodDesc 0x66474328 +0x22 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)), calling (MethodDesc 0x66f1bcd0 +0 System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs))
...

Here we can see that the recursive loop seems to involve EvalBinaryOp calling Eval which again calls EvalBinaryOp.

Note: In the stack it looks like EvalBinaryOp calls Eval which calls Eval which calls EvalBinaryOp. This is not true, instead the call tree actually looks like this

  • EvalBinaryOp
    • Eval (left)
      • EvalBinaryOp
        • Eval (left) …
        • Eval (right) …
    • Eval (right)
      • EvalBinaryOp
        • Eval (left) …
        • Eval (right) …

but because !dumpstack shows all the functions on the stack it looks like the above.

From the stack though we can see that what happens here is that DataSetPage.Page_Load sets the row filter on a DataView/DataTable, and in doing so it will bind the filter expression and start evaluating it.

If the filter looks like this ID=0 or ID=1 or ID=2 it will do Eval(ID=0) or Eval(ID=1 or ID=2) => Eval(ID=0) or (Eval (ID=1) or Eval(ID=2)) etc. and if this filter or expression is very complex such that there are a lot of operands, the recursion goes very deep. If it is long enough or has enough operands it will be so deep that the stack space will be exhausted before the whole expression is evaluated.

If we look at !dso to see the objects on the stack we can find the expression to see what it looks like.

0:016> !dso
OS Thread Id: 0x2848 (16)
ESP/REG  Object   Name
ebx      03035cc0 System.Data.BinaryNode
esi      03035be8 System.Data.BinaryNode
024f37c8 03035cc0 System.Data.BinaryNode
024f37cc 03035be8 System.Data.BinaryNode
024f37e4 06f37854 System.Data.DataRow
024f37e8 03035bd0 System.Data.BinaryNode
024f37ec 03035af8 System.Data.BinaryNode
024f37f0 06eb07c4 System.DBNull
024f3824 06eb07c4 System.DBNull
024f3828 06eb07c4 System.DBNull
024f382c 03035cd8 System.Data.BinaryNode
024f4224 03035db0 System.Data.BinaryNode
024f4228 03035cd8 System.Data.BinaryNode
024f4240 06f37854 System.Data.DataRow
024f4244 03035cc0 System.Data.BinaryNode
024f4248 03035be8 System.Data.BinaryNode
024f424c 06eb07c4 System.DBNull
...
0252f278 0301ad3c System.Data.DataExpression
0252f280 030179d8 System.String    ID=0 OR ID=1 OR ID=2 OR ID=3 OR ID=4 OR ID=5 OR ID=6 OR ID=7 OR ID=8 OR ID=9 OR ID=10 OR ID=11 OR ID=12 OR ID=13 OR ID=14 OR ID=15 OR ID=16 OR ID=17 OR ID=18 OR ID=19 OR ID=20 OR ID=21 OR ID=22 OR ID=23 OR ID=24 OR ID=25 OR ID=26 OR ID=27 OR ID=28 OR ID=29 OR ID=30 OR ID=31 OR ID=32 OR ID=33 OR ID=34 OR ID=35 OR ID=36 OR ID=37 OR ID=38 OR ID=39 OR ID=40 OR ID=41 OR ID=42 OR ID=43 OR ID=44 ...
...
0252f55c 06f14538 ASP.datasetpage_aspx
0252f560 06f14538 ASP.datasetpage_aspx
0252f574 02ebcab0 System.AsyncCallback
0252f578 06f14538 ASP.datasetpage_aspx
0252f5b0 06f122ec System.Web.HttpContext
0252f5c8 02ebcab0 System.AsyncCallback
...

We could have dumped out the DataExpression object by doing !do 0301ad3c but it is really not necessary here since the filter expression is already printed out in the string that precedes the DataExpression (the ID=0 or ID=1 or…)

Resolution and final words

I could get up to around 75 operands before getting a stack overflow but mileage may vary as it depends on the size of the objects on the stack and how deep the call chain is before you get to the expression evaluation.

Typically when I get these issues the expression wasn’t meant to be that long, so there was some logic issue causing the app to add on more and more filters. If however you need an extremely long expression you may want to spawn off a new thread (with a bigger call stack) to perform the setting of the row filter.

Laters,

Tess

You May Also Enjoy

Naming tests

7 minute read

I’m a sucker for clean, minimalistic code, that is easy to read, understand and maintain.

Organizing ASP.NET Core Minimal APIs

4 minute read

I often hear (on social media or direct comments) people saying you can’t use minimal APIs for “real production” apps.