ASP.NET Memory - Identifying pages with high Viewstate
From time to time we get issues with high memory and performance issues due to massive viewstate. I have talked about it before here, but I thought I’d show some techniques for finding out which pages have high viewstate.
If you look at a memory dump of your process in windbg and notice that your large object heap is riddled with viewstate strings, that doesn’t really tell you much, except for that you have some pages that have high viewstate, so the goose chase begins trying to find the source of these viewstate strings.
Btw, if the sentence above made no sense to you at all, you might want to check out the viewstate debugging article, to figure out how you identify that you have lots of large viewstate strings in the first place…
In this post I will show two ways of figuring out which page is generating the viewstate
- Using the IIS logs
- Inspecting the viewstate strings
To inspect the viewstate in the pages found in the IIS logs and the viewstate strings I am using Fritz Onion’s viewstate decoder which you can download here
Getting the viewstate strings and identifying that we have large viewstate
The scenario here is usually that we have slow running pages and/or memory bloat in the asp.net process, and while looking at the memory we notice lots of large strings on the large object heap
0:026> !dumpheap -min 85000
------------------------------
Heap 0
Address MT Size
0aec1b38 790fd8c4 130984
0aee1af0 790fd8c4 131048
0af01ae8 790fd8c4 131048
0af21ae0 790fd8c4 329928
0af723b8 790fd8c4 329928
0afc2c90 790fd8c4 329928
0b013568 790fd8c4 329928
total 7 objects
------------------------------
Heap 1
Address MT Size
0ceb0048 7912dae8 131088
0ced0068 790fd8c4 263616
0cf10638 790fd8c4 130984
0cf305f0 790fd8c4 131048
0cf505e8 790fd8c4 329920
0cfa0eb8 790fd8c4 329928
0cff1790 790fd8c4 329928
0d042068 790fd8c4 329928
0d092940 790fd8c4 329928
0d0e3218 790fd8c4 329928
total 10 objects
------------------------------
total 17 objects
Statistics:
MT Count TotalSize Class Name
7912dae8 1 131088 System.Byte[]
790fd8c4 16 4218000 System.String
Total 17 objects
SOS will not automatically print out the strings since this takes some time and clutters the image a bit, but we can print out all the large strings with a foreach command.
In this example below I loop through all the strings over 85000 bytes (i.e. the ones on the large object heap), and then we print them with -nofields since we are only interested in the strings themselves.
0:026> .foreach (longstring {!dumpheap -type System.String -min 85000 -short}){.echo ***;.echo ${longstring};!do -nofields ${longstring} }
***
0aec1b38
Name: System.String
MethodTable: 790fd8c4
EEClass: 790fd824
Size: 130978(0x1ffa2) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: /wEPDwULLTExMzQzMzE4NDIPZBYCAgMPZBYEAgEPPCsADQBkAgMPEA8WAh4LXyFEYXRhQm91bmRnZBAV6AcXU3lzdGVtLkRhdGEuRGF0YVJvd1ZpZXcXU
...
I am just showing partial output to save some space, but what we see here is a string starting with /wEP which is typical for viewstate strings in 2.0. In 1.1 they usually start with dDW. Pretty much all the large strings in this dump look about the same, so from here we can tell that we have some pages emitting viewstate > 85000 bytes which means that each time the user views these pages they will have to download this data.
If the viewstate is large enough and we have enough concurrent users viewstate like this will cause both bandwidth issues and memory issues.
Identifying the pages with large viewstate in the IIS logs
The simplest way to identify which pages have large viewstate is to start logging Bytes Sent and Bytes Received in the IIS logs and go through the IIS logs to identify pages that have a lot of bytes sent (sc-bytes).
Starting with the ones with the highest bytes sent you can then use Fitz Onion’s Viewstate decoder to decode the viewstate by entering the URL, extracting the viewstate and decoding it…
In this example I have a page http://localhost/HighViewstate/Default.aspx that contains a grid view (grdProducts) and I have filled this up with 1000 rows, where each row has a product name (Product 1, Product 2, etc.)
When we decode it, we can see that my page has a viewstate of 164952 characters (~329904 bytes) which corresponds nicely to some of the strings on the LOH
0cfa0eb8 790fd8c4 329928
0cff1790 790fd8c4 329928
Further, in the viewstate decoder (ControlState) we can see that we have one dictionary entry, grdProducts, so the viewstate seems to mostly pertain to this control, in the Viewstate box we can also see what the viewstate contains and in this case it is the different values in the data grid (Product 443, Product 444 etc.).
So based on this we can conclude that if we want to lower the amount of viewstate this page is emitting we would have to do something about the grid view, like removing the viewstate, filtering the data or similar.
Inspecting the viewstate strings from the dump
Inspecting the strings takes a lot longer and requires debugging, but it is also a more precise method.
I would recommend that you start by inspecting the IIS logs, but if you already have a dump and have identified large viewstate strings, then this is the way to figure out where they came from.
For this we need to extract the whole string, plug it into the decoder and click Decode->.
If you run !do on the string, this will print out the string, but unfortunately it will only print out a part of the string so we need to figure out a way to get the full string so that we can plug it into the viewstate decoder.
Looking at this string for example
0:026> !do 0d0e3218
Name: System.String
MethodTable: 790fd8c4
EEClass: 790fd824
Size: 329922(0x508c2) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: /wEPDwULLTE3MDA2MjU5MjQPZBYCAgMPZBYEAgEPEA8WBh4NRGF0YVRleHRGaWVsZAULUHJvZHVjdE5hbW...
We can see that the size is 329922 bytes or 0x508c2 hex bytes, and we can print out the string using du <address> <address>+size, or to be more specific we would print it out using du <address>+0xc <address>+size because the actual string starts at an offset of 0xc. du means dump unicode, i.e. du <start> <end> will print the contents between <start> and <end> in unicode, which is what we want.
0:026> du 0d0e3218+0xc 0d0e3218+0x508c2
0d0e3224 "/wEPDwULLTE3MDA2MjU5MjQPZBYCAgMP"
0d0e3264 "ZBYEAgEPEA8WBh4NRGF0YVRleHRGaWVs"
0d0e32a4 "ZAULUHJvZHVjdE5hbWUeDkRhdGFWYWx1"
0d0e32e4 "ZUZpZWxkBQtQcm9kdWN0TmFtZR4LXyFE"
0d0e3324 "YXRhQm91bmRnZBAV6AcJUHJvZHVjdCAw"
0d0e3364 "CVByb2R1Y3QgMQlQcm9kdWN0IDIJUHJv"
0d0e33a4 "ZHVjdCAzCVByb2R1Y3QgNAlQcm9kdWN0"
0d0e33e4 "IDUJUHJvZHVjdCA2CVByb2R1Y3QgNwlQ"
0d0e3424 "cm9kdWN0IDgJUHJvZHVjdCA5ClByb2R1"
0d0e3464 "Y3QgMTAKUHJvZHVjdCAxMQpQcm9kdWN0"
0d0e34a4 "IDEyClByb2R1Y3QgMTMKUHJvZHVjdCAx"
0d0e34e4 "NApQcm9kdWN0IDE1ClByb2R1Y3QgMTYK"
0d0e3524 "UHJvZHVjdCAxNwpQcm9kdWN0IDE4ClBy"
0d0e3564 "b2R1Y3QgMTkKUHJvZHVjdCAyMApQcm9k"
0d0e35a4 "dWN0IDIxClByb2R1Y3QgMjIKUHJvZHVj"
0d0e35e4 "dCAyMwpQcm9kdWN0IDI0ClByb2R1Y3Qg"
0d0e3624 "MjUKUHJvZHVjdCAyNgpQcm9kdWN0IDI3"
0d0e3664 "ClByb2R1Y3QgMjgKUHJvZHVjdCAyOQpQ"
0d0e36a4 "cm9kdWN0IDMwClByb2R1Y3QgMzEKUHJv"
...
ok, so this gives us the string, and we can copy and plug it into the viewstate decoder if it wasn’t for the fact that we would have to manipulate the output, removing the first column and the double quotes which is a bit of a hassle.
To get a clean string, we can execute the same command, and encapsulate it in a .foreach so that we skip the first column by adding /pS 1 /ps 1
0:026> .foreach /pS 1 /ps 1 (token {du 0d0e3218+0xc 0d0e3218+0x508c2}){.echo ${token}}
/wEPDwULLTE3MDA2MjU5MjQPZBYCAgMP
ZBYEAgEPEA8WBh4NRGF0YVRleHRGaWVs
ZAULUHJvZHVjdE5hbWUeDkRhdGFWYWx1
ZUZpZWxkBQtQcm9kdWN0TmFtZR4LXyFE
YXRhQm91bmRnZBAV6AcJUHJvZHVjdCAw
CVByb2R1Y3QgMQlQcm9kdWN0IDIJUHJv
ZHVjdCAzCVByb2R1Y3QgNAlQcm9kdWN0
IDUJUHJvZHVjdCA2CVByb2R1Y3QgNwlQ
cm9kdWN0IDgJUHJvZHVjdCA5ClByb2R1
Y3QgMTAKUHJvZHVjdCAxMQpQcm9kdWN0
IDEyClByb2R1Y3QgMTMKUHJvZHVjdCAx
NApQcm9kdWN0IDE1ClByb2R1Y3QgMTYK
UHJvZHVjdCAxNwpQcm9kdWN0IDE4ClBy
b2R1Y3QgMTkKUHJvZHVjdCAyMApQcm9k
dWN0IDIxClByb2R1Y3QgMjIKUHJvZHVj
...
And now, finally we have a clean string that we can copy into the Viewstate String: textbox in the viewstate decoder. When we decode it we will get a very similar output to the one we got when decoding viewstate from an URL, and based on the contents of the viewstate (i.e. finding out that it belongs to grdProducts, and has the contents Product 1, Product 2 etc. ) we can narrow down the page that it belongs to.
And that is pretty much all there is to it…
Laters, Tess
