SPListAnalysis: A sample DebugDiag v2.0 Analysis Rule

DebugDiag is a tool for collecting and analyzing memory dumps. Version 2.0 was just released and makes it extremely easy to write your own custom analysis rules using any language that target the CLR.

SPListAnalysis is a sample analysis rule I implemented to show the basic steps involved in the process. The rule does the following:

  • Iterates over all threads found in the process dumps to identify any threads that are querying an SPList object (or causing it to load the items contained therein).
  • Identifies the CAML query associated with the operation.
  • Generates a warning if the query contains more than 10 fields, loads all the fields in the list, or if the query fails to specify a RowFilter.
  • Prints the stack traces and query details for each matching thread.

The code for this sample can be found here.

This particular sample if very simple, consisting of a single class called SPListAnalysis, which implements the IHangDumpRule interface to process one dump at a time. It also implements the IAnalysisRuleMetadata interface to define the category and description for this analysis.

DebugDiag Analysis

DebugDiag Analysis

The basis for the analysis is looking at all threads that contain a significant function in the call stack, and then looking for a matching object in the stack frames (similar to doing a !DumpStackObjects) command):

private void RunSPListAnalysis(NetDbgObj debugger)
{
  // initialize report
  foreach ( var thread in debugger.Threads )
  {
    AnalyzeThread(thread);
  }
  // report findings
}

private void AnalyzeThread(NetDbgThread thread)
{
  // ...
  if ( ContainsFrame(thread, SPLIST_FILL_FRAME) )
  {
    //...
    dynamic obj = thread.FindFirstStackObject("Microsoft.SharePoint.SPListItemCollection");
    if ( obj != null )
    {
      String viewXml = (string)obj.m_Query.m_strViewXml;
      //...
      XDocument doc = XDocument.Parse(viewXml);
      AnalyzeViewXml(thread, doc);
      //...
    }
    PrintThreadStack(thread);
  }
}

Reporting the results is mostly a matter of generating HTML. However, we can also generate warnings that are included at the top of the report, allowing us to quickly alert the user that something of interest was found in the dump:

private void ReportThreadsWithLargeSPQueries()
{
  if ( threadsWithLargeSPQueries.Count > 0 )
  {
    StringBuilder sb = new StringBuilder();
    sb.Append("The following threads appear to be executing SPList queries requesting many fields.");
    sb.Append("<br/><ul>");
    foreach ( int threadId in threadsWithLargeSPQueries.Keys )
    {
      sb.AppendFormat("<li><a href='#thread{0}'>{0}</a> ({1} fields)</li>",
          threadId, threadsWithLargeSPQueries[threadId]);
    }
    sb.Append("</ul>");
    this.manager.ReportWarning(sb.ToString(), "");
  }
}

The resulting report will look like this:

DebugDiag Warnings

DebugDiag Warnings

Enjoy!

Trying out Resharper 5.0

There are many .NET developers that can’t live without Jetbrain’s R# product. I’m not really one of them. Don’t get me wrong, I like the idea of R# and some of the features it offers, it’s just that it could be so much better if it didn’t keep getting in my way!

Let’s get something out of the way before I dive into what I’d like to see improved in R#: I bought my own R# license out of my own money. I do think it’s a fairly reasonably priced product for all the features it offers.

Improvements I’d like to see

  • More modularity: I work in a fairly specific environment where some of the tools R# provides don’t help at all, and in fact, I don’t need them or want them in my way.
    Hence, I’d love to see top-level options to turn off broad features. Right now pretty much only two features have this: IntelliSense and Code Analysis. I’d love to see the same for things like code formatting and the like.
  • Better options for saving/sharing settings: It’s a pain in the neck to configure R# after installing it (or keeping it configured after that). The Options dialog is a mess, and there are way too many settings and no easy way (that I can find) to export your settings to a file I can save and keep around for later installations.
    Yes, I’m aware that R# saves its settings in an external XML file you can save, but it’s stored some place I can never remember, gets overwritten and messed up all the time when running multiple VS options at the same time and I always forget to fish it out before dumping a VM or something, so it’s as if it was never there.
    Just give me a way to export/import settings; that’s a lot easier to live with.
  • Unit Testing: I’m rather happy using the test runner in R#, use it all the time. However, it would be really nice if there was a way to set it up so that the Debug option worked for mixed managed/unmanaged solutions.
    Right now, the Debug option only enables the managed debugger, but I also work with lots of code that includes managed tests over a mixture of managed and unmanaged C++ code and to debug that I always have to use the NUnit test runner instead.

Things that just piss me off

There are a few things that I can’t stand when using R#, however; to the point where I have to disable it just to get any work done without losing my mind:

  • Crashes: R# 4.5 was notoriously unstable. It would crash all the freaking time for me. R# 5.0 seems better on VS2010, but not on VS2008 to the point where the simplest reproducible crash that R# 4.5 would cause is still present in 5.0. Here’s just an example: Launch a new, empty VS2008 instance (with no auto-project opening or anything), then drag a C# file from explorer to VS. Watch VS crash and burn.
  • Stop messing with ViEmu: I’m a long time user of ViEmu, absolutely love it. However, R# often messes with ViEmu on VS2008, in particularly with the keyboard handling. The one that always causes me trouble is firing a new empty VS2008 instance and loading a solution that contains only unmanaged C++ projects, where R# shouldn’t even get involved, and watch the keyboard go all crazy on you.
    Only way to work when this happens is to disable the R# add-in temporarily, which is really annoying. I know people love R#, but for me ViEmu >>> R#. Mess with ViEmu and it’s you that’s getting canned.
  • Smarter Settings: Given my concerns about making it easier to export/import my R# settings, and my desire to disable code formatting, it’s not surprising this is an issue for me.
    See, I already have all my VS code formatting rules properly configured and persisted in my .VSSETTINGS file. Install VS again and in seconds its there. Install R# and instead of defaulting its own code formatting rules to my existing VS settings, it applies its own coding conventions. That’s just disrespectful and annoying, to be honest.

Anyway, let’s see how life with R# 5.0 goes. So far, the VS2010 integration seems nice, but I will still be primarily a VS2008 user for a long time, so it sucks a lot that stability there hasn’t improved.

Downloading WinDBG

I only use WinDBG every once in a while, but, when I need it, I really need it and need it now. Kevin Dente pointed out earlier today that apparently, the latest versions of WinDBG was not available as a standalone installer, but only as part of the Windows Driver Kit ISO download.

That’s right. You now need to download a 620MB ISO, find a tool to open it up with (since Windows still lacks native support for opening ISO files directly) to extract a 17MB installer for WinDBG. From the WinDBG download page:

This current version of Debugging Tools for Windows is available as part of the Windows Driver Kit (WDK) release 7.1.0. To download the WDK and manually install Debugging Tools for Windows:

1. Download the WDK and burn the ISO to a DVD or other installation media. (The WDK ISO file is approximately 620 MB to download.)

2. Open the ISO from the installation media and open the Debuggers directory.

3. In the Debuggers directory, run the appropriate MSI or setup EXE for x86 and follow the instructions in the Setup Wizard.

4. After installation is complete, you can find the debugger shortcuts by clicking Start, pointing to All Programs, and then pointing to Debugging Tools for Windows (x86).

This really sounds awesome, on par with other MS utilities that get distributed as an MSI installer that installs an MSI installer that you can install. Yes, I’m being sarcastic.

Get it together, Microsoft; WinDBG is too important for support to screw it up like this.

Converting a .PPTX File to .ODP Format

I had a PowerPoint 2007 presentation this morning that I needed to convert into the OpenDocument Presentation format, to make it usable in OpenOffice 3.0. It was a fairly simple presentation, with no animations or anything, just text and some figures.

My first attempt was to simply use the PPTX importer in Impress, and ended up with mixed results. While the presentation was readable, it had a bunch of things that were not quite right:

  • Bullets were missing (or used the wrong kind)
  • Images in the master slide background had borders around them
  • Titles were the wrong color

Being lazy, I looked around for another option, and remembered that Office 2007 SP1 included some functionality to export documents in OpenDocument format. I decided to give it a try and went into PowerPoint 2007, and executed the Save As –> OpenDocument Presentation command.

I was pleasantly surprised by the results, which were pretty much perfect, except for a couple minor differences due to font size. Definitely much better than I expected, and better than OpenOffice itself did. Not bad!

VirtualBox, VirtualPC and Crashes

I take advantage of both VirtualBox and VirtualPC as core parts of my development environment. I keep several sets of Virtual Machines around on both platforms that I use for core development (writing code) and testing on different platforms.

Because of this, I’m constantly switching between the two, depending on which VM I need to use to try something. I’m finding myself using VirtualBox a lot lately because of the need to run a couple of VMs at the same time, which is something that it does quite a bit better than VirtualPC.

I don’t try to run both of them at the same time, nor do I need them to. I take advantage of hardware virtualization support in my desktop machine (which has an Intel Q6600 processor) in both platforms, which can speed things quite a bit.

The problem, however, is that both tools seem to pretty much assume that they are the only ones that will try to probe the processor virtualization support.

Occasionally, I’ll make a mistake and click on the wrong icon in the Quick Launch toolbar and end up starting the wrong application. Sometimes I end up starting VPC when I’m already working on VBox, and sometimes I’ll end up starting VBox when I’ve already got a VM in VPC open.

The end result is almost always the same: Windows crashing right away with a Blue Screen of Death (BSOD), usually with a fault in vmm.sys.

Again, I have no need to use both tools at the same time, but it sure would be nice if accidentally clicking the wrong icon didn’t force me to spend 10 minutes while my machine comes back up and I can setup my environment as it was before the crash, and probably lose some recent work, as well!

Dumping Static Fields with WinDBG

Here’s something that I very ocassionally have to do and always forget it’s done: Getting the values of static fields in a managed class when debugging using WinDBG with the SOS extensions:

Get the address where the CLASS object is located in memory. There are several ways to do this, but an easy one is to use the Name2EE function, passing in the assembly or module name and the fully qualified type name:

0:007> !name2ee mscorlib.dll System.Console
Module: 0000064278854000 (mscorlib.dll)
Token: 0x000000000200008b
MethodTable: 00000642788c8d10
EEClass: 0000064278a271a8
Name: System.Console

Now, we can use the DumpClass function, passing in the EEClass value returned by the previous command:

0:007> !dumpclass 0000064278a271a8
Class Name: System.Console
mdToken: 000000000200008b (C:\WINDOWS\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Parent Class: 00000642788c0c30
Module: 0000064278854000
Method Table: 00000642788c8d10
Vtable Slots: 4
Total Method Slots: 78
Class Attributes: 100181  Abstract, 
NumInstanceFields: 0
NumStaticFields: d
              MT    Field   Offset                 Type VT     Attr            Value Name
00000642788f5aa0  40002ae       d8 System.IO.TextReader  0   shared           static _in
                                 >> Domain:Value  000000000015b120:NotInit  <<
00000642788cdd08  40002af       e0 System.IO.TextWriter  0   shared           static _out
                                 >> Domain:Value  000000000015b120:NotInit  <<
00000642788cdd08  40002b0       e8 System.IO.TextWriter  0   shared           static _error
                                 >> Domain:Value  000000000015b120:NotInit  <<
000006427895dcb0  40002b1       f0 ...ancelEventHandler  0   shared           static _cancelCallbacks
                                 >> Domain:Value  000000000015b120:NotInit  <<
000006427895db88  40002b2       f8 ...le+ControlCHooker  0   shared           static _hooker
                                 >> Domain:Value  000000000015b120:NotInit  <<
00000642788c91e0  40002b3       d0 ...ative+InputRecord  1   shared           static _cachedInputRecord
                                 >> Domain:Value  000000000015b120:NotInit  <<
00000642788dca98  40002b4      990       System.Boolean  1   shared           static _haveReadDefaultColors
                                 >> Domain:Value  000000000015b120:NotInit  <<
00000642788c9f78  40002b5      994          System.Byte  1   shared           static _defaultColors
                                 >> Domain:Value  000000000015b120:NotInit  <<
00000642788dca98  40002b6      998       System.Boolean  1   shared           static _wasOutRedirected
                                 >> Domain:Value  000000000015b120:NotInit  <<
00000642788dca98  40002b7      99c       System.Boolean  1   shared           static _wasErrorRedirected
                                 >> Domain:Value  000000000015b120:NotInit  <<
00000642788c0cd8  40002b8      100        System.Object  0   shared           static s_InternalSyncObject
                                 >> Domain:Value  000000000015b120:NotInit  <<
00000642788c9618  40002b9      980        System.IntPtr  1   shared           static _consoleInputHandle
                                 >> Domain:Value  000000000015b120:NotInit  <<
00000642788c9618  40002ba      988        System.IntPtr  1   shared           static _consoleOutputHandle
                                 >> Domain:Value  000000000015b120:NotInit  <<

Notice that this was on an x86-64 system running 64-bit Windows, which is why all pointer values are 64-bit long.

Configuring Putty with Tango

I use Putty relatively often, and it’s a great tool to have on your Windows utilities if you do any kind of SSH-connectivity every now and then. However, the way customizations are made leaves somewhat to be desired at times.

In particular, one of the things that annoys me about Putty is that pretty much all customizations seem to be done only at the profile level, even those that relate to the appearance of the terminal. And lets face it, the default Putty configuration is ugly and hard to use on a screen with a large resolution.

Meanwhile, I’ve gotten pretty used on my Linux boxes to using gnome-terminal with the Tango color scheme, which is relatively nice as far as these things go. It’s perfectly possible to configure Putty to use this colorscheme, but let’s face it: it’s cumbersome. Plus, moving your profiles around by exporting/importing registry keys ain’t great, either.

Putty with Tango scheme

So I built this very simple PowerShell script (puttyconf.vim) to fill in all the necessary registry values for the Tango scheme on an existing Putty profile in the registry. Enjoy!

param([string]$puttyConf)

$sessionKey = "HKCU:\Software\SimonTatham\PuTTY\Sessions\$puttyConf"
if ( test-path $sessionKey ) {
   $values = @{
      "Xterm256Colour" = 0x00000001
      "BoldAsColour" = 0x00000001
      "Colour0" = "187,187,187"
      "Colour1" = "255,255,255"
      "Colour2" = "8,8,8"
      "Colour3" = "85,85,85"
      "Colour4" = "0,0,0"
      "Colour5" = "0,255,0"
      "Colour6" = "46,52,54"
      "Colour7" = "85,87,83"
      "Colour8" = "204,0,0"
      "Colour9" = "239,41,41"
      "Colour10" = "78,154,6"
      "Colour11" = "138,226,52"
      "Colour12" = "196,160,0"
      "Colour13" = "252,233,79"
      "Colour14" = "52,101,164"
      "Colour15" = "114,159,207"
      "Colour16" = "117,80,123"
      "Colour17" = "173,127,168"
      "Colour18" = "6,152,154"
      "Colour19" = "52,226,226"
      "Colour20" = "211,215,207"
      "Colour21" = "238,238,236"
   }
   $values.Keys | %{
      set-itemproperty $sessionKey $_ $values[$_]
   }
} else {
   write-error "Profile '$puttyConf' does not exist in the Putty configuration"
}

ViEmu’s Keybindings Dialog

ViEmu is a Vi/Vim emulator add-in for Visual Studio that is the only one I consider a must-have in my Visual Studio arsenal: I love it. I’ve mentioned ViEmu on this blog in the past, and I still consider ViEmu a wonderful addition to make Visual Studio usable for us Vim users.

That’s not to say that ViEmu is perfect. I’m very happy with ViEmu most of the time, but there are a few issues that could be improved. One of those, is the Keyboard Management Dialog.

The main purpose of this dialog is to handle conflicts between keyboard mappings in your selected Visual Studio keyboard layout and Vim shortcuts/commands that ViEmu supports. You can get to it during the first few uses, or by going to Tools –> Options –> ViEmu –> General and clicking on the Keyboard button on the right.

In general terms, ViEmu does a good job of finding the conflicts and telling you about them. My problem, however, is with the dialog itself, which looks like this:

viemu-keys

I’m going to be brutally honest here: I have no freaking clue how this dialog is supposed to work. I somehow manage to set up my settings to something workable each time I re-install ViEmu, but to this day, I don’t know exactly how I accomplish that.

There are a couple of problems with this dialog:

  1. It’s not clear (to me, at least) what the buttons do. If I click on the Remove button on the right, does it remove the keybinding clashes? or does it leave the ViEmu one? What does the Forget button do? (Seriously, I have no idea).
  2. The clashing bindings appear to be only manageable as a single batch. Sometimes, I’d like an easier way to manage individual clashing bindings. While I’m fairly sure you can do that from VS’ own keyboard layout options screen, these will get lost in all the noise, so it’s not as obvious what you’re doing.

All this said, I do think the “Keys Scanned” textbox is a nice touch; it’s fairly obvious what it does and seems to work well.

Anyway, ViEmu still rocks, and I’m looking forward to future versions of the product to see what they come up with.

Some Linux Apps I Discovered Recently

It’s no secret that I’ve been slowly getting back into the Linux world. I’ve always liked a lot of things about Unix in general, and though I spent most of my work-day doing Windows development, I still like to fool around with Linux a bit in my free time.

Recently, however, I’ve been spending a lot more time on Linux as it’s been very helpful for my work: I put all kinds of test servers and tools there that I use during my java/.NET work. Because of this, I’ve been looking to improve my list of useful Linux applications, and I’ve found a few that I’ve liked well enough.

I’m listing them here so that I don’t forget about them later on :-):

GNU Screen

Screen is a “terminal multiplexor” that allows you to have multiple windows inside an ANSI terminal or do split-screens and run separate programs on it.

I knew about it but always dismissed as I wasn’t exactly sure what benefit it would provide for my work. I run X most of the time and like gnome-terminal or konsole well enough and for many tasks, the tab support in those is more than enough.

After seeing a bunch of screen-related links in reddit, however, I decided there might be something there worth looking into after all, and decided to give it a try. I’m just starting to get the hang of it and it’s certainly providing a bit useful, though is it an absolute must to run all the time? Not sure yet. I did found a very colorful tutorial on screen :-).

screen

There are a couple of features I’ve been finding interesting:

  • Detach/Reattach and session sharing: Allows you to detach from a session and reattach later (even connecting through SSH), or have multiple connections (even from different users) attached to the same screen sessions and viewing the same content/applications. Being able to just reattach to a session gone when I accidentally closed the gnome-terminal window also seems pretty useful!
  • Vertical Splits: Modern Screen version can split your terminal window into two vertical panes, which is incredibly useful on large screens and something you can’t replicate using the regular tabbed interfaces in other terminals (you could use multiple terminal windows, but it’s a lot uglier to configure).

Vifm

Vifm is a console-based, but graphical file manager that uses Vi-like key bindings. The first time I ran it I immediately knew how to use it; it’s very intuitive for Vim users.

vifmI mostly stay in the shell on Linux for file operations, particularly since most graphical file managers on Linux suck imho, but for some tasks, they are easier to use. Now I have one I won’t mind using so much, which is pretty nice.

Before anyone else brings it up, yes, I’m aware of Midnight Commander. First time I ran into it I thought it would be awesome to get that good old’ Norton Commander feeling (I loved NC!). But when I ran it, I realized I had completely forgotten how to NC was used and found Midnight Commander totally unintuitive. Go figure.

Irssi

Irssi is another terminal-based IRC client (though there’s a GUI based one too, I believe). I hadn’t used IRC in ages, and only started hanging in it occasionally around last year or so. Despite the horrible documentation, I found Irssi to be fairly intuitive and was able to connect to a server and join a room without even going though the docs, so it’s not so bad.

Some people love to script and do all kinds of things with their IRC client, but, to be honest, I don’t care about it at all. All I want is a simple client where I could write text and get it sent and read the text others write comfortably, and Irssi seems to do the trick for now.

Only thing that I found tricky at first until I finally found a page commenting how it’s done was switching between windows and scrolling. So that I don’t forget about it

  • Switch to Window #: Alt + #
  • Scroll through Windows: Ctrl + P / Ctrl + N
  • Scroll window text up/down : Alt + P / Alt + N

Before Irssi, I used to use X-chat and Besirc on Windows, but found myself constantly annoyed with both of them, but can’t pinpoint exactly why.

So far, I only lurk around #powershell on irc.freenode.org a bit, but if anyone has any other suggestions for other groups that might suit my interests, please do let me know!

Using NetBeans

A few months ago I mentioned I had started using NetBeans 6.1 for my java development instead of Eclipse, which was my tool of choice until then. The reason for the change at that time was that I needed to look into some JBI stuff and using NetBeans + Glassfish was a natural choice for that.

By the time I was through with that project and had started to work on other things I realized that I had grown rather used to NetBeans and actually preferred it over Eclipse! Both are excellent developer environments, but I actually like the NetBeans UI a little better and sort of found it nicer to use overall.

A couple of weeks ago I finally moved to NetBeans 6.5, the latest release, and it’s been working pretty nicely for me. A few things have contributed to this:

A decent color scheme

The default NetBeans configuration isn’t very usable for someone that prefers dark color schemes on their editor like me; and doesn’t include many default choices.

However, I found the additional Ruby themes plugin, which includes the Aloha and Dark Ruby Pastels schemes. I disliked the second (too similar to Vibrant Ink, which I hate), but Aloha was fairly nice.

With a bit of customization of Aloha to tweak it a bit for Java code I ended up with something looking like this:

netbeans

You can install the extra ruby themes from the Tools –> Plugins menu option.

jVi

It’s not secret I’m a Vim user, and I rather like the way Vim handles editing text and the its modal behavior. In Visual Studio, I use the most excellent Viemu plugin, which is the only plugin I consider a must-have these days. I’ve written about Viemu before.

NetBeans, however, didn’t have anything like this, or so I thought, until I ran into jVi on sourceforge. What a fantastic tool! It’s actually a subset of Vim embedded into NetBeans and supporting the modal editor behavior and all the common key bindings and operations.

I downloaded the alpha for NetBeans 6.5 and it’s been extremely stable for me and working great! I love it!

jvi

One downside with jVi I ran into is that it’s configured through it’s own set of settings, some of which overlap (and take precedence) over the NetBeans one, like the tab behavior. But once you’ve configured it to your liking it works very nicely, though occasionally conflicts a little bit with NetBeans functionality (not often, though).

What I don’t like about NetBeans

NetBeans does have a couple of issues that nag me. One is that memory usage is still fairly high (usually around 200MB), however, the slowdowns that I used to see after extended use with 6.1 don’t seem to happen with 6.5 anymore, but I’m not sure if that’s because of changes in NetBeans itself or because I finally bit the bullet and installed JDK 1.6 to run it.

The second issue is startup time: It still takes forever to load from scratch. Fortunately, performance once it finishes loading is quite acceptable and the editor is usually very responsive, so it hasn’t been that much of an issue. It helps a lot that I usually just leave it open and save the state of my VMs instead.

I still turn to Vim whenever I need to do some quick editing and don’t need the full IDE, but then again, I also do that for C# code. That said, VS 2008 usually loads a lot faster than VS2005 for me, so this is less of an issue there (it helps that I don’t install as much crap as I did for 2005).