Ayende has been doing some interesting stuff recently with threading and using the .NET Thread Pool which is based on the I/O Completion Ports (IOCP) model in the Win32 API and the NT kernel. IOCPs are a very efficient way of handling threads and thread pools and doing async I/O, so they are definitely a good fit, and it's good that the .NET framework encapsulates some of the complexity of using them (though as they are exposed to .NET reduces a bit what you can do with them and how you do so).
In one of those posts, Ayende mentioned trying to emulate the parallel structures in Erlang using the ThreadPool class, and specifically taking advantage of ThreadPool.QueueUserWorkItem() and ThreadPool.RegisterWaitForSingleObject(), and he mentioned he was running into a "Not enough storage available to process this command" error as he increased the size of the dataset he was testing with.
On a comment on the post, I mentioned that perhaps the use of IOCPs here was causing the kernel to exhaust its Non-Paged Pool memory [1], though Ayende mentioned he wasn't convinced that ThreadPool.RegisterWaitForSingleObject() would do this.
I decided to give it a try, so I added a bit of simple code around Ayende's sample and ran it, and sure enough, usage of non-paged pool kernel memory starts going off like a rocket. This is easy to verify using the Task Manager performance tab (under Kernel Memory - Non paged) or using Performance Monitor and the Memory performance counters.
That said, I don't think ThreadPool.RegisterWaitForSingleObject() is the culprit. Rather, I think what causes heavy usage of the non-paged pool memory is ThreadPool.QueueUserWorkItem(). I had vague recollections of a discussion long ago in the MS newsgroups about threading, IOCPs and finding out that IOCPs used non-paged pool memory to keep track of pending request. After searching for a bit on google, I was able to find the original thread on microsoft.public.win32.programmer.kernel. It's a long, but interesting discussion (and do note this was 1999!)
[1] The non-paged pool is a region of physical memory allocated by the kernel that cannot be swapped out to disk. Because of this, it is a very precious and limited resource, and abusing it can impact performance in a big way. There;s some discussion as to how the different memory pools are allocated by the kernel here.
