Console2 and x64

I recently mentioned that version 2.00b140 of the great Console utility had come out. Eddie Velasquez commented on that post that, unfortunately, Console didn't work correctly on Windows x64.

I had the opportunity to verify this during the last week. Console2 runs, but all text drawn on the console is invisible, which makes it a bit hard to read :-). I looked around the web, and although a few people mentioned seeing the same problem, no one offered any solutions. So I decided to take a look.

I installed Visual C++ 2008 on my x64 machine, downloaded the Console2 source code package and went hunting around for all dependencies needed to actually build it, which took a couple of hours. Off the top of my head, you need:

• The boost libraries
• The Windows Template Library (WTL)
• The FreeImage libraries
• A bunch of isolated WTL extensions found all over the place that I don't even remember.

After a bit of tweaking the project files, I was able to build an x86 version of the code and started looking around the code to see where the problem might be. I checked a few things, like:

• Was it possible that it was interacting incorrectly with the underlying system console and not picking up the text colors? Nope. Debugger showed me it was indeed picking up the right color for each character drawn.
• Was it possible that it was mapping colors incorrectly? nope, not that either.
• Was it possible it was drawing the text in the wrong locations? Didn't seem to be that either, as you could even copy paste the text from Console correctly, and the caret seemed to be at the right location either.
• Could the off-screen surfaces be somehow invalid? Nope, a few calls to GetDeviceCaps() showed it had all the correct number of planes and bits per pixel that you'd expect.

After a few hours of tinkering, I was able to narrow it down to one issue: the use of the AlphaBlend() function when blitting between the off-screen and on-screen surfaces. In fact, I'm fairly confident that the issue Console2 is running into is the same one described on this newsgroup post. Unfortunately, that didn't seem to point to a good workaround for the issue.

Now, my GDI programming isn't very fresh on my mind, and frankly, I didn't want to sit around for a long time trying to get this fixed. However, I was able to make a quick change to the code that would make it work. The changes are setting up the BlendFunction for AlphaBlend slightly different.

The change I did was around line 1979, changing the setup before AlphaBlend like this:

Now, to be honest, this is just a hack, but it does the trick for now, using ClearType for font rendering and even alpha transparency on the entire window. I was not able (at least not in my quick look at the code) to fix the code path that gets executed when you're using an image as the console background, since that actually does require full alpha-blending to render the right output.

So it's not a great fix, but it means I can use Console2 on my x64 box now, which is good enough for me at this point. YMMV.