MSMQ and External Certificates

I’ve spent some time lately playing around with MSMQ, Authentication and the use of External Certificates. This is an interesting scenario, but one that I found to be documented in relatively unclear terms, with somewhat conflicting information all over the place. Plus, the whole process isn’t very transparent at first.

Normally, when you’re using MSMQ Authentication, you’ll have MSMQ configured with Active Directory Integration. When this setup, MSMQ will normally create an internal, self-signed certificate for each user/machine pair and register it in Active Directory. The sender would then request that the message be sent authenticated, which will cause the local MSMQ runtime to sign the message body + a set of properties using this certificate, and then include the public key of the certificate and the signature (encrypted with the certificate’s private key) alongside the message.

The MSMQ service where the authenticated queue is defined would then verify the signature using the certificate public key to ensure it wasn’t tampered with. If that check passed, it could then lookup a matching certificate in AD, allowing it to identity the user sending the message for authentication (there are a few more checks in there, but this is the basic process).

This basic setup runs fairly well and is easy to test. Normally all you have to do here is:

• Mark the target queue with the Authenticated property:
  
• Have the sender make sure to request authentication. For System.Messaging, this would be the MessageQueue.Authenticate property, while using the C/C++ API it would mean setting the PROPID_M_AUTH_LEVEL property to an appropriate value (normally MQMSG_AUTH_LEVEL_ALWAYS).

External Certificates

So what if you didn’t want to use the MSMQ internal certificate, for some reason? Well, MSMQ also allows you to use your own certificates, which could be, for example, generated using your Certificate Server CA installation trusted by your domain, or some other well known Certification Authority. To do this you need to:

1. Register your certificate + private key on the sender machine for the user that is going to be sending the messages. The docs all talk about registering the certificate on the “Microsoft Internet Explorer Certificate Store” which confused me at first, but turns out it just meant that it would normally be the CurrentUser\MY store.
2. Register the certificate (public key only) on the corresponding AD object so that the receiving MSMQ server can find it. You can do this using the MSMQ admin console from the MSMQ server properties window, or using the MQRegisterCertificate() API.

I decided to try this using a simple self-signed certificate generated using MakeCert (a tool I hate having to use because I can never remember the right arguments to use!). I created my certificate into CurrentUser\MY, then registed the public part of the cert using with the MSMQ management console.

The next part was changing my sender application code to use the external certificate, which is done by setting MessageQueue.SenderCertificate (or the PROPID_M_SENDER_CERT property). It wasn’t immediately obvious at first what format the certificate was to be provided, but turns out it’s just the DER-encoded representation of the X.509 certificate. In .NET, this means you can just use X509Certificate.GetRawCertData()/X509Certificate2.RawData or X509Certificate.Export() with X509ContentType.Cert value, which is equivalent. Here’s the basic code:

X509Certificate cert = ...;

Message msg = new Message();

msg.Body = data;

msg.UseDeadLetterQueue = true;

msg.UseAuthentication = true;

msg.SenderCertificate = cert.GetRawCertData();

queue.Send(msg, MessageQueueTransactionType.Single);

I ran this on my Windows 7 machine and all I got was this error:

System.Messaging.MessageQueueException (0x80004005): Cryptographic function has failed.

   at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)

   at System.Messaging.MessageQueue.Send(Object obj, MessageQueueTransactionType transactionType)

   at MsmqTest1.Program.Main(String[] args) in C:\tomasr\tests\MSMQ\MsmqTest1\Program.cs:line 73

What follows is a nice chase down the rabbit hole in order to figure it out.

The Problem

After bashing my head against this error for a few hours, I have to admit I was stumped and had no clue what was going on. My first impression was that I had been specifying the certificate incorrectly (false) or that I was deploying my certificate to the wrong stores. That wasn't it, either. Then I thought perhaps there was something in my self-signed certificate that was incorrect, so I started comparing its properties with those of the origina MSMQ internal certificate used in the first test. As far as I could see, the only meaningful difference was that the MSMQ one had 2048-bit keys while mine had 1024-bit ones, but that was hardly relevant at this point.

After some spelunking and lots of searching I ran into the MQTrace script [1]. With that in hand, I enabled MSMQ tracing and notice a 0x80090008 error code, which means NTE_BAD_ALGID ("Invalid algorithm specified"). So obviously somehow MSMQ was using a "wrong" algorithm, but which one, and why?

In the middle of all this I decided to try something: I switched my sender application to delivering the message over HTTP instead of the native MSMQ TCP-based protocol; all it required was changing the Format Name used to reference the queue. This failed with the same error, but the MSMQ log gave me another bit of information: The NTE_BAD_ALGID error was being returned by a CryptCreateHash() function call!

Armed with this dangerous knowledge, I whipped out trusty WinDBG, set up a breakpoint in CryptCreateHash() and ran into this:

003ee174 50b6453f 005e14d0 0000800e 00000000 CRYPTSP!CryptCreateHash

0x800e is CALG_SHA512. So MSMQ was trying to use the SHA-512 algorithm for the hash; good information! Logically, the next thing I tried was to force MSMQ to use the more common SHA-1 algorithm instead by setting the appropriate property in the message (PROPID_M_HASH_ALG):

msg.HashAlgorithm = HashAlgorithm.Sha;

And it worked! Well, for HTTP anyhow, as it broke again as soon as I switched the sender app back to the native transport. It turns out that the PROPID_M_HASH_ALG property is ignored when using the native transport.

Changes in MSMQ 4.0 and 5.0

Around this time I ran into a couple of interesting documents:

• What's new in Message Queuing 5.0
• MSMQ 5.0 - Changes introduced with Windows 7 and Windows Server 2008 R2

The key part that came up from both of these is that in MSMQ 4.0 and 5.0 there were security changes around the Hash algorithm used for signing messages. In MSMQ 4.0, support for using SHA-2 was added and MAC/MD2/MD4/MD5 were disabled by default, while keeping SHA-1 as the default one. In MSMQ 5.0, however, SHA-1 was disabled by default again and SHA-2 (specifically SHA-512) was made the default [2]. So this explains readily why my test case was using SHA-512 as the hashing algorithm.

At this point I went back to using internal MSMQ certificates and checked what Hash algorithm was being used in that case and, sure enough, it was using SHA-512 as well. Obviously then something in the certificate I was providing was triggering the problem, but what?

And then it hit me that the problem had nothing to do with the certificate itself, but with the private/public key pair associated with the certificate. When you create/request a certificate, one aspect is what Cryptographic (CAPI) Provider to use to generate (and store it?) and I had been using the default, which is the "Microsoft Strong Cryptographic Provider"; according to this page it apparently isn't quite strong enough to support SHA-2 [3].

So I created a new certificate, this time explicitly using a cryptographic provider that supports SHA-2, by providing the following arguments to my MakeCert.exe call

-sp "Microsoft Enhanced RSA and AES Cryptographic Provider" -sy 12

And sure enough, sending authenticated messages over the native protocol succeeded now. Sending over HTTP also worked with the updated certificate without having to set the HashAlgorithm property.

Exporting and Using Certificates

Something to watch out for that might not be entirely obvious: If you export a certificate + private key from a system store into a PFX (PKCS#12) file, your original choice of cryptographic service provider is stored alongside the key in the file:

Once you import the PFX file into a second machine, it will use the same crypto provider as the original one. This might be important if you're generating certificates in one machine for using in another. The lesson so far seems to be:

When using external certificates for use with MSMQ 5.0, ensure you choose a cryptographic provider that supports SHA-2 when generating the public/private key pair of the certificate, like the "Microsoft Enhanced RSA and AES Cryptographic Provider".

At this point I'm not entirely sure how much control you have over this part if you're generating the certificates in, say, Unix machines where this concept doesn't apply (or how exactly it might work with external certification authorities).

External Certificates in Workgroup Mode

Another aspect I was interested in was the use of External Certificates when MSMQ is in Workgroup mode. In this mode, authenticated queues aren't all that useful (at least as far as I understand things right now), because without AD the receiving MSMQ server has no way to match the certificate used to sign the message with a Windows identity to use for the access check on the queue. In this scenario messages appear to be rejected when they reach the queue with a "The signature is invalid" error.

However, if the queue does not have the Authenticated option checked, then messages signed with external certificates will reach the queue successfully. The receiving application can then check that the message was indeed signed because the Message.DigitalSignature (PROPID_M_SIGNATURE)property will contain the hash encrypted with the certificate private key as expected. The application could then simply retrieve the public key and look it up in whatever application-specific store it had to check it against known certificates.

My understanding here is that despite that it cannot lookup the certificate in DS, the MSMQ receiving server will still verify that the signature is indeed valid according to the certificate attached to the message. That's only half the work so that's why the application should then verify that the certificate is known and trusted.

[1] John Breakwell's MSMQ blog on MSDN is a godsend when troubleshooting and understanding MSMQ. Glad he continued posting about this stuff on his alternate blog after he left MS.

[2] The System.Messaging stuff does not provide a way to specify SHA-2 algorithms in the message properties in .NET 4.0. No idea if this will be improved in the future.

[3] I always thought the CAPI providers seemed to be named by someone strongly bent on confusing his/her enemies...