OK, time to post some technical content again :-)
If you've used BizTalk server 2004/2006, you've probably seen the configuration dialog where you enter the adapter-specific settings on a send port or receive location. Have you noticed how some adapters simply have a property grid with values, while others have a full-blown set of custom property pages with custom dialogs? Ever wondered why?
The standard property-grid based settings dialog is provided by the BizTalk Adapter Framework by default, all you need to do have your adapter's MgmtClass provide the proper XML Schema (XSD) that defines the adapter options.
In a lot of ways this is very good, as it makes adding design-time configuration support for your adapters very easy, and, while it might not be the best looking UI ever, it is fairly usable and fairly standard, so it works fairly well.
Even more, the adapter framework provides you some useful ways to extend this model, with, for example, custom value editors for specific properties. There's good documentation on this on the BizTalk Documentation on creating adapters, which can be found here.
Other adapters, however, have a completely different configuration UI. A good example of this is the standard FILE adapter, which has several different options and two property pages. This is a lot more usable, but it does mean a lot more work on the part of the developer (after all, you get the regular UI almost for free!).
Some people have asked me recently how you could do this for your own adapters, and I really didn't have a good answer for that (that is, I had no clue!). If you look at the BizTalk documentation you'll notice that there's virtually no discussion of this topic whatsoever. I also realized that the custom UI dialogs are used almost exclusively by unmanaged adapters: FILE, SMTP, Receive Side HTTP, and so on.
Unmanaged adapter development is something supported in BizTalk right from 2004, but I can't say I know anyone doing it except Microsoft itself. It's also noteworthy that no where on the documentation is the topic of unmanaged adapter development explicitly covered, and certainly the design-time configuration aspect isn't even mentioned. So I was under the impression that both topics were related somehow.
Managed Adapters can have custom configuration UIs
As it turns out, however, it is possible for a regular, managed adapter to have it's own custom adapter configuration design-time experience in BizTalk 2006 R2, though it isn't very obvious how to do it.
I first realized this when I noticed that all the WCF adapters in BizTalk Server 2006 R2 had custom dialogs . Looking a bit deeper I realized that the HTTP send-side adapter also had one, but the HTTP adapter is a rare beast in the sense that the receive-side adapter is unmanaged (an ISAPI extension, actually), while the send-side adapter is managed (built on top of System.Net.HttpWebRequest). Because of this, the management infrastructure for both receive and send side adapters is unmanaged, so it doesn't really count.
Sidebar: I haven't seen a way in BizTalk 2006/4 to support this yet. In particular, the interface used in R2 to support this is not even available in previous versions, and as far as I can see, unmanaged adapters implement this in a completely different way, using IPropertyPage and IPropertyPageSite, I think.
So, how does the WCF adapter do it? I haven't traced yet all the details necessary, but looking a bit with reflector and poking in the registry revealed some interesting details. Here are some things that might get you started if you decide to try this for yourself:
MgmtClass: Apparently you still need a MgmtClass for your adapter that implements IAdapterConfig (and likely IAdapterConfigValidation as well). However, your implementation of IAdapterConfig.GetConfigSchema() can simply return null, as it isn't apparently relevant.
Custom UI: You implement your custom UI by implementing a class that provides BizTalk a way to load/save your adapter settings as well as provide it with the information to create your custom dialog:
- It needs to implement the Microsoft.BizTalk.Admin.IPropertyPageFrame interface . This interface has a single ShowPropertyFrame() method which gets handed the HWND of the window you should use as your custom dialog parent (so that you can show it in modal form).
- It needs to implement IPersistPropertyBag. Apparently, however, you need to implement it in two variations: Microsoft.BizTalk.Admin.IPersistPropertyBag as well as Microsoft.BizTalk.ExplorerOM.IPersistPropertyBag. This is probably done to support both the management console as well as the older BizTalk Explorer inside Visual Studio. This is used to both load/save the adapter configuration as well as some other stuff I'm not quite sure about yet.
- It needs to be visible to COM and registered in the registry with a proper GUID you can reference.
Registry Settings: For your custom UI class to be used by BizTalk, you need to add a few new values in your adapter's Registry key:
- ReceiveLocation_PageProv: This will be the GUID of the class implementing IPropertyPageFrame for the receive-side of your adapter.
- TransmitLocation_PageProv: This will be the GUID of the class implementing IPropertyPageFrame for the send-side of your adapter.
- InboundProtocol_PageProv: This will be the GUID of the class implementing IPropertyPageFrame for the receive handler configuration.
- OutboundProtocol_PageProv:This will be the GUID of the class implementing IPropertyPageFrame for the send handler configuration.
I'm in the process of trying this now and I'll let you how it works.
 Actually, the WCF adapter creates it's own model internally to handle the custom configuration UI on top of the original extensibility model in the adapter framework, which is then leveraged by all the WCF built-in adapters and custom WCF adapters.
 This one is defined in the copy of Microsoft.BizTalk.Admin.dll that's in the GAC, which, oddly enough, is different from the Microsoft.BizTalk.Admin.Dll that can be found in the BizTalk installation folder.