Back to the Past

Every so often I see people asking in the newsgroups how to solve certain challenges they encounter while working on their BizTalk applications. One common question revolves around being able to go "back to the past" when an error happens during processing of a message.

This isn't a bad question at all, and usually revolves around how to simulate the behavior of atomic transactions in an environment where transactions can be a lot more complex and not always as natural.

The question usually goes like this: "I'm receiving a message in BizTalk, which is triggering an orchestration instance. The orchestration does this and that, and if it any of those things fail, I want to put the message back where I got it from".

This-That-There

The question might seem simple, but it's not always necessarily so. In fact, sometimes you have to stop a moment and ask yourself whether this really makes sense. There are several aspects you need to consider:

  1. Handling the case where "this" causes an error is probably not a big deal. Handling the case where "this" succeeded but "that" failed, however, might not be that simple. Not all actions your orchestration might do can be undone.
  2. Most of the time you'll find that both actions can't be done as a single unit in a single atomic transaction. Fortunately, BizTalk provides very good support for long-running transactions and compensation which can help quite a bit.

    Unfortunately, long-running transactions and compensation models are often misunderstood (cue in the inevitable "How long does a transaction have to last to be a long-running transaction?" jokes/questions).

    Here are a few articles that do a great job of describing the BizTalk Transaction features and how to use them effectively:
  3. The sentence "put the message back where I got it from" can be either a very good thing, or a very problematic thing. It basically relates to leaving stuff as you found it; in particular, leaving the message back into its origin (thus relating to the transactional concept of "nothing happened here, move along")so that you can try processing it again later on and hopefully it will succeed at that time.

The problem with number 3 is that it (a) isn't always possible, and (b) it isn't always a good idea.

It might not be possible to put the message back where you got it if someone was pushing the message to you instead of you pulling it from somewhere. If you had a SOAP/HTTP WebService exposed that received a message from someone else, then you probably can't put the message back where you got it from!

On the other hand, this is a very common model for queued messaging systems: If you run into an error processing the message, you put it back into the queue and try again later. And this works great many times and can simplify error handling a great deal.

The point where this becomes a problem is when you rely on this as your only error handling mechanism. If you blindly send the message back to its origin to retry processing for any and all errors and a message comes in that always fails, you've got yourself a poison message!

toxic I've already talked about Poison Messages in the past, so I won't comment much more on them. But there are other things you can keep in mind to improve the "back to the past" error handling technique, particularly if you don't care about message processing:

  1. If you can identify and classify the source/cause of the errors, you can make your orchestration smarter about how to handle them. For example:
    • Can you distinguish transient error conditions? For example, a timeout connecting to the database might be a temporary condition because of a network fluke or a server being restarted. Sometimes retrying the operation after a short while is enough to deal with this situation effectively.
    • Can you distinguish errors that might require manual intervention to fix? Example: Validating an operation fails because some configuration data is missing. This is a case where you want to be proactive and raise an appropriate alert so that someone can get in there and fix the issue. Extra points if you can tell apart conditions that require intervention from a business users and those that require it from a systems administrator.

      Notice, however, that in this case putting the message back at the start right after creating the notification is not the right thing to do. People don't react that fast. You need to set the message aside until such time as the corrective measure has been taken and it is safe to try processing it again.
  2. Can you control when the retry might happen? Can you throttle it if necessary? If the answer is no, then you might want to be very careful about using this technique. You could easily increase the system load substantially if lots of messages fail in a short time and you try reprocessing them in a tight loop.
  3. Be mindful of adapters that provide no ordering semantics. For example, if your original location used the FILE adapter and you put the message back in the original folder, it will likely get picked up very soon again for processing; which can quickly get you back to step 2.

    At least with an adapter like MSMQ you can push the message to the end of the queue, which might buy you some time.
  4. Even if you take 1, 2 and 3 into account, you still need to provide a way to deal with poison messages. Keep in mind that what started as a transient error condition can suddenly escalate to a full-blown problem you can't do nothing about, like when that temporary network fluke turns into a days-long outage after some idiot digging a whole outside snaps your network fiber cable in two.

    In fact, sometimes you might need to go so far as to completely shut down processing. Sometimes being able to detect that some things that should be working keep failing after an extended period of time and alerting about it can help get things sorted out before they spiral out of control.

These are just some ideas that might help make your system more reliable and more manageable. Some of them do cost money; that is, you have to invest time and development/testing efforts in getting them done, and that's where you're going to have to evaluate what makes sense and what not.

Technorati tags: , ,

The other side of the bridge

06May07-110308

Some readers of this blog might have noticed that my posting frequency has been somewhat reduced during the past few months. Some of this is the result of working on several different projects, some of which isn't stuff I can talk about much.

Besides my usual .NET/BizTalk work, I've been increasingly spending time working on some Java stuff as well. I would say that I probably spend my time 50/50 between both technologies nowadays. There are good and bad things about this, but that's a topic for a different post.

One of the interesting things I've worked on lately on the Java side has been  implementing custom Binding Components based on the Java Business Integration (JBI) specification. JBI isn't a new technology; it's been around for a while, but until now I had not had the opportunity of working with it.

Mostly, I've been working with Sun's OpenESB, which is actually not that bad, though I've also used JBoss/ServiceMix a bit.

It took me a little while to fully grok the core of the JBI model. There are many similarities in the concepts it uses to other messaging technologies, but the terminology used can be somewhat different, which can be confusing sometimes. In case you're not familiar with JBI, it defines two different kinds of components developers might wish to implement:

  • Service Engines, which are just components that do some processing and send and/or receive messages.
  • Binding Components, which are mostly transport-level adapters. These are the equivalent components in JBI to BizTalk adapters and WCF's transport channels.

I'm not going to do a comprehensive comparison of JBI to BizTalk or WCF, but did want to make a few observations about some aspects that I found interesting.

Abstraction

The JBI spec is a fairly low-level specification. Much of the spec is really concerned with how components interact with the JBI server and how class loading and deployment work, as opposed to defining a more higher-level messaging model.

Actually, maybe I should say that the JBI spec doesn't really define abstraction layers. It's all a single layer that addresses a number of different concerns.

Most of the JBI messaging model is really defined to match the WSDL model, and there are advantages and disadvantages to this. On the plus side, the WSDL model is generic enough to be useful in a broad set of scenarios, and it is fairly well known. It also makes a lot of sense considering the terminology used throughout the spec (once you get familiar with it).

On the bad side, it means that components also have to get intimate knowledge of the WSDL contract model and this forces some compromises I personally don't quite agree with.

For example, in WCF and BizTalk, if I'm writing a transport-level adapter/channel, I may need to be intimately aware of the contract definition being used by the client/server. But it's not really necessary most of the time, particularly when you're working with network-level transports. Mostly, you care about the Message Exchange Pattern (MEP) that the contract demands.

In JBI, however, you have to understand the entire contract. This might be fine for Service Engines, but it can be rather inconvenient for Binding Components, as now you have to figure out out how to map contract operations to your transport, and even how to marshal data back and forth based on the contract definition.

Configuration

This leads me to my second gripe about JBI: Configuration.

Endpoint configuration in JBI is done through a combination of two things:

  • The WSDL that defines the contract you'll want to receive or send messages to (extended with your binding component specific port/binding/operation configuration data)
  • The deployment descriptors which tell your component if you want to receive or send messages for the corresponding WSDL file.

wsdl

In reality, WSDL permeates the entire JBI development experience, including how messages are represented internally between the JBI runtime and binding components / service engines.

A huge downside of this, though is that the code needs to "understand" the contracts being used, so parsing the WSDL files and extracting your component's configuration data is your responsibility. Libraries like WSDL4J help, but it can still be a drag.

What does really irk me a bit about JBI though is not that you have to manually craft WSDL files (not that it's a lot of fun). It's that the JBI spec doesn't address the design time aspect of JBI components at all besides, you guessed it, manually crafting WSDL files and deployment descriptors.

So if you're creating JBI components and want a decent design time experience, then you need to target each different server/tooling specifically. For example, for OpenESB you'd craft a custom NetBeans module, while for another tool you'd do something different. It's just not nice.

What's worse is that some servers decided this model wasn't good enough either and added their own configuration models for components and contracts. For example, while ServiceMix supports the stuff mandated by the JBI spec, it also supports an alternative XML-based configuration syntax. While it's more compact, it's hard to argue it really is a significant improvement.

Encoding

This is one aspect where I thought the JBI spec really blew it. As I said above, the JBI spec uses an XML representation for messages internally, that matches the WSDL message definition (parts and all).

There's no layering here that allows binding components to worry about sending and receiving messages without really caring about message contents. They have to care about the message contents and how to map that to however that message was defined in the contract WDSL definition. This might mean looking at which parts are defined and how to break the message down to them, or how to decode complex XSD types defined in the contract.

Instead of being delegated explicitly in the spec to separate components, it's all the responsibility of the binding components themselves. The obvious benefit of having this layered would've been more reusability of said encoding/decoding components, particularly if a minimum set of encoders were mandated by the spec.

Just to take one example: basic XML encoding/decoding is such a basic part of the system, that it's just really odd each component writer gets to redo it from scratch."

I think this is one aspect that BizTalk in particular gets very right. WCF does keep encoding/decoding an explicitly separate task, even if transport channels are responsible for using the encoders directly, but it is still a much better solution than what JBI came up with.

Technorati tags: ,

What Irks Me About Visual DSLs

There's a lot of talk about Domain Specific Languages lately. The exact definition of what a DSL is, however, might change depending on who you ask. Microsoft itself tends to favor significantly Visual DSLs, that is, domain specific languages that are made of visual components (as opposed to Text-based DSLs that are made of some kind of text driven representation).

Frankly, I don't expect MS to change their direction, nor am I sure it would be the wisest decision given their target audience, but I do tend to favor text-based DSLs myself, for several reasons:

  1. Text-based DSLs work best during development. We have a significant amount of experience and a rich set of tools available to deal with text in an effective fashion: Source control and comparison tools, good editors, diff'ing and merging, and so on.
  2. A text-based DSL is illustrative in and of itself. Anyone with a text editor can look at it, so it only requires special tooling during execution, unlike their visual counterparts.
  3. If you're spending significant time using a DSL to create new things (versus, say, simply visualizing existing stuff), then a textual DSL is usually more effective.

I should say at this point that XML-based languages don't necessarily fit this descriptions. XML can be clunky at times, and a lot of people hate having to manually crank XML to do something. For example, many people dislike manually editing NAnt or MSBuild build files.

What's not to like about Visual DSLs

Many Visual DSLs are very appealing at first to create new things when you're unfamiliar with the language, as they can be very didactic. But once you're familiar with the language, Visual DSLs, as implemented by most tooling out there, will usually get in the way instead of boosting your productivity.

Don't get me wrong; there are a lot of things to like about Visual languages. In particular, they can be great tools for visualizing things. In some cases, they are great tools to editing existing things and occasionally, even creating new things.

The last one, however, is pretty rare. I've been thinking a lot about this, and I've started to think that one of the reasons this is so is that there's a fundamental disconnect in how we usually think about Visual Languages and Tools.

The disconnect is that we tend to assume that the visual representation of the underlying domain that is best for visualizing and describing the language is actually an acceptable choice for "writing" in that language.

For example, let's consider Windows Workflow Foundation workflows or BizTalk Orchestrations. Both could be consider DSLs for building processes and workflows, and they are actually pretty effective at that. Both use a visual representation that feels more or less natural to people used to working with processes (or state machines, in the case of WF). Both of those representations are great for working with existing processes, as they allow the reader to quickly grasp the flow of it, and it even works very well when debugging a running process.

But, to be honest, both leave a lot to be desired when you're actually sitting down to create and define a new process, and both tend to get a lot in the way. I personally feel that WF is a lot worse in this respect.

XAML

I should mention that I do not consider XAML a text-based DSL (even if it is "just text"). Fundamentally, XAML is just a serialization format, and that shows in a number of places. It is build to be created and consumed by software tools, not the human developer (though it is possible to do so, as many people found out with WPF in the early days).

More importantly, these kinds of XML/XAML languages that are aimed at tools don't necessarily work great with the tooling we have for dealing with text (see the all-important point 1 above). For example, a lot of people have found the hard way that trying to do a diff or trying to merge two copies of a tool-generated XML/XAML file can be nearly impossible at times.

It's pretty evident that Microsoft is working on a lot more tools based on XAML, so that's here to stay, but it remains to be seen yet how that is going to work out. I'm sure there's going to be good Visual tooling around it, but, as usual, the problem is that it just isn't enough.

What about Oslo?

A lot of my fellow MVPs and a bunch of people that attended the recent SOA and Business Process Conference have mentioned Microsoft's Oslo initiative that was announced at the conference.

From what little I know of it, it is a far reaching initiative, touching multiple key products in the Microsoft development platform. A significant component of this effort is investment in models and, you guessed it, modeling tools around them. I think it's obvious to everyone by now that a substantial set of those tools will be built around visual DSLs and visual designer tools (that XAML's in there somewhere is probably also a safe bet). Some people will think this is a key advantage, others will probably hate it.

The one conclusion I've reached so far regarding Oslo is that will likely mean a significant shift in how we do development on the MS platform (at least for those of us involved in connected systems). However, I'm holding my thoughts on what will be good or bad about those changes until we know more precisely what the Oslo technologies will be delivering and we have a clearer picture of how we will interact with them. Also, it's clear that this is an initiative that will be gradually rolled out, so it will probably be a long transition period around it (which is both good and bad in itself).

As customers, and users of those technologies, however, we have a big opportunity, and a big responsibility in letting Microsoft know what kind of tooling we want/need to have around the modeling tools and other technologies. Like I told someone at MS recently: "I don't expect MS to shift its position on visual tooling and Visual DSLs, but I do wish the hooks and infrastructure was there for us in the community, creating our own, non-visual DSL tools around it that allow us to work more effectively with the technology". Hopefully, that little thought will not be forgotten.

Technorati tags: , ,

System.Transactions and Workflows

Scott Bellware discusses some interesting things here regarding transactions and extensibility hooks and even considering Ruby on Rails as a web front end. I'll ignore the Ruby thing, but I do want to talk about his comments on transactions and workflows.

If I understand right, Scott is basically providing a way to extend his application through user-defined workflows that are executed at specific and controlled extensibility points. This is a very cool scenario for WF that enables very interesting possibilities.

His concern, however, seems to center around controlling what happens when one of those user-defined workflows fails, to ensure the entire system is not left in an inconsistent state. This is certainly a valid concern, but it is one that I feel isn't quite as simple to solve as simply "use transactions".

Let's take WF out of the picture for a moment and assume we were using a good old code-based extensibility model (or even a script-based one). Even with the help of System.Transactions and distributed transactions, there's really no way to guarantee that whatever code users of the system put into the extensibility hooks would still work correctly in the event of failure. This revolves around the fundamental fact that there's no guarantee that whatever is put in there is even transaction-aware. Granted, there's a big chance that code put in there is indeed transaction aware, but that's only because a lot of what comes directly into the core .NET framework is tightly integrated in itself. Indeed, for all you know, the user might have even explicitly bring out his operations out of the transactional context (for example using "Enlist=false" in his connection strings).

So even if you wrapped your extensibility hooks with a transaction scope built with System.Transactions, there's no guarantee that a transaction rollback will really bring the system to a consistent state. That's a constant danger with extensible systems, and one that it is not easily addressed. If you've noticed, most framework facilities focused on extensibility only concern themselves with the security problems, but not with extension behavior in the general sense.

This fundamental issue doesn't really get any better just because your extensions are now written as workflows. Indeed, it can become a bit more complicated. To me, using System.Transactions with WF Workflows used in this matter doesn't really make much sense, for several reasons:

  1. Yes, WF supports System.Transactions (indeed this is what a TransactionScope activity uses underneath), but the same danger exists of someone using an activity that's not transaction-aware.
  2. Wrapping the entire workflow in a System.Transactions transaction limits substantially what the user can do with the extension workflows. Are you willing to restrict your users to running only short-lived, simple workflows?

The second one is to me the most significant one, because putting that restriction in place limits substantially the power of WF for these scenarios, particularly when you allow workflow persistence and tracking as part of your application extensibility.

The one benefit that WF gives you over code-based extensibility for handling errors is indeed mentioned by Scott: Compensations. This is a very powerful mechanism, particularly for long running workflows, and can certainly be leveraged by extension workflows to put the system back into a consistent state.

In the end, just like with code-based extensibility, you still need to put guidelines in place to make sure users extending your system do so in such a way that they don't jeopardize the behavior and consistency of the entire system, but even with WF, it is still hard to put those guidelines in executable form.

In some cases, however, WF can indeed make it easier, if you're willing to live with some restrictions. For example, here are some ideas to think about to improve your chances:

  1. Restrict the vocabulary: WF can be extended through the use of custom Activities. Ideally, you will have devised a set of custom activities specific to your problem domain that the users can use to compose their extension workflows. If you're willing to restrict users a bit, and your activity set is complex enough, consider putting restrictions in place so that users can only compose extension workflows using a set of activities you've "approved" (those on your own set and some basic flow control ones, for example). This won't be a perfect model, but can improve the chances that users don't extend the system incorrectly, and can easily be done.
  2. Use validators: Take advantage of WF's ActivityValidators to enforce rules on your extension workflows. If you've also restricted the vocabulary, you can certainly take advantage of domain=specific knowledge to write the validation rules. For example, if you've got an UpdateXXX activity, you could validate that it always is used within a TransactionScope activity.
  3. Create templates your users can use to get started writing extension workflows. Ideally, make these into top-level workflow activities your users can derive from. So, for example, you could have users create a workflow derived from InsuranceExtensionWorkflowActivity instead of a simple SequenceWorkflowActivity, Your base workflow could also have custom validators enforcing, for example, that a global compensation flow is implemented by the users (if necessary).

None of these are foolproof, but they can improve the odds in your favor when used correctly.

Visual DSLs

Scott Bellware wrote yesterday on disambiguating fluent interfaces (aka APIs) from domain-specific languages. Rant aside it's an interesting read. Something, however, gave me a chuckle: Scott says:

"At some point in the near future, Microsoft will tell the .NET mainstream that DSL's are visual, that they are drag-and-drop, and that they require a new plug-in to Visual Studio in order to make them and use them."

Hummmm..... sorry to say this Scott, but they already did a couple of years ago :-). I think it started with the whole Software Factories [1] idea that Microsoft pushed some time ago.

I must say, though, that I don't hold any grudges to the idea of Visual DSLs. In fact, they can be very useful tools both to create and visualize software. The problem appears when a) they are not the most productive way of working either because of limitations in the visual language itself, or b) because of poor implementation. The Windows Workflow Foundation Designer strikes me as an example of a concept where the visual thing works very well, though it has an unfortunate implementation (it's slow as molasses, almost unbearable to use at times!).

For many of these tools, what I'd really like to have is the best of both worlds: Having a visual experience can be very useful, but having a productive, effective and efficient textual dsl is, to me, a core requirement. Despite what some people might think, writing text is still a far more efficient means of writing software if you have the right language at your disposal.

[1] I think the Software Factories idea has some good merits as a conceptual concept and vision, but it was unfortunately named (and I believe the term has made substantial damage to our software industry, by the way).

Technorati tags: ,

Web Services Interoperability

Udi Dahan posted a few days ago an entry (rant?) on Web Services as an interoperability option between .NET and Java and the potential pitfalls in that. Based on my personal experience, I can't help but agree with him that there's still a ways to go.

Now, don't get me wrong; using Web Services is still simpler in many cases than the other options available, but it's no silver bullet. More to the point, interoperability is not something you get for free; it's something you have to architect, design, code and test for.

Here are some (random) observations on this from my own personal experience:

Know who your clients are and what tools they use

Sometimes you will have to make trade offs in the design of your service contract based on who your service consumers will be and what tools they are using. In an ideal world this wouldn't be necessary, but alas, that's not where we live. For example, Java tools tend to favor certain specific patterns. Or maybe your consumer will be an integration tool or maybe even a legacy platform using one of the many, lesser known SOAP stacks available. Those usually have specific restrictions.

If you know your consumers will be using Java, try to find out what specific SOAP stack they are using (from the plethora available for Java) and explicitly test against it. It will be a worthwhile use of your time. If your consumers are LOB applications, modern third party packages with built-in WS support, or desktop applications (i.e. MS Office) then again special restrictions might apply, so try to find out before hand.

Understand SOAP Encoding and Messaging Patterns

A lot of people still don't know the difference between SOAP RPC/Encoded and Document/Literal. Microsoft took the initiative back in 2001/2002 when the .NET ASMX stack turned on Doc/Literal encoding by default (a wise move, imho), but Java toolkits took a long while to catch up.

Even today, a lot of Java toolkits will default to RPC/Encoded even if they support Doc/Literal and, unfortunately, a lot of Java developers don't know their kits well enough to turn on the right settings. To be fair, a lot of .NET developers don't know how to change the encoding settings either, but we forgive them because of the sensible defaults in the platform :-)

Know the basics

Study and learn how to apply the basics of the WS-I Basic Profile in your platform of choice. Take good care in ensuring you follow the basic guidelines for service/message contract design, and learn what options are available in your platform to control service and message generation.

Here are some good resources to keep in mind for .NET:

Beyond the Basics

If you thought interop with plain old SOAP (POS?) could be hard, then be double wary when you get into the WS-* specs, which are sure to complicate things further and bring a lot of extra interop challenges. WS-Security and its related specs, in particular, can give you some nasty headaches. Some thing to watch out for:

  • Again, check which toolkits and clients your consumers will be using. Make sure they support the exact protocol versions you expect to be using.
  • Make sure you've tested interoperability using the specific protocol features you expect to use. For example, if you're using WS-Security, make sure you can interop at a basic level with the message confidentiality, message authentication options and token kinds you plan to enable.
  • WS-Security interoperability problems, in particular, can be nasty to diagnose because most implementations throw extremely uninformative error messages that don't tell you much. Yes, even WCF falls into this trap.
  • Watch out for interrelated specs in use. For example, when using WSE to add advanced security to your services, it would by default add a bunch of WS-Addressing stuff that caused extra interoperability issues, because there were several [incompatible] WS-Addressing versions. Add to this that WS-Security was a lot farther down the standardization path than WS-Addressing.
  • Know your platform limitations: If you're using .NET 1.1, then you've got very limited options (WSE 2.0); If .NET 2.0/3.0 then at least you have a much more modern WSE 3.0 and WCF options at your disposal.

One hand giveth, the other taketh away

If you're using WCF, understand the choices it makes for you and how to work around them. For example, Data Contracts in WCF aim to improve interoperability by severely limiting the kind of XML structure you can use (elements only and so on). This, however, doesn't help you if you have to work with specific schemas designed by someone else (like standard industry schemas), so be ready to fall back to the old trusty XmlSerializer.

Another thing that might impact you is WCF's default behavior in generating split WSDL descriptions where the schemas are imported (and possibly other WSDL documents as well if your service elements have different namespaces); a lot of external tools choke on those service descriptions. Again, let me point to Christian Weyer's excellent article and code sample to ease the pain of this.

Technorati tags: ,

Bounded Mort

Nick Malik has posted another thoughtful piece on the whole Mort topic that's been going around these past few days. While I appreciate where Nick is coming from, I can't agree at all with his perception. Ayende also voiced his opinion on the topic here.

Trying to control and bound a developer and intentionally doing everything possible to keep him blissfully ignorant about the tools he's working on has not proven very successful so far, and frankly, I think it's not only a bad idea (as in, it doesn't work at all, abstractions are leaky and all that), but I also find the idea incredibly depressing and dreadful.

Here's just a few things that are incredibly wrong with this whole idea, imho:

  • It is a self-defeating attitude: It means you stop focusing on improving the situation and purposefully keeping it in a bad, unsustainable position.
  • It is really elitist, far more than any of the times the whole ALT.NET idea has been brought forward. It essentially means you just split the developer profession in two mutually exclusive, segregating communities (casts?): Those that know and do, and those who don't know and do.
  • There's no way to cross the bridge: Once you've started as one of these Morts guys, you stay there. The whole proposition is based on the idea that Mort has no need to learn anything new on the job (and possibly didn't even need to know much to start with). Indeed Mort should not try to learn anything or become smarter lest he tries to work around the walls of his prison (thus causing damage).
  • This, in effect means that once a Mort, always a Mort. Mort is not likely to look for a new work place, since he'd only be able to work as Mort anywhere else, anyway. No need to keep working hard trying to attract and keep those pesky developer types, whooohooo!.

Incidentally, this is why this whole idea is the wet dream of most project managers and IT shop owner's I've met: it promises cheap, untrained, replaceable labor, and low cost to ensure high personnel retention. Goodness gracious great balls of crap!

Now, is it good to try to make it easier for the developer to focus on solving the business needs? Absolutely. Is it good to try to raise the level of abstractions we work for so that we can concentrate most of the time in the parts of applications that actually matter? Again, absolutely agree. But making the leap from here that the obvious end result we should aspire to is to this? I mean, if that's the best we can do, we might as well just abandon all hope and give up.

An interesting aspect of Nick's proposition is that it is indeed a win-win situation for a company like Microsoft, while I'd say pretty much all IT shops (partners) in the MS ecosystem would come out loosing.  See, in Nick's ideal world, Mort would just drag and drop till he drops (sorry, couldn't resist :-)), while someone smart would create the tools and services he use. The next proposition in line would be that those tools would be done by MS (of course) and the services well, by none other than MCS [1]. Partners around could put in those Morts instead of supplying development talent, but, well, what do you know! Since Mort doesn't really have any special skill set (particularly none that's very software specific), well, the business themselves might decide to supply it! [2] 

And now we've just gotten round all the way back to the old idea of CASE tools and 4GL languages.

[1] I have no grudges against MCS; indeed I know many people working there. It's just the logical conclusion :-)

[2] This would also likely mean that those lucky enough to actually work on the Developer side would probably be able to get paid a lot more for their work, being a scarce resource and all...

Technorati tags: ,

It’s not the tools, it’s how they’re used

In my previous entry about Morts, Nick Malik left an interesting response. What caught my eye the most was his comment that:

"If you find yourself having to maintain a program that was not written to be maintained, don't blame the maker of the tools!"

I agree with this sentiment to a large degree. How people use the tools, is critical. But, that's only part of the story. See, a lot of tools don't lend themselves towards wiring maintainable code. Some of Microsoft's current tools fall in that category, unfortunately. For example, some don't lend themselves towards creating testable code. That's a big problem all by itself.

So I'd like to take Nick's comment and turn it an opportunity. Sure, it's the people using the tool that take most of the blame, however, the maker of the tool may (should!) have a significant responsibility in creating tools that encourage the creation of maintainable code. (Notice I use the word encourage here).

There's a thin, but significant line between tools that allow you to create crappy code quickly and then leave you high and dry, versus tools that encourage you towards creating good code. The key here is that the tools should make it as painless and natural as possible to do the right thing. This is not about wizards. It's not about visual designers. It's about code.

Microsoft is in a unique position to provide this, but so far some teams within it insist on making the default way of creating applications to be the one that favors quick and dirty over the one that favors maintainable code. The latter does not need to take longer, be more complicated, or be harder than the former, but some tools insist on making it that way. Choosing the right abstractions, by the way, is a key step here.

Technorati tags: ,

Developer Skill and Writing Maintainable Code

There are some discussions going on at this time on the community around the topic of
writing maintainable code and the different trade-offs a given "project reality"
might force on the developers involved in it. Very much worth reading and highly
recommended everyone reads and participates. Jeremy Miller beat me to it with some
good comments, as well.

My own personal opinion on the matter is a bit of a mixed bag (and so at times I might
seem to contradict myself).  On one hand, I find myself in total agreement with Ayende and Sam (just to name some of the most vocal proponents) that raising the level and knowledge of the developers that are going to be receiving the application is a critical step. I am a strong believer, learned from my own personal experience, that there's no amount of tooling and supervision/control that can "fix" the deficiencies of the team members themselves. It just simply doesn't cut it.

I believe says it best when he says that "Because my view is that having the developers learn a better way to build software is much less costly than continuing to produce software that is hard to maintain". I couldn't have said it better.

I am, however, sensitive to the dilemmas brought up by JDN and Frans Bouma, both of which have articulated their concerns and point of view in very provocative and thoughtful ways. I'm sensitive because I've found myself in their position a few times, and I've made the same choice they've argued in some of those occasions. However, my experience also shows that making that trade off, at those times I've made it, has eventually shown to have been a less than optimal decision.

Given this, I'm firmly in the camp that a) you should strive to avoid making that choice if at all possible, and b) you've got to be pragmatic about it. Sometimes, yes, organizational dynamics and politics play a role here, even if we disagree with it. So sometimes that might be the right choice to make, but rarely (if ever) for purely
technical reasons. And the whole "other developers might not get it" is, in the end, a red herring and tends to obscure the real reasons and motives behind it.

Why is it a red herring? Because it is usually a sign of the "we want to have a few highly skilled developers and lots of unskilled, cheap developers for the 'easy' stuff" mentality that pervades a lot of the companies in our profession, and it's just one I can't agree with. It's not that everyone should have 10 years of experience building extremely complex applications, but rather that everyone should have the opportunity to do their work to their best of their abilities. Inevitably, I find that this sort of mentality tends to set bounds that developers can't later cross easily without a lot of push back from management and the organization, leaving unskilled developers without a path they can collaborate with their more skilled, more experienced peers to grow their own skill set.

In many ways, this is a topic that strikes close to home for me, for several reasons. First of all, this kind of discussion is one of the reasons I decided to quit my last job and start my own company. In my point of view, the whole concept that the people are the key ingredient that make/break a project is absolute key. It's not about how much experience and knowledge the developers have. This is a significant component, for sure, but one that can be acquired with time. What is absolute key is the attitude: 

  • A willingness to constantly learn and look for new opportunities for personal/team/product improvement. 
  • A commitment to satisfying the client/costumer/user.
  • A commitment to being ethical, honest and professional about your work and your own capabilities and limitations.
  • Awareness of your context and that of your project/product.
  • Caring about what your doing and being enthusiastic about it.
  • Being critical about your own work and those of others, even if your believe others are above you or have different responsibilities (say, project manager or architect). Note this is not about constantly criticizing other people's work; it's about thinking critically about what the work that's being done by the team as a whole and the results.

I've seen many developers that lack one of this. Some are just 9-5 guys [1] that just want to get through the day without caring about the end result. Others have no desire for learning new things or getting out of their comfort zone. Others simply don't care about what other people do or what they do, usually manifested by them saying they're done implementing something and when you go look it doesn't even compile. Others will just do what their told to do and never even stop considering if it makes sense or if indeed it is the best way to do it.

Like Ayende, I refuse to be just a replaceable cog in a big machine; I want to make a difference and continuously learn and grow my skill set. I wouldn't have it any other way.

So it's about lots of different things. I love working with people that are critical about my work; they make me be a far better developer, and by listening to their opinions and sometimes different alternatives they propose I myself learn a lot. Sure, it's sometimes hard to listen to it all, but it's much worth it.

Arguing about the right things
I think one aspect that has tended to mud a bit the discussion is that we put a lot of different things into the same bag, which makes everything more confusing. For example, some people have mentioned MVP/MVC, ORM and TDD as important aspects in achieving maintainability, which some people have reacted a bit against.

From my point of view, the problem here is that we're mixing practices with tools. For example, TDD is a practice, and one that indeed is (imho) very important in facilitating the creation of maintainable code. But by itself it is no silver bullet (my friend Sam, for example, always talks about using the full set of 12 practices proposed by eXtreme Programming, not just TDD).

On the other hand, ORM and MVP/MVC are tools (yes, patterns are just design/coding tools, in the end), and sure, they can significantly help in producing maintainable code. But they are just tools, and you have to evaluate their impact and usefulness in each case. Some projects might simply not benefit from using ORM
(for example, this is rarely a useful component in the integration project's I've worked on), while it might reduce the amount of hand written code in your typical business application (and less code will always mean more maintainable code). But others might prefer for certain applications the use of code generation instead of an ORM and, you know what? That might be a perfectly valid choice in the context they're in. What's key here is
that they've done the analysis and choose what they believe it's best for them given their context.

Note: It's perfectly possible that the choice still turns out to have been the wrong one in the end, given that many times you start out with limited information from which to make your decision; but that's life.

It's about the future

One key issue I've seen again and again is that many developers don't even care about writing maintainable software, simply because they're not the ones that will have to maintain it. It's a very pervasive mentality in some circles that has been brought, in part, by the outsourcing of software development (something I make my living off, by the way). For example, in Colombia, a lot of companies have no internal software developers anymore, so external providers come in, create the initial release of an application and go away, many times never to see that code again except for one or two fixes during stabilization.

This means many developers don't really learn about the value of writing maintainable software because they it's easier to justify to yourself the writing of poor code when you think "I'm not going to maintain this later on, so why should I care?". This is not only a very egotistical point of view (and easy to fall on) but also ethically challenged. But what's worse is that it is a complete fallacy: you're still maintaining your code, and every day you write poor code in a project it's more code you have to maintain. It's just a lot of people seem to think otherwise and lie to themselves as a defense mechanism against the pain their going though every day on their project.

[1] there's a difference between wanting to have a life outside of work (good) and not giving a shit (bad).

The Wrong Mort

My good friend Sam Gentile pointed to me Nick Malik's post on "Tools for Mort", in which Nick responds to some comments made by Sam a few days ago. I don't particularly agree with Nick's point of view. Several things bothered me about this post.

Why are we talking about Mort again?

First of all, I've said in the past that I don't particularly care in the Mort/Elvis/Einstein personas. Besides the fact that they tend to split users/developers around what I believe are the wrong areas, they are way too misinterpreted all around. Just bringing up the whole Mort topic is a sure way to turn a discussion in an unproductive direction.

What's worse, Nick apparently thinks Morts are the great Agile developers fighting the corporate world. Frankly, I have no clue where he got that idea from, but what do I know.

Look, it's great that there are people leveraging office and other platforms to solve business problems and get work done. There's a lot of really good use cases for that technology. However, it doesn't mean all software projects should be solved that way. In some scenarios, these kind of tools are great for solving immediate issues, but many times they are not suitable for building long-lasting, scalable solutions. Both sides of the software spectrum have different needs and deserve different solutions.

However, let's be clear about why the term Mort got a bad reputation: It was typically applied to people who didn't quite understand the tools/environment they were working on, creating half-assed applications that while did the trick initially, were a living hell to maintain because they didn't respect the most basic principles of good software development (in many cases, were not even aware they existed). Hey Nick, let's be honest here: your "Mort" friend was definitely no Mort if he was doing continuous integration and automated unit testing [1].

Sam doesn't get Agile?

Nick goes on to say:

"Do you really mean that Microsoft should make a priority of serving top-down project managers who believe in BDUF by including big modeling tools in Visual Studio, because the MDD people are more uber-geeky than most of us will ever be?  I hate to point this out, Sam, but Alpha Geeks are not the ones using TDD.  It's the Morts of the programming world.  Alpha geeks are using Domain Specific Languages."

From this sentence alone it's clear to me that Nick is obviously not aware of Sam's extensive and continuous efforts talking about "agile architects" and railing against BDFU. Really, you might want to look a bit around before making those kind of comments.

Now, Morts are the ones doing TDD? Wow. color me surprised. Alpha Geeks use DSLs? Really? AND THEY DO NOT DO TEST DRIVEN DEVELOPMENT? WOW.

Hummm...not quite. Most of the so called Morts have no freaking clue what Test Driven Development. People like the Code Better guys, Ayende and a lot others are precisely doing that: Trying to raise the level of awareness in the community about TDD/DDD and improving the way they write software. They are the ones trying to improve our field by making developers more aware of how they work and get them to write better software that not only works and solves the business problem it is intended to solve, but that is also maintainable. At least grant them that much.

I think Nick is also a bit confused about the whole Alpha Geek business. A lot of them do use dynamic languages and yes, a lot of them use them to create domain specific languages. But what makes you think they don't need TDD? Quite the contrary, Nick, they are far more reliant on TDD and automated unit testing.

Nick then says that "We [Microsoft] are the singlehanded leaders in the space of bringing BASIC developers up to modern computing languages.". Humm.... certainly true, you moved them over to VB.NET. It's certainly possible to write Object Oriented code in VB.NET. That, however, doesn't mean they are. Let's face it, there are millions of developers out there using VB.NET and (yes) C# that are still writing essentially procedural code (if they are lucky). Just because you declare classes doesn't mean you're doing OOD.

It's the tooling

But that's all mostly irrelevant to the discussion. I got the chance to speak about this with Sam and others during the past MVP summit, so I do think I have an idea what the fuzz is about (and because I happen to agree with the consensus). See, none of us are arguing that Microsoft has not done significant contributions to the community and even to the Software Developer profession, because they have.

What we do argue about is the tooling coming out of Microsoft and the way it is meant to be used. A lot of the software coming out of DevDiv these days seems to be very focused on building something quick. That's not a bad thing by itself, we all want software could be developed faster and that we could deliver it to the business as soon as possible.

What is a problem is that the tools enable that quick result at the expense of something else; something fundamentally important if you're not writing sample applications or "quick-n'-dirties". They are fundamentally non scalable (and I'm not talking about performance here), and unmaintainable if you just stick to the whole straight from the db databinding and drag and drop style of development Microsoft is so proud about. Better people than me have argued this more vehemently that I ever could so I'm not going to repeat it here.

I think that, at a fundamental level, there's a significant disconnect here. What some part of the community spends its efforts in is to try to raise the awareness, knowledge and capabilities of developers everywhere as a means. For them this is a key point in getting better software, with better and faster results. Instead of dumbing down the tools to suit the unwary developer, they are trying to smart up the developer to use the available tools more effectively.

[1] Somehow, I can't imagine Nick's Mort doing TDD while writing Excel macros, but maybe I'm just being cynic...

Technorati tags: , ,