Here's one interesting issue I ran into while trying a few things out: Conversions.
Given the dynamic nature of the DLR and the languages implemented on top of it, and the statically typed nature of the CLR and most .NET languages, you need to be able to ensure that the right type conversions happen at the right places.
This involves, among other things, ensuring that the right casts and boxing/unboxing operations happen, as well as ensuring that language-specific conversions can be applied (such as, say, converting an arbitrary object to a CLR string through an explicit method call).
The DLR does a bunch of these automatically in a number of places, like variable writes and assignments, call return values and so on. Most of these can be made to follow language-specific behaviors because the conversion expressions are generated through calls to your language ActionBinder implementation (like the RubyBinder class in the case of IronRuby).
For example, many of these scenarios will generate calls to your ActionBinder.ConvertExpression() method. Here's one case I found where this does not happen, making it a bit more tedious:
Today I was debugging some code because of an InvalidCastException getting thrown. Indeed, the problem was that I had a method call that required converting one of the parameters, and it was a set of types I had not tried before. In this case, the necessary conversion was language specific, so I added the rule into my ActionBinder.ConvertExpression() implementation and made sure as well that ActionBinder.CanConvertFrom() handled the case as well.
But it was still not working. After a bit of debugging, I realized that in this particular case, the ActionBinder wasn't getting involved at all for the conversion! What was the deal?
The issue turns out to be that the call was being done through one of my IDynamicObject implementations. In particular, this one handled a Call action by generating a rule target containing a call to the target delegate using the DLR"s Ast.ComplexCallHelper() method.
The problem is that ComplexCallHelper() doesn't really deal with language-specific conversion needs at all. Instead, it relies on Ast.Convert() to convert the actual argument values to the types declared in the target method/delegate signature.
The problem with Ast.Convert() should be obvious by now: All Ast.Convert() and Ast.ConvertHelper() do is insert an explicit Cast expression into the code.
This works great if all that is needed is a box/unbox operation or an upcast/downcast, but not enough in our case, so it is something to watch out for! I still haven't decided yet how to best work around the issue