Processing IIS ETW events using Azure Stream Analytics

This blog post continues my series of blog posts about ETW logging in IIS 8.5. One of the things that I wanted to do from the start was using this as an excuse to dig into some Azure services. In the last entry, I updated the sample code to push ETW events collected from IIS in near real-time to an Azure Event Hub.

Now, I’d like to take that a bit further and actually process the events that make it to the hub. Naturally, I could write my own Event Hub listener that processes said events, but the Azure Stream Analytics service (in preview) sounded like a pretty interesting way to do this, so I decided to give it a try!

Before I actually detail how to make this happen, I had to make a change in the collector code. My original design used a “nice” JSON representation of the collected ETW events. That is, an event was represented by a complex JSON object that contained 3 properties (header, event, payload), of which the first two where objects containing properties themselves.

Unfortunately, this does not appear to be supported by Stream Analytics. Though the documentation doesn’t explain it as clearly, Stream Analytics requires that input JSON objects be “flat” (that is, they are simple name-value dictionaries). This is a very understandable requirement, and makes sense given that the other supported format is CSV, but it is something to keep in mind.

Based on this, I modified the collector code to use a essentially the same format, but flattening the header and event members using the header_ and event_ prefixes as necessary.

Configuring the Stream Analytics job.

The first step to use Stream Analytics is creating a job in the management portal:

clip_image002

Once your Stream Analytics job is created, we will want to connect it to our existing Event Hub that is receiving events from the collector service.

Creating an Input

Before doing this, we need to add a new SAS policy that Stream Analytics can use to read the events from the hub.

To do this, go to your Event Hub configuration page, look for the “shared access policies” section and add a new one with Manage permissions:

clip_image002[10]

Then click on the Save button to persist the changes.

Now go back to the Stream Analytics instance and select the Add Inputs option. Here’re we will want to connect it to our existing Event Hub instance:

clip_image002[12]

The type of the input should be Data Stream, Event Hub, and then enter an alias for the input and select the existing event hub and the right shared access policy:

clip_image004

Then select the serialization format and encoding:

clip_image002[14]

Creating a Query

Now that we have a way to get data into our Stream Analytics job, we can define one or more queries on top of the data. For example, let’s say we want to create a summary of the successful hits within a 5 minute window:

clip_image001

Creating an Output

Now let’s define an output for our query. In this case, I want to store the job output into a SQL Database on azure, which I have already created. Notice that you will want your DB to be on the same region as the Stream Analytics job! In this database, we’ll create a simple table to store the results:

clip_image001[7]

Go to the Output section of your Stream Analytics job and select the Add an Output option. Then select “SQL Database” as the destination, and enter your database details:

clip_image002

Note: If you use a database that has ‘-’ characters as part of the name, trying to create the output will result in an error, at least when I tried it.

clip_image002[16]Running the job

Now we can start our Stream Analytics job and test the results!

After the job has been running for a while, and we’re feeding data to the Event Hub, we start seeing the results in our database table:

clip_image004[6]

Overall, this was surprisingly easier than I expected it to be, but at the same time, it took me longer than it should have to get it to work. Mostly, due to the fact that I wasted a lot of time trying to get it to accept my output database and understanding some of the limitations and the query syntax. However, I expect this will get a lot better once the documentation is improved!

Playing with Roslyn and VSIX

I have been playing a bit with the Roslyn APIs in Visual Studio "14". Since I am a big fan of VS editor extensions, my first attempt was building something simple: A Visual Studio tagger that would colorize uses of parameter variables and field variables in code:

Roslyn Colorizer

The code is ugly as it is, and does not use the Roslyn Async API, which is something I’d like to look into later on. I’m also not entirely sure if there’s a better way to build such a thing (Roslyn has it’s own concept of classification, but not sure if that is extensible). It was fun, though!

Code can be found here

ETW Logging in IIS 8.5 (2)

A few days ago I posted a sample on capturing ETW log events generated by IIS 8.5.

This turned out to be a good excuse to play around with the new Event Hubs service in Microsoft Azure, so I’ve now extended the sample with a collector application that can capture the ETW log events generated by IIS and push them in JSON format to an Azure Event Hub.

IIS - Event Hub

From here, I could process events in a number of ways, and it will be interesting to play with other Azure services to do just that later on. The sample code is updated on the github repository, but be aware it is very rough, but I’ll keep improving it.

Invalid application pool name error in WebDeploy

I was helping out a customer a few weeks ago to use the Web Deployment Tool to migrate applications and sites from IIS 6.0 to IIS 7.x/8.x, and ran into an error I had not seen before. I thought I'd document it here in case I, or anyone else, runs into it again.

During our work, we exported an entire IIS 8.5 server to a ZIP package using the Web Deployment Tool tasks integrated into the IIS Management Console (inetmgr).

When we tried to restore the server archive on the same server, we got the following error:

Error: Invalid application pool name

After investigating the issue for a while, I realized the problem was that the IIS configuration of the server was invalid before we exported the server package! IIS is very robust at dealing with some invalid configurations, and this was such a case: The server appeared to work perfectly fine even though there was one slight configuration error, that would have only caused trouble under very specific circumstances!

To figure it out I had to attempt the restore operation using the command line msdeploy.exe tool rather than the UI. Capturing the full log, I noticed that the last relevant entry was:

Info: Updating applicationDefaults (MSDeploy.webServer/webServer/appHostConfig[@path='']/location[@path='']/section[@name='system.applicationHost/sites']/sites/site[@name='Default Web Site']/applicationDefaults).

The problem turned out to be that for some reason, the applicationPool attribute of the section for all sites was not set. When the full server archive was created, Web Deploy exported a blank value for this property. When I looked at the archive.xml included in the ZIP file, this could be seen clearly:

<applicationDefaults
    path='' MSDeploy.path='1'
    applicationPool=''
    MSDeploy.applicationPool='1'
    enabledProtocols='http'
    MSDeploy.enabledProtocols='1'
    serviceAutoStartEnabled='false'
    MSDeploy.serviceAutoStartEnabled='1'
    serviceAutoStartProvider=''
    MSDeploy.serviceAutoStartProvider='1'
    preloadEnabled='false'
    MSDeploy.preloadEnabled='1'
    MSDeploy.MSDeployLinkName='applicationDefaults' />

How to fix it

Once I had this nailed down, working around it was easy: I simply extracted the entire ZIP archive to a folder, modified the archive.xml file to set a applicationPool='DefaultAppPool', and then simply imported the server backup directly from the extracted folder by using the archiveDir provider as the source. Alternatively, I could've created the ZIP file again and import it as usual.

Viasfora 2.0 Released!

Today I’ve released version 2.0 of my Viasfora extension for Visual Studio, which supports VS2010, VS2012 and VS2013. Lots of work went into this release, not only to add some cool new features, but also to clean up the existing codebase a little bit and fix bugs reported by users.

Two new features in this release extend Rainbow Braces to make it more useful:

  • Commands to navigate from the caret position to the opening/closing brace
  • Rainbow tooltips, which allow you to quickly check the line where the matching brace is located

Rainbow tooltip

Working on Rainbow ToolTips was very interesting because I wanted to end up with something that looked similar to the preview tooltip in the map mode scrollbar in VS2013. The first problem was figuring out the right way to create the text view for the tooltip, and then making it look good. One particularly frustrating part of it was having to write a custom IIntellisensePresenter to avoid having a very wide border around the tooltip!

Another significant change in this release is that I have removed the support for Visual Studio “14″ CTP, which was introduced in v1.9. Unfortunately, there is problem with the Visual Studio Gallery at this time, which makes extensions ‘invisible’ to Visual Studio 2013 if support for VS14 is present in the manifest [1].

Because of this, very few people running VS2013 have been able to update to v1.9 unless they explicitly went to the VS Gallery and downloaded the VSIX manually.

Unfortunately, it is not clear if and when this problem might be fixed. In the worst case, extension developers will just have to make separate releases for VS 14 and publish them as separate extensions in the VS Gallery. This is a lot of work that, at this point, is not worth the effort for me. I will reevaluate this when VS 14 is officially released if the problem still exists.

For now, if I get enough interest in VS 14 support, I’ll figure some out of band mechanism for sharing VS14-enabled builds.

Finally, I’d like to say that working on Viasfora has been very rewarding personally. It has given me an excellent opportunity to learn a lot about Visual Studio extensibility, and both the Web Essentials and VsVim sources have been extremely useful for this. I’d also like to say thanks to all my twitter contacts, not only for putting up with my frequent rants, but also for often helping me figure out how to accomplish something in VS (in particular, the feedback from Jared Parsons was always very appreciated).

[1] Disclaimer: I do not know if the problem is enabling VS 14 support in the manifest, or enabling support for 4 VS versions at the same time. I also do not know what plans the team responsible for the VS Gallery has regarding the issue, or even if any exist at this point.

IIS AppPool Identity SIDs

Since Windows Vista came out, a bunch of different sets of "virtual account"e; types have been added to the operating system. The most obvious one being Service accounts managed by the Service Control Manager, which belong to the "NET Service" authority and have the SID prefix "S-1-5-80". If we look at the documentation for these SIDs, we'll find that they are basically defined as "S-1-5-80-{SHA1(UNICODE(<service name in upper case>)}".

IIS 7.X/8.X also has its own set of virtual accounts, associated with the AppPoolIdentity. These belong to the "IIS AppPool" authority with base SID prefix "S-1-5-82". The way these application pool names convert to SIDs is basically the same as that of service accounts, with one little difference: The input to the SHA1 function is not the application pool name in uppercase, but in lowercase.

Here's a very simple PowerShell function I wrote to make it easier to do these conversions:

function Get-SIDHash([String]$sidPrefix, [String]$user) {
  $userToHash = switch ( $sidPrefix ) {
    'S-1-5-82' { $user.ToLower() }
    default { $user.ToUpper() }
  }
  $userBytes = [Text.Encoding]::Unicode.GetBytes($userToHash)
  $hash = Convert-FromBinHex (Get-Hash $userBytes 'SHA1')
  $sid = $sidPrefix
  for ( $i=0; $i -lt 5; $i++ ) {
    $sid += '-' + [BitConverter]::ToUInt32($hash, $i*4)
  }
  $sid
}

Viasfora v1.6 Released

Today I published a new update to my Viasfora extension for Visual Studio 2010-2013. One of the new features in this build is a text editor margin that could be useful to other fellow developers working on extending the Visual Studio Text Editor.

Developer Margin

Developer Margin

One of the reasons why I implemented this was that I when I needed to extend Viasfora to support new file formats and other document types, I always had to resort to setting breakpoints in the debugger to figure out things such as the content type and how the editor worked, and this gets tiresome real quick!

In this first iteration, the margin exposes the following information:

  • What buffers compose the buffer graph: Complex editors in VS often require layering multiple buffers with different content types on top of each other, and this will let you easily see what those buffers are. For example, an ASP.NET MVC Razor editor might contain several buffers, such as the main Razor document, one containing the underlying CSharp code generated for the document, another containing the JavaScript code, plus a few other projection/elision buffers.
    Because the content type is the key to extending the text editor, it is very useful to be able to examine how the buffer graph is made up and making sure you don’t have multiple instances of your extension on multiple buffers interacting poorly with one another.
  • Exploring individual buffers: Besides visualizing the buffer graph, you can also explore the contents of the each of the buffers in a separate editor window. One side benefit of this is that it makes it easy to check out what the generated code for many text editors is like :).
  • Exploring Content Types: In Visual Studio, content types can inherit from one another. For example, the “RazorCSharp” content type inherits from “Razor”, which in turn inherits from “htmlx”, and so on. With Viasfora, you can now easily inspect the content type tree.
  • Keeping track of the caret position: Visual Studio does a nice job of telling you where you’re located in the document at any given time with the line/column slots in the status bar. However, if you’re implementing text editor extensions such as classifiers, you often will be tracking things based on the actual position in the text buffer, rather than as line/column pairs. The margin adds such a counter on the bottom right.

Eventually, I hope to add other useful features to the margin, but suggestions are always welcome. Enjoy!

Pretty QuickInfo Tooltips and VS Themes

A Visual Studio Extension can extend quick info tooltips by implementing the IQuickInfoSource interface (and a few other things).

Most of the samples implement the AugmentQuickInfoSession() method by adding string values into the quickInfoContent collection. However, you can also add UIElement objects when you want to present tooltips that have complex content.

I implemented this for Viasfora as part of the XmlQuickInfoSource class, in order to present a much nicer tooltip for XML namespace prefixes. The end result was this:

Tooltip on Light Theme

Tooltip on Light Theme

It looked great. That is, until I tried it on Visual Studio 2012/2013 using the Dark Theme, which turned out to be unreadable:

Tooltip on Dark Theme

Tooltip on Dark Theme

After a lot of researching, I ran into the answer: There is an EnvironmentColors class included in the Microsoft.VisualStudio.Shell.(11|12).0.dll assembly, which will contains styles you can use to theme your WPF elements to properly match the selected theme.

For my needs, I could just use the ToolTipBrushKey, ToolTipTextBrushKey and PanelHyperlinkBrushKey properties to style my controls. However, I ran into a snag: The EnvironmentColors class is not available on the Visual Studio 2010 assemblies, which was a problem for me in order to support VS2010/2012/2013 with a single assembly.

While not a very pretty solution, I resorted to using Reflection coupled with the major Visual Studio version number in order to support this feature. The relevant pieces of code are:

private UIElement CreateInfoText(String xmlns, String url) {
  var textBlock = new TextBlock();
  Hyperlink hl = new Hyperlink(new Run(url));
  textBlock.Inlines.AddRange(new Inline[] {
    new Bold(new Run("Prefix: ")),
    new Run(xmlns),
    new LineBreak(),
    new Bold(new Run("Namespace: ")),
    hl
  });
  // set styles in order to support other 
  // visual studio themes on 2012/2013
  object tooltipBrushKey = VsfPackage.Instance.FindColorResource("ToolTipBrushKey");
  if ( tooltipBrushKey != null  ) {
    textBlock.SetResourceReference(TextBlock.BackgroundProperty, tooltipBrushKey);
  }
  object tooltipTextBrushKey = VsfPackage.Instance.FindColorResource("ToolTipTextBrushKey");
  if ( tooltipTextBrushKey != null ) {
    textBlock.SetResourceReference(TextBlock.ForegroundProperty, tooltipTextBrushKey);
  }
  object hlBrushKey = VsfPackage.Instance.FindColorResource("PanelHyperlinkBrushKey");
  if ( hlBrushKey != null ) {
    hl.SetResourceReference(Hyperlink.ForegroundProperty, hlBrushKey);
  }
  return textBlock;
}
private Version vsVersion;

public object FindColorResource(String name) {
  if ( vsVersion.Major <= 10 ) {
    return null;
  }
  Assembly assembly = Assembly.Load(String.Format(
    "Microsoft.VisualStudio.Shell.{0}.0, Version={0}.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a",
    vsVersion.Major
    ));
  if ( assembly != null ) {
    Type type = assembly.GetType("Microsoft.VisualStudio.PlatformUI.EnvironmentColors");
    var prop = type.GetProperty(name);
    return prop.GetValue(null, null);
  }
  return null;
}

Hope someone else finds this useful. Enjoy!

IClassifier and IDisposable

I’ve been troubleshooting some issues with Viasfora and noticed something that I definitely was not expecting:

It appears that if you implement an IClassifier extension, and also implement IDisposable, it will not be called by Visual Studio once the text buffer is released.

I have noticed this on Visual Studio 2013, but so far, I haven’t fully verified if this is indeed the expected behavior, or if it is caused by something else I am doing. I do know of many other Visual Studio extensions that combine IClassifier with IDisposable, so if this turns out to be the default behavior, it could signal other extensions may be leaking resources.