Saturday, September 16, 2006

.NET 2.0, HttpWebRequest.KeepAlive And ServicePoint.ConnectionLimit

Since switching over to .NET 2.0, one of our applications occasionally threw "The request was aborted: The request was canceled." resp. "The underlying connection was closed: The request was canceled." errors. This happened maybe 1 out of 25 times when connecting to another server using HTTP, either directly via HttpWebRequest or on webservice calls, which also apply HttpWebRequest internally.

Digging further into the issue, it turned out that several people suggested to set HttpWebRequest.KeepAlive to false. And as matter of fact this worked. But KeepAlive=false will most likely imply a performance penalty as there won't be any re-use (also known as HTTP pipelining) of TCP-connections. TCP-connections are now closed immediately and reopened on each HTTP-request. To make things worse, we have SSL turned on, with both server and client certificates, so KeepAlive=false will then cause a complete SSL-Handshake now for each and every HTTP request to the same server. As a sidenote, KeepAlive=true with client authentication requires the initial authentication to be shared between all following requests, hence it is necessary to set HttpWebRequest.UnsafeAuthenticatedConnectionSharing=true. Please consult the MSDN documentation for more information about what that implies.

Back to the original topic, our application used to work fine under .NET 1.1, only .NET 2.0 seems to have this problem. And despite the fact that people have posted this issue over and over again, there is no Microsoft hotfix up to this day. KeepAlive=false is the only remedy.

By the way, another property that is essential for network throughput under .NET is HttpWebRequest.ServicePoint.ConnectionLimit. Microsoft started limiting the maximum number of parallel network connections some time ago (I once read in order to prevent malware from flooding the network - which sounds unlikely given that one can simply raise this limit programmatically, at least with admin credentials), and while this might not harm the single user when surfing the web, it surely is not suited for other kinds of applications. After the limit is reached, further request will be queued and have to wait for a running request to terminate. I am sure there are quite some applications out there which run into performance problems because of this.

The preset connection limit might vary depending on the underlying operating system and hosting environment. It can be changed programmatically by setting HttpWebRequest.ServicePoint.ConnectionLimit to a higher value, or in one of .NET's .config files (machine.config, app.config resp. web.config):

<add address="" maxconnection="32">

This is just one of many possible explanations, but that unfortunate limitation might boil down to a particular interpretation of RFC 2616 (HTTP1.1), which states:

Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy. A proxy SHOULD use up to 2*N connections to another server or proxy, where N is the number of simultaneously active users. These guidelines are intended to improve HTTP response times and avoid congestion.

So is any .NET application considered "single-user client" by default? Our system surely isn't a single-user client, it's more like a kind of proxy, with plenty of concurrent users and HTTP connections. And how many developers might now even have heard about this setting, and will find out once their application went into production?

Also note that phrase of "persistent connections". Of course it is a bad idea for the client to hold hundreds of connections captive and thus bringing the server to its knee. But as I have mentioned, we cannot pool any connections anyway, not even a few of them, thank's to .NET 2.0's "The request was aborted: The request was canceled."-bug.

Anyway, it's the server's responsiblitlity to determine when too many connections are kept alive. After all, HTTP-KeepAlive is something that the client might ask for, but that the server may always decline.

Wednesday, September 13, 2006

Jeff Atwood Takes On Joel Spolsky

Jeff Atwood openly asks if Joel Spolsky "jumped the shark" because of implementing their his company's flagship product FogBugz in an in-house Basic-dialect called "Wasabi". Wasabi can then be compiled to VBScript or PHP, depending on customers' requirements.

Back when I read about this approach, it made me raise an eyebrow as well. I remember thinking, "boy I thought I'd like to work at Fog Creek, but having to code in such a proprietary language would be no-go". And I was wondering if that wouldn't contradict their frequently mentioned efforts of trying to attract great programmers. Actually, of the best programmers I know, nobody would ever voluntarily program in something like VB or VBScript, let go Wasabi.

But he then went on describing how this decision made sense from a business point (platform independence at a minimum of effort, no complete rewrite necessary), so I said OK, I might not like it from a developer's perspective, but from a business perspective it could make sense.

Now Jeff Atwood probably is a more censorious person than me. He decided to rip apart Joel's argumentation and connect the Wasabi-story with Joel's criticism of Ruby. He continued to make some pretty hefty statements like "fifty thousand programmers' heads simultaneously exploding", "absolutely beyond the pale" or "amplifies the insanity". I usually like Jeff's artwork, but attaching a big red WTF label on Joel's forehead is not really subtle either. And the flaming continued in the comment sections of both blogs.

My recommendation: Please settle down! From reading one or two paragraphs in some Joel on Software article, nobody possesses the same amount of information that Joel Spolsky was basing his decision on. Also, recognizing that what seems right from a developer's view must not always be right from a business standpoint is something that many programmers still have to get into their minds.

But hey, let's not hang on each of Joel Spolsky's words either. He certainly is an extraordinary programmer and writer, but he might be mistaken as well at times (a possibility he openly admits).