Failing The Turing Test

While We Were Doing That...

2011-12-19T00:45:00.000+01:00

JAX-WS: WSDL Fetched During Service Proxy Creation

2011-12-18T22:49:00.016+01:00

I did some webservice client programming again lately, and was surprised to learn that by default, the JAX-WS service client library fetches the WSDL from its original location again each time a service proxy is created. I am pretty sure that not everyone is aware of that, and that this leads to production system problems once WSDLs become unavailable.

The JAX-WS FAQs mention runtime binding as the reason behind this, but as the WSDL is not supposed to change, I wonder why it isn't simply mapped into a generated Java class, or added as a local resource. Anyway, this can be done manually using the JAX WS wsgen tool (commandline option "-wsdl"), adding the local WSDL file to your JAR file, and then setting the service proxy's WSDL_LOCATION constant to:

MYSERVICE_WSDL_LOCATION = 
    new URL(
        MyClass.class.getResource(""), 
        "MyWsdl.wsdl");

Chickens Can Do Your Job

2011-12-14T22:00:00.016+01:00

And that's why we don't verify a candidate's skills by asking technical questions in interviews. That might hurt his feelings after all! We probably have heard of studies stating that, from a group of people with identical formal qualification, the top-notch folks can outperform their counterparts by a factor of 10 in efficiency (that is coding productivity, code quality, correctness and performance, whatever measure you choose). More than that, even people with lower formal qualification are more efficient by a factor of 10, compared to the other end of the spectrum.

But does that mean we will be searching for great developers, and once we have found them, pay them accordingly, and give them tasks that match their skills? No! Developers are developers are developers. They are interchangable, and what they don't know now, they will simply learn on-the-job. It's that easy!

COM Programming

2011-11-22T07:17:00.003+01:00

Luckily, with all those ATL macros and wizards available, resp. with COM-specific annotations in case of .NET, COM development became a lot easier over the years. Still, I would recommend everyone who is programming in COM, or even just reusing an existing COM component, to at least once walk through the complete process of manually implementing and registrating a COM component (including ClassFactory and hand-written registration code), to find out what's going on under the hood. A very nice step-by-step tuorial can be found at codeguru.com.

Tips For Lightning-Fast Insert Performance On SQL Server

2011-10-03T22:18:00.042+02:00

1. Increase ADO.NET BatchSize to eliminate unnecessary network roundtrips, e.g. SqlDataAdapter.UpdateBatchSize when working with DataAdapters, or SqlBulkCopy.BatchSize when applying SqlBulkCopy.
2. Limit the number of indices on the target table to what is really essential for query performance.
3. Place indices on master table columns referenced by the target table's foreign keys.
4. Choose the target table's clustered index wisely, so that inserts won't lead to clustered index node splits. Usually an identity column (AKA "autoinc") is a good choice. If you don't have autoinc primary keys, consider introducing a new identity column just for the sake of making it your clustered index.
5. Let the client insert into a temporary heap table first (that is, a table that has no clustered index, resp. no index at all). Then, issue one big "insert-into-select" statement to push all that staging table data into the actual target table. The "insert-into-select"-statement must contain an "order-by"-clause which guarantees ordering by clustered index.
6. Apply SqlBulkCopy.
7. Decrease transaction logging by choosing bulk-logged recovery model, resp. setting SqlServer traceflag 610.
8. If your business scenario allows for it, place a table lock before inserting. This will make any further locking unnecessary, and is especially a viable option on staging tables as described in (5). SqlBulkCopy also supports table locks via SqlBulkCopyOptions.
9. Place database datafile and logfile on two physically separated devices, e.g. on two disks or two SAN LUNs configured for different spindles.
10. Prefer server-side processsing (e.g. by means of "insert-into-select") to client-to-server-roundtrips wherever possible.
11. This is probably the fastest insert-approach I have ever heard of (taken from this sqlbi whitepaper, see final paragraph): Create a new heap table just for the current insert batch, SqlBulk-Copy data into that table, then build a suited clustered index, and add the table as a new table partition to an existing partitioned table.
12. Check execution plan when inserting, and go sure it does not contain anything unexpected or dispensable that might slow down your inserts, e.g. UDF-calls during check constraint execution, heavyweight trigger code, referential integrity checks without index usage or indexed view updates.

How To Make A Borderless WPF Windows Movable

2011-09-04T13:51:00.006+02:00

Say you have a borderless WPF window (WindowStyle="None"), the problem arises, how can one still move/drag the window around on the desktop (as there simply is no more window title bar)?

It's actuelly pretty simple:

MouseDown += delegate { DragMove(); };

Thanks to Marlon for this hint.

Is Your WPF ProgressBar Eating Up Unnecessary CPU Cycles?

2011-09-03T07:46:00.003+02:00

When your WPF ProgressBar is causing lots of CPU usage, even if it is not visible any more, this most likely is caused by the IsIndeterminate property, which - when set to true - simply continues the animation to be running in the background.

One solution is to bind the IsIndeterminate property to the same underlying value as the Visibility property, for example something like this:

<ProgressBar IsIndeterminate="{Binding IsBusy}" 

Visibility="{Binding IsBusy, 

Converter={StaticResource VisibilityConverter}}"/>

The IsBusy property of the bound DataSource can then represent whatever condition is suited in this scenario.

Thanks to Adam for this tip!

Compuware Acquires DynaTrace

2011-07-06T23:44:00.001+02:00

Congrats to the folks at DynaTrace: Compuware acquires DynaTrace.

Confessions Of An IT Manager

2011-07-01T23:49:00.002+02:00

As I stumbled back out into the light of the car-park, I wondered, for the life of me, how the IT industry had got in such a mess that unqualified and untrained people were in responsible positions within organisations all over the country, managing the databases that are the lifeblood of the enterprise.

What would happen if the same state of affairs infected Surgery, so that there was a chance of being operated on by someone whose only qualification was that he'd cut open his teddy bear as a child, using his 'My Little Doctor' kit, but managed to bullshit his way into a job armed with a bogus CV and a string of false references.

From: "Confessions of an IT Manager"

Free EBooks On Redgate.Com

2011-06-30T23:45:00.005+02:00

RedGate offers several free eBooks on SqlServer and other topics (some of them only announced and yet to become downloadable) on their bookstore. I had a look at "SQL Server Execution Plans" by Grant Fritchey, which seems to be pretty good. So now we can probably forgive RedGate for turning Reflector into a commercial product, after promising the opposite two years before, right?

When DataTable.Select() Is Slow, Use DataView.FindRows()

2011-02-17T22:03:00.016+01:00

ADO.NET's DataTable.Select() comes with a runtime complexity of O(N). This turns into a problem when the DataTable contains a lot of rows, and DataTable.Select() is invoked many times. It's pretty obvious why - just like in LinqToObjects-queries - there is no indexed lookup. DataTable.Select() needs to scan through all rows, and compare all values.

What I usually ended up doing in those cases, was to loop over the DataTable once, and fill a Dictionary using the Select's lookup-criteria as the entry's key, and the DataRow as its value, and use the Dictionary for lookups from this point on. This works fine, but it clutters the code with all those Dictionaries, which - in addition - must be synchronized manually each time changes occur in the underlying DataTable.

The DataView class offers an alternative. By defining DataView's Sort-property in accordance to the search criteria, DataView.FindRows() then allows indexed lookup with the usual binary search tree lookup runtime complexity of O(log(n)) (yes I know, Hashtable lookup might even execute at O(1), given an excellent hashing algorithm, but O(log(n)) will do here). For example:

// do this just one time
DataView view = new DataView(table);
view.Sort = "name, city";

// do this N times
DataRowView[] res = view.FindRows(
    new object[] { "Earp", "Tombstone" });

I should probably mention that DataTables and DataRows actually do offer indexed searching in two special cases:

(1) When there is a primary key defined on a DataTable, DataTable.Rows.Find() (resp. DataTable.FindBy[KeyColName]() on typed DataSets) allows for indexed primary key loookup.
(2) When there is a child relation defined on a DataTable, DataRow.GetChildRows() (resp. DataRow.Get[ChildRelName]() on typed DataSets) will return the parent row's children by indexed lookup as well.

When SQL IS Broken

2011-02-06T21:40:00.032+01:00

I think it was Mr. Benedict in "Last Action Hero", who said:

"Eliminate all logical solutions to a problem - and the illogical becomes invariably true."

I received a distress call the other day, concerning a legacy application locking a SqlServer 2000 database, together with some profiler tracefiles.

Looking at the traces I indeed saw how certain statements were blocked for minutes, until other connections committed their transactions. Nothing extradordinary one might think, of course uncommitted data updates lead to data locks, which will block ensuing statements, that are trying to read or write the same data. Maybe those transactions were just taking too long. But here comes the catch: They did not contain any Inserts/Updates/Deletes, just Selects (the only reason for the existence of the transaction being, that a programmer some fifteen years ago had decided to activate implicit transactions). And with Isolation Level "Read Committed", and those queries lasting only a few milliseconds, there should not be any long-living locks.

Unfortunately, SqlServer 2000 tracing does not support "Blocked process report" events. This tracing event was only introduced in SqlServer 2005, and delivers in detail which SQL statements block which other statements (you can find it in the "Errors and Warnings" event category. Hint: don't forget to activate 'show advanced options', and set an appropriate 'blocked process threshold' via sp_configure()). Of course one can always have a look at the Management Studio Acitivity Monitor, but the statement information there is not so fine-grained (all it displayes is the last statement within a batch, not necessarily the one causing blocking). And I hadn't received any activity monitor data for my analysis.

So when I took a closer at the tracefile, all I could do was to scan through all of those Lock Acquired / Lock Released events. This is a really tedious task, not only because there might be hundreds of locks being acquired and released on any scope from row to table, but most of all because you only see which locks get acquired, and not which locks actually fail to get acquired (and hence lead to blocking).

The statement being blocked turned out to be a single row update. I saw a key lock and a page lock being acquired, and then things stood still. So what I did was to look for an identical statement within the trace, but one which would not be blocked. Luckily, I found one, and by closer examination, I noticed that the next lock acquisition would have been a page level lock mode upgrade from "Intent Update (IU)" to "Intent Exclusive (IE)". That's also not surprising, just standard procedure when a row is about to be updated.

The only thing that collides with such a lock upgrade would have been an existing "Shared (S)" lock on the same page (or a row within that page). But how could there be a shared lock, when no other query accessing the same data was currently running (taking in consideration Isolation Level "Read Committed")? OK, some minutes before, a query accessing the same table had been executed, and its transaction was still open (due to implicit transactions), but the statement's shared locks should be gone by now. OK, it didn't make a lot of sense to use a transaction for a read-only operation (again: Isolation Level "Read Committed", not "Repeatable Read"), and applying a clientside cursor was also kind of weird. But the cursor had been closed at this point, so it's locks should be gone. Legacy applications sometimes act weird, and we should still expect proper database locking.

I built a repro testcase, and sure enough, could reproduce it on SqlServer 2000. There were several shared locks on the query's connection, even once the query's cursor had been closed. In comparison, running the same testcase on SqlServer 2005 resulted to no dagling IS locks, hence no blocking. Well, it's anyway about time to upgrade ten year old database systems. You see, "SQL(Server 2000) WAS broken" in this case.

SqlServer: When querying previously inserted rows within the same transaction is slow...

2011-01-28T20:30:00.019+01:00

... this might have to do with table statistics not having been updated yet!

I was banging my head on this problem today: One and the same query took minutes when run within a transaction, and only seconds when run right after transaction commit. What's special about that case is, that the query reads from a table, that undergoes massive inserting just the statement before (insert-by-select).

As I was testing in an isolated environment without any other connections, it was pretty clear from that start, that this was not a locking issue. I suspected a query optimizer problem from the beginning, and tried "option(recompile)", rephrasing the query (it was pretty complex, three levels of subqueries and the like) and even indexed views, but to no avail.

I was about to dig into a huge profiled transaction batch execution plan (the same query's post-transaction execution plan had looked fine), when I found this posting, and scales fell from my eyes. My immediate guess: this was not an index issue (as the original poster had presumed), this was an "outdated statistics" issue! See, every index has statistics attached, and rebuilding an index automatically updates its statistics as well.

But instead of calling some heavyweight "dbcc dbreindex" as explained in the posting (index rebuild should go to a maintenance plan, but not be placed in an application runtime SQL batch!), I simply invoked "update statistics" on the table in question, right between the insert statement and the query. Voila! Query optimizer is back on track, and now executes the same query about a hundred times faster. I should note though, that "update statistics" also comes with a certain overhead (two to three seconds in our case).

I was aware that SqlServer decides on its own when to update statistics. Seems as if 50,000 new rows and a complex follow-up query isn't necessarily considered sufficient reason for a statistics update, at least not when being part of the same transaction.

Is "Select Distinct" a Code Smell?

2011-01-26T00:10:00.007+01:00

From my experience, "Select distinct" indeed is a code smell (and I am not alone). Why would I fetch something, just to get rid of it again in the next step? There are situations when "distinct" makes sense, but they are rare...

In 9 out of 10 cases when I encountered a "Select distinct", the programmer tried to remove duplicate rows caused by joins along multiple too-many associations (AKA cartesian product). This not only is bad programming style, it might also kill performance.

There is always a way to rephrase such a query in order to avoid join duplicates, e.g. by replacing joins with exist-clauses, using subselects within the select-clause, derived tables in the from-clause, etc.

Hibernate Performance Tuning Tips

2011-01-23T20:52:00.065+01:00

(1) Avoid join duplicates (AKA cartesian products) due to joins along two or more parallel to-many associations; use Exists-subqueries, multiple queries or fetch="subselect" (see (2)) instead - whatever is most appropriate in the specific situation. Join duplicates are already pretty bad in plain SQL, but things get even worse when they occur within Hibernate, because of unnecessary mapping workload and child collections containing duplicates.

(2) Define lazy loading as the preferred association loading strategy, and consider applying fetch="subselect" rather than "select" resp. "batch-size". Configure eager loading only for special associations, but join-fetch selectively on a per-query basis.

(3) In case of read-only services with huge query resultsets, use projections and fetch into flat DTOs (e.g. via AliasToBean-ResultTransformer), instead of loading thousands of mapped objects into the Session.

(4) Take advantage of HQL Bulk Update and Delete statements, as well as Insert-By-Select (supported by HQL as well).

(5) Set FlushMode to "Never" on Queries and Criteria, when flushing is not necessary at this point.

(6) Set ReadOnly to "true" on Queries and Criteria, when objects returned will never be modified.

(7) Consider clearing the Session after flushing, when objects are not longer needed.

(8) Define a suitable value for jdbc.batch_size (resp. adonet.batch_size under NHibernate).

(9) Use Hibernate Query-Cache and Second Level Caching where appropriate (but go sure you are aware of the consequences).

(10) Set hibernate.show_sql to "false" and ensure that Hibernate logging is running at the lowest possible loglevel (also check log4j/log4net root logger configuration).

Special Addendum:

NHibernate's 2.x ProjectionList implementation added considerable runtime-overhead, namely by invoking some reflection API calls for every resultset row. This might slow down queries with large resultsets.

As those reflection API calls were invariant in regard to the resultset rows being iterated over (as a matter of fact, what these calls actually did was to gather resultset column schema information), and I needed an immediate fix and did not have time to wait for any NHibernate patch, I forked my own ProjectionList from NHibernate's code, did all those reflection calls once, and re-used the result from that point on. Now I doubt this is still an issue under NHibernate 3.x (and I think Hibernate under Java was never affected), but if you are using NHibernate 2.x and your profiling tool shows lots of CPU cycles being burnt in ProjectionList, you might want to give this approach a try.

All those advices are Hibernate-specific. For additional database tuning posts on this blog, please read:

Fetching Strategies In Hibernate

2011-01-23T20:16:00.004+01:00

One thing always to remember is that the fetching strategies - as defined within Hibernate mapping configuration - only affect data retrieval via Session.get(), Session.load() and association navigation, plus Criteria queries, as long as they do not override fetch behavior via Criteria.setFetchMode(). HQL in contrast works in a kind of WYSIWYG-manner - whatever fetching is defined by the query will be done, not more, and not less, with mapping configuration being ignored as far as fetching is concerned.

More Hibernate postings:

Will The Real Programmers Please Stand Up?

2010-11-30T22:26:00.002+01:00

Arousing posting on the RethinkDB blog.

In the interest of openness, we’ll post the smoke test that makes us turn away 19 out of 20 candidates within half an hour of a phone conversation (and that’s after screening the resumes). We don’t ask people to code a solution to a complex algorithms problem. We don’t ask to solve tricky puzzle questions. We don’t ask to do complex pointer arithmetic or manipulation. Here is the question that the vast majority of candidates are unable to successfully solve, even in half an hour, even with a lot of nudging in the right direction:

Write a C function that reverses a singly-linked list.

That’s it. We’ve turned away people with incredibly impressive resumes (including kernel developers, compiler designers, and many a Ph.D. candidate) because they were unable to code a solution to this problem in any reasonable amount of time.

Using Database Snapshots For Fast Testautomation DB Restore

2010-11-25T07:02:00.017+01:00

The textbooks basically suggest the following alternatives for restoring databases to their original state after individual automated test runs:

(1) Let each test insert its own testdata on initialization, and when done, revert those changes, as well as everything the test did update.
(2) Embed the whole test run in a transaction, and rollback the transaction at the end.
(3) Restore the database from its original state before each test.

Unfortunately, none of those options worked for us. (1) is not practicable on complex database operations. (2) is not possible if there is no access to the database connection / transaction applied by the tested component. Plus we us a different data access layer for verifying test results (as the same data access layer could camouflage errors), so there is no way to work within the same transaction scope. Option (3) - when taken conventionally (applying traditional backup/restore) - was just too slow; because of several reasons, that very test database is about 150MB of size, and we have over 2000 automated tests to run.

What helped me out was SqlServer's wonderful Database Snapshot feature (available in Developer and Enterprise Editions). Snapshots are readonly-views of the database at a certain point in time. Their copy-on-write approach allows for much faster database restore, as the database now keeps track of which data pages have really changed.

So what I basically did was to create a snapshot at the beginning, then run the test on the original database, and then restore that database from the snapshot. Lightning fast!

Hibernate: Joins Without Associations

2010-10-16T21:37:00.038+02:00

I think I have seen this topic come up dozens of times on the Hibernate discussion forums:

How can I join a table in Hibernate when there is no association in the object model?

And the usual answer is:

Use Theta-style Joins in HQL, and it can't be done in Criteria API.

While this is basically true, I would like to elaborate a little bit more on that.

First of all, we should distinguish between joining and join fetching in Hibernate. Are we using the join in order to fetch a child collection or a single entity within the same query, or not? Because if we are not, there is the chance we don't need a Join at all, and a simple Exists-subquery will do as well, because the joined entity only appears in the From-/Where-clauses, not in the Select-clause. And in a correlated subquery, we can connect pretty much everything to the main query. This has other advantages too, e.g. avoiding unwanted join duplicates and achieving better performance.

Thinking about join-fetching, this might also give us an idea (at least by my interpretation) on why Theta-Joins are required in HQL. They basically look like this:

select foo, bar
from   FooEntity as foo, BarEntity as bar
where  foo.someothercol = 'foo' and 
       foo.somecol = bar.somecol

So we will end up with a resultset that contains a list, with each list element being an object-array of size 2, with obj[0] representing a FooEntity and obj[1] representing a BarEntity .

If we had by contrast an association between FooEntity and BarEntity, we could right something like:

select foo
from   FooEntity foo
       inner join fetch foo.Bars
where  foo.someothercol = 'foo'

Looking at the query alone makes it pretty clear, that there must be a distinct definition on how the association between FooEntity and BarEntity looks like. This definition, usually representing a foreign key relation, will be applied on join fetches, lazy loading and - maybe most importantly - when updating associations. That is what Hibernate associations are all about. FooEntity.Bars will always contain child-rows following that association definition (either all child-rows, or a subset). This is why we can not choose any other arbitrary join criteria in order to fetch other BarEntities into FooEntity.Bars. It just does not make sense.

"But I want to use Criteria API", I can hear you say. OK, I'll come to that in a moment.

First I should mention at that point, that Hibernate also offers nice support for legacy databases in this context. Associations can be defined even when referencing non-primary keys. Unfortunately the NHibernate-guys have not ported that feature yet.

And I want to note that the restriction on associated rows can be narrowed down even more by providing an additional "With"-expression in de Join-Clause or within the Where-clause in HQL.

But i digress. So we have a solution for HQL, and we know why associations are needed for join fetching, and when there is no join fetching but only joining, we might be better off using an Exists subquery anyway. So what about Criteria API?

Fact is: Joining requires an association under Criteria API. Fact is also: There are join criterias that can not be expressed by an association. So the solution I came up with was: Use a database view as a kind of mediator. This view, let's call it FooBar, contains the join criteria expression in it's From/Where-clause, and selects the primary keys of the two associated tables Foo and Bar:

create view FooBar as
select Foo.Id as FooId,
       Bar.Id as BarId
from   Foo
       inner join Bar on foo.somecol = bar.somecol

The view can as well contain further columns, and even a primary key (constructed on-the-fly), or might be materialized for performance reasons.

It is mapped into the Hibernate object model just as any table would be in case of a 1:1 or 1:N relation (and we can then define associations both to Foo and to Bar), or as a N:M mapping table. Hibernate won't even know (nor care), that it's a view. In our object model, we make those associations read-only (as we rather won't try to update the view), simply by providing public access only to an enumerator/iterator, resp. a getter but no setter for a single associated entity.

We can now join-fetch formerly non-associated entities under Criteria API as well, or project some of their columns into a flat resultset (AKA DTO-fetching). And we are able to provide further criteria in our queries' Where-clauses. Works like charm!

From a design viewpoint, we should not pollute our domain model with too many of those view-associations. What we can do is to provide an additional table mapping for those cases, e.g. by subclassing our main entities.

With all those options at hand, there is very little we can NOT do with Joins in Hibernate. And in that case, there is always native SQL support in Hibernate as well.

More Hibernate postings:

Hash Tables Vs. Binary Search Trees

2010-10-15T21:33:00.075+02:00

This is one of my favorite programming interview questions: "What's the difference between Hash Tables and Binary Search Trees?". Let's see...

	Hash Tables	Binary Search Trees
Algorithm	Keys are mapped to values by using hash functions. Hash functions transform the key to a numeric index (usually by calculating an integer value from the key first, and then applying a "modulo arraysize" operation). This index identifies an array element ("bucket"), where all corresponding values reside (values which all stem from keys with equal hash value). The key must still be looked up within the bucket, but as hashing algorithms are supposed to ensure a good hash value distribution, hence small bucket size, this lookup is very fast. The bucket array may have to be resized dynamically when a certain load factor is reached.	Node-based tree data structure, where every node contains one record (key-only, or key and value), and has a left subtree (with smaller keys) and a right subtree (with larger keys); hence keys must be comparable. Self-balancing trees, like red-black-trees, keep their height as small as possible for fast searching and inserting.
Java Implementations	Hashtable HashMap HashSet	TreeMap
.NET Implementations	Hashtable Dictionary HashSet	SortedDictionary SortedList
C++ STL Implementations	std::unordered_map std::unordered_set	std::map
Sorted	No	Yes
Range Search	No	Yes
Runtime Complexity	Inserting: O(1) (normal case), O(N) (worst case, only with bad hash algorithm) Searching: O(1) (normal case), O(N) (worst case, only with bad hash algorithm). More details.	Inserting: O(log(n)) (when balanced) Searching: O(log(n)) (when balanced) More details.
Implementation Notes	Equal objects must produce the same hash code, however unequal objects need not produce distinct hash codes. Equals() implementations must be reflexive, symmetric, transitive, consistent. More details. .NET's ValueType.Equals() and GetHashCode() methods are based on reflection, hence slow; you should provide your own implementation in structs.	CompareTo() implementations must be reflexive, symmetric, transitive, consistent. More details.

The Story Behind Balsamiq Mockups

2010-10-14T21:15:00.004+02:00

Peldi Guilizzoni writes on the creation of Balsamiq Mockups. They have indeed built a wonderful tool, elegant in its design, simple in its usage, yet extremely powerful in what can be achieved by applying it. Our UI folks have been using Balsamiq Mockups since summer 2008, but I just happened to read the history of its creation today. What a truly inspiring story - sometimes one has to take risks to fulfill his dream.

The Best Programmers...

2010-10-02T00:04:00.001+02:00

The best programmers are not marginally better than merely good ones. They are an order-of-magnitude better, measured by whatever standard: conceptual creativity, speed, ingenuity of design, or problem-solving ability.

(Randall E. Stross)

SqlServer: Favor Inline UDFs Over Table Variables

2010-09-03T22:10:00.019+02:00

T-SQL's Table Variables surely do have their benefits, but several drawbacks as well. We used to apply them in the past within larger T-SQL batches, in order to save and reuse some smaller query results, and to avoid code duplication:

declare @MyTable table (...)

insert into @MyTable(...)
select ...

select *
from   @MyTable MyTable
       inner join ...
where  ...

But some databases grew in size far beyond original expectations, and those table variables that used to hold 100 rows, now hold 10.000 rows or more, and spill over into TempDB, hence perform badly.

So I replaced them with Inline User Defined Functions (these are table-valued UDFs, which only consist of a single Select statement):

create function dbo.MyUdf(...)
returns table
as
return (
    select ...
)
go

They can then be inlined like this:

select *
from   (dbo.MyUdf(someparam)) MyVirtualTable
       inner join ...
where  ...

This means high performance (due to the UDF query being inlined, query optimizer can do quite some magic) AND avoiding code duplication. Nice!

SqlServer Standard Edition DOES Support Indexed Views

2010-09-02T22:44:00.017+02:00

Why on earth do nine out of ten articles on SqlServer Indexed Views state, that they are only available in Enterprise and Developer Edition? This is blatantly wrong, and prevents developers from using a great performance tuning feature.

Standard Edition DOES support Indexed Views (and from what I have heard, but not tried: even so does Express Edition). The only difference is, that one has to provide a "with (noexpand)" query hint in order for the Indexed View to be applied in Standard Edition (I checked execution plans to go sure). Plus Enterprise Edition can choose to use an Indexed View for optimization, even if it does not appear in the original query.

I think this whole confusion might be caused by Microsoft's SqlServer feature matrices, which are pretty unclear on that topic, too.

.NET ValueType Equals()/GetHashCode() Performance

2010-09-01T23:53:00.030+02:00

As most people are aware of, .NET Framework's builtin implementations of Equals() and GetHashCode() for ValueTypes are based on Value Equality (as opposed to many ReferenceTypes, where those methods are based on Reference Equality). The problem is, those ValueType.Equals() and GetHashCode() implementations are quite slow for user-defined structs - which is not surprising, as they must apply reflection for obtaining field and property values.

That's why it's better to code your own Equals() and GetHashCode() methods in these cases. The task is pretty much straightforward - just compare those fields and properties, that define equality, in Equals(), and XOR their values (or numeric representations of their values) in GetHashCode():

public override bool Equals(object obj) {
    Nullable<MyType> other = obj as Nullable<MyType>;
    return other != null && 
           this.X == other.Value.X && 
           this.Y == other.Value.Y;
}

public override int GetHashCode() {
    return this.X ^ this.Y;
}

I recently was asked to tune some code, and replaced several nested loops with dictionary lookups - only to find out that this was not as fast as I had expected, due to structs being applied as dictionary keys. Provided my own Equals() and GetHashCode() implementations, and voila - ten-fold speedup!

Developer Candidate Interviews

2010-08-02T15:36:00.004+02:00

Uncle Bob / Robert Martin writes in Developer Certification WTF?:

So what if we… interviewed ... candidate developers? What if we asked them what accomplishments they’d achieved in previous employment. What if we asked them who they used to work with. And… ok, this is even wilder ... what if we called those references and checked them out?

Yeah, and what if we would additionally let them write some code during the interview? I know this sounds crazy to many software development companies, with some notable exceptions (Microsoft, Google and Apple for example - hmmm, maybe this should make us think?)

How to improve Visual Studio Build Performance

2010-05-02T21:26:00.026+02:00

Funny how compile duration is sometimes seen as being carved in stone.

"Building our solution takes 3 minutes, so each time I change a line of code in a base component and run the app, I must wait 3 minutes. Period."

Is this really how software development should work these days?

As projects grow over time, compilation gets slower and slower. Productivity goes south when people are watching the compiler do its work half the time. The decline creeps in and people think this is just normal, until a new developer arrives and throws his hands up in horror.

Also, Visual Studio seems to lack behind other IDEs (most notably Eclipse) on this topic. Eclipse has built-in support for REAL incremental background compilation for ages now (it compiles while the developer types (or during idle time), hence can start the application with no or little delay once he hits <Run>). Visual Studio still doesn't.

So let's see what we can do to speed up things in Visual Studio.

(1) Decrease File IO

Use the same output path for every project in your solution, e.g. "..\Output\Debug" for "Debug" configuration (Project / Properties / Build / Output Path). All assemblies will be created there, so no need to copy files back and forth between project folders.

If there is only one output path, project references can be set to "Copy Local = False" (References / Properties / Copy Local). This furthermore decreases File IO.

I have experienced up to three-fold buildtime improvements by applying that simple configuration change.

In addition, you might want to mitigate virus scanner policies for scans on devenv.exe file access.

(2) Prevent non-dependent project compilation

If a solution contains projects, that the current startup project does not depend upon (e.g. test projects), compiling those projects can be avoided by selecting the option "Only build startup projects and dependencies on Run" under Tools / Options / Projects and Solutions / Build and Run.

(3) Prevent project dependency tree compilation

Visual Studio resp. MSBuild compiles all projects with changes, plus the complete project dependency tree underneath those projects. This is all good and fine, but it also implies that once only one line of code is changed in a commonly used base class, this might cause a complete solution recompilation. But why should we recompile everything, when say only the interior of a single method implementation has changed?

When you know that you are going to change something deep down in your dependency tree, but won't be altering non-private method signatures, here is what you can do:

Option 3.1: Additional solution configurations
Create additional solution configurations for partial builds, and apply identical output paths as defined in (1) (Solution / Properties / Configuration Properties / Configuration Manager / Active Solution Configuration / New).

Solution configurations allow for defining which projects should be built, and which should not; this can lead to much smaller build dependency trees. We might provide configurations for framework class libraries, for functional modules or for data access projects, and so on. And once we know we only are going to change the inner workings of certain components, we switch to the suitable solution configuration first.

Other dependent projects are not going to be compiled if they are not part of the active configuration. What does that imply at runtime? As there are no newly built dependent assemblies, the previously built assemblies (still residing in "..\Output\Debug") are going to be loaded, which is just what we want.

Option 3.2: Manually avoid dependent project recompile
Under Tools / Options / Projects and Solutions / Build and Run, choose "On run, when projects are out of date: Never build". Then, manually build the project only where sourcecode has been edited (Build <projectname> resp. Shift-F6), and finally run the startup project. This too only works when all projects use the same output folder (see (1)).

Option 3.3: Decouple projects, use assembly references instead of project references
Decoupling is not only recommended for software architecture in general, it might also boost compiletime. When application layers are separated (e.g. coding against interfaces, and acquiring implementation classes only at runtime by using factories, dependency injection and the like), code changes other than interface modifications will not lead to dependent project compilations any more.

Also, choose solution structure carefully. You don't want one project containing 80% of all code, and the rest spread on ten other projects. Or consider having Team A work on projectset #1, and Team B working on projectset #2, which depends on what Team A has produced. Provide separate solution files for each Team, let Team A deliver libraries to Team B, and integrate them in Team B's solution by using assembly references (instead of project references).

Take those options with a grain of salt, though. Once signatures of methods invoked by other assemblies are actually altered, the compiler will not be able to pick that up, and runtime errors will occure. In addition, these approaches require stable assembly version numbers.

And I shall call it "Foreign Key Cascade Delete"

2010-03-31T21:28:00.013+02:00

I know it sounds kind of stupid trying to implement cascade delete in T-SQL, when all you have to do is to activate the "cascade delete" option on each foreign key within the association graph. Similarly stupid as implementing referential integrity by hand.

But imagine a case where you generally don't want cascade delete in normal production mode (because no user should be able to delete a parent row which has any child rows still referencing it). And unfortunately there are those rare occasions, when just for the moment, for one single parent row, cascade delete would make sense again, because you really have to get rid of it, only exceptionally.

Of course one could simply activate cascade deletes for this single transaction. But I don't like to change database schemas in production.

The guys at sqlteam.com came up with a different solution: A stored procedure, invoked recursively, traversing over all foreign key relations and deleting child rows bottom-up.

Sounded nice, but did not work in our case. The approach was based on nested subqueries, and from about 8 nested queries on, SqlServer refused to even calculate a query execution plan (I do have sympathy for SqlServer here, I wouldn't enjoy building an execution plan on queries like this either. And even with a plan, the performance would most likely be extremely poor). Plus the script could not handle foreign key relation cycles (I mean cycles on a table-basis, not on a row-basis; the latter can't be solved anyway), a fact that caused a stored procedure "stack overflow" (the limit is 32 recursive calls) on our database.

I then started to adapt the sqlteam.com script in order to make it work, and was about halfway done, when I noticed that commentator GreySky had already provided a solution which was following the same approach I had planned for. He solved the execution plan / performance issue by storing parent primary key values in a temptable, together with information about the current call level, so that the delete criteria would be ONE simple IN-subselect instead of many nested subselects.

So GreySky's script worked fine, with the exception that it still did not handle cycles in foreign key relations. But at this point, with a temptable holding all parent primary keys at hand, this was a simple task. I only had to add an extra column storing the table name that the primary keys belonged to. Thus it was possible to extend the statement's where-criteria to "visit" only those child rows, that had not been marked for deletion before. (Nearly) like Dijkstra's algorithm, you know ;-) Works like a charm!

Composite WPF ("Prism") DelegateCommand.CanExecuteChanged Memory Leak

2009-06-24T18:17:00.005+02:00

I was happy to hear that the Microsoft Patterns and Practices Team has picked up this Composite WPF ("Prism") bug report that I submitted several weeks ago. From the issue description:

When profiling my application I noticed that plenty of EventHandlers had never been deregistered from DelegateCommand's CanExecuteChanged-Event. So those EventHandlers were never been garbage-collected, which caused a severe memory leak.

As registering CanExecuteChanged-EventHandles is done outside application code scope I had expected them to be deregistered automatically as well. At this point I thought this might as well be a ThirdParty WPF control issue, but digging further I read a blog post stating that "WPF expects the ICommand.CanExecuteChanged-Event to apply WeakReferences for EventHandlers". I had a look into RoutedCommand, and noticed it uses WeakReferences as well.

Now this is no showstopper for us as we are very early in the development cycle, and we simply patched DelegateCommand in the meantime by using WeakReferences for its CanExecuteChanged-EventHandlers.

As far as I know the issue has been fixed already, and it is currently going through testing.

NHibernate Criteria-Query: Child Collection Not Populated Despite FetchMode.Join When Criteria Exists For Child Table

2009-06-17T01:51:00.050+02:00

Example taken from NHibernate Bugreport #381:


session.CreateCriteria(typeof(Contact))
   .Add(Expression.Eq("Name", "Bob")
   .SetFetchMode("Emails", FetchMode.Join)
   .CreateCriteria("Emails")
      .Add(Expression.Eq("EmailAddress", 
      "Bob@hotmail.com"))
   .List();

The resulting SQL will include a Join to Emails as expected, the resultset returned by the database is OK, but within the object model Contact.Emails is not going to be populated with data. Which means once Contact.Emails is being accessed in code, lazy loading occurs, which probably was not the coder's intention. This is not the case when

CreateCriteria("Emails")
      .Add(Expression.Eq("EmailAddress", 
      "Bob@hotmail.com"))

is omitted.

The bug report was closed without fix, but contained a comment that "According to Hibernate guys this is correct behavior" and a link to Hibernate bug report #918.

To me that does not sound completely implausible. Hibernate's interpretation ot this criteria tree is that the EMail-criteria is meant to narrow down die Contact parent-row, not the EMail child-rows. HQL queries act just the other way around. Under HQL, additionaly Join-With- or Where-expressions can limit which child rows are loaded into the child collection. I know that HQL - in contrast to Criteria queries - does not apply the fetching strategy defined in the mapping configuration. But with an explicit FetchMode.Join I would have expected Criteria query to do the same.

Apparently under Criteria API this can be worked around by applying an outer Join (which of course is somewhat semantically different):

session.CreateCriteria(typeof(Contact))
   .Add(Expression.Eq("Name", "Bob")
   .CreateCriteria("Emails", JoinType.LeftOuterJoin)
      .Add(Expression.Eq("EmailAddress", 
      "Bob@hotmail.com"))
   .List();

Which seems kind of inconsistent compared to the inner join scenario, and there is even a Hibernate bug report on that.

What I would recommend anyway: If the goal is to narrow the parent data, but then fetch all the children, why not apply an Exists-subquery for narrowing, and in the same query fetch-join all children without further narrowing. Or, if you prefer lazy loading, simply define fetchmode="subselect" on the association.

On a related topic, eagerly joining several child associations has the drawback that the resultset consists of a cartesian product over all children - lots of rows with duplicate data. Let's say there are three child associations A, B and C with 10 rows each for a given parent row, joining all three associations will blast up the resultset to 1 x 10 x 10 x 10 = 1000 rows, when only 1 + 10 + 10 + 10 = 31 rows would be needed.

And while those duplicates will only lead to duplicate references in the object model (and not to duplicate instances), and even those duplicate references can be eliminated again by using Maps or Sets for child collections, these Joins impose a severe performance and memory drawback on the database resp. ADO.NET level.

Of course one could simply issue N single select statements, one for each table, with equaivalent where-clauses. But that implies N database roundtrips as well. Not so good.

The means to avoid this are Hibernate Criteria- and HQL-MultiQueries. Gabriel Schenker has posted a really nice article on MultiQueries with NHibernate.

More Hibernate postings:

My Amazon Listmania Lists

2009-06-16T01:34:00.005+02:00

I had nearly forgotten about them, and was surprised to see that over time nearly 25,000 people have viewed my Amazon Listmania Book Recommendation Lists. Hey I should have received commission! ;-) Subtle hint: The J2EE list is a little bit out-dated by now.

Visualizing TFS Work Items With Treemaps

2009-06-12T20:23:00.058+02:00

Microsoft Team System / Team Foundation Server is a really nice line of products. Besides version control we heavily rely on TFS Work Items for organizing development tasks. One of our largest project is conducted using Scrum, so we are utilizing Conchango's Scrum Plugin for Team System, plus Conchango Taskboard for Sprint planning. Taskboard is better suited than the general purpose Work Item lists and forms that are part of Visual Studio Team Explorer. Let's compare.

Visual Studio Work Item list:

Conchango Taskboard:

From a certain project size on the Visual Studio Work Item lists just won't scale and end up with heaps of data that one can scroll through forever. Don't get me wrong, those lists are sufficient for standard tasks, but they are cumbersome for gaining insight into the project's big picture. And Conchango Taskboard is for Sprint planning and Product Backlog maintenance only. The Conchango Scrum Plugin does have a set of really nice reports though.

So this is where I decided to ramp up my own little solution, which would be based on rendering Work Item data into Treemaps. This week I hacked out a little prototype in my sparetime (working title "Aurora"):

This configuration example provides an overall impression of the sample project's progress: Green tasks are done, blue tasks are not done, and their size represents their complexity. And this is by no means limited to Scrum projects, it works for all kinds of TFS project templates.

Three simple input parameters are all it takes:

Work Item type (e.g. Product Backlog Item, Sprint Backlog Item, Bug)
Work Item attribute defining Treemap size (e.g. Storypoints or any other numeric data, or none in case all items should be rendered with equal size)
Work Item attribute defining Treemap color (e.g. State, Sprint ID, etc)

Plus an optional query string for narrowing the list of items.

This approach possibly allows to visualize about 70% of the reports I could think of. I am still wondering how to implement the missing 30%, as they cannot be covered that easily. For instance I want to group Area Paths with equal prefixes by rendering them with the same color. Or simplify the creation of queries (can't expect everyone to know WIQL by heart). And I don't want to over-complicate things either. Any suggestions regarding those matters are highly welcome! Another requirement is to let the user define color mapping. And item hierarchies are still missing, too (that's the "Tree" in "Treemaps" after all).

By the way, I am using woopef's WPF TreeMap control, thanks a lot for making it publicly available. I am also going to open-source Aurora once it provides basic functionality and reaches a certain level of stability, most likely on CodePlex.

Vanilla Data Access Layer Library 0.6.0 Released

2009-06-08T00:15:00.012+02:00

I have just updated Vanilla DAL on Sourceforge. Release 0.6.0 is still in Alpha state, and comes with improved automatic transaction management. I now chose am approach similar to System.Transactions.TransactionScope. Of course nothing as sophisticated, Vanilla DAL's TransactionScope is a simple IDisposable object that consists of a thread-bound transaction and a does some reference-counting. The transaction will be commited when the last TransactionScope is being disposed, resp. rollbacked in case of any exception during execution:


    using (accessor.CreateTransactionScope()) 
    {
        accessor.Update(new UpdateParameter(northwindDataset.Customers));
        accessor.ExecuteNonQuery(new NonQueryParameter(new Statement("DeleteTestCustomers")));
    }

My main problem was how to find out whether the current call to IDisposable.Dispose() happens within the process of exception unwinding. Several people have recommended Marshal.GetExceptionPointers(), which is the only working solution I have found so far. But I consider this a semi-hack. Any better ideas?

By the way, Luis Ramirez has written a nice article on Codeproject.Com, comparing Vanilla DAL to plain ADO.NET, Microsoft Data Access Application Block and SqlNetFramework.

Code Coverage Analysis With QmCover

2009-06-07T21:35:00.000+02:00

My cousin has developed a graphical code coverage analysis tool for the GNU toolchain. Now I am not really up-to-date regarding the current state of code coverage tools in Unix land (during my old Solaris days all I needed was vi and gcc), but QmCover certainly looks cool!

Implementing Equals And HashCode In Hibernate POCOs/POJOs

2009-05-30T21:15:00.017+02:00

Nearly everybody who uses (N)Hibernate stumbles over this question sooner or later: How to implement Equals() And (Get)HashCode() in Hibernate POCOs/POJOs?

What's the problem? At first sight a naturaly approach seems to apply the object's primary key in Equals And HashCode implementations - simply delegate the calls to Equals and HashCode to the primary key type. But think about it for a second: Most primary keys are AutoIncrement / Identity columns which the database will set when inserting rows. So the primary key's value becomes available only after that - which is too late in most cases. E.g. if there is more than one newly created object (Hibernate state "transient") at one point in time, all these objects would appear to be "equal". Additionally when the newly created object has been added to a child collection of type Map or Set, it will be replaced by the next newly created object that comes along. Undoubtedly this is unwanted behavior.

Moreover this breaks the Equals/HashCode contract, because the HashCode value changes once the primary key has been set by the database. This may lead to all kinds of weird things, e.g. Maps / IDictionaries whose implementations depend on HashCode will not be able to find their objects any more.

So what to do about it? Articles on hibernate.org and the holy book of Hibernate "Java Persistence With Hibernate" recommend applying business keys instead of surrogate keys in Equals / HashCode. Business key values are unique too, plus they are available BEFORE the data has been inserted into the database.

I used to follow that advice in former projects, but at times found it difficult to determine which would be my business keys. Sometimes they just don't exist in the application domain. Plus business key values are supposed to never change. This might be the case today, but change tomorrow (admitted, then those columns probably never were real business keys in the first place).

Plenty of postings point out similar problems. Some people came up with quite sophisticated solutions, e.g. a combined surrogate / business key approach. In case an object is newly created and has not been assigned its database surrogate key yet, it will apply a business key. The business key values and hashcodes will be cached and re-used for the lifetime of the object, even after inserting it into the database table. Hence the Equals / HashCode contract is not broken.

This should work, but seems unnecessarily complicated for me, at least in my scenario. Reading hibernate.org's famous 109-article once again, I figured that my current project falls into the first of the following three categories:

We don't have composite keys, and there is no need to compare objects from different Hibernate sessions. We do have multiple new instances in Sets, and collections must stay intact after saving. Which means I can omit any custom Equals / HashCode implementations altogether, and can go along with the default reference-based implementations (instead of value-based). And it works like a charm!

More Hibernate postings:

Codegeneration With CodeSmith

2009-04-23T23:32:00.022+02:00

It has been a while since I blogged about the sad state of NHibernate tool support in .NET land, compared with Hibernate Tools for Eclipse.

I then discovered Tom DuPont's Codesmith templates for NHibernate, which was a nice starting point. I added some functionality, e.g. support for one-to-one associations, self-referring tables, and of course I had to adapt the output to our needs. By now our develpoment team can create:

NHibernate POCOs
NHibernate mapping configuration
DAOs (yeah I know some folks will complain at this point that with Hibernate there is no need for DAOs - but I mean DAOs in a Spring HibernateDAOSupport sense), DAO-Interfaces
Additionally - and although I generally try to avoid CRUDY Service interfaces - I provided the option for generating entity-based Service implementations and interfaces (but hey, at least they are based on DTOs). We have some situations where this is necessary
DTOs, incl. validation attributes
Mapping code for NHibernate POCO <=> DTO conversion
Spring.NET XML-configuration for DAOs and Services
Unit Test Templates

All of this is generated based on the database schema information plus some SqlServer extended properties on a table- and column-level. Those properties may define name aliases, whether associations are bidirectional or not, which validations to apply, how to do the DTO-assembling, which DB-columns to ignore or which .NET type to map to (if it differs from the default type), stuff like that. They are optional, so without extended properties we end up with a default generation style.

CodeSmith also comes with a nice integration into Visual Studio. So our developers just right-click the code generation project configuration, choose some database tables and which code fragments to generate, and seconds later the generated components are either added or updated within Visual Studio. Some of those fragments are split to partial classes and are then attached as code-behind files, so one can easily overwrite those parts in follow-up code-generations without losing hand-crafted code. Sweet! This is saving my development team tons of time.

Programming Contests

2009-03-31T01:35:00.004+02:00

After last year's Cubido C# Pirates Programming Competition I consider entering the Catalysts Coding Contest this time. That one is about implementation speed, so I was wondering which development environment to use. Most of my programming lately has been in C#, so Visual Studio would be a natural choice. But I consider Eclipse to be a more productive IDE. And there is still the option for a real radical approach - applying LISP for example (oh well, two months left for intensive LISP training then).

Teaching C++

2009-02-28T18:43:00.001+01:00

I recently taught two sessions of C++ programming to a group of mobile computing students. Noticing my C++ knowledge was getting a little bit rusty this kind of refurbishing was really welcome. I was also half sick, and found it hard to concentrate, but at the end all of my students passed their exam, so it didn't seem to have been THAT bad...

Stressed!

2009-01-21T22:22:00.000+01:00

Lots of work currently, and my kids keep me busy on the weekends. So unfortunately no more postings this months.

Programming: Love It Or Leave It?

2008-12-29T20:03:00.015+01:00

Nice post here from Jeff Atwood:

Unless you're fortunate enough to work for a top tier software development company, like Google, Microsoft, or Apple, you've probably experienced first hand the huge skill disparities in your fellow programmers. I'm betting you've also wondered more than once why some of your coworkers can't, well, program. Even if that's what their job description says.

Over the last twenty years, I've worked with far too many programmers who honestly had no business being paid to be a programmer. Now, I'm not talking about your average programmer here. We're all human, and we all make mistakes. I'm talking about the Daily WTF crew. People that actively give programming a bad name, and you, as their coworker, a constant headache.

On the other hand, I would not go as far as Jeff when he states:

So if a programmer ever hints, even in passing, that they might possibly want to exit the field -- they probably should.

Some of the most talented people I know have had thoughts like this once in a while. And the reason was mainly one of two things: Horrible management mistakes or incompetent co-workers (or even worse, both).

Which brings us back to Jeff's original argument: Most programmers don't know how to program. And not only does that screw the associated work of the highly-skilled folks, it kills their motivation too.

If just more bozos would decide to leave the field, well - problem solved! Unfortunately they don't. As they are missing one of the most importat qualifications for this job, namely a healthy bit of self-reflection and the drive to improve, it's quite unlikely most of them have even noticed their deficiencies yet.

I was tempted to start another long rambling article on this whole topic, but instead of repeating myself let me just shortly refer to several of my older postings:

Robin Sharp writes:

A lot of research on programmer productivity shows that the best programmers are up to 20 times more productive than the worst programmers. If the income differential between the best and worst programmers is even 5 times, it means employers are getting incredible value for money hiring the best programmers.

Why then don't companies hire a few very good programmers and leave the rest flipping Big Macs? There is one very good reason: the psychology of managers. Managers simply can't believe that one programmer can be as productive as 3, let alone 5 or even 20 times. Managers believe that productivity is a management issue.

They believe that simply by re-organising their human resources it is they who can gain leaps in productivity, and reap the rewards. But the reality of management, as we all know, is that most projects are late and over budget.

And from Jeff's older articles (Why Can't Programmers.. Program?, Skill Disparities in Programming):

In programming specifically, many studies have shown order of magnitude differences in the quality of the programs written, the sizes of the programs written, and the productivity of the programmers. [...] They studied professional programmers with an average of 7 years' experience and found that the ratio of intitial coding time between the best and worst programmers was about 20:1; the ratio of debugging times over 25:1; of program sizes 5:1; and of program execution speed about 10:1.

Unless you truly enjoy programming you should seek another profession. Be realistic: are you programming to collect a paycheck, or are you programming because you are driven to? I know this sounds harsh, but it's an economic reality-- in an enviroment of global offshoring, the world simply can't support any more highly paid mediocre coders. There are a hundred thousand well educated Indian developers who will do what you do at a fraction of the price, and thousands more coming of age in other third world countries. Blame the Internet if you want, but just being "good with computers" is no longer a free ticket to a high paying tech job.

Finally, here is a very good quote from the comment section on Jeff's site:

Actually most programmers are bad programmers. No really, they are. The problem is that everyone can be a programmer. This isn't true for other jobs. Not everyone can work as a lawyer or doctor for example. However nobody forbids you to work as a programmer, you don't need a special license, there is no mandatory education either. If a company hires you as a coder, no matter if you ever had any education on this matter or not, you work as a coder; period. And this itself is not even bad. Some of the best coders on earth never had any education in computer science, still they are brilliant coders. However, many people had no education and are just horrible coders.

Disappearing Permissions On MS Team Foundation Server 2008

2008-12-03T21:31:00.018+01:00

We had just finished migrating one of our Team System projects from TFS 2005 to a new TFS 2008 machine, when we suddenly noticed that all Team Foundation Server group permissions that had been set on certain project folders via Properties / Security would not show up in the Security tab the next time the Property dialog was opened. The funny thing is - the permissions themselves seemed to work! There was just no way to view, edit or remove them. Same was true for the tf.exe commandline tool ("tf permission ...") - it did not list any existing permissions.

After hours of internet research I finally found this posting. And sure enough, after checking which .NET 3.5 Framework servicepack level was installed on the server (SP1 in our case), and identifying the Team Foundation Server 2008 servicepack level (TFS 2008 RTM, which is not so easy to find out and includes checking the assembly version number of Microsoft.TeamFoundation.Server.dll located in %PROGRAMFILES%\Microsoft Visual Studio 2008 Team Foundation Server\Web Services\Services\Bin (simply google the version number to obtain the release definition)), we had confirmation that it was really that combination of product releaes which caused the group permissions to not show up.

It's highly recommended to have matching .NET Framework and Team System releases installed side-by-side (.NET 3.5 SP1 and TS 2008 SP1 in this example).

Nietzsche On Software Development

2008-12-01T21:30:00.001+01:00

What we do is never understood, but only praised and blamed.

(Friedrich Nietzsche)

Update And Quotes On Programming

2008-09-30T16:48:00.040+02:00

It has been a while since my last posting. We are currently starting a new project and I deeply buried myself into some technology evaluation. So for the meantime, just some random thoughts...

How come there is no usable Database-to-HBM/POCO reverse engineering tool for NHibernate out there? I must have tried five or six of them, some didn't work at all (but one Visual Studio Add-In screwed my IDE altogether), some managed do produce some kind of output, but were missing essential parts like relationship definitions, etc. I am really tempted to write one on my own.

Entity Framework is nice from an API point of view, but lacks some first-class support for disconnected / n-tier scenarios (yes you can detach entities, but it gets kind of complicated when re-attaching them including change state).

Microsoft's new IoC-framework Unity is a big step ahead in comparison to ObjectBuilder. Nice IoC core. Spring.NET is another IoC framework out there, and includes an additional feature stack: support for AOP (e.g. declarative transaction management), NHibernate, Quartz.NET, etc.

I especially like .NET 3.5, Visual Studio 2008 and Team System. Great stuff. WPF is outstanding - what an improvement in productivity!

I am also involved in some old legacy projects. I don't want to go into much detail, but can only reinforce the importance of:

Plan for the future, choose wisely which path to take both in a microcosm of software components as in the realm of the whole project.

Avoid mistakes at the beginning, or pay for them big time later.

People make products. No tools, no process model, no extra hours will counterbalance for poor staffing.

Time and time again I am amazed by the difference in developers' productivity and the variance in code quality. I wonder why our profession is so prone to that dilemma?

These quotes from a stackoverflow.com thread might provide some further insights:

"A computer lets you make more mistakes faster than any other invention in human history, with the possible exceptions of handguns and tequila."

"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult."

"The trouble with programmers is that you can never tell what a programmer is doing until it's too late."

"Most software today is very much like an Egyptian pyramid with millions of bricks piled on top of each other, with no structural integrity, but just done by brute force and thousands of slaves."

"It is practically impossible to teach good programming style to students that have had prior exposure to BASIC. As potential programmers, they are mentally mutilated beyond hope of regeneration."

"Three virtues of a programmer:

Laziness - The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful, and document what you wrote so you don't have to answer so many questions about it. Hence, the first great virtue of a programmer. Also hence, this book. See also impatience and hubris.

Impatience - The anger you feel when the computer is being lazy. This makes you write programs that don't just react to your needs, but actually anticipate them. Or at least pretend to. Hence, the second great virtue of a programmer. See also laziness and hubris.

Hubris - Excessive pride, the sort of thing Zeus zaps you for. Also the quality that makes you write (and maintain) programs that other people won't want to say bad things about. Hence, the third great virtue of a programmer. See also laziness and impatience."

"Any fool can write code that a computer can understand. Good programmers write code that humans can understand."

ADO.NET Entity Framework Vote of No Confidence?

2008-07-04T21:53:00.029+02:00

Several MVPs and other .NET Entity Framework testers (partly from the NHibernate camp) signed an open letter some weeks ago describing their concerns on some design decisions and the current state of what is about to become version 1.0 of the ADO.NET Entity Framework. Here is Microsoft's Tim Mallalieu's response.

Now I won't pretend to be an expert on Entity Framework (I spent two or three days playing around with it). I do have some Hibernate experience though (admitted, in the Java space). So this clearly qualifies me to throw in my 5 cents too, now doesn't it? ;-)

Here is what I think:

INORDINATE FOCUS THE DATA ASPECT OF ENTITIES LEADS TO DEGRADED ENTITY ARCHITECTURES
To be honest, my Hibernate POJOs never contained any logic code, they were plain data holders. This was a design decision that always worked fine for me, even if luminaries like Martin Fowler tend to criticize it. I think the requirement to combine data and logic code in entity objects, although comprehendible from an OOP perspective, is kind of overrated. Anyway one can still place that code in partial classes, what else can we ask for? What might be problematic in some application scenarios is that entity classes are linked to the whole Entity Framework by inheritance and code. They cannot exist "standalone" (e.g. in another tier without EF being around). One alternative are dynamic proxy classes being generated at runtime, but they have some drawbacks as well.

EXCESS CODE NEEDED TO DEAL WITH LACK OF LAZY LOADING
Once again, some people can't live without lazy loading. But lazy loading also might open a can of worms, e.g. in distributed environments like the ones I have to deal with. LazyInitializationException, anybody? I mostly prefer other loading strategies, like "Join Fetch". It boils down whether you (A) want to abstract the fact that there is a SQL database behind the hoods and only want to work on a cloud of objects or (B) the reason you use an ORM is to prevent tight coupling, database dependency and writing loads of mapping, routine SQL and infrastructure code (transaction management, etc) on your own, but still need to be in control of when and how data access actually happens. I am more of kind (B). EntityCollection.Load is just fine for me!

SHARED, CANONICAL MODEL CONTRADICTS SOFTWARE BEST PRACTICES
In this case the "best practice" depends on the type of application one is building. Nobody is being forced to have only "one model to rule them all" - one can as well apply the EF model within a single layer only.

LACK OF PERSISTENCE IGNORANCE CAUSES BUSINESS LOGIC TO BE HARDER TO READ, WRITE, AND MODIFY, CAUSING DEVELOPMENT AND MAINTENANCE COSTS TO INCREASE AT AN EXAGGERATED RATE
While I agree that PI-support is desirable in this area, this is not really a killer argument in my current project scenario. And from what I have heard this issue is going to be addressed in EF V2.

EXCESSIVE MERGE CONFLICTS WITH SOURCE CONTROL IN TEAM ENVIRONMENTS
This is true, that was one of the first things I noticed when I got introduced to the Entity Framework.

But hey guys, this is only version 1.0 of the Entity Framework. To me it still looks promising. Please keep up the good work. And you have a Vote of Confidence from me!

Battleship Game Algorithm Explained (Part 4)

2008-07-01T23:37:00.018+02:00

This is part 4 of the mini-series explaining the inner workings of the Cubido C# Pirates Winner Algorithm.

Please note: This code was not optimized for speed and has plenty of potential of performance improvement, esp. regarding nested looping. The competition rules included an upper time limit of 500ms per shot. This implementation runs at 1ms per shot in average, so there was no point in tuning that esp. as readability might have suffered.

Here is the resulting sourcecode:

using System;
using System.Collections.Generic;
using SoftwareArchitects.Battleships;

namespace Battleships
{

  [Player(Name = "The Privateer")]
  public class ThePrivateer : Player
  {
    private class ShipPositionOption
    {
      private int free;
      private bool done;
      private Direction dir;
      private Coordinates nextCoord;

      public ShipPositionOption(Direction posDir)
      {
        dir = posDir;
        free = 0;
        done = false;
        nextCoord = null;
      }

      public void Reset()
      {
        free = 0;
        done = false;
        nextCoord = null;
      }

      public void MarkInvalid()
      {
        free = 0;
        done = true;
        nextCoord = null;
      }

      public Coordinates NextCoord
      {
        get { return nextCoord; }
        set { nextCoord = value; }
      }

      public int Free
      {
        get { return free; }
        set { free = value; }
      }

      public bool Done
      {
        get { return done; }
        set { done = value; }
      }

      public Direction Dir
      {
        get { return dir; }
      }

    }

    private class Direction
    {
      private Orientation orient;
      private int dX;
      private int dY;

      public Direction(Orientation or, int x, int y)
      {
        orient = or;
        dX = x;
        dY = y;
      }

      public Orientation Orient
      {
        get { return orient; }
      }

      public int DX
      {
        get { return dX; }
      }

      public int DY
      {
        get { return dY; }
      }
    }

    private class SinkTryResult
    {
      private Direction dir;
      private Coordinates sinkCoord;

      public SinkTryResult(Coordinates sinkCoordParam, Direction dirParam)
      {
        sinkCoord = sinkCoordParam;
        dir = dirParam;
      }

      public Coordinates SinkCoord
      {
        get { return sinkCoord; }
        set { sinkCoord = value; }
      }

      public Direction Dir
      {
        get { return dir; }
        set { dir = value; }
      }
    }

    private enum Orientation : int
    {
      EAST = 0,
      WEST = 1,
      NORTH = 2,
      SOUTH = 3
    }

    private enum Mode { ScanMode, SinkMode };

    private static readonly IDictionary<Orientation, Direction> DIRECTIONS = new Dictionary<Orientation, Direction>();
    private const int FIELD_DIM_X = 10;
    private const int FIELD_DIM_Y = 10;

    private const int EAST_IDX = (int)Orientation.EAST;
    private const int WEST_IDX = (int)Orientation.WEST;
    private const int SOUTH_IDX = (int)Orientation.SOUTH;
    private const int NORTH_IDX = (int)Orientation.NORTH;

    private Random rnd = new Random();
    private Mode mode = Mode.ScanMode;
    private Coordinates sinkCoord = null;
    private bool[,] impossibleCoords = new bool[10, 10];

    private int seekedShipSize = 5;
    private int hitCount = 0;

    private int[] sinksRequired = new int[] { 0, 0, 1, 2, 1, 1 };

    static ThePrivateer()
    {
      DIRECTIONS[Orientation.EAST] = new Direction(Orientation.EAST, 1, 0);
      DIRECTIONS[Orientation.WEST] = new Direction(Orientation.WEST, -1, 0);
      DIRECTIONS[Orientation.SOUTH] = new Direction(Orientation.SOUTH, 0, 1);
      DIRECTIONS[Orientation.NORTH] = new Direction(Orientation.NORTH, 0, -1);
    }

    public ThePrivateer()
    {
    }

    public override void Move(PlayingField playingField)
    {
      if (mode == Mode.ScanMode)
      {
        Coordinates coord = GetNextScanCoord(playingField);
        if (playingField.Fire(coord.X, coord.Y) == State.HitShip)
        {
          sinkCoord = coord;
          mode = Mode.SinkMode;
          hitCount = 1;
        }
      }
      else if (mode == Mode.SinkMode)
      {
        SinkTryResult sinkTry = GetNextSinkTry(playingField);

        State res = playingField.Fire(sinkTry.SinkCoord.X, sinkTry.SinkCoord.Y);

        if (res == State.HitShip || res == State.SunkShip)
        {
          sinkCoord = sinkTry.SinkCoord;
          hitCount++;

          if (res == State.SunkShip)
          {

            sinksRequired[hitCount] = sinksRequired[hitCount] - 1;

            int sunkX = sinkCoord.X;
            int sunkY = sinkCoord.Y;

            for (int i = 0; i < hitCount; i++)
            {
              SetImpossibleCoord(sunkX - 1, sunkY - 1);
              SetImpossibleCoord(sunkX - 1, sunkY);
              SetImpossibleCoord(sunkX - 1, sunkY + 1);
              SetImpossibleCoord(sunkX, sunkY - 1);
              SetImpossibleCoord(sunkX, sunkY);
              SetImpossibleCoord(sunkX, sunkY + 1);
              SetImpossibleCoord(sunkX + 1, sunkY - 1);
              SetImpossibleCoord(sunkX + 1, sunkY);
              SetImpossibleCoord(sunkX + 1, sunkY + 1);

              sunkX = sunkX - sinkTry.Dir.DX;
              sunkY = sunkY - sinkTry.Dir.DY;
            }

            while (seekedShipSize > 0 && sinksRequired[seekedShipSize] == 0)
            {
              seekedShipSize--;
            }

            hitCount = 0;
            sinkCoord = null;
            mode = Mode.ScanMode;
          }
        }
      }
    }

    private ShipPositionOption[] CreateShipPositionOptions()
    {
      ShipPositionOption[] options = new ShipPositionOption[4];
      options[EAST_IDX] = new ShipPositionOption(DIRECTIONS[Orientation.EAST]);
      options[WEST_IDX] = new ShipPositionOption(DIRECTIONS[Orientation.WEST]);
      options[SOUTH_IDX] = new ShipPositionOption(DIRECTIONS[Orientation.SOUTH]);
      options[NORTH_IDX] = new ShipPositionOption(DIRECTIONS[Orientation.NORTH]);
      return options;
    }

    private Coordinates GetNextScanCoord(PlayingField playingField)
    {
      ShipPositionOption[] options = CreateShipPositionOptions();
      Coordinates bestCoord = null;
      int bestCnt = 0;
      int bestNeighborFact = 0;
      bool bestMatching = false;

      for (int x = 0; x < FIELD_DIM_X; x++)
      {
        for (int y = 0; y < FIELD_DIM_Y; y++)
        {
          if (IsUnknown(playingField, x, y))
          {
            int cnt = 0;
            int neighborFact = 0;

            for (int shipSize = 0; shipSize < sinksRequired.Length; shipSize++)
            {
              if (sinksRequired[shipSize] > 0)
              {
                for (int optIdx = 0; optIdx < options.Length; optIdx++)
                {
                  ShipPositionOption option = options[optIdx];
                  option.Reset();
                  for (int d = 1; d < shipSize && !option.Done; d++)
                  {
                    if (IsUnknown(playingField, x + d * option.Dir.DX, y + d * option.Dir.DY))
                    {
                      option.Free = option.Free + 1;
                    }
                    else
                    {
                      option.Done = true;
                    }

                  }

                }
                int cntHor = Math.Max(0, options[EAST_IDX].Free + options[WEST_IDX].Free + 2 - shipSize);
                int cntVer = Math.Max(0, options[SOUTH_IDX].Free + options[NORTH_IDX].Free + 2 - shipSize);
                cnt += cntHor + cntVer;

              }

            }

            foreach (Direction dir1 in DIRECTIONS.Values)
            {
              int x1 = x + dir1.DX;
              int y1 = y + dir1.DY;
              if (IsUnknown(playingField, x1, y1))
              {
                foreach (Direction dir2 in DIRECTIONS.Values)
                {
                  int x2 = x1 + dir2.DX;
                  int y2 = y1 + dir2.DY;
                  if (x2 != x || y2 != y)
                  {
                    if (IsWaterOrImpossible(playingField, x2, y2))
                    {
                      neighborFact++;
                    }
                  }
                }
              }
            }

            bool matching = MatchesScanPattern(x, y);
            if (cnt > bestCnt || (cnt == bestCnt && (neighborFact > bestNeighborFact || (neighborFact == bestNeighborFact && ((!bestMatching && matching) || (bestMatching == matching && rnd.Next(2) == 1))))))
            {
              bestCnt = cnt;
              bestNeighborFact = neighborFact;
              bestMatching = matching;
              bestCoord = new Coordinates(x, y);
            }
          }
        }
      }
      return bestCoord;
    }

    private SinkTryResult GetNextSinkTry(PlayingField playingField)
    {
      ShipPositionOption[] options = CreateShipPositionOptions();

      for (int i = 1; i < seekedShipSize && (!options[EAST_IDX].Done || !options[WEST_IDX].Done || !options[SOUTH_IDX].Done || !options[NORTH_IDX].Done); i++)
      {
        for (int optIdx = 0; optIdx < options.Length; optIdx++)
        {
          ShipPositionOption option = options[optIdx];
          if (!option.Done)
          {
            Coordinates nextCoord = new Coordinates(sinkCoord.X + i * option.Dir.DX, sinkCoord.Y + i * option.Dir.DY);
            if (!IsValidCoord(nextCoord) || impossibleCoords[nextCoord.X, nextCoord.Y])
            {
              option.Done = true;
            }
            else
            {
              State state = playingField.GetState(nextCoord.X, nextCoord.Y);
              if (state == State.Unknown)
              {
                if (option.NextCoord == null)
                {
                  option.NextCoord = nextCoord;
                }
                option.Free = option.Free + 1;
              }
              else if (state == State.HitShip)
              {
                if (option.Dir.Orient == Orientation.EAST || option.Dir.Orient == Orientation.WEST)
                {
                  options[SOUTH_IDX].MarkInvalid();
                  options[NORTH_IDX].MarkInvalid();
                }
                else
                {
                  options[EAST_IDX].MarkInvalid();
                  options[WEST_IDX].MarkInvalid();
                }
              }
              else
              {
                option.Done = true;
              }
            }
          }
        }
      }

      ShipPositionOption bestOpt;
      ShipPositionOption option1;
      ShipPositionOption option2;

      int horFree = options[EAST_IDX].Free + options[WEST_IDX].Free;
      int verFree = options[SOUTH_IDX].Free + options[NORTH_IDX].Free;
      int minHorFree = Math.Min(options[EAST_IDX].Free, options[WEST_IDX].Free);
      int minVerFree = Math.Min(options[SOUTH_IDX].Free, options[NORTH_IDX].Free);

      if (horFree > verFree || (horFree == verFree && (minHorFree > minVerFree || (minHorFree == minVerFree && rnd.Next(2) == 1))))
      {
        option1 = options[EAST_IDX];
        option2 = options[WEST_IDX];
      }
      else
      {
        option1 = options[SOUTH_IDX];
        option2 = options[NORTH_IDX];
      }

      if (option1.Free > option2.Free)
      {
        bestOpt = option1;
      }
      else if (option2.Free > option1.Free)
      {
        bestOpt = option2;
      }
      else
      {
        bestOpt = rnd.Next(2) == 1 ? option1 : option2;
      }

      return new SinkTryResult(bestOpt.NextCoord, bestOpt.Dir);
    }

    private bool IsUnknown(PlayingField playingField, int x, int y)
    {
      return IsValidCoord(x, y) && !impossibleCoords[x, y] && playingField.GetState(x, y) == State.Unknown;
    }

    private bool IsWater(PlayingField playingField, int x, int y)
    {
      return IsValidCoord(x, y) && playingField.GetState(x, y) == State.Water;
    }

    private bool IsWaterOrImpossible(PlayingField playingField, int x, int y)
    {
      return IsValidCoord(x, y) && (impossibleCoords[x, y] || playingField.GetState(x, y) == State.Water);
    }

    private bool MatchesScanPattern(Coordinates coord)
    {
      return coord != null && MatchesScanPattern(coord.X, coord.Y);
    }

    private bool MatchesScanPattern(int x, int y)
    {
      return (x + y) % 2 == 0;
    }

    private bool IsValidCoord(Coordinates coord)
    {
      return IsValidCoord(coord.X, coord.Y);
    }

    private bool IsValidCoord(int x, int y)
    {
      return x >= 0 && x < FIELD_DIM_X && y >= 0 && y < FIELD_DIM_Y;
    }

    private void SetImpossibleCoord(int x, int y)
    {
      if (IsValidCoord(x, y))
      {
        impossibleCoords[x, y] = true;
      }
    }

  }
}

Previous Postings:

Where Jeff Atwood Is Wrong On Stackoverflow.Com

2008-06-21T23:48:00.083+02:00

I have been reading Jeff Atwood's Coding Horror Blog for many years and have high regards for it, but I am starting to get a little disillusioned by some of his later writings and also several comments on the Stackoverflow.Com Podcast. Anyway, considering all that interesting and entertaining stuff he has been blogging over time I suppose it's also OK if I can't identify with his opinion once in a while.

There is this blog entry on the Model/View/Controller design pattern, and Jeff and Joel Spolsky also discussed that during their podcast. I consider the "Model=HTML, View=CSS and Controller=Browser"-explanation misleading (I would not go so far to call it "completely wrong", it's just not a great analogy). HTML and CSS are declarative languages, not Models/Views in an OOP sense. The browser application's code and data structures are the building blocks for Model, View and Controller. So the browser IS Model, View and Controller.

By definition, the Model represents application state, the View defines how Model data is rendered into the UI and the Controller takes user input and changes the Model accordingly, resp. invokes application functionality.

One could favorably argue that HTML is a source of data for the browser's Model, and that CSS also affects the way the View does the rendering. But Model and View themselves are something else, they are browser-internal software components that represent web content (Model) resp. are responsible for visualizing that web content on a UI platform (View). Remember the View serves as an Observer on Model data - that just doesn't fit with CSS and HTML. And that's why I think the comparison is a little bit far-fetched.

Take a CAD application as another example of a Model that contains some kind of graphical artefacts simply because that's the nature of the application (just like web content in case of a browser). The CAD Model might consist of shapes and polygon definitions, and the CAD View then renders that into a two-dimensional UI.

Here is a diagram taken from the J2EE Architecture blueprints that pretty much sums up things (yeah I know this article refers to N-tier architecture, but the diagram is valid for all MVC-secnarios).

Jeff Atwood noted that he could not find any MVC documentation that really pleased him and that he could do better (hey, why not take the GoF book - MVC is described in the introduction chapter). Sorry to say that in this case he didn't comply with his own high standards.

Then there was a remark about SqlServer as being pretty much standalone self-tuning. As much as I appreciate SqlServer, it's really not self-tuning. Don't get me started on all that query optimizer issues I have been through (admitted this has gotten better in the latest versions). It's also a must to define a customized maintenance plan on SqlServer. But maintenance plans are kind of cumbersome or even impossible on desktop/express engines, and on those engines self-tuning would be essential (as there is no database admin to take care of such things). If I had to name a self-tuning database, it would be Sybase SQLAnywhere Personal Server - no big surprise as it has been designed for smaller-than-enterprise usage.

Finally Jeff stated that he sees no difference between computer science and software engineering. I'd like to quote Steve McConnell at this point (and by the way, coincidently the term "Coding Horror" descends from Steve McConnell's book "Code Complete"):

Scientists learn what is true, how to test hypotheses, and how to extend knowledge in their field. Engineers learn what is true, what is useful, and how to apply well-understood knowledge to solve practical problems.

Scientists must keep up to date with the latest research. Engineers must be familiar with knowledge that has already proven to be reliable and effective. If you are doing science, you can afford to be narrow and specialized. If you are doing engineering, you need a broad understanding of all the factors that affect the product you are designing.

Scientists don't have to be regulated because they are chiefly accountable to other scientists. Engineers do have to be regulated because they are chiefly accountable to the public.

An undergraduate science education prepares students to continue their studies. An undergraduate engineering education prepares students to enter the workforce immediately after completing their studies.

But hey, just as Jeff remarked repeatedly, it's also fine to disagree every now and then.

Battleship Game Algorithm Explained (Part 3)

2008-06-19T01:44:00.032+02:00

This is part 3 of the mini-series explaining the inner workings of the Cubido C# Pirates Winner Algorithm. We have talked about how to find the cell best suited for a trial shot. Now let's see what we can do to improve the accuracy of sinking shots, that is the follow-up shots once a trial shot hit on a ship.

So what needs to be decided at this point is in which direction to continue shooting after the first hit. The stakes are high - after all it's possible to produce three unnecessary misses until we know for sure were the ship is really located.

Let's have a look at the following scenario, and suppose that there is one ship left of size 4, one ship of size 3 and one ship of size 2:

It somehow seems tempting to select the eastern neighbor cell for the next sink shot, because of the large unknown space in that direction. But as it turns out that would be a mistake.

A better approach is to calculate the probabilities for the ships that are still left on the battlefield to be located on each unknown neighbor cell (of course considering the fact where we have hit the ship already). This is very similar to our strategy when placing trial shots.

The ship of size 2 can be located in either of the four directions, so this does not really help.

There is 1 possibility for the ship of size 3 to be located on the western neighbor cell, 2 possibilities on the northern neighbor cell, 2 possibilities on the eastern neighbor cell and 2 possibilities on the southern neighbor cell. So the western cell is out of the game.

Regarding the ship of size 4, there is 1 possibility to the west, 2 to the north, 2 to the east and 3 to the south. The southern neighbor is our target of choice.

In case we miss we'll give it another try the next round by applying the same mechanism. After all the probabilities have changed by then (the ship is then more likely to be positioned horizontally).

Once we hit a second ship cell we can be sure about its orientation, but may still not know about the exact location. So we continue to calculate probabilities for neighbor cells on each side until the ship has been sunk. After that we store the information about which ship size just has been sunk, mark all cells surrounding the ship with State Impossible, and switch back to trialshot mode (as explained in part 1).

In part 4 we are going to put all of that together, and have a look at the resulting sourcecode.

Previous Postings:

Followup Postings:

Battleship Game Algorithm Explained (Part 4)

Battleship Game Algorithm Explained (Part 2)

2008-06-18T18:13:00.038+02:00

This is part 2 of the mini-series explaining the inner workings of the Cubido C# Pirates Winner Algorithm. We previously examined how to basically choose the next battlefield cell to shoot at, that is by calculating each cell's probability to contain a ship. If there is more than one top-ranked cell, we need to decide among those cells.

So far we have done our best to hit a ship. But let's face it, it's more likely to hit the water. How can we benefit from a missed shot as well?

By applying a checkerboard-like shooting pattern we will find each ship by only having to hit only 50% of the cells (and that's a worst case scenario). E.g. there is no way for a ship of size 2 to hide anywhere here:

Great, we now have an additional selection criteria for choosing the next target cell: the checkerboard pattern.

Still there might be plenty of cells left with equal ship location probabilities which also fit into the checkerboard. Which other value can be gained from a missed shot? The value of learning something about unknown neighbor cells! When an unknown cell A has a water cell neighbor B (or an impossible cell neigbor, which is a cell that adjoins a sunk ship), this clearly limits the options for a ship to be located on cell A - because cell B is out of the game for a joined ship location.

If cell A is surrounded by two water/impossible cells the probabilities decrease further, and even more of course with three water/impossible neighbors (just one direction left to place a ship). Finally if we happen to shoot at the last remaining unknown neighbor cell of A (and hit water only), we can abandon that cell A for future consideration all together.

So what we are going to do is to count the water and impossible neighbors for each of our potential target's unknown neighbors (our target's unknown neighbors' water/impossible neighbors so to speak), and prefer the one target with the highest score.

This cell here might be a good choice for a shot (neighbor score is 7, 2 water neighbors for the western unknown neighbor + 2 water/impossible neighbors for the northern unknown neighbor + 0 for the eastern impossible neighbor + 3 water/impossible neighbors for the southern unknown neighbor):

In the following case that cell would score even higher (neighbor score is 9, 3 water neighbors for the western unknown neighbor + 3 water/impossible neighbors for the northern unknown neighbor + 0 for the eastern impossible neighbor + 3 water/impossible neighbors for the southern unknown neighbor):

So the neigbor score becomes our second target selection criteria, right after to the ship location probability.

If there are cells with equal ship location probability and equal neighbor score, we choose one that fits the checkerboard (by the way, applying the neighbor score almost naturally leads to a checkerboard as well, so this is just to go sure for special cases, e.g. a more or less empty battlefield). If more than one of the top-ranked cells fits the checkerboard, we'll randomly select any of them.

In part 3 we will discuss how to proceed once the first cell of a ship has been hit. There is a lot of potential for tuning as well.

Previous Postings:

Battleship Game Algorithm Explained (Part 1)

Followup Postings:

Battleship Game Algorithm Explained (Part 1)

2008-06-17T16:22:00.026+02:00

Several people have approached me about this, so here is the first article in a mini-series describing the Cubido C# Pirates Winner Algorithm. It's final score was 38.19 shots (average shot count on 10,000 battleship games). 17 contentors scored under 39.00 shots, so this clearly was a very close match. 83 algorithms took part overall.

Here is the spec: The battlefield consists of 10x10 cells. Ships are laid out either horizontally or vertically, with a margin of at least one cell between each ship. There is one ship of size 5, one of size 4, two of size 3 and one of size 2. Battleship algorithms are being called back on each round and must return the cell to shoot at next. The state of each cell can be queried through the C# Pirates API. Those states are: Unknown (cell has not been shot at yet), Water (cell has been shot at before and contains no ship) and Ship (cell has been shot at before and contains a ship).

Unknown cell

Water cell

Ship cell

So on each round of the game there is a decision to be made: at which cell to shoot at next. It soon becomes clear that we need two different modes of operation: Trialshot mode and Sinkshot mode. Trial shots occure while scanning the battlefield for ships. Sink shots happen once a ship has been hit in order to continue sinking it. We are going to talk about trial shots in part 1 and part 2, while sink shots will be covered in part 3.

In trialshot mode it's clearly a good idea to choose the cell that is most likely to be occupied by a ship. The probability of a ship residing at a certain cell can be calculated by scanning for unknown neighbor cells in each direction.

Let's have a look at an example. Say that there are the following ships left to be sunk: One of size 5, one of size 3 and one of size 2 (we know that because we kept track on which ships we sank already), and we are about to examine the cell marked with a shot impact.

There are following possibilities for this cell to contain a ship: 1 option for a ship of size 5 (filling out the whole horizontal space), 1 option for a ship of size 3 (to the east), and 2 options for a ship of size 2 (north and east). So this gives us 4 options for ships to be placed on that very cell.

What about the next unknown cell?

Again there is 1 option for a ship of size 5 (horizontally), 2 options for a ship of size 3 (both horizontally, one to the east, the other one with the proposed target cell located in the center of the ship), and 3 options for a ship of size 2 (east, west and south). There is no way for a ship to fit to the north although the northern neighbor has not been shot at yet, as it's itself restricted by the sunken ship next to it (one cell margin between ships). Still, 6 options beat 4 options, so the algorithm would choose the second cell among those two cells.

The fact that there is a margin of one cell between ships creates another cell state that we must consider: the cell has not been shot at, but is next to a ship, hence cannot be part of a ship itself. Let's call that state Impossible, and keep this information in an internal data structure between callbacks, because the C# Pirates API does not support that state, and querying all neighbors of state Unknown each time is kind of cumbersome.

So what we basically do is to iterate over all cells of state Unknown, calculate their probabilities to contain a ship, and select the highest-rated option. The question is what to do when there are several target cells with the same probability (which by the way is quite likely to happen). Making a good decision at that point is crucial and will differentiate our algorithm from competing ones which apply the same basic strategy. We will discuss that in part 2.

Followup Postings:

Victory!

2008-05-23T15:28:00.019+02:00

As luck would have it I won the Cubido C# Pirates Programming Competition for the best Battleship Game Algorithm last week, and took home an XBox 360 Premium Edition as first prize.

In general my algorithm pretty much stayed the same since I last blogged about it a while ago, but some fine-tuning brought the average winning shot count down to about 38 (compared to 39 shots achieved by the initial version) - that is the number of shots to sink five ships of size 2, 3 (2 times), 4 and 5 on a 10x10 battlefield.

Tuning was absolutely necessary as 17 algorithms scored under 39 shots at the end (around 80 players took part over all). I had submitted several slightly different versions, hence the victory was manifold. My best algorithm variation attained 38.19 shots, with the closest competitor finishing at 38.33.

The final competition was a public event (a big thank you goes to the kind folks at Cubido), and many contestants dropped by at the Cubido Headquarters and watched the progress on a large scoreboard (which was also available online). The battle lasted nearly four hours. 10,000 games had been played eventually (each algorithm was required to finish 10,000 identical battlefields). With results so close this large number was a must to avoid any "accidental winner", a winner that would have benefited by certain random number distributions (as a matter of fact during the preliminary rounds, which only consisted of 50 games each, my algorithms scored at between 34.5 and 40.5 shots, so that's quite a spread - a spread that was caused both by the random distribution of ships on the battlefield, as well as the algorithm's internal randomizer, which is applied when choices are evaluated as being equal otherwise).

As I have promised before, I am going to talk about the ideas behind the algorithm and post the complete sourcecode over the next few days.

Battleship Game Algorithm Competition

2008-04-07T21:25:00.024+02:00

I am one of the C# Pirates taking part in Cubido's Battleship Game Algorithm Competition. Dozens of computer programs battling it out against each other - in batch mode on the server (current ranking), as well as viewable in any Silverlight-enabled web browser.

Last week I attended Microsoft Austria's Big Days 2008 and found out about the contest - two nights of coding later, and Captain Alex (namesake of my older son) reached the top position. Captain Ahab, the old swashbuckler, is still a tough competitor though.

And by the way kudos to Cubido. What a great idea! This is the kind of stuff that excites most software developers. And their Silverlight client does have really cool visuals.

I am going to blog about the Captain Alex algorithm once the contest has ended (it is quite simple).

DrawCircle Performance Tuning

2008-03-14T23:07:00.138+01:00

A while ago a friend of mine asked me an interesting programming question: "How would you implement a DrawCircle() function?". "I'd loop and invoke Sin() and Cos() in order to obtain coordinates" I answered. "OK", he replied, "and how would you make it fast? There should be something like an incremental way, maybe without the need for floating point operations at all.".

This kept me thinking. So why not give it a try right here and now! Let's start with something like this (C#, .NET 2.0 - using a managed environment is OK, as .NET is guaranteed to JIT this code immediately anyway):

private static void DrawCircle1(int radius)
{
  double angle = 0;

  do
  {
    double rad = angle * Math.PI / 180.0;
    double x = Math.Cos(rad);
    double y = Math.Sin(rad);

    DrawPixel(x, y);

    angle++;
  } while (angle < 360);
}

Sure enough, a circle appears on the screen. It looks weird - the line is dotted, not solid. Right, incrementing the angle by 1 each time might imply advancing too fast. An increment value better suited is the angle between two pixels on the circle. Something like this:

private static void DrawCircle2(int radius)
{
  double angle = 0;

  double step = Math.Asin(1.0 / radius) * 
                180.0 / Math.PI;

  do
  {
    double rad = angle * Math.PI / 180.0;
    double x = Math.Cos(rad) * radius;
    double y = Math.Sin(rad) * radius;

    DrawPixel(x, y);

    angle += step;
  } while (angle < 360);
}

This works, so that's something to start with. To get an idea about its speed, let's ignore the call to DrawPixel() for a second (and assume that we have a lightning-fast DrawPixel() implementation at that point - and by lightning-fast I mean inlined native code that just does something like an indexed array lookup, a little pointer arithmetic, and one or two bitwise operations on a bitmap bytearray, not slow stuff like SetPixel(), Bitmap.SetPixel() or BufferedImage.setRGB()). Oh yeah, and no synchronous screen refreshing ;-)

Some profiling tells us that his function can calculate the pixels for 160,000 circles (of radius 100) per second (all numbers refer to release mode).

Even with floating point processing units, floating point operations can be expensive in a certain sense - that is, avoiding them altogether, could still cause a measurable improvement. Sure we can build up a large lookup array with pre-calculates Cos- and Sin-values, and interpolate in between them. I used to do that on other occasions, e.g. in my 3D chat project. But here there is no way of telling in advance how precise the lookup array needs to be, and with increasing precision the table will grow in size too.

Another way of speeding up the algorithm is to take advantage of the fact that a circle is symmetric. Wouldn't it be enough to calculate one quarter of the circle, and then just mirror it on the x- and y-axis. But wait - what about calculating only one eighth of the circle, and mirroring it on the diagonal, too? No sooner said than done:

private static void DrawCircle3(int radius)
{
  double angle = 0;

  double step = Math.Asin(1.0 / radius) * 
                180.0 / Math.PI;

  do
  {
    double rad = angle * Math.PI / 180.0;
    double x = Math.Cos(rad) * radius;
    double y = Math.Sin(rad) * radius;

    DrawPixel(x, y);
    DrawPixel(x, -y);
    DrawPixel(-x, y);
    DrawPixel(-x, -y);
    DrawPixel(y, x);
    DrawPixel(y, -x);
    DrawPixel(-y, x);
    DrawPixel(-y, -x);

    angle += step;
  } while (angle <= 45);
}

This gives us something close to the expected 8-fold speedup (1,200,000 circle calculations per second to be exact). Still it seems like a dead end. We need a different approach.

Let's search for a way to draw a circle without the need of invoking Sin() and Cos(). What does a circle function look like? It's

x^2 + y^2 = 1

so:

y = sqrt(1 - x^2)

Let's also take advantage of the fact that we know that x is always incremented while looping. How do we know that? Because we just calculate 1/8th of the circle. We are starting at coordinate (x = 0, y = radius), hence on each step will either move right only, or move right and down.

private static void DrawCircle4(int radius)
{
  double x = 0;
  double y = radius;

  do
  {
    DrawPixel(x, y);
    DrawPixel(x, -y);
    DrawPixel(-x, y);
    DrawPixel(-x, -y);
    DrawPixel(y, x);
    DrawPixel(y, -x);
    DrawPixel(-y, x);
    DrawPixel(-y, -x);

    x++;
    //cast to get rid of decimal places
    y = (int)((Math.Sqrt(1.0 - 
        Math.Pow(x / radius, 2)) * radius));
  } while (x <= y);
}

100,000 circles per second, what a disappointment. I did some quick check, it seems that Math.Pow() does not have dedicated FPU-support, so this might be a reason.

Still this might open a new perspective. Having a look at two neighboring pixels on the circle, what does make them neighbors? Their x- and y-coordinates differ by 1, either x does, or y, or both. Starting at a well-known pixel, e.g. (x = 0, y = radius) we could move along the circle just be deciding whether we increment x and leave y as it is, or decrement y and leave x as it is, or increment x and decrement y.

Once more the circle function:

x^2 + y^2 = 1

which equates to:

x^2 + y^2 - 1 = 0

Drawing a perfect circle on a discrete field of pixels, there is always a deviation, an error. An error e that can be expressed like this:

x^2 + y^2 - 1 = e

The pixel we draw is either inside the perfect circle or outside. When we have moved outside the perfect circle, we have to pull back to the inside, and vice versa. In each iteration, we must keep e as close to 0 as possible.

How does e change on each iteration? Let's assume we increment x and decrement y, so the next pixel is at (x + 1, y - 1):

e = x^2 + y^2 - 1
nextE = (x + 1)^2 + (y - 1)^2 - 1 
      = (x^2 + 2x + 1) + (y^2 - 2y + 1) - 1

The difference between nextE and e is influenced by the following factors:

dx = (x^2 + 2x + 1) - x^2 = 2x + 1
dy = (y^2 - 2y + 1) - y^2 = -2y + 1

This is how nextE looks like when x is incremented and y is decremented:

nextE = e + dx + dy

If we decide only to increment x and leave y as it is, nextE would be:

nextE = e + dx

And in case we only decrement y, nextE is:

nextE = e + dy

At this point we just have to figure out whether e stays smallest by moving along the x-axis only, or the y-axis only, or both.

Wow, no need to invoke Sin(), Cos(), Sqrt() or Pow(), no need for floating point functions or values, we can do it all with integers only. Note that we could even avoid the multiplications by a factor of 2 simply by writing (x + x) and (- y - y), but compilers normally replace that by a left-shift anyway. Also, we do not invoke Math.Abs(), but get rid of minus signs on our own.

private static void DrawCircle5(int radius)
{

  int e = 0;
  int x = 0;
  int y = radius;

  do
  {
    DrawPixel(x, y);
    DrawPixel(x, -y);
    DrawPixel(-x, y);
    DrawPixel(-x, -y);
    DrawPixel(y, x);
    DrawPixel(y, -x);
    DrawPixel(-y, x);
    DrawPixel(-y, -x);

    int dx = 2*x + 1;
    int dy = -(2*y) + 1;

    int e1 = e + dx;
    int e2 = e + dx + dy;
    int e3 = e + dy;

    e1 = e1 < 0 ? -e1 : e1;
    e2 = e2 < 0 ? -e2 : e2;
    e3 = e3 < 0 ? -e3 : e3;

    if (e1 <= e2 && e1 <= e3)
    {
      x++;
      e += dx;
    }
    else if (e2 <= e1 && e2 <= e3)
    {
      y--;
      x++;
      e += dx + dy;
    }
    else
    {
      y--;
      e += dy;
    }
  } while (x <= y);

}

2,100,000 circle calculations a second. It cannot possibly get better, can it?

Yes it can. Similar to DrawCircle4(), when we always incremented x because of calculating 1/8th of the circle only, we can assume here that we will never enter the last else-block in DrawCircle5(), as it does not increment x either.

And since we are always moving to the right incrementing x, the delta of e depends entirely on whether we also move down and decrement y. The change in the error value caused by decrementing y is (-2*y + 1), so let's choose to decrement y each time e is greater or equal to y, as this will move e closer to 0 again.

All of this allows us to get rid of several statements, and some local variables too. Also, let's do the left-shifting explicitly this time instead of multiplying by 2 (as mentioned above, we may expect our compiler to take care of that, but I have experienced cases when that did not happen, e.g. when writing -2*y instead of -(2*y)).

private static void DrawCircle6(int radius) 
{
  int e = 0;
  int x = 0;
  int y = radius;

  do
  {
    DrawPixel(x, y);
    DrawPixel(x, -y);
    DrawPixel(-x, y);
    DrawPixel(-x, -y);
    DrawPixel(y, x);
    DrawPixel(y, -x);
    DrawPixel(-y, x);
    DrawPixel(-y, -x);

    if (e < y) 
    {
      e += (x<<1) + 1;
    }
    else
    {
      e += (x - y + 1)<<1;
      y--;
    }
    x++;
  } while (x <= y);

}

6,400,000 circles per second. It is now 40 times faster than at the beginning.

I guess this is a good example that there is always room for improvement, and the solution that comes to mind first probably is not the quickest.

By the way, if you know an even faster algorithm, please drop me a line, and I will post a follow-up.

Update 2011:

I did some further research recently, and found out that what I developed back then is kind of a variant of the Midpoint Circle Algorithm, dating back to 1967. So I can't really claim ownership [ ;-) ].

As mentioned above, I had received two initial hints, namely that there should be an algorithm without the need of floating point operations, and that it might work in a kind of incremental way. This is were I started, and then kept going. DrawCircle6() came to my mind only after sleeping over the problem for another night.

Windows Forms DataBound-Controls On TabPages

2008-02-27T22:40:00.001+01:00

I stumble over this issue every once in a while: Windows Forms controls will only data-bind when they are visible or have been visible once - this can become an issue when controls are located on a TabPage that has never been shown. The control's bound value then has never been pushed into the UI.

Especially validation-code that checks on properties like TextBox.Text is vulnerable to this - e.g., no matter what the bound value is, TextBox.Text returns an empty string as long as it has not been visible (anyway, in my opinion validation code should always work on model values, not on view properties).

One way to work around this is to check on Control.Created before validating a control's content.

.NET Webcasts

2008-01-21T21:39:00.006+01:00

I am currently watching lots of .NET Webcasts. Some of the best can be found at:

DotNetRocks TV
MSDN On-Demand Webcasts
Channel 9 Screencasts
Mike Taulty's Screencasts from Microsoft UK

Especially Mike Taulty's screencasts are great. They have the highest entropy - tons of information in a relatively short amount of time. He exactly delivers all the facts I need to know - straight to the point.

I also try to save time when watching screencasts by switching to double playback speed in MediaPlayer.

With some concentration its possible to do so without missing any piece of information.

Arno's Software Development Bookshelf

2007-12-23T20:41:00.000+01:00

Time for a picture of my software development bookshelf. It's basically divided into three sections: (A) Software development technology (by far the largest part), (B) Software development project management and (C) The history of the software industry. Please click on the image to zoom in.

Switching Source Control Providers Under Visual Studio

2007-11-08T00:19:00.001+01:00

Visual Studio 2005 let's you switch between different source control providers (e.g. Team Foundation Server and another SCC provider for Subversion or CVS) during runtime under Tools / Options / Source Control / Plug-In Selection. But if you still have to maintain some old .NET 1.1 code in Visual Studio 2003, you may be out of luck.

E.g. I recently had to access an old Sourcesafe repository, but since I had installed Team Foundation Server, Visual Studio 2003 would try to connect to the TFS server using a Sourcesafe URI. Not a good idea. And there is not way to change the provider back to VSS temporarily from within Visual Studio 2003.

Luckily I found this nice little tool SCCSwitch which did the job.

Poker Calculator

2007-11-04T20:39:00.001+01:00

My friend and colleague Josef developed a pokerhand odds calculator in Java, and I asked him to throw it into an applet so I could host it on this blog as well. Here we go:

Great stuff, Josef! Handling the input is very easy, just click the cards of choice for the input field having the focus (yellow background), and press "Calculate" to compute the odds, resp. to finish dealing out. You can also undo/redo any action.

The odds calculator uses a brute force approach. It looks quite fast, considering the fact it needs to process millions of combinations before the flop, and finishes in about a second on my PC. You can take a closer look at the code as well - it's open source.

My blog stylesheet won't resize the content area's width, that's why the applet is cut off on the right side, otherwise my navigation panel would have disappeared (I'll probably ask Josef to let his LayoutManager do some resizing instead, so the applet will fit within smaller screen estate as well). You can see it here in full size.

The applet requires Java 5, which - if missing - should be installed automatically on MSIE, I hope also on Firefox (for Mozilla browsers I had to replace the <embed> tag by an <applet> tag in order for it to run at all inside a blogspot.com page).

And by the way Josef, here are the odds for our game last week, when you went all in. Sorry to mention that 84% sometimes still is not enough. ;-)

Update: Some proxies seem to filter away <object> tags altogether, so I replaced it with an <applet> tag for all browsers.

How To Trim A .NET Application's Memory Workingset

2007-10-31T21:26:00.003+01:00

.NET developers who have monitored their application's memory consumption in Windows Taskmanager, or slightly more sophisticated performance monitors like PerfMon, might have noticed the effect when memory usage slowly rises, and hardly ever drops. Once the whole operating systems starts to run low on memory, the CLR finally seems to give back memory as well. And as some people have noted, memory usage also goes down once an application's main window is minimized.

First of all it's important to note that by default the Windows Taskmanager only shows the amount of physical memory acquired. There is another column for displaying virtual memory usage, but it's not visible originally. So when physical memory usage drops, it's not always necessarily the CLR returning memory, but probably physical memory being swapped out to disk.

So memory consumption drops at some point in time - just probably too late. Those symptoms give us a first clue that we are not dealing with memory leaks here (of course memory leaks are more unlikely to happen in managed environments than in unmanaged ones, still it's possible - e.g. static variables holding whole trees of objects that could otherwise be reclaimed, or that EventListener that should have been unregistered but wasn't). Also, whatever amount of native heap the CLR has allocated, the size of the managed heap within that native heap is a whole different story. The CLR might just have decided to keep native memory allocated even if it could be free'd after a garbage collection pass.

And this does not look like a big deal at first glance - so what if the CLR keeps some more memory than necessary, as long as it's being returned once in a while? But the thing is, the CLR's decisions on when the right moment for freeing memory has arrived (or for that matter, the OS swapping unused memory pages to disk) might not always coincide with the users' expectations. And I have also seen Citrix installations with plenty of .NET Winforms applications running in parallel, soaking up a lot more resources than necessary, hence restraining the whole system.

Some customers tend to get nervous when they watch a simple client process holding 500MB or more of memory. "Your application is leaking memory" is the first thing they will scream. And nervous programmers will profile their code, unable to find a leak, and then start invoking GC.Collect() manually - which not only doesn't help, but is a bad idea generally speaking.

Under Java there is a way to limit the default maximum heap size (the default value depends on the Java VM), which can be overridden by passing the "-Xmx" commandline parameter to the runtime. Once the limit is reached, the garbage collector will be forced to run once more, and if that doesn't help any more either, an OutOfMemoryError is thrown. This might be bad news for the Java application, but at least it will not bring down the whole system.

I don't know of a counterpart to "-Xmx" in the .NET world. Process.MaxWorkingSet property allows for limiting the physical memory a process may occupy. I have read several postings recommending this approach to keep the whole .NET memory footprint low, but I am not so sure, plus setting Process.MaxWorkingSet requires admin privileges - something that application users will not (and should not) have.

A better choice is the Win32 API function SetProcessWorkingSetSize() with two special paramater values: -1.

From MSDN:

BOOL WINAPI SetProcessWorkingSetSize(
__in HANDLE hProcess,
__in SIZE_T dwMinimumWorkingSetSize,
__in SIZE_T dwMaximumWorkingSetSize
);

If both dwMinimumWorkingSetSize and dwMaximumWorkingSetSize have the value (SIZE_T)-1, the function temporarily trims the working set of the specified process to zero. This essentially swaps the process out of physical RAM memory.

What SetProcessWorkingSetSize() does is to invalidate the process's memory pages. what we have achieved at this point, is that our application's physical memory usage is limited to the bare minimum. And all that unused memory - as long as it is not being accessed, it will not be reloaded into physical memory. The same is true for .NET Assemblies which have been loaded, but are not currently used.

And the good news: this does not require the user to have admin rights. By the way, SetProcessWorkingSetSize is what's being invoked when an application window is minimized, which explains the effect described above.

I should not that there might be a performance penalty associated with that approach, as it might lead to a higher number of page faults following following the invocation, in case other processes regain physical memory in the meantime.

Obviously Windows' virtual memory implementation can not always swap out unused memory as aggressively. And it's my guess that what might hinder it furthermore is the constant relocation of objects within the native heap caused by garbage collection (which means a lot of different memory pages are being accessed over time, hence hardly ever paged to disk).

A Timer can be applied for repeated invocations of SetProcessWorkingSetSize(), with a reasonable interval between two calls of maybe 15 or 30 minutes (this depends heavily on the kind of application and its workload). Another possibility is to check from time to time on the physical memory being used, and once a certain amount has been reached the call to SetProcessWorkingSetSize() will occurr. A word of warning though - I do not advocate to invoke it too often either. Also, don't set the minimum and maximum working sizes (let the CLR take care of that), just use the -1 parameter values in order to swap out memory, after all that's what we are trying to achieve.

The complete code:


[DllImport("kernel32")]
static extern bool SetProcessWorkingSetSize(IntPtr handle, int minSize, int maxSize);

SetProcessWorkingSetSize(Process.GetCurrentProcess().Handle, -1, -1);

Anyway, our Citrix customers are happy again, and no one has ever screamed "Memory leak!" since we implemented that workaround.

Five Easy Ways To Fail

2007-10-25T20:45:00.002+02:00

Joel Spolsky describes the most common reasons for software projects to go awry in his latest articel "How Hard Could It Be? Five Easy Ways to Fail".

As kind of expected, a "mediocre team of developers" comes up as number one, and as usual Joel Spolsky describes it much more eloquently than I ever could:

#1: Start with a mediocre team of developers

Designing software is hard, and unfortunately, a lot of the people who call themselves programmers can't really do it. But even though a bad team of developers tends to be the No. 1 cause of software project failures, you'd never know it from reading official postmortems.

In all fields, from software to logistics to customer service, people are too nice to talk about their co-workers' lack of competence. You'll never hear anyone say "the team was just not smart enough or talented enough to pull this off." Why hurt their feelings?

The simple fact is that if the people on a given project team aren't very good at what they do, they're going to come into work every day and yet--behold!--the software won't get created. And don't worry too much about HR standing in your way of hiring a bunch of duds. In most cases, I assure you they will do nothing to prevent you from hiring untalented people.

I tend to question the four other reasons he mentions though (mainly estimating and scheduling issues). Don't get me wrong, he surely got his points, but I would rank other problem fields higher than that, lack of management support, amateurish requirements analysis or suffering from the NIH-syndrome among them.

Hints And Pitfalls In Database Development (Part 5): The Importance Of Choosing The Right Clustered Index

2007-10-15T20:30:00.000+02:00

In database design, a clustered index defines the physical order in which data rows are stored on disk (Note: the most common data structure for storing rows both in memory and on disk are B-trees, so the term "page" can also be interpretated as "B-tree leaf node" in the following text, although it's not necessarily a 1:1 match - but you get the point). In most cases the default clustered index is the primary key. The trouble starts when people don't spend any further thought and stick with that setting no matter whether the primary key is a good choice for physical ordering or not...

File I/O happens at a page level, so reading a row implies that all other rows stored within the same physical disk page are read as well. Wouldn't it make sense to align those rows together which are most likely to be fetched en bloc too? This limits the number of page reads, and avoids having to switch disk tracks (which would be a costly operation).

Image courtesy of MSDN / SqlServer 2005 Database Engine Books Online

So the secret is to choose an attribute for clustering which causes the least overhead for I/O. Those rows that are most likely going to be accessed together should reside within the same page, or at least in pages next to each other.

Usually an auto-increment primary key is a good choice for a clustered index. Rows that have been created consecutively will then be stored consecutively, which fits in case they are likely to be accessed along with each other as well. On the other hand if a row contains a date column, and data is mainly being selected based on these date values, this column might be the right option for clustering. And for child rows it's probably a good idea to choose the foreign key column referencing the parent row for the table's clustered index - a parent row's child rows can then be fetched in one pass.

I work on a project that applies unique identifiers for primary keys. This has several advantages, the client being able to create primary keys in advance among them. But unique identifier primary keys are a bad choice for a clustered index, as their values disperse more or less randomly, hence the physical order on disk will be just as random. We have experienced a many-fold performance speedup by choosing more suitable columns for clustered indexing.

Previous Posts:

Software Development Projects: The Road To Success

2007-10-10T22:42:00.001+02:00

Fun With WinDbg

2007-10-05T22:14:00.001+02:00

I did some debugging on an old legacy reporting system this week, using WinDbg. The reporting engine terminated prematurely after something like 1000 printouts.

After attaching WinDbg and letting the reporter run for half an hour, a first chance exception breakpoint hit because of this memory access violation:

(aa8.a14): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000000 ebx=665b0006 ecx=7c80ff98 edx=00000000 esi=00000000 edi=00000000
eip=665a384f esp=0012bdc4 ebp=00000005 iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010246
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for GEEI11.dll - 
GEEI11!FUNPower+0x15f:
665a384f 668b7804        mov     di,word ptr [eax+4]      ds:0023:00000004=????

Trying to access address 0x0000004 ([EAX+4]), one of the reporting DLLs was obviously doing some pointer arithmetics on a NULL pointer. The previous command was a call to GEEI11!WEP+0xb47c, which happened to be the fixup for GlobalAlloc:

665a3849 ff157c445b66    call    dword ptr [GEEI11!WEP+0xb47c (665b447c)]
665a384f 668b7804        mov     di,word ptr [eax+4]      ds:0023:00000004=????

GlobalLock takes a global memory handle, locks it, and returns a pointer to the actual memory block, or NULL in case of an error. According to the Win32 API calling conventions (stdcall), EAX is used for 32bit return values.

The reporting engine code calling into GlobalLock was too optimistic and did not test for a NULL return value.

The next question was, why would GlobalLock return NULL? Most likely because of an invalid handle passed in. Where could the parameter be found? At the ESI register - it was the one pushed onto the stack before the call to GlobalAlloc, thus must be the one and only function parameter, and it is callee-saved, so GlobalAlloc had restored it in its epilog.

665a3848 56              push    esi
665a3849 ff157c445b66    call    dword ptr [GEEI11!WEP+0xb47c (665b447c)]

0:000> r
eax=00000000 ebx=665b0006 ecx=7c80ff98 edx=00000000 esi=00000000 edi=00000000
eip=665a384f esp=0012bdc4 ebp=00000005 iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010246

As expected, ESI was 0x00000000, and GetLastError confirmed this as well:

0:000> !gle
LastErrorValue: (Win32) 0x6 (6) - Das Handle ist ungültig.
LastStatusValue: (NTSTATUS) 0xc0000034 - Der Objektname wurde nicht gefunden.

Doing some further research, I found out that the global memory handle was NULL because a prior invocation of GlobalAlloc had been unsuccessful. Again, the caller had not checked for NULL at that point. And GlobalAlloc failed because the system had run out of global memory handles, as there is an upper limit of 65535. The reporting engine leaked those handles, neglecting to call GlobalDelete() on time, and after a while (1000 reports) had run out of handles.

By the way, I could not figure out how to dump the global memory handle table in WinDbg. It seems to support all kinds of Windows handles, with the exception of global memory handles. Please drop me a line in case you know how to do that.

Now, there is no way to patch the reporting engine as it's an old third party binary, so the solution we will most likely implement is to restart the engine process after a while, so all handles are free'd by terminating the old process.

Disdain Mediocrity

2007-09-29T21:39:00.000+02:00

Ben Rady writes:

I do know some traits that good software developers all seem to share. One of them is a healthy disdain for mediocrity.

Good developers cannot stand sloppiness (in software, anyway). Apathy, haste, and carelessness send shivers down their spines. They may disagree on the best way to do things, but they all agree that things should be done the best way. And they’re constantly looking and learning to find exactly what the best way is. They realize that seeking it is an ever-changing, lifelong quest.

I couldn't agree more...

Three Years Of Blogging

2007-09-27T01:14:00.000+02:00

Three years of blogging, 290 postings full of insight and wisdom (yeah, right), and ranked within the top 5 on hit count is the entry about the old Arcade joystick?!? I am flabbergasted... ;-)

The People Factor In Software Development

2007-09-26T23:10:00.000+02:00

You may want to have a look at a presentation I assembled on "The People Factor in Software Development" (german only).

Finding Unused Classes Under .NET

2007-09-14T22:51:00.000+02:00

While integrating some .NET libraries (which by the way came from an external development partner) in our main project, we noticed several classes that never were utilized. Getting suspicious, we decided to search for all unused classes. The question was: How to do that?

The .NET compiler is no big help on this, which is understandable - it can't emit warnings on apparently unused public classes within a class library, as they are most likely part of the library's public API, but happen not to referenced inside the library itself. The same is true for IDE-integrated refactoring tools like Resharper. Resharper points out private/internal methods never called and private/internal types never referenced, but public classes are another story.

So my next bet was on static code analysis tools. They usually let you define the system boundaries, hence it should be possible to identify classes never referenced within those boundaries.

FXCop was one of the most widely used tools in the early days of .NET, but seems a little bit abandoned now, and did not have any matching analysis rule (or at least I didn't find any).

Total .NET Analyzer on the other hand looked very promising and supposedly includes this feature. In contrast to FXCop it parses the sourcecode as well, thus has the means for a more powerful breakdown. Unfortunately it ran out of memory when scanning our Visual Studio solution on my 2GB developer workstation.

Finally I ended up applying NDepend. NDepend has extensive code analysis capabilities, including the highly-anticipated search for unused classes. It also calculates all kinds of other metrics. I have only scratched the surface so far, but what I have seen is very convincing.

Leading Software Development Teams: The Human Factor

2007-09-09T22:18:00.000+02:00

I am currently preparing a presentation on the topic "Leading Software Development Teams: The Human Factor". Once I find time I will translate it to English and post it on this blog, for the moment here is my list of sources:

DeMarco, Lister: "Peopleware"
McBreen, P.: "Software craftsmanship - The new imperative"
Rainwater, H.: "Herding cats: A primer for programmers who lead programmers"
Spolsky, J.: "Joel on software"
Spolsky, J.: "Smart and gets things done - Joel Spolsky's concise guide to finding the best technical talent"
Weinberg, G.: "Becoming a technical leader - An organic problem-solving approach"
Weinberg, G.: "The psychology of computer programming"
Yourdon, E.: "Death March"

Declare War On Your Enemies

2007-09-06T18:03:00.000+02:00

While Joel Spolsky - as recently mentioned - prefers to bury bozos under a pile of bug reports, Ted Neward goes on the offensive and declares war on such and other enemies of project success:

Software is endless battle and conflict, and you cannot develop effectively unless you can identify the enemies of your project. Obstacles are subtle and evasive, sometimes appearing to be strengths and not distractions. You need clarity. Learn to smoke out your obstacles, to spot them by the signs and patterns that reveal hostility and opposition to your success. Then, once you have them in your sights, have your team declare war. As the opposite poles of a magnet create motion, your enemies - your opposites - can fill you with purpose and direction. As people and problems that stand in your way, who represent what you loathe, oppositions to react against, they are a source of energy. Do not be naive: with some problems, there can be no compromise, no middle ground.

Highly recommended reading material!

Word COM Interop: Problems With Application.Selection

2007-08-31T22:07:00.000+02:00

Most code samples out there use the Application.Selection component for creating Word documents with COM Interop. From the Word object model documentation:

The Selection object represents the area that is currently selected. When you perform an operation in the Word user interface, such as bolding text, you select, or highlight, the text and then apply the formatting. The Selection object is always present in a document. If nothing is selected, then it represents the insertion point. In addition, it can also be multiple blocks of text that are not contiguous.

Often those code samples do not even bother to set the selection beforehand (e.g. by invoking Document.Range(begin, end).Select()), as they misconceive that Application.Selection just keeps on matching while they insert content to the end of the document. Even with Range.Select() there are potantial race conditions, as we will soon see...

The problem here becomes obvious when taking a closer look at the object model: the Selection object is attached to the Application object, not to the Document object. If the application's active document changes, its selection will refer to a different document than before.

Now it still seems to be save at first sight as long as the caller operates on one document only, but not even this is the case: If the user opens a Word document from his desktop in the meantime, this is not going to fork a new Word process, but the new document will be attached to the running Word Interop process (hence the same Application). This Application has run window-less in the background so far, but now will pop up a Word window, where the user can watch in real-time how the Interop code suddenly manipulates the WRONG document, because from this moment on Application.Selection refers to the newly opened document.

How come this simple fact is not mentioned in the API documentation? I have found several newsgroup postings on that issue, people are struggling with this. So what might be a possible solution? Working with the Range API on document ranges (e.g. Document.Range(begin, end)), instead of application selections.

Previous Posts:

COM Interop Performance

2007-08-23T23:11:00.000+02:00

COM Interop method calls are slow. When you think about it, that does not come as a surprise. Managed parameter types need to be marshalled to unmanaged parameter types and vice versa. This hurts performance especially with plenty of calls to methods which only accomplish a tiny piece of work, like setting or getting property values. 99% percent of the time might be lost on call overhead then.

There are more issues, e.g. incompatible threading models that may lead to message posting and thread context switching, memory fragmentation due to all the parameter marshalling effort, unnecessarily long-lived COM objects which are not disposed by the caller, but by the garbage collector much, much later. So when handled improperly, systems applying COM Interop not only tend to be slow, but also degrade over time regarding performance and system resources.

Most of our COM Interop code is related to MS Office integration, assemblying word documents (content and formatting) for example, this kind of stuff. We had to do quite some tuning to kind of reach a level of acceptable runtime behavior. Limiting the number of method calls and keeping references to COM objects (respectively their COM interop wrappers) for reuse was one step towards improvement, while explicitly disposing them as soon as not needed any more was another one. And never forget to invoke Application.Quit() when done, unless no one cares for zombie Word processes sucking up system resources.

Previous Posts:

Follow-Ups:

Word COM Interop: Problems With Application.Selection

Clippy For Software Architects

2007-08-18T22:24:00.001+02:00

Handling The Bozo Invasion

2007-08-12T22:45:00.000+02:00

Invaluable advice from Joel Spolsky in "Getting Things Done When You're Only a Grunt":

Strategy 4: Neutralize The Bozos

Even the best teams can have a bozo or two. The frustrating part about having bad programmers on your team is when their bad code breaks your good code, or good programmers have to spend time cleaning up after the bad programmers.

As a grunt, your goal is damage-minimization, a.k.a. containment. At some point, one of these geniuses will spend two weeks writing a bit of code that is so unbelievably bad that it can never work. You're tempted to spend the fifteen minutes that it takes to rewrite the thing correctly from scratch. Resist the temptation. You've got a perfect opportunity to neutralize this moron for several months. Just keep reporting bugs against their code. They will have no choice but to keep slogging away at it for months until you can't find any more bugs. Those are months in which they can't do any damage anywhere else.

I agree for the scenario described above, that is if you have to deal with one or two bozos under your coworkers. There might be other situations, though. For example bozos placed in strategic positions, so they can highly influence your work. Or company growth based on hiring legions of morons. An invasion of bozos usually is a sign that an organization is doomed anyway. The path to downfall might be long and winding, and being doomed doesn't necessarily mean going out of business - just a slow and steady decline with failed projects paving the way. This is just inevitable. It's better to leave the sinking ship earlier than later in those cases.

DSOFramer Now Compatible To Office 2007

2007-08-08T08:05:00.000+02:00

Just a short follow-up to my posting about hosting MS Office inside .NET WinForms applications, Microsoft's DSOFramer is now available in an Office2007-compatible version.

Previous Posts:

Follow-Ups:

My Computer Museum (Update 2)

2007-07-22T00:58:00.000+02:00

Main table (left to right)
(1) Sun SparcStation 5, NEC 21" VGA Monitor
(2) Commodore 128D, Commodore 1901 Color Monitor
(3) Commodore Amiga 500, Commodore 1085S Color Monitor
(4) Atari 1040ST, Atari SM124 Monochrome Monitor, Atari SC1224 Color Monitor
(5) Apple Macintosh LC, Apple Monochrome Monitor
(6) Commodore 64, Commodore 1541 Floppy Drive, Commodore 1081 Color Monitor

Back cross table (left to right)
(1) Commodore CBM8032
(2) Apple IIc, Apple II Green Composite Monitor
(3) Sinclair ZX81, B+W TV Set

Middle cross table (left to right)
(1) Atari 2600
(2) Magnavox Odyssey 2
(3) Atari PONG
(4) Magnavox Odyssey 1

Front cross table (left to right)
(1) Apple Macintosh Plus
(2) IBM PC 5150, IBM 5151 Green Composite Monitor

Parkinson's Law

2007-07-18T00:39:00.000+02:00

In project management, individual tasks with end dates rarely finish early because the people doing the work expand the work to finish approximately at the end date. Coupled with deferment or avoidance of an action or task to a later time, individual tasks are nearly guaranteed to be late.

From: Parkinson's Law

Very true, especially for software projects.

Vintage Computer / Video Console Purchases On Ebay

2007-07-14T00:46:00.001+02:00

My latest vintage computer / video console purchases on Ebay include:

There were other auctions on rare products, e.g. an Apple Lisa (went away for USD 700) and a NeXTStation (sold for USD 200 - man, I should have bought it!).

I am kind of running out of table space in my cellar, that's why I cannot provide any actual images yet of the entire collection, so for the meantime there is this photo of a previous state.

My IBM 5150 Is Booting

2007-07-13T23:25:00.000+02:00

The DOS 2.10 disks - which by the way I purchased in Australia - arrived today, so I tried to convince my IBM 5150 to boot. Unfortunately it did not recognize the disk at first, and continued to jump to the built-in Basic interpreter. I had no possibility to check the disk on any other system. What to do next? I mean this thing doesn't even have a BIOS, it has... DIP switches on the motherboard! I should note that my IBM did not come with the two standard 5.25" floppy drives, but one floppy drive and one 10MB hard drive, so I figured there might be some misconfiguration. Luckily I still found documentation on the 5150 DIP switch settings, and after flipping one switch I was able to boot.

Current date is Tue 1-01-1980
Enter new date:
Current time is 0:00:12.89
Enter new time:

The IBM Personal Computer DOS Version 2.10
(C)Copyright IBM Corp 1981, 1982, 1983

A> _

Next steps: Backup the only DOS 2.10 disk I have, and try to install DOS on the hard disk. Hard disks were supported from DOS 2.0 on - so this should work as well. Oh yeah, DOS 2.0 also introduced subdirectories...

Hosting .NET Winforms Applications In MSIE

2007-07-10T23:52:00.000+02:00

I built a little .NET 2.0 Winforms frontend for my brute-force Sudoku Solver, and played around with hosting it inside MSIE. In case you are using MSIE and have .NET Framework 2.0 installed, you can give it a try here.

This is how it looks like:

Weird, isn't it? What is that, the Black and White Look & Feel? When running as a standalone application (download link), of course everything is just fine:

I have seen the same effect on other sites as well. Maybe that's Microsoft's way of visually marking MSIE-hosted .NET applications. Won't be too beneficial to the success of that approach I am afraid.

BTW, the Sudoku puzzle you see here is somehow famous for its problem depth. I compared the 62ms it took the algorithm on my old 2.4GHz Athlon to find the solution with other brute force Sudoku solvers, and it came off quite well in comparison.

Previos Posts:

More Performance Tuning

2007-06-29T22:17:00.002+02:00

I was occupied with some performance tuning this week on your .NET client/server solution, here are some of the conclusions I have drawn.

One of the bottlenecks turned out to be caused by some missing database indices. This again reminds me that I have hardly experienced a case when there were too many indices, but plenty of times when there were too few. Foreign keys AND where-criteria attributes are primary candidates for indices, and I'd rather have a real good reason when leaving out an index on those columns. I know some folks will disagree, and point out the performance penalty for maintaining indices, and that database size will grow. While I agree those implications exist, they are negligible in comparison to queries that run a hundred times faster with an index at the right place.

Also, the reasoning that relatively small tables don't need indices at all simply does not catch up with reality. While an index seek might not be much faster than a scan on a small number of rows, missing an index on a column that is being searched for also implies that there are no statistics attached to this column (unless statistics are manually created - and guess how many developers will do so), hence the query optimizer might be wrong about that data distribution which can lead to ill-performing query plans. Also, the optimizer might make different decisions because it considers the fact that there is no index on certain columns.

There are several ways to find out about missing indices: Running the Tuning Advisor, checking execution plans for anomalies, and of course some excellent tool scripts from SqlServerCentral and from Microsoft.

A query construct that turns out problematic at times under SqlServer looks something like

where (table.attribute = @param or @param is null)

While these expressions help to keep SQL statements simple when applying optional parameters, they can confuse the query optimizer. Think about it - with @param holding a value, the optimizer has a restraining criteria at hand, but with @param being null it doesn't. When the optimizer re-uses an execution plan previously compiled for the same statement, trouble is around the corner. This is specifically true for stored procedures. Invoking stored procedures "with recompile" usually solves this issue. But this may happen within client-generated SQL-code as well. Options then are forcing an explicit execution plan for the query (tedious, plus if you still have to support SqlServer 2000 you are out of luck), using join hints, or finally re-phrasing the query (in one case we moved some criteria from the where- to the join-clause, which led the optimizer back to the right path again - also additional criterias might help).

By the way I can only emphasize the importance of mass data tests and of database profiling. I don't want my customers to find out about my performance problems, and I want to know what is being shoveled hence and forth my database at any time.

I was also investigating another performance problem that appeared when reloading a certain set of records with plenty of child data. Convinced of just having to profile for the slowest SQL, I was surprised that they all performed well. In reality, the time got lost within the UI, with too much refreshing going on (some over-eagerly implemented toolbar wrapping code which re-instantiated a complete Infragistics Toolbar Manager being the reason - external code, not ours).

I enjoy runtime tuning. A clear goal, followed through all obstacles, and at the end watching the same application running faster multiple times - those are the best moments for a developer.

Pure Ignorance Or The Not-Invented-Here Syndrome?

2007-06-23T12:08:00.000+02:00

I just finished listening to this .NET Rocks episode with guest Jeff Atwood from Coding Horror fame. I can really identify with a lot of what he says and what he propagates on his blog. One topic was brought up by Richard Campbell I think, when he mentioned that nothing else hurts him more than watching a 60K or 80K developer working on problems for months or years (not to mention whether successfully or not), when there are existing solutions available either free in terms of open source projects, or as commerical products priced at some hundred bucks. Jeff wholeheartedly agreed.

They hit the nail on the head. I don't know whether it's pure ignorance or just the infamous Not-Invented-Here Syndrome, but this just seems to happen again and again. For my part, I blame decision makers approving budgets for such projects just as much as developers who try to solve problems that have been solved a hundred times better a hundred times before.

Here are some examples:

Why not implement my own Database Diff tool? See RedGate SQL Compare and SQL data Compare.
Why not implement my own O/R mapper? See Hibernate.
Why not implement my own XML parser? See Apache Xerces.
Why not implement my own XML binding? See JAXB.
Why not implement my own HTTP protocol? See package sun.net.www.http or Jakarta Commons HttpClient.
Why not implement my own Java Enterprise Framework? See Spring Framework.

Bidding For An Atari Pong Videogame

2007-06-22T01:35:00.000+02:00

Please don't tell anybody, but I am currently bidding for an Atari Pong videogame (the so called Sears edition) on EBay.

COM Registration In .NET

2007-06-21T21:01:00.000+02:00

Just a short follow-up to my posting about Hosting MS Office inside .NET WinForms applications: In case there is the need to install ActiveX controls like DSOFramer or other COM components, but no Installer that could do so (e.g. because of XCopy deployment), this sample code shows how to accomplish on-the-fly COM registration in .NET.

Previous Posts:

Hosting MS Office Inside .NET Windows Forms Applications

Follow-Ups:

EBay'ed An IBM 5150

2007-06-20T21:27:00.000+02:00

After weeks of trying I finally managed to buy an IBM 5150 on EBay (yes, that's the first IBM PC). EUR 100 plus EUR 70 for shipping - but hey, it's worth the price.

I found original IBM DOS 2.0 disks as well, which complete the whole purchase (IBM DOS 1.1 or 1.0 would have been even better, but they are also more rare).

In Defense Of Data Transfer Objects

2007-06-15T21:37:00.000+02:00

In the J2EE world, Data Transfer Objects have been branded as bad design or even as an anti-pattern by several Enterprise Java luminiaries. Their bad image also stems back to the time when DTOs were a must at EJB Container boundaries, as many developers ended up using Session EJBs at the middle tier which received/passed DTOs from/to the client, tediously mapping them to Entity Beans internally.

Then came Hibernate, which alongside its excellent O/R-mapping capabilities allowed for passing entities through all tiers, attached in case of a Open-Session-In-View scenario, detached in case of a loosely coupled service layer. So no need of DTOs and that cumbersome mapping approach anymore, this seems to be the widely accepted opinion.

And it's true, DTOs might be a bad idea in many, maybe even in most of the cases. E.g. it doesn't not make a lot of sense to map Hibernate POJOs to DTOs when POJO and DTO classes would just look the same.

But what if internal and external domain models would differ? One probably does not want to propagate certain internal attributes to the client, because they only matter inside the business layer. Or some attributes just must be sliced off for certain services, because they are of no interest in their context.

What if a system had been designed with a physical separation between web and middle tier (e.g. due to security and scalability reasons)? An EJB container hosting stateless session beans is still first class citizens in this case. Other services might be published as webservices. It's problematic to transfer Hibernate-specific classes over RMI/IIOP or SOAP. Even if it's possible (as it is the case under RMI/IIOP) this necessarily makes the client Hibernate-aware.

While it is true that Hibernate (and as well EJB3 resp. the Java Persistence API) are based on lightweight POJOs for entities, Hibernate has to inject it's own collection classes (PersistentSet) and CGLib-generated entity subclasses. That's logical due the nature of O/R-mapping, but having these classes being transferred over service layer boundaries is not a great thing to happen. And there are more little pitfalls, for example state management on detached entities - how can an object be marked for deletion when it is detached from it's Hibernate session?

Sorry, but I have to stand up and defend DTOs for these scenarios. Don't get me wrong, I appreciate Hibernate a lot and use it thoroughly within the middle tier, but I also don't see the problem of mapping Hibernate POJOs to DTOs on external service boundaries, especially when done in a non-invasive way. No mapping code has to pollute the business logic, no hardwiring is necessary, it can all be achieved by applying mapping frameworks like Dozer, using predefined mapping configurations. What goes over the wire at runtime is exactly the same as declared at compiletime, a clear service contract, no obsolete entity attributes, no object trees without knowing where the boundaries are, and no surprising LazyInitializationExceptions on the client.

Learning From Experience (Introduction)

2007-06-04T10:02:00.000+02:00

Last weekend I digged out an old university project of mine - Visual Chat, a graphical Java chat system that I built for a class assignment exactly a decade ago. I was allowed to open-source Visual Chat back then, and I still receive questions from people today who are working on extending the application. Back in 1997 I was clearly still an apprentice, as this was one of my first Java resp. OOP projects at all. When I look at the sourcecode today, I remember what I learned in the process of developing it, but I also see I was still missing a lot of experience.

Learning not only in theory but also from real projects is particularly important in software development, and this is something that - in my opinion - is done far too little during education. Designing a system at a larger scale is a completely different task than solving isolated algorithmic problems (which is what you normally get to do for homework).

Class assignments that simulate real projects are great because one has the freedom to make mistakes and learn from them, much more than in professional life. Around 500.000 people have signed up at the Visual Chat test installation site so far, so I could learn a lot from monitoring what was going on at runtime. The worst thing that might have happened was that people would not like my chat program and would switch to another one (as I am sure, many did). There was no danger of financial losses (or worse) for any user in case of an application mistake. I am glad I had this opportunity - it helped me to build better products once the chips were down at work, e.g. when developing banking or hospital information systems.

Receiving a solid training is important in our profession, but only as long as it is accompanied by applying what one has learned in classroom. Half-baked knowledge is dangerous, and it often takes two or three tries to make it right. Only a small fraction of people is able to do a perfect job from the beginning (and even a genius like Linus Torvalds openly admits that he also wrote some ugly code once upon a time).

So I decided to do two things. I am going to provide a little series of what I learned back then, and since then (looking at the code today), hoping that novices reading those articles can benefit by taking this path as a short-cut instead of walking through the same of experience on their own (which would be far more time-consuming). And I will do some refactoring on the old code, so everyone who decided to continue development work on Visual Chat will be able to take advantage of that as well.

Here is my topic list at this time of writing:

1. OOP Design - On lose coupling, the sense of interfaces and the concept of Singletons
2. Multithreading on the server - Gaining stability and performance through queuing
3. Multithreading on the client - The secrets of the AWT EventDispatch thread, EventQueues and repainting
4. How not to serialize a java.awt.Image
5. What are asynchronous Sockets (and why are they not supported in JDK1.1)?
6. Conclusions

Bill Gates And Steve Jobs Interview

2007-05-31T19:10:00.000+02:00

This recent interview is really a must see - I think they did not have that much fun since the Macintosh Dating Game in 1984 and the Macworld Expo Keynote in 1997.

Not to forget these previously unknown scenes.

BTW, I highly recommend the movie "Pirates Of Silicon Valley", which - as far as I can tell - provides a quite accurate picture of the PC revolution, plus it's very entertaining. Years ago I bought both the english and the german movie version, still back on VHS. I must have watched it a dozen times since then.

Hints And Pitfalls In Database Development (Part 4): Do Not String-Concatenate SQL Parameter Values

2007-05-15T00:46:00.000+02:00

Once again, this should be obvious. Unfortunately there are still plenty of articles and books out there with code examples that look something like this:

string sql = "select * from orders where orderdate = '" + myDate.ToShortDateString() + "'";

I don't know why this is the case. Most authors are well aware of the problems that might arise, and actually comment that this should not be done in real-life projects, but guess what - it will end up in real-life projects if examples like that are floating around.

So it's not really surprising people keep on writing this kind of statements. There are many reasons why this is just plain wrong.

Instead of concatenating parameter values, it's much better to use parameter placeholders, e.g.:

string sql = "select * from orders where orderdate = @myDate";

The placeholder syntax might be varying and depends on whether we are talking about the JDBC or ODBC / OLEDB / ADO.NET world (under OLEDB the syntax is defined by the underlying database specific driver).

The database API then provides some functionality to pass the actual parameter values , something like:

SqlCommand cmd = new SqlCommand();
cmd.CommandText = sql;
cmd.Parameters.Add("@myDate", myDate);

This has several advantages:

Avoiding SQL injection: No need to worry that any user will enter something like "'; drop database;" into an input field. The database vendor's driver is going to take care of that.

Taking advantage of prepared statements: As the SQL code always stays the same, and only parameter values change on consecutive invocations, the statement can be precompiled, which guarantees better performance.

Independence from database language settings: Expressions like myDate.ToShortDateString() produce language-specific formats. If your customer's database was set up with another language setting, or this code is being executed from a client with different language settings, you are out of luck.

Improved readability: SQL code can be defined in one piece at one place, maybe inside XML, without the need of cluttering it with of string-concatenation. Often those statements can just be copied and executed 1:1 within a database query tool for testing purposes. All that is necessary is to provide the parameters and set their values manually.

Follow-Ups:

The Importance Of Choosing The Right Clustered Index

Hints And Pitfalls In Database Development (Part 3): Database Programming Requires More Than SQL Knowledge

2007-04-29T00:45:00.000+02:00

First of all, "knowing SQL" is a rather broad term. SQL basics can be taught within a day, but gaining real in-depth SQL know-how, including all kind of database-specific extensions, might take years of practice.

Then there are always plenty of ways of how to build a solution in SQL, and only a small subset of those are really good ones. It is important to be aware of the implications of certain SQL constructs - which kind of actions the database engine has to undertake in order to fulfill a given task.

Let me provide a little checklist - those are things every database programmer should know about in my opinion:

ANSI SQL and vendor-specific additions (syntax, functions, procedural extensions, etc). What can be done in SQL, and when should it be done like that.
Database basics (ACID, transactions, locking, indexing, stored procedures, triggers, and so on).
Database design (normalization plus a dose of pragmatism, referential integrity, indices, table constraints, stuff like that).
Internal functioning (for instance B-trees, transaction logs, temp databases, caching, statistics, execution plans, prepared statements, file structure and requirements for physical media, clustering, mirroring, and so on).
How do certain tasks impact I/O, memory and CPU usage.
Query optimizer: what it can do, and what it can't do.
Error handling, security (for example how to avoid SQL injection, ...).
Database tools: profiling, index tuning, maintenance plans (e.g. backup and reindexing), server monitoring.
Interpretation of execution plans.

Follow-Ups:

Hints And Pitfalls In Database Development (Part 2): Let The Database Enforce Data Constraints

2007-04-05T11:17:00.000+02:00

Letting the database enforce data constraints might sound obvious, but as a matter of fact I had to listen to some people just a few months ago who advocated checking for referential integrity within business code instead of letting the database do so, because "foreign keys are slow" - I mean geez, what kind of nonsense is that?! I don't know, they probably forgot to define their primary keys as well, hence didn't have any index for fast lookup (*sarcasm*).

No other entity is better suited to safeguard data integrity than the database. It's closest to the data, it knows most about the data, and it's the single point every chunk of data has to pass through before being persisted.

And I am not only talking about primary keys and foreign key constraints. I put all kinds of checks on my databases, for example unique compound indices whenever I know that a certain combination of column values can only occur once. Or table constraints that enforce certain rules on the data which are being checked by the database on each insert and update. Triggers are another possibility, but care must be taken - they might be slower than table constraints.

Setting up constraints usually is quite easy, easier than implementing the same rules in business code. Of course performance matters, and it is important to balance costs and benefits. I do not recommend to code complex SQL-statements within constraint checks - they should be limited to simple logical expressions.

During application development, the database will scream immediately when data integrity rules are being violated. In the unlikely case that such a programming mistake slips through into production code, the customer will be confronted with some error message, but his data is unendangered, any you will find out about it immediately and can provide a fix at no time - not months or years later, when data corruption would have occurred, which then might have been impossible to repair. This has saved us on several occasions.

Plus you never know if other clients are going to be manipulating data in the future, clients that might not be aware of the existence of those data rules. Data normally lives longer than application code.

Previous Posts:

Missing Indices

Follow-Ups:

Hints And Pitfalls In Database Development (Part 1): Missing Indices

2007-03-19T20:43:00.001+01:00

I have been involved in a lot of database tuning lately. This has partly to do with the fact that some of our databases simply outgrew the originally anticipated data dimensions. Furthermore I sometimes have to deal with application modules from "external contributors".

So I am trying to sum up some of my experiences on database development over the years in a little mini-series. Most of those things should be quite obvious to many developers, others might be piece of news for one or the other. I will fall back into some SqlServer terminology at times, but many principles should apply to any major database.

So let me start with the number one reason for bad database performance:

Missing Indices

It's just amazing how creating indices tends to be forgotten again and again. I don't know, it might have to do with the fact that on 4th dimension platforms like Microsoft Access, where a lot of folks come from, they did not have to worry too much about indexing. Databases were kind of smaller there, and the database schema wizard most likely made any foreign key an index by default anyway.

I often hear the argument that "indices are going to slow down our updates". Well I don't know what is worse, slowing down updates by a factor of 0.1, or slowing down queries a factor of 100. The numbers vary of course. And it is true, indices that will not be applied by the query optimizer hence won't lead to any performance gain should not be introduced in the first place. The best way to find out is to have a look at the query execution plans, and see if the indices are being actually being used.

So in general, foreign keys and other query criteria fields are candidates for indices, with the exception of attributes with small value ranges like booleans or a tiny set of numbers, large varchar columns and tables with a small number of rows.

But sometimes even "golden rules" of when to create or not to create indices can turn out to be wrong. One of those rules is to avoid indices on columns with sparse value distribution, like a status attribute with 5 different values. But I experienced a case when the query optimizer made completely wrong assumptions about row counts for one of those status values. I added an index on this column, which implicitly induced statistic maintenance, and that in turn helped the optimizer to make the right decisions.

That being said, it is undoubtedly important to avoid excessive index creation, as this can lead to performance penalties during inserts, updates and deletes.

In addition, composite indices can lead to major speedups as well, when applied to tuples of columns which are typically jointly queried (note: leftmost index columns must all appear in a query for the index to be applied).

The good news is: you are not alone. There are plenty of tools which help finding a good indexing strategy. I start up SqlServer's Database Engine Tuning Advisor (formerly known as Index Tuning Wizard) quite frequently, and in addition I have several schema check scripts at hand which - among other things - look for missing indices, e.g. on foreign key columns.

And: Indices must be rebuilt frequently. This serves two purposes: (1) The index nodes then have a fill factor for optimal balance between read and write performance, and (2) Index rebuilding updates index statistics as well. Outdated statistics can be a major cause for slow query performance.

Follow-Ups:

Automatic CompSci Research Paper Generator

2007-03-13T20:50:00.000+01:00

SCIgen is a program that generates random Computer Science research papers, including graphs, figures, and citations. While the content is completely gibberish, this did not stop SCIgen's creators from submitting to and presenting the generator's work at conferences. Conferences with such low submission standards that they accepted the papers.

Those folks did not even bother to take a look at the auto-generated slides before giving their talk, they just went along - absolutely priceless!

Windows For Warships 3.11

2007-03-05T22:49:00.000+01:00

News of the day: British Royal Navy T45 destroyers to be equipped with Windows 2000.

From the text:

Windows platforms may be troublesome to maintain, but most civilian sysadmins simply wouldn't believe the resources the navy can throw at problems. A present-day Type 42 destroyer carries at least four people who have absolutely nothing else to do but care for the ship's command system. As of just a few years ago, this was still a pair of antique 24-bit, 1MHz machines each with about 25KB of RAM.

There may also be perfectly valid criticisms to be made regarding Windows usability. When triggering missile decoys with seconds to spare, one doesn't need a superfluous pop-up box saying "Do you want to use soft kill?" with "Confirm" and "Cancel" buttons. But this kind of thing isn't going to faze you if you're used to entering instruction sets such as "PE L5414.10N L00335.67E R6000 TMDA [INJECT]" from memory without backspace or delete. During combat, mind. The one group of users to whom Windows 2000 might look pretty marvellous are RN warfare operators.

Why Great Coders Get Paid Far Too Little

2007-03-02T00:53:00.000+01:00

Kevin Barnes picks up a similar topic I was talking about the other day - the question why great coders get paid far too little. Very insightful posting, just as some statements from the comments section. Let me quote:

Also, the industry keeps complaining about a labor shortage, while employees complain about a job shortage. I finally got to the bottom of this. Industry is finding a shortage of good people, whereas crappy people can’t get jobs. Universities, with dropping enrollment in engineering programs, are lowering standards so that more people can train to be engineers. Poor engineers enter the job market, can't get jobs, and when they do, they lower the overall pay range. As a result, good people see low wages and a job shortage, and go into other fields, increasing the problem.

Great coders do not fit in typical organizational hierarchies. Because the hierarchy rarely recognizes the value produced by higher quality developers until they are gone.

I've been in this market for over 20 years. What I noticed is that in 90% of the cases, the manager has no clue who is good and who's mediocre. Mostly, he criteria is based on how each one advertise himself.

The difference between programmers, doctors and lawyers is that if a doctor or a lawyer is mediocre you will notice it pretty soon. With the programmers, only time will show the truth.

Why Can't Programmers... Program?

2007-02-27T22:46:00.000+01:00

Excellent piece of writing today from Jeff Atwood, who poses the question how on earth the majority of applicants for a programming job can't even solve the most basic tasks, let's say implement a linked list, or a simple SQL join. Yeah I know there is java.util.LinkedList, but guess what, I have seen people iterating over a LinkedList in a counting loop using get(int index) on each element. Can you say O(N²)? That's just my point! That's why sites like The Daily WTF never run short of stories on "curious perversions in information technology".

So I openly admit that just like Jeff I am getting increasingly tired of being affiliated with a profession where let's say

4 out of 10 people don't know how to code at all

another 4 out of 10 people know how to code (kind of), but don't know how to design software systems

which leaves 2 out of 10 people who are actually qualified for what they are doing

What's really ironic is that underperformers often slip through the recruitment process quite easily, while some of top notch folks won't because they expect a somewhat higher salary, a nice working environment or an interesting project assignment.

This usually happens when interviewing is exclusively done by human resource folks or other people who are not capable of letting candidates do some sample coding right on the spot, and who have never heard about Joel Spolsky's "The Guerrilla Guide to Interviewing" or the fact that the best developers are ten times more productive (that is ten times more features or the same feature-set ten times faster, with ten times fewer errors) than the worst (which refers to "the worst qualified" who might still be considered for doing the job - and many empirical studies compare people coming from the same training or working in the same team).

I can't imagine this is usual in other areas (imagine that e.g. in the medical field). It must have to do with the unique nature of software development. And the fact that many decision makers don't quite grasp how it works.

EBay'ed A PDP 11 CPU

2007-02-19T22:24:00.000+01:00

Yet another show-piece for my computer museum, and that for only 25GBP, a real bargain! ;-)

I think this chipset comes from the PDP-11/84 series.

The Power Of Google

2007-02-14T21:08:00.000+01:00

Two weeks ago, my car began bucking. First only when starting to go or when switching gears, later on it also happened randomly just while driving. At my repair shop they changed the spark plugs, and also informed me that the clutch was a little loose, but fixing it would turn out quite expensive, and might actually not pay off any more considering the age of the car.

Unfortunately the bucking did not stop, it got worse. I asked for advice again, and once more the mechanic recommended repairing the clutch. Originally I had been planning to buy a new car in the second half of 2007, so that would probably have meant to prepone that purchase. So I started skimming through car sales notes already...

In a final attempt, I invoked a Google search on my car's model and construction generation, as well as several keywords describing the behavior. Hundreds of online forum posts showed up. I took a look at the first results which included possible explanations, and did a quick count on which parts might be involved. Seven out of ten postings listed "exhaust gas regulation valve" (I hope that's the term in in English), others such as "lambda probe", "fuel filter", "spark plugs" or "injector" scored between one and four (some articles specified several causes).

At this point I should probably mention that I have absolutely no clue about cars in general, and just a very rough idea what an "exhaust gas regulation valve" might even be. But I went back to my repair shop and told them to give the exhaust gas regulation valve thingy a shot instead of fiddling with the clutch. They did, and sure enough the valve turned out to be the root cause.

Thank you Google!

System Calls On HTTP Request

2007-02-07T21:55:00.000+01:00

I slightly doubt the claim that comparing the number of system calls while processing two HTTP requests on Linux/Apache and Windows/IIS inherently proves that Windows is less secure than Linux, but those two call graphs alone are worth looking at.

Linux/Apache

Windows/IIS

Thanks to Richard Stiennon for this visualization, nice idea.

SOAP: The S Stands For "Simple"

2007-02-05T21:47:00.000+01:00

Pete Lacey pretty much sums up the horror I have gone through during the last five years when it came to web service integration.

[...]

Developer: (Reads XML Schema spec). Saints preserve us! Alexander the Great couldn't unravel that.

SOAP Guy: Don’t worry about it. Your tools will create the schema for you. Really, its all about the tooling.

[...]

Developer: This is getting ugly. The WSDL my tools generated can't be consumed by the tools my partners use. Not only that, the schemas it generates are impenetrable and can't be reused. And no tool seems to have agreed on how best to handle the SOAPAction header.

SOAP Guy: Sorry to hear that, buddy. On the bright side, nobody uses the Doc/Lit style anymore. In order to get transport independence back we’re all using wrapped-doc/lit now. Doesn't that sound cool: wrapped-doc/lit?

Debugging The .NET Framework

2007-01-31T21:27:00.000+01:00

Over the last weeks, a small fraction of our customers reported some application instability. The client, a .NET 1.1 WinForms program (due to .NET framework backward incompatibilities resp. several third party tool problems we have not been able to migrate to .NET 2.0 yet), would simply terminate occasionally without any error message, dump or whatsoever. Several modules were affected, and there seemed to be no systematic scheme behind it.

After talking with some customers we figured out what those crashes had in common: they all occurred when a worker thread had been spawned to do some DB stuff, while the UI thread would display a modal progress dialog. Moving to a single thread (namely the UI thread) was no alternative, as the worker thread's task included some long running DB stuff, which would block the client, hence make the UI non-responsive, even when applying Application.DoEvents in between).

I suspected a concurrency issue, and had a closer look at the code - but there was nothing wrong about it, data access was cleanly separated between the two threads, which would only communicate with each other on a message basis.

So the next thing we tried was to reproduce the error on one of our test machines. But bad luck again. We had recorded a macro which represented the scenario our customers had described, but our application wouldn't crash, not in hours, not in days.

Finally we digged out an old developer PC which had actually been scrapped already, and there it happened, just as the customer had reported it - repeatedly, and within minutes. The .NET runtime simply terminated, do dump, no system log entry, nothing. Attaching the debugger didn't help either. Next I invoked several profilers (Compuware BoundsChecker, Rational Purify and JetBrains DotTrace). Profiling also did not provide new insights. BoundsChecker and Purify originally are unmanaged code profilers - they do support .NET profiling, but still it doesn't seem to be a first class citizen, and DotTrace requires .NET 2.0 for memory profiling.

So time for WinDbg, Microsoft's system level debugger. I didn't really have a lot of necessity to apply WinDbg before, but this seemed to be one of those cases. Luckily I read Tess' "If broken it is, fix it you should"-blog on Windows debugging at regular intervals, so this turned out to be helpful. Impatiently anticipating some kind of progress, I attached WinDbg to our application, and sure enough some time later it broke at a second chance exception. The call stack was full of entries like

mscorwks!GetCompileInfo

Yes the .NET runtime crashed, and it was clear I needed some runtime symbol files ("GetCompileInfo" is just a placeholder).

Fortunately there is the so called Microsoft symbol server, from which one can either download all kinds of system symbol files, or simply let WinDbg connect to it and do the job (File / Symbol File Path / "srv*c:\symbols*http://msdl.microsoft.com/download/symbols"). Furthermore I wanted better support for .NET debugging, so I loaded the SOS (Son of Strike) extension into WinDbg:

0:000> .load clr10\sos

On the next crash, the callstack made more sense:

0:000> kb
00 0012d3d0 79216288 0162652c 00166c8c 00000001 mscorwks!gc_heap::mark_object_simple+0x77 (FPO: [Non-Fpo])
01 0012d44c 79213753 79211d81 00000000 00000000 mscorwks!gc_heap::mark_through_cards_for_segments+0x28f (FPO: [Non-Fpo])
02 0012d478 7921248e 00000001 00000000 00000000 mscorwks!gc_heap::mark_phase+0xb6 (FPO: [Non-Fpo])
03 0012d4d4 7921301a 00000000 00000000 00000000 mscorwks!gc_heap::gc1+0x96 (FPO: [Non-Fpo])
04 0012d514 79213c37 00000000 00000000 00000000 mscorwks!gc_heap::garbage_collect+0x1bf (FPO: [Non-Fpo])
05 0012d534 79214e83 00000000 00000000 00000044 mscorwks!GCHeap::GarbageCollectGeneration+0x11b (FPO: [Non-Fpo])
06 0012d564 792d176b 793df808 00000044 00000000 mscorwks!gc_heap::allocate_more_space+0x13a (FPO: [Non-Fpo])
07 0012d788 791b3af0 00000044 00000002 0012d7b8 mscorwks!GCHeap::Alloc+0x77 (FPO: [Non-Fpo])
08 0012d798 791d6280 00000044 00000000 00040000 mscorwks!Alloc+0x3a (FPO: [Non-Fpo])

It was the garbage collector, and it looked like some managed heap corruption. "verifyheap" would tell me for sure:

0:000> !verifyheap
VerifyHeap will only produce output if there are errors in the heap
object cbad00: bad member 162652c at 00cbad30
curr_object : 0xcbad00 size = 0
Last good object: 0xcbac94

OK, maybe the last good object would supply some further clues:

0:000> !do 0xcbac94
Name: System.Data.SqlClient.SqlCommand

I tried this several times, the last good object always was either System.Data.SqlClient.SqlCommand or System.Data.SqlClient.SqlDataAdapter. I had a look at our code and checked which other objects were instantiated in this context. Besides SqlCommand and SqlDataAdapter there was System.Data.SqlClient.SqlConnection. So I googled for "managed heap corruption" for each of those three classes, and there it was: "Managed Heap Corruption When You Use SqlConnection". From the text: "[...] This causes the Microsoft .NET application to quit unexpectedly during the garbage collection phase".

This is an MDAC (Microsoft Data Access Components) 2.7 problem (MDAC 2.7 SP1 solves it), which occurrs when database connections are being rapidly opened and closed (logically, not physically) on multiple threads. As .NET 1.1 itself only requires MDAC 2.6 or above, older clients that have not been updated are most likely affected.

It had been a hard day's work.

Sudoku Benchmarking / .NET 2.0, Java 5 and Java 6 Results

2007-01-15T00:30:00.001+01:00

Brian Deacon asked me to add Sudoku Solver benchmark results for .NET 2.0 and the latest Java releases, so I am happy to provide those numbers (plus this finally gave me a reason to download and install Java 6).

.NET 2.0 and Java 6 both have improved a lot, and for the first time .NET without NGEN overtakes Java (not applying NGEN is the fairest-possible comparison between .NET and Java in my opinion - and in this case it's a very close match), but both still don't quite reach C++ performance:

Rank	Platform	Execution Time (for several thousand valid solutions)
1.	C++ (MSVC)	250ms
2.	C# (.NET 2.0) with NGEN	375ms
3.	C# (.NET 1.1) with NGEN	390ms
4.	C# (.NET 2.0)	406ms
5.	Java (Java 6)	422ms
6.	Java (Java 5 Update 10)	657ms
7.	Java (Java 1.4.2)	680ms
8.	C# (.NET 1.1)	790ms

What I did here was to feed the Sudoku solver with a sparsely populated puzzle that has thousands of valid solutions (Sudoku purists will point out that this is a non-well-formed puzzle). The algorithm uses a slightly optimized brute force approach (my intention was runtime comparison, not building a hyperfast solver algorithm) - so it finds all the solutions. Of course I took care to apply exactly the equivalent language constructs on all platforms, most importantly when it came to dynamic memory allocation.

The usual one-solution, human-solvable puzzles are being processed at a rate of about 60,000 valid solutions per second (C++ version on my 2.4GHz Athlon) - but the time it takes to solve one of those can hardly be measured, and I didn't want to inject the same puzzle a thousand times for the benchmark. There are harder puzzles though with one solution but a larger search space, which means it takes longer to solve them.

And as I have mentioned before: I doubt that there is a lot of optimization potential for Java Hotspot other but compiling the algorithm's main method (which only consists of a few lines of code anyway) to native code as soon as possible. Dynamic memory allocation only happens for storing away solution arrays, and those are not going to be garbage collected until the end, so there is not a lot of optimizing potential on the memory management side. The .NET CLR should JIT all of the code at startup anyway. I did some tests to go sure, and the numbers did not change neither under Java and nor under .NET even after running the same stack of puzzles many times in a row.

Previos Posts:

Follow-Ups:

Hosting .NET Winforms Applications In MSIE