There are two factors working against you learning good commenting technique in college.

1.  Unlike the real world, you do a lot of small one-off projects as a solo developer.  There’s no one out there fantasizing about dropping a boulder on you for making them decipher your coding atrocity.
2.  That commenting style you are emulating from your textbook is only a good practice when the comments are intended for a student learning to program. It is downright annoying to professional programmers.

These tips are primarily intended for upstart programmers who are transitioning into the real world of programming, and hopefully will prevent a few from looking quite so n00bish during their first code review. Code Review? Oh yeah, that’s something else they didn’t teach you in school, but that’s a whole other article, I’ll defer to Jason Cohen on that one.

So let’s get started…

(1) Comments are not subtitles

It’s easy to project your own worldview that code is a foreign language understood only by computers, and that you are doing the reader a service by explaining what each line does in some form of human language. Or perhaps you are doing it for the benefit of that non-programmer manager who will certainly want to read your code (Spoiler: He won’t).

Look, in the not too distant future, you will be able to read code almost as easily as your native language, and everyone else who will even glance at it almost certainly already can. By then you will realize how silly it is to write comments like these:

// Loop through all bananas in the bunch
foreach(banana b in bunch) {
    monkey.eat(b);  //make the monkey eat one banana
}

You may have been taught to program by first writing  pseudo-code comments then writing the real code into that wire-frame. This is a perfectly reasonable approach for a novice programmer. Just be sure to replace the comments with the code, and don’t leave them in there.

Computer: Enemy is matching velocity.
Gwen DeMarco: The enemy is matching velocity!
Sir Alexander Dane: We heard it the first time!
Gwen DeMarco: Gosh, I’m doing it. I’m repeating the darn computer!

-Galaxy Quest

Exceptions:

• Code examples used to teach a concept or new programming language.
• Programming languages that aren’t remotely human readable (Assembly, Perl)

(2) Comments are not an art project

This is a bad habit propagated by code samples in programing books and open source copyright notices that are desperate to make you pay attention to them.

/*
   _     _      _     _      _     _      _     _      _     _      _     _
  (c).-.(c)    (c).-.(c)    (c).-.(c)    (c).-.(c)    (c).-.(c)    (c).-.(c)
   / ._. \      / ._. \      / ._. \      / ._. \      / ._. \      / ._. \
 __\( Y )/__  __\( Y )/__  __\( Y )/__  __\( Y )/__  __\( Y )/__  __\( Y )/__
(_.-/'-'\-._)(_.-/'-'\-._)(_.-/'-'\-._)(_.-/'-'\-._)(_.-/'-'\-._)(_.-/'-'\-._)
   || M ||      || O ||      || N ||      || K ||      || E ||      || Y ||
 _.' `-' '._  _.' `-' '._  _.' `-' '._  _.' `-' '._  _.' `-' '._  _.' `-' '._
(.-./`-'\.-.)(.-./`-'\.-.)(.-./`-'\.-.)(.-./`-'\.-.)(.-./`-'\.-.)(.-./`-'\.-.)
 `-'     `-'  `-'     `-'  `-'     `-'  `-'     `-'  `-'     `-'  `-'     `-'

                 -It's Monkey Business Time! (Version 1.5)
*/

Why, that’s silly. You’d never do something so silly in your comments.

ORLY? Does this look familiar?

   +------------------------------------------------------------+
   | Module Name: classMonkey                                   |
   | Module Purpose: emulate a monkey                           |
   | Inputs: Bananas                                              |
   | Outputs: Grunts                                            |
   | Throws: Poop                                               |
   +------------------------------------------------------------+

Programmers love to go “touch up” their code to make it look good when their brain hurts and they want something easy to do for a while. It may be a waste of time, but at least they are wasting it during periods where they wouldn’t be productive anyway.

The trouble is that it creates a time-wasting maintenance tax imposed on anyone working with the code in the future just to keep the pretty little box intact when the text ruins the symmetry of it. Even programmers who hate these header blocks tend to take the time to maintain them because they like consistency and every other method in the project has one.

How much is it bugging you that the right border on that block is misaligned? Yeah. That’s the point.

(3) Header Blocks: Nuisance or Menace?

This one is going to be controversial, but I’m holding my ground. I don’t like blocks of header comments at the top of every file, method or class.

Not in a boat, not with a goat.
Why? Well let me tell you, George McFly…

They are enablers for badly named objects/methods – Of course, header blocks aren’t the cause for badly named identifiers, but they are an easy excuse to not  put in the work to come up with meaningful names, an often deceptively difficult task. It provides too much slack to just assume the consumer can just read the “inline documentation” to solve the mystery of what the DoTheMonkeyThing method is all about.

JohnFx’s Commandment:
The consumer of thy code should never have to see its source code to use it, not even the comments.

They never get updated: We all know that methods are supposed to remain short and sweet, but real life gets in the way and before you know it you have a 4K line class and the header block is scrolled off of the screen in the IDE 83% of the time. Out of sight, out of mind, never updated.

The bad news is that they are usually out of date. The good news is that people rarely read them so the opportunity for confusion is mitigated somewhat. Either way, why waste your time on something that is more likely to hurt than help?

JohnFx’s Maxim of Plagiarized Ideas :
Bad Documentation is worse than no documentation.

Exception: Some languages (Java/C#) have tools that can digest specially formatted header block comments into documentation or Intellisense/Autocomplete hints. Still, remember rule (2) and stick to the minimum required by the tool and draw the line at creating any form of ASCII art.

(4) Comments are not source control

This issue is so common that I have to assume that programmers (a) don’t know how to use source control; or  (b) don’t trust it.

Archetype 1: “The Historian”

     // method name: pityTheFoo (included for the sake of redundancy)
     // created: Feb 18, 2009 11:33PM
     // Author: Bob
     // Revisions: Sue (2/19/2009) - Lengthened monkey's arms
     //            Bob (2/20/2009) - Solved drooling issue

     void pityTheFoo() {
          ...
     }

The programmers involved in the evolution of this method probably checked this code into a source control system designed to track the change history of every file, but decided to clutter up the code anyway. These same programmers more than likely always leave the Check-In Comments box empty on their commits.

I think I hate this type of comment worst of all, because it imposes a duty on other programmers to keep up the tradition of duplicating effort and wasting time maintaining this chaff. I almost always delete this mess from any code I touch without an ounce of guilt. I encourage you to do the same.

Archetype 2: “The Code Hoarder”

     void monkeyShines() {
          if (monkeysOnBed(Jumping).count > max) {
             monkeysOnBed.breakHead();

             // code removed, smoothie shop closed.
             // leaving it in case a new one opens.
             // monkeysOnBed.Drink(BananaSmoothie);
          }
     }

Another feature of any tool that has any right to call itself a SCM is the ability to recover old versions of code, including the parts you removed. If you want to be triple super extra sure, create a branch to help you with your trust issues.

(5) Comments are a code smell

Comments are little signposts in your code explaining it to future archaeologists that desperately need to understand how 21st century man sorted lists of purchase orders.

Unfortunately, as Donald Norman explained so brilliantly in The Design of Everyday Things, things generally need signs because their affordances have failed. In plain English, when you add a comment you are admitting that you have written code that doesn’t communicate its purpose well.

Sign:"This is a mop sink." Why would that be necces... oh.

Despite what your prof told you in college, a high comment to code ratio is not a good thing.  I’m not saying to avoid them completely, but if you have a 1-1 or even a 5-1 ratio of LOC to comments, you are probably overdoing it. The need for excessive comments is a good indicator that your code needs refactoring.

Whenever you think, “This code needs a comment” follow that thought with, “How could I modify the code so its purpose is obvious?”
Talk with your code, not your comments.

Technique 1: Use meaningful identifiers and constants (even if they are single use)

     // Before
     // Calculate monkey's arm length
     // using it's height and the magic monkey arm ratio
     double length = h * 1.845; //magic numbers are EVIL!

    // After - No comment required
    double armLength = height * MONKEY_ARM_HEIGHT_RATIO;

Technique 2: Use strongly typed input and output parameters

      // Before
      // input parameter monkeysToFeed:
      // DataSet with one table containing two columns
      //     MonkeyID (int) the monkey to feed
      //     MonkeyDiet (string) that monkey's diet
      void feedMonkeys(DataSet monkeysToFeed) {
      }

     //  After: No comment required
     void feedMonkeys(Monkeys monkeysToFeed) {
     }

Technique 3: Extract commented blocks into another method

      // Before

      // Begin: flip the "hairy" bit on monkeys
      foreach(monkey m in theseMonkeys) {
          // 5-6 steps to flip bit.
      }
      // End: flip the "hairy" bit on monkeys

     // After No comment required
     flipHairyBit(theseMonkeys);

As an added bonus, technique 3 will tend to reduce the size of your methods and minimizing the nesting depth (see also “Flattening Arrow Code”) all of which contribute to eliminating the need for commenting the closing tags of blocks like this:

            } // ... if see evil
         } // ... while monkey do.
      } // ... if monkey see.
    } // ... class monkey
  } // ... namespace primate

Acknowledgments

Several of the ideas presented here, and a good deal of the fundamental things I know about programming as part of a team, and not as a lone-wolf working on a college project I learned from the book Code Complete by Steve McConnell. If you are a working programmer and have not read this book year, stop what you are doing and read it before you write another line of code.

Do you ever wish you could use the SQL IN operator in your C# code to make your conditional blocks more concise and your code easier to read?

Perhaps it’s just my persnickety nature, but I believe that line-wrapped conditional expressions like this are a code smell.

if (animal.Equals("Cow") ||
   animal.Equals("Horse") ||
   animal.Equals("Hen"))
{
   Console.WriteLine("We must be on the farm.");
}

This would be so much cleaner…

if (animal.CompareMultiple("Cow","Horse","Hen")
{
   Console.WriteLine("We must be on the farm.");
}

The Code

With a simple extension class you can upgrade your string classes to do this very thing.

Step 1: Create an extension class as demonstrated here.

C#

namespace extenders.strings
{
  public static class StringExtender {

    public static bool CompareMultiple(this string data, StringComparison compareType, params string[] compareValues) {
      foreach (string s in compareValues) {
        if (data.Equals(s, compareType)) {
          return true;
        }
      }
      return false;
   }
  }
}

VB.NET

Imports System.Runtime.CompilerServices

Namespace Extenders.Strings

  Public Module StringExtender

    <Extension()> _
    Public Function CompareMultiple(ByVal this As String, compareType As StringComparison, ParamArray compareValues As String()) As Boolean

       Dim s As String

       For Each s In compareValues
         If (this.Equals(s, compareType)) Then
            Return True
         End If
       Next s

       Return False
    End Function

  End Module

End Namespace

Step 2: Add a reference to the extension namespace and use it.

C#

using extenders.strings;

namespace MyProgram
{
  static class program {
    static void Main() {

      string foodItem = "Bacon";

      if (foodItem.CompareMultiple(StringComparison.CurrentCultureIgnoreCase, "bacon", "eggs", "biscuit")) {
         System.Console.WriteLine("Breakfast!");
      }
      else {
         System.Console.Write("Dinner");
      }

    }
  }
}

VB.NET

Imports StringExtenderExampleVB.Extenders.Strings

Module Program

  Sub Main()
    Dim foodItem As String = "Bacon"

    If (foodItem.CompareMultiple(System.StringComparison.CurrentCultureIgnoreCase, "bacon", "eggs", "biscuit")) Then
      System.Console.WriteLine("Breakfast!")
    Else
      System.Console.Write("Dinner")
    End If

    System.Console.ReadLine()
  End Sub

End Module

I’m happy to announce that I’ve released a new jQuery plugin today that will enable web page authors to create a richer user experience than is offered by the standard implementation of the SELECT element in poplar browser platforms. The plugin will supercharge the dropdown controls into combo-box style controls that allow filtering the selection list based on keyboard input.

The following two screen shots tell the story of this plugin better than a 100 words ever could.

Headings and disabled items are fully supported.

This list is filtered to only matching items as the user types.

Another auto-complete plugin? Why?

Okay. I am fully aware that there exist a number of freely available jQuery plugins with similar functionality. However, none of them seemed to fit exactly what I needed on one of my development projects so I built ImprovedDropDown originally for my own use, and later decided it was worth sharing with the developer community.

Key differences between ImprovedDropdown and the typical auto-complete plugin:

1. It allows free-text entry to filter the list, but forces the user to pick an item that is in the list.
2. When filtering/suggesting items it uses open-ended contains logic, not begins-with logic.
3. It pulls the suggestion list from plain-old client side HTML rather than requiring an AJAX call for suggestions.
4. It supports non-selectable group headers in the list similar to an OPTGROUP tag in a SELECT control.

Item #3 at first glance may look like a step back since AJAX is a cool way to serve up suggestions for a long list of choices. However, I was looking for a super simple way to add functionality to a dropdown without adding the additional complexity of  extra server-side code.

For small to medium sized lists it seemed overkill to implement AJAX handlers on the server all the time. As a bonus, this control will work just fine for static web-sites where the hosting provider doesn’t make available a server-side scripting capabilities.

Do I have to pay for it?

Nope. I’m releasing this project as open-source freeware as a thank you to all the other developers that have contributed greatly to many of my development projects by  contributing their own code to the community. I’ve licensed it under the Apache 2.0 License, which is fairly liberal in terms of what you can do with the code either in commercial or non-commercial software.

Although it isn’t mandated by the license, the one thing I ask in return for using the software is to let me know that you are using it. A shout-out on twitter (@JohnFx), an e-mail, blog post, or comment somewhere on this blog will suffice.  You don’t have to tell me any specifics of your project, unless you just want to brag on it. Primarily, I am just using this request as a way to gauge how much use it is actually getting in the wild.

Is it complicated to use?

If you are new to jQuery, there is a small learning curve there, but even if you don’t know jQuery you can pretty much just copy the included examples to get up and running.

Step 1: Copy the files to your site (jQuery + the 3 ImprovedDropdown files)

Step 2: Add a few references to the HEAD section of the page where you want the control.

Step 3: In the $(document).ready event, call the improveDropdown method to upgrade one or more controls on that page.

Example code (steps 2-3) to upgrade all SELECT elements on the  page to Improved Dropdowns:

<head>
<script type="text/javascript" src="js/jquery.js"></script>
<script type="text/javascript" src="js/improvedDropDown.js"></script>
<link rel="StyleSheet" href="css/improvedDropdown.css" type="text/css" />

    <script type="text/javascript">
       $(document).ready(function () {
            $("select").improveDropDown();
       });
    </script>

</head>

Sounds Great. Where can I get it?

You can download all the files you need to get started from the Improved Dropdown page in the jQuery plugin library.

The project is also hosted on the Improved Dropdown project page at BitBucket (Go Mercurial!) in case you’d like to contribute, follow the project, or just laugh at all the bug-fix changesets that it took to get to the 1.x version.

What do you think?

I put considerable thought into making  the integration of this functionality into existing web pages super quick and easy. I hope that you will find it easy to work with and your users will like the functionality it offers. If you have any comments, complaints, or suggestions for improvement I would love to hear them. Just post a comment somewhere on the blog or drop me an e-mail (see “About” tab for my address).

So today’s project was to find another way to skin that cat without  inserting bogus characters into the data.

Statement of The Problem

For those who haven’t read Jeff’s article, here is a basic statement of the task:

Replace any continuous series of repeating spaces of in a database column with a single space.

Constraints:

1. Although a CLR UDF using .NET’s regular expression library is the most straightforward way to do this, the original article went for a pure SQL approach so I did the same.
2. My solution is based language features available in MS SQL Server 2005 and later.

Test Data

I used a permanent table to test this instead of a temporary table, but used the same sample data set from Jeff’s article.  Also, I inserted these same rows 100K times so I had a large enough data set to be useful for performance testing.

CREATE TABLE SpacesTest(
ID int IDENTITY(1,1) NOT NULL,
SomeText varchar(max) NULL,
)

go
DECLARE @loopCounter int
set @test=0

WHILE (@loopCounter<100000)
BEGIN
  INSERT INTO spacestest (sometext)
    SELECT '  This      has multiple   unknown                 spaces in        it.   '
    UNION ALL SELECT 'So                     does                      this!'
    UNION ALL SELECT 'As                                does                        this'
    UNION ALL SELECT 'This, that, and the other  thing.'
    UNION ALL SELECT 'This needs no repair.'
END

Solutions

If we could assume that no series of spaces in the string consisted of more than two spaces, this challenge could be met with a simple application of the REPLACE statement to replace all double spaces with single spaces. However, given series of arbitrary length you need would need to repeat this operation until you had collapsed the longer series down to single spaces.

You could do it iteratively like this:

WHILE EXISTS(SELECT * FROM SpacesTest WHERE (SomeText like '%  %'))
BEGIN
 UPDATE SpacesTest
 SET SomeText=REPLACE(SomeText,'  ',' ')
 WHERE (SomeText like '%  %')
END

While the iterative approach works just fine, it does do of repetitive updates to each row and requires you to write the intermediate output to a temporary table if you just want to query the data rather than updating the source table.  I came up with the following recursive solution that doesn’t require the intermediate updates and is more of  “set based” approach.

Recursive Solution:

WITH spacesCollapsed (ID,SomeText) AS
(
   SELECT ID, cast(LTRIM(RTRIM(SomeText)) as varchar(max)) FROM SpacesTest
   UNION ALL
   SELECT ID, cast(REPLACE(SomeText,'  ',' ') as varchar(max)) as SomeText FROM spacesCollapsed WHERE (SomeText like '%  %')
)
SELECT ID, SomeText
FROM (
   SELECT ID,SomeText,  ROW_NUMBER() OVER(PARTITION BY ID ORDER BY min(len(sometext))) as GroupRowNum
   FROM spacesCollapsed
   GROUP BY ID,SomeText
 ) as reductions
WHERE reductions.GroupRowNum=1;

Conclusions

My new recursive approach did generate valid output and does solve the dilemma of choosing a character that won’t occur in the data. The bad news is that it didn’t perform all that well. In fact, it was over 26x slower than Jeff’s approach. The iterative approach performed a lot better, but was still almost 6x slower than the benchmark. I suspect the performance problems with both of my alternatives come from the need to make multiple passes at the source data using a non-SARGable expression in the where clause.

Jeff's Approach (for comparison)

Recursive Approach (Ugh!)

So, although it was a fun exercise, I have to concede this one. If performance is a major consideration (and when isn’t it?) then Jeff actually has a superior technique. Here is an example query using that technique, just be careful to understand the potential side-effects if you use it.

 SELECT REPLACE(REPLACE(REPLACE(LTRIM(RTRIM(sometext)),'  ',' '+ CHAR(7)) , CHAR(7)+' ',''), CHAR(7),'') AS CleanString
 FROM SpacesTest
 WHERE CHARINDEX('  ',sometext) > 0

Feedback

Given how dramatic the difference is for a similar operation, I have to think that maybe I am missing an optimization on my recursive solution that could reign in the performance. If you see anything I’m missing that might help my solution in terms of performance please let me know in the comments.

This week I got a request troubleshoot a legacy Microsoft Access application that has been floating around our company for ages, but still gets used daily because dang it, it does the job and always has. Seems like most companies that are standardized on MS Office have a few of these lurking out on the network.

Earlier in my career I did a ton of work in MS Access and have garnered a reputation within my company for being an expert in this oft maligned platform so I got the call to look into the problem. It had been quite a while since I’d done any real work on MS Access and I’d forgotten about how quirky it could be. Also, I am more than a little disappointed at how Microsoft has mangled the UI of my old friend Access in the 2007 version. It is almost painful to work with it as a power-user in the current incarnation.

The Problem

So anyway, the issue turned out to be that someone increased the length of a field in the underlying SQL Server table linked into the Access application. They increased it past the magical border (255 characters) between what Access considers a text and a memo field, which imposed new limits on how it could be used. In particular, Access doesn’t allow either end of a join in a query to be a memo field.

This won't fly, McFly

The Solution

The solution is painfully simple. So much so that I have to wonder why Access doesn’t just do it behind the scenes. Perhaps it is just trying to discourage you from building databases that link on big text fields for your own good (see “The Caveat” below).

The trick is to move the join into the WHERE clause of the query  like so:

SELECT Table1.*, Table2.*
FROM Table1, Table2
WHERE (Table1.MemoField=table2.MemoField);

Here is the same query in the query builder for those of you who prefer it to the SQL view:

Remove the join between the tables and add a criterion

Access will raise nary a complaint if you run this query which is logically equivalent to the one it abhorred. That’s all there is to it.

The Caveat

A final note. It is a definite database smell for an application to be joining tables on long text fields and will likely be the source of some performance issues in a database of non-trivial size. However, as was the case for the application I was tweaking, joining on long text fields is sometimes necessary in queries used for data clean-up, validation, or replication.  Still, use this type of join with caution avoiding it whenever possible.

Q: How do you know when your software has a usability problem?

A: When someone creates an entire humor site dedicated to mocking it.

Okay, you might have said something involving reports from the usability testing, or maybe even some axiomatic list of UX smells. However, if you ever find that a piece of software you wrote has inspired an entire new category of parodic humor, there just might be a problem.

How has it come to this?

The emergence of this UX gaffe leaves me incredulous for a number of reasons. First, one of the biggest offending implementations is baked into iOS, a product of a company that gets a lot of respect for designing great interfaces. Secondly, and more importantly, most of the Mobile OS platforms with this issue have gone through several major upgrades and not fixed the problem despite all the bad press about it.

Fail x 439,000

What’s the Problem?

So to reel this conversation back in, and make it instructional and not just another ranting voice on the Internet that the developers of this software will ignore, let’s talk about what it is about the auto-correct feature on your cell phone that makes it such a usability disaster.

The core problem is that it violates the usability principle that the user should always feel like they are in control of the software.

The users of your software should be the drivers, not the passengers. Ask Toyota how their ‘users’ felt about their products deciding independently that more speed might be nice .

If you are going to make your spelling checker jump in and autonomously decide it knows what the typist meant to say, it had better be right 100% of the time.  Is that bar too high? Well then stop making your app take over the controls. A well behaved app is heard and not stomping all over my carefully written prose and/or review of the wing place where I am currently receiving inferior service.

Not only does this implementation of auto-correct make the user feel impotent, it also violates the usability commandment that the user’s input is sacred. Violations of this principle contribute mightily to the computer-phobic person’s notion that the machine is going all SkyNet on them.

Extra Credit: Why do you think applications like MS Office that move your menu items around by usage frequency so infuriating? How about browser pop-ups?

Reinventing the Wheel – Let’s try square this time!

This would all be more excusable if a perfectly user-friendly auto-correct interface didn’t already exist. What’s more aggravating is that the existing workable system is so close to the new broken version that it is inconceivable that the designers of the new version weren’t familiar with the existing one.

Consider for a moment how auto-correct is implemented in MS Office, Blackberry OS, and the Firefox spell-checker. In almost identical fashion it gives you a list of suggested alternatives for the suspected misspelled word.  The key difference is that it SUGGESTS, but doesn’t CORRECT the user’s text unless the user takes affirmative action. This puts the user fully in control of the system, which makes them happy. Sure some typos may slip by the user, but if you are giving good enough visual indicators that should be minimized.

If only auto-correct had an anti-troll feature...

Is Anyone Listening?

Perhaps I’ve completely missed the point here, and there is a good reason why something that was routine on my old Blackberry is nigh impossible on the other smart phone platforms. If you have any inside scoop, or just have a convincing theory to share I’d love to hear about it in the comments. If you just want to complain about it, that’s fine too, maybe we can make this square wheel squeaky enough to get some satisfaction.

But it works on race cars....

I’m not writing this article to chastise the fledgling programmer who has fallen into this trap, nor is this intended as a how-to article on optimizing your code. Heaven knows there the internet is lousy with articles about the most efficient way to cram strings together. I will address the problems associated with some of the mythology about string concatenation, but my primary goal will be to encourage critical thinking and healthy skepticism for silver bullet programming techniques.

The Symptom

Though I possess no paranormal super-powers, I do believe I can read the mind of another human when I see code that looks like this:

    StringBuilder WelcomeMessage = new StringBuilder();
    WelcomeMessage.Append("Hello ");
    WelcomeMessage.Append(firstName);
    WelcomeMessage.Append(" ");
    WelcomeMessage.Append(LastName);
    WelcomeMessage.Append("!\n")

My spirit guide informs me that that programmer responsible for this code remembers reading somewhere that <insert programming language> is really inefficient at concatenating strings, but you can overcome that limitation by using the StringBuilder class. Based on this information he/she replaces every string concatenation operator with this clever technique leaving a scent trail through the code that experienced programmers can smell from miles away.

Problem 1: Premature Optimization is Procrastination

Sure, you want ALL of your code to perform well, but experienced programmers understand that their time is valuable and best spent on activities that deliver actual business value. Premature optimization and its ugly cousin Micro Optimization are almost always a waste of time. I understand how tempting it is to justify in your head that you could squeeze a little bit more performance out of your app by re-factoring the whole thing you learned in the blog post you read today, especially since it can be like a mini-vacation for your mind from the really complex issues you really should be working on, but be strong and resist!

As a rule of thumb: If it isn’t worth creating a jig to profile the performance gains you expect to get from optimization re-factoring, then it isn’t worth the time to do the re-factoring in the first place, plus it is risky because you won’t notice that your supposed “optimization” actually hurt performance.

More on that later.

Problem 2: Cookie Cutter Optimizations Assume the Compiler is Stupid

If you could universally make string concatenation faster by applying a simple formula then the compiler would probably already be applying the transformation anyway. Granted, I think this point is lost on some novice programmers who only have experience in higher level languages.

For them, I’ll clarify with this point.

Your program is running the compiler’s interpretation of your code. Not your actual code.

With that in mind, to think that the StringBuilder approach always runs faster would require you to believe that the people who wrote the programming language were smart enough to make string concatenation fast when they created the StringBuilder class, but forgot how to do it when they built the concatenation operator.

Pop Quiz: Does this code give you any heartburn? Why?

    string querySQL = "SELECT * " +
                      "FROM myTable " +
                      "WHERE (ID=5)";

If you said yes because it isn’t worth incurring the cost of concatenation for code readability then you aren’t giving the compiler enough credit.

Here is the MSIL output for the above statement:

ldstr      "SELECT * FROM myTable WHERE (ID=5)"

Amazing, huh?

Compilers are written to do the complex task of reading your code and interpreting what it means. Figuring out that a series of constant strings can be combined is child’s play.

Problem 3: If you don’t understand it, you’ll do it wrong

Cargo Cult programming is a derisive term for doing things in your program because you think you need to, but don’t understand (or have a vague notion of)  the underlying reason. It is really bad practice to adopt a technique without asking enough “why” questions to grasp the reason using it is desirable.

As an example, let’s dissect the premise that string concatenation using operators is slow and should be replaced by StringBuilders.

Q: Why do some claim that string concatenation with operators is slow?
In many garbage collected languages (Java/.NET) string objects are  immutable, meaning you can’t change them. So when you append more content into an existing string the program must internally create a new string and copy the old and new contents into it. The extra effort to create, destroy and garbage collect the extra string objects has the potential to create more work for your program and can degrade performance if done excessively.

Q: How does the StringBuilder help?
The StringBuilder class is implemented as a mutable memory buffer that typically has extra unused space allocated so that concatenations can be made in place without the need to create extra objects to juggle the data.

Q: How much extra space does it reserve? What happens if I append more content than will fit in the unused space?
By default (in .NET) 16 characters, unless you specify differently in the constructor. If you append more data than there is space, the StringBuilder will behave much like a String object creating a new StringBuilder object with double the existing capacity then copy over the data.

You: Wait, what?

You mean that you have been using StringBuilder with the default constructor and then appending more than 16 characters to it?

Yeah, well if you are lucky you’ll be no worse off than if you just used the “evil” string concatenation operators. However, due to that neat capacity doubling side-effect, your program might actually be locking up unnecessarily large chunks of memory on top of the additional work required to wrangle all the intermediary objects. Perhaps it is worth investigating and setting the initial capacity of that StringBuilder to avoid such nastiness.

Bonus: Now that you understand the potential performance benefit is based (at least partly) on mutability, you will see that other string optimization opportunities may exist whenever existing strings need to be modified, not just appended to.

Final Thoughts

Again, the point of all this isn’t about strings, or optimization, or any of that. It is about taking the time to understand what you are doing to avoid falling prey to the potentially harmful myths that are enthusiastically passed around by programmers (see also “The database sorts by the clustered index if you don’t specify an Ordering“).

In any event, I’m curious as to how many of my readers actually have at one time subscribed to the cargo cult programing meme of “StringBuilder is always better”. Please let me know in the comments.

In my previous article, Collaborative Problem Solving, I talked about the importance of questioning assumptions and reconfirming your understanding of the problem when you hit a brick wall during design or troubleshooting of a system. In fact, one of my favorite strategies for tackling a tough design problem is take the  nihilist approach. In the words of the Zen Master…

Remember that there is no code faster than no code.

- Taligent’s Guide to Designing Programs

My corrolary to this is that there is also no code simpler than no code. So when confronted with a complex design challenge or optimization problem, my first question is always “How can we get away without implementing this at all?” Granted, this only works about 20% of the time, but when we can step back and realize that a particularly pernicious problem has just vaporized it is a wonderful thing.

I’m sure that some may be  incredulous about the idea that this ever works at all. I mean, why would anyone have even started working on a problem that didn’t need to be solved? The answer is that only in rare cases does this approach reveal that the entire project was unnecessary, although I have that happen. More often it just reveals that the developer has ventured too far down a dead-end path and is working exclusively on meta-problems that can be eliminated by just backing up a few steps.

For example, this Stack Overflow Question, is a good example of a meta-problem that can be eliminated by taking a step back and reconsidering how to avoid throwing errors instead of how to ignore them as explained in this answer.

Re-evaluating the Problem

What got me started on this two day rant about problem solving approaches was this really insightful idea for solving some of the core issues with implementing an voting system. It was a really great example of how re-examining the core problem led to a solution to a seemingly intractable problem.  Here’s a quick breakdown on the dilemma.

Goals: An voting system must satisfy these criteria (among others)

1. Voters must be able to anonymously cast their votes.
2. Voters can cast zero or one ballot, but never more than one.
3. Voters must not be able to prove who they voted for (to preclude vote buying).
4. Voters should be able to confirm their vote was properly recorded for each candidate they picked.

Problem: If a voter can confirm who they voted for then they have the means to prove (and receive remuneration) for voting for a particular candidate.

The problem that has stumped many, including myself in analyzing this problem is that requirements (3) and (4) are apparently incompatible.  How can you give the person a way to confirm their vote without being able to prove their vote to another person?

The Solution: The solution that David Bismark came up with was fairly straightforward once he realized that we were looking at requirement (4) incorrectly. Consider this clarification of that requirement.

4. Voters should be able to confirm their vote was properly recorded for each candidate they picked the way they marked it.

Under this revised understanding of the problem, the solution became more straightforward. Essentially his solution involves randomizing the order of the names on the ballot, after voting the ballot is torn in half and the names part is shredded. The bubble-candidate correlation is encrypted on the ballot to prevent tampering, but allow the vote to be tabulated.

The voter then can check (perhaps online) that the bubbles they marked were counted, but no longer see which candidate corresponds with each bubble. Assuming they remember the order of the candidates, they can confirm their voted counted but not prove who they voted for. Voila! Brilliant!

I’m sure everyone reading this can remember an instance where you went to ask someone a question and simply the act of asking it aloud made you realize the answer. Consider for a moment why this technique works, and why it often works better when the other person has little advance knowledge of  the issue. To bring that other person into the discussion you are forced to rewind your brain back to when you started working on the problem eschewing all of the clutter and complexity that you have added since you’ve been working on it. Then you must revisit each of the steps you have taken justifying each decision and restating each assumption.

So why should revisiting the history of the problem be effective? After all, having the same person answer the same intermediate questions ought to bring that person to the same conclusion, right? If this were true, then the sounding board approach would be a waste of time, which we know is not the case from anecdotal evidence. The reason you arrive at a different answer is that unlike the first time through, you now have the benefit of foresight and are far less likely to take  a path that you know will lead you through the brambles.

The longer you walk down those brambly paths the easier it is to mentally commit to pressing forward. On an intellectual level we understand the sunk cost fallacy, but are still susceptible to it and can form mental blocks that make it discomforting to backtrack for more than one or two forks in the road. This is why we need the guide to force us to rethink the whole journey and each decision along the way.

Tips for Being a Good Sounding Board

I'm Listening.

Face it, there are times when a poster of Justin Bieber would be exactly as productive as a sounding board as you or anyone else on the team. The core problem is sometimes just so obvious that saying it out loud is enough to expose the flaws in the current approach. I’m going to save the advice on how to be more like Justin for a future article and focus now on how you can be a better guide for helping out on the thornier problems.

1. Establish yourself as a devil’s advocate – The rest of these tips taken in the wrong context can easily put the person defensive. Establish that you are going to ask a bunch of questions, some naive, some pointed, but that your goal is to help them re-verify their assumptions, not to promote an alternate idea or criticize their approach.
   
2. Be Kind, Rewind - Even if you know the back-story, ask lots of “why” questions and continue pushing back towards the root problem. Help them fight the tendency to revisit only the last few decisions. “You are de-normalizing the table for performance? Why did you need to do that?”
3. Don’t let them linger on meta-problems – When a problem seems too big to hold in your brain, it’s tempting to seek respite by focusing on manageable meta-problems. While those problems may need solving, they are distractions from the real issue that is blocking progress. When the person lingers on details or raises meta-problems, keep rewinding.
4. Ask Probing Questions - Make them talk through each decision and ask stupid questions. Apply extra scrutiny when the developer clearly thinks a decision was easy or obvious. When we think things are obvious, we take mental shortcuts by avoiding thinking things through as thoroughly. Challenge assumptions, this is where problems hide.
5. Use Reflective Listening – This is a communication technique where you repeat back  a summary of what the other person just said to you to confirm understanding. Another benefit in this situation is that having the person hear their own ideas in another person’s voice/words may make it easier for them to be objective.
6. Avoid injecting your own ideas – At some point you may have a great idea for a better approach. Keep it to yourself. It will just make them defensive and will undermine their sense of ownership of the problem, and inhibit their ability to understand the solution.
7. Lead them to the answer – If they simply aren’t making progress and you know a good answer consider  leading them to the answer with a line of questioning that directs them instead of just hand feeding the information to them. They are more likely to take ownership if they feel they reached the conclusion, and people generally retain information more readily if they arrived at it by logic on their own. If you must resort to this, tread lightly with your tone and take extra care not to come off as pedantic.

It may seem strange that my approach advocates being somewhat evasive with information or potential solutions. However, it is given in the context of a manager/mentor acting as a sounding board. In this role, your primary focus should be to create leverage by making things happen through your team. Any veteran manager will tell you that handing out answers routinely to technical problems is only going to make the line to your office longer and make you the bottleneck.  I strongly favor approaches that encourage critical thinking among developers and gives the glory to the developer instead of the esteemed leader. Trust me, it pays off in the long run.

He claimed they knew what needed to be done and did it without any official methodology. In these parts we might call that “Cowboy Coding.” His premise that following a standard methodology was meant for novice teams or those with immature programmers who hadn’t gotten “into the groove” of their careers.

Giddyup!

Before you trash the guy,  consider the following:

• The guy and his team really were hardcore. It was chock full of developers that I’d love to be able to lure over to my shop.
• This guy had bootstrapped several start-up organizations and is extremely technically competent for a suit.
• He based this statement on demonstrable results, working software that apparently was popular with his customers.

How can you argue with results? Of course, I’m not absolutely sure about the veracity of his claims of success. In the context of our conversation it was conceivable he might have been exaggerating given that I currently work for a company that is a potential customer for his company’s product.

Personally, and despite these points I still tend to disagree. Even though he may be seeing success despite an unstructured development environment. I’d argue that the team may still be performing below their full potential and would still benefit from some flavor of an Agile approach.

It seems to me that Agile is better suited to teams with a lot of talent and experience, and if anything would be more problematic with a bunch of rookies. A core Agile concept is to let the development team self-organize, which implies a great deal of trust.  Agile is not about babysitting, and certainly not about command-and-control management. It is about communication, coordination, and frequent re-alignment with an emerging picture of the customer’s needs.

I’m curious what you guys think?