Problem Solving 101: First, Understand the Problem

The Null Solution

In my previous article, Collaborative Problem Solving, I talked about the importance of questioning assumptions and reconfirming your understanding of the problem when you hit a brick wall during design or troubleshooting of a system. In fact, one of my favorite strategies for tackling a tough design problem is take the  nihilist approach. In the words of the Zen Master…

Remember that there is no code faster than no code.

Taligent’s Guide to Designing Programs

My corrolary to this is that there is also no code simpler than no code. So when confronted with a complex design challenge or optimization problem, my first question is always “How can we get away without implementing this at all?” Granted, this only works about 20% of the time, but when we can step back and realize that a particularly pernicious problem has just vaporized it is a wonderful thing.

I’m sure that some may be  incredulous about the idea that this ever works at all. I mean, why would anyone have even started working on a problem that didn’t need to be solved? The answer is that only in rare cases does this approach reveal that the entire project was unnecessary, although I have that happen. More often it just reveals that the developer has ventured too far down a dead-end path and is working exclusively on meta-problems that can be eliminated by just backing up a few steps.

For example, this Stack Overflow Question, is a good example of a meta-problem that can be eliminated by taking a step back and reconsidering how to avoid throwing errors instead of how to ignore them as explained in this answer.

Re-evaluating the Problem

What got me started on this two day rant about problem solving approaches was this really insightful idea for solving some of the core issues with implementing an voting system. It was a really great example of how re-examining the core problem led to a solution to a seemingly intractable problem.  Here’s a quick breakdown on the dilemma.

Goals: An voting system must satisfy these criteria (among others)

  1. Voters must be able to anonymously cast their votes.
  2. Voters can cast zero or one ballot, but never more than one.
  3. Voters must not be able to prove who they voted for (to preclude vote buying).
  4. Voters should be able to confirm their vote was properly recorded for each candidate they picked.

Problem: If a voter can confirm who they voted for then they have the means to prove (and receive remuneration) for voting for a particular candidate.

The problem that has stumped many, including myself in analyzing this problem is that requirements (3) and (4) are apparently incompatible.  How can you give the person a way to confirm their vote without being able to prove their vote to another person?

The Solution: The solution that David Bismark came up with was fairly straightforward once he realized that we were looking at requirement (4) incorrectly. Consider this clarification of that requirement.

4. Voters should be able to confirm their vote was properly recorded for each candidate they picked the way they marked it.

Under this revised understanding of the problem, the solution became more straightforward. Essentially his solution involves randomizing the order of the names on the ballot, after voting the ballot is torn in half and the names part is shredded. The bubble-candidate correlation is encrypted on the ballot to prevent tampering, but allow the vote to be tabulated.

The voter then can check (perhaps online) that the bubbles they marked were counted, but no longer see which candidate corresponds with each bubble. Assuming they remember the order of the candidates, they can confirm their voted counted but not prove who they voted for. Voila! Brilliant!

Problem Solving 101: Collaborative Problem Solving

As a software development manager, I am frequently visited by developers who are spinning their wheels on a design problem or running out of ideas while troubleshooting an application.  I’ve developed a reputation as a problem solver because most of these visits end with a eureka moment that clears the logjam for the developer. Although I do like to think that I’m a reasonably good troubleshooter/architect,  I have accrued quite a bit of unearned reputation from these impromptu brainstorming sessions as a problem solver. A more fair assessment of my contribution might be that I am just a really good sounding board.

I’m sure everyone reading this can remember an instance where you went to ask someone a question and simply the act of asking it aloud made you realize the answer. Consider for a moment why this technique works, and why it often works better when the other person has little advance knowledge of  the issue. To bring that other person into the discussion you are forced to rewind your brain back to when you started working on the problem eschewing all of the clutter and complexity that you have added since you’ve been working on it. Then you must revisit each of the steps you have taken justifying each decision and restating each assumption.

So why should revisiting the history of the problem be effective? After all, having the same person answer the same intermediate questions ought to bring that person to the same conclusion, right? If this were true, then the sounding board approach would be a waste of time, which we know is not the case from anecdotal evidence. The reason you arrive at a different answer is that unlike the first time through, you now have the benefit of foresight and are far less likely to take  a path that you know will lead you through the brambles.

The longer you walk down those brambly paths the easier it is to mentally commit to pressing forward. On an intellectual level we understand the sunk cost fallacy, but are still susceptible to it and can form mental blocks that make it discomforting to backtrack for more than one or two forks in the road. This is why we need the guide to force us to rethink the whole journey and each decision along the way.

Tips for Being a Good Sounding Board

I'm Listening.

Face it, there are times when a poster of Justin Bieber would be exactly as productive as a sounding board as you or anyone else on the team. The core problem is sometimes just so obvious that saying it out loud is enough to expose the flaws in the current approach. I’m going to save the advice on how to be more like Justin for a future article and focus now on how you can be a better guide for helping out on the thornier problems.

  1. Establish yourself as a devil’s advocate – The rest of these tips taken in the wrong context can easily put the person defensive. Establish that you are going to ask a bunch of questions, some naive, some pointed, but that your goal is to help them re-verify their assumptions, not to promote an alternate idea or criticize their approach.
  2. Be Kind, Rewind – Even if you know the back-story, ask lots of “why” questions and continue pushing back towards the root problem. Help them fight the tendency to revisit only the last few decisions. “You are de-normalizing the table for performance? Why did you need to do that?”
  3. Don’t let them linger on meta-problems – When a problem seems too big to hold in your brain, it’s tempting to seek respite by focusing on manageable meta-problems. While those problems may need solving, they are distractions from the real issue that is blocking progress. When the person lingers on details or raises meta-problems, keep rewinding.
  4. Ask Probing Questions – Make them talk through each decision and ask stupid questions. Apply extra scrutiny when the developer clearly thinks a decision was easy or obvious. When we think things are obvious, we take mental shortcuts by avoiding thinking things through as thoroughly. Challenge assumptions, this is where problems hide.
  5. Use Reflective Listening – This is a communication technique where you repeat back  a summary of what the other person just said to you to confirm understanding. Another benefit in this situation is that having the person hear their own ideas in another person’s voice/words may make it easier for them to be objective.
  6. Avoid injecting your own ideas – At some point you may have a great idea for a better approach. Keep it to yourself. It will just make them defensive and will undermine their sense of ownership of the problem, and inhibit their ability to understand the solution.
  7. Lead them to the answer – If they simply aren’t making progress and you know a good answer consider  leading them to the answer with a line of questioning that directs them instead of just hand feeding the information to them. They are more likely to take ownership if they feel they reached the conclusion, and people generally retain information more readily if they arrived at it by logic on their own. If you must resort to this, tread lightly with your tone and take extra care not to come off as pedantic.

It may seem strange that my approach advocates being somewhat evasive with information or potential solutions. However, it is given in the context of a manager/mentor acting as a sounding board. In this role, your primary focus should be to create leverage by making things happen through your team. Any veteran manager will tell you that handing out answers routinely to technical problems is only going to make the line to your office longer and make you the bottleneck.  I strongly favor approaches that encourage critical thinking among developers and gives the glory to the developer instead of the esteemed leader. Trust me, it pays off in the long run.


Grokking Distributed Version Control Systems

As a longtime fan of  FogBugz, I’ve been dying to start moving some of my personal projects over to Fog Creek’s Kiln. Although I won’t get much use from Kiln’s code review functionality as the lone developer on these projects, I like the idea of storing my code in the same tool that I use to manage the workflow and other meta-information about those applications.

I’d like to say this preference is based on the potential value-add of the integration features between Kiln and FogBugz, but frankly it is more about my core belief in keeping things as simple as possible.  Also, I make a habit of not storing anything that I can’t afford to lose on the hard drive of my development computer where backups are a tad sporadic. So using a hosted source code repository is attractive in that respect also.

I was sold. All I needed to do to kick off “Project Simplify” was to import my source code into Kiln from my Vault repository and … hmmm. It seems that Kiln is based on an SCM tool called Mercurial that I was unfamiliar with. No worries, I’d adapted to new source control tools before, it would just be a matter of translating the jargon from Vaulteese to Mercurian, right?

 

Wrong.

As it happens, distributed version control systems (DVCS) such as Mercurial represent a considerable paradigm shift from their traditional centralized cousins. At least this is what its evangelists are saying. Despite the claim on the Mercurial Home page that it is “Easy to Learn”, all the anecdotal evidence I’ve seen seems to indicate a significant learning curve.  However, it also appears that once DVCS finally clicks, it is extremely popular with developers. That’s enough for me, I’m still on board, but cautiously optimistic.

So I downloaded and installed the “Kiln Client”, a bundled copy of Mercurial, TortiseHg, and a few Kiln specific extensions, and started working my way through Joel’s Mercurial Primer. Unfortunately, it still wasn’t clicking for me. I got the mechanics down pretty quickly, but I still couldn’t GROK what all the hoopla was about or how I should adapt my current workflow. Things got hectic at work, so I set it aside until the Fog Creek 2010 FogBugz and Kiln World Tour came to town promising to drop some knowledge on us poor schmucks who hadn’t come to Jesus yet.

It is no secret why Joel is hawking Mercurial so fiercely given that to have any use for his new Kiln software you pretty much have to first migrate to Mercurial. Although I appreciate his zeal, I am admittedly skeptical of his decision to tie the fate of Kiln to a technology that (1) is still in the early adopter phase; (2)  involves a substantial barrier for his customers to buy his solution; and (3) requires a religion change for many users of competing SCM tools.

Still, I’m not going to pass up on free training, even if there might be a self-serving agenda behind it. So I went and I think I am better off for it, but I’m still don’t feel confident that I fully understand the use case. Maybe it is because DVCS is always pitched as the solution to the nightmare that is branching/merging in Subversion. I haven’t used Subversion for years, sparingly use branching in Vault  and work on a relatively small development team. Perhaps if I was working with hundreds of developers on big OSS projects, that have seen high adoption of DVCS tools,  I’d see the appeal more easily, but for now it is still evasive.

One aspect that I totally get is the idea of a two phase commit. I may not have been involved in a monster merge, but I have on numerous occasions had developer’s machines go down taking several weeks worth of uncommitted code changes with them. The idea of being able to check-in code privately resonates very much with me. However,  this feature (“Shelving”) already exists on existing centralized version control systems, e.g. Vault. Further, most DVCS workflow examples I’ve seen seem to encourage keeping the working version of the repository on the developer’s workstation, which doesn’t seem to provide much protection from data loss for uncommitted changes.

I also have a very positive opinion of the notion of revision control on entire change-sets as opposed to individual files, but again, while this is a common feature of DVCS systems it need not necessarily be unique to those systems and could be added to more traditional centralized SCM tools.

I Want to Believe

Having said all of that, I’m not giving up yet, but I’m still waiting for that eureka moment where the necessity of DVCS becomes so obvious that I can’t imagine ever going back. For now, I’ll use it for my projects that I want to store in Kiln and continue to use vault for my other projects. If you have a really good use case for DVCS or a really good way of explaining the benefits that will help me along, I’d definitely appreciate you leaving a comment on this post. I want to believe!

The He-Man Muggle Haters Club for Programmers (part 2)

This is the second article in a two-part series. If you haven’t already, please read Part One first.

 


 Protecting Our Interests

Although we may be loathe to admit it, we want and need programming to be sufficiently complex to keep the unwashed masses out of our clubhouse. The trouble is, we need a barrier to entry to support our value as programmers. Unlike lawyers and doctors who have expensive degrees and trade associations to protect their earning potential and aura of authority, it is not uncommon for a programmer to make a good living without the benefit of academic credentials or a formal endorsement of an organization. It is the magic of our craft that makes us feel special and separates us from the muggles and wanna-be’s like your boss’s nephew who is really good with computers.

 

Shhh! Muggles Are Everywhere

Fundamentally, I think this is a quite valid concern and admit that I sometimes succumb to these same territorial instincts. Especially when provoked by ignorant statements that undermine the investment programmers continually make in a Red Queen’s Race to stay on top of their game.

 

 

 

 

The Salman Rushdie of Programming?

Be Honest. How many of you have the urge to burn  this guy as a heretic?

There’s no reason why office workers, homemakers, professional engineers and pizza delivery persons shouldn’t be able to take advantage of their own hand crafted custom computer programs to work faster and smarter. It shouldn’t take a ‘professional programmer’ (whatever that is) to do the job. You know what needs to be done better than anyone else. You can do it yourself! (And I say this as someone who has spent many years writing programs for other people … ‘professionally’.)
VB Programming for the non-programmer!

Before everyone starts jumping on the VB dogpile in the comments, I should clarify that the referenced article is actually about VBA, and not VB proper. I’m sure, however, that many of my readers will question whether that distinction is meaningful inspiring yet another battle in the platform wars. This brings me back to my point.

 Why do many programmers despise tools like Access and VB, and even those that use VB look down on those who do too much drag-drop programming. The concept of a non-technical person using tools like this to build a useful software drives some of us nucking futs. Why? Shouldn’t we welcome usability to simplify the task of software construction?

Perhaps because it opens that clubhouse door a bit too wide for comfort.

 

"Sir, you can't let him in here. He'll see everything. He'll see the big board!"- General "Buck" Turgenson

Sir, you can’t let him in here. He’ll see everything. He’ll see the big board!

– General “Buck” Turgenson

 

 

 

 

Gold Plating The Secret Handshake

It is my contention that this effect has played a role in driving recent trends in programming technology. After years of considerable progress towards simplifying the physical act of programming (ASP.NET), recent developments in technology built for programmers has decidedly shifted towards better implementation of computer science-y stuff like functional programming, MVC and lambda expressions into the more pedestrian programming languages.

Also, the attention of developers is increasingly turning to the haute couture*technology of the day. Ruby? That is so two years ago, don’t you know we are using Scala now? (Haskell)

* The use of haute couture in this blog is not sanctioned by the French Government. For my French or Bostononian readers, please substitute the phrase “Wicked Awesome.”

This seems to drive programmers, who like woodworkers often build their own tools/jigs, to devote our time to constantly moving goalposts rather than reducing the required expertise to build functional software.

 

The Consequence and My Conclusions

I’m not saying that new programming languages don’t add value, just that this type of progress is more about fine tuning than making programmers generally more productive. My inner pragmatist makes me question the need to sharpen our tools to such a degree when the majority of the programming work seems to be centered on developing CRUD apps.  Additionally, given the very real threat of “low-cost” outsourcing options for programmers, it also makes me wonder if perhaps our priorities are not aligned with our best interests.

 

Don’t Go Away Angry

I fully realize that I have engaged in some very controversial conjecture in this series, and want to emphasize that the intent of this article is is not to drive a particular agenda. Rather, I’m merely trying to encourage introspection into some aspects of our trade that may have otherwise gone under-examined. I am not certain that I even agree with all of the opinions I have offered here, but found musing on them interesting and hope you will too.

The He-Man Muggle Haters Club for Programmers (part 1)

Little Rascals - He-Man Woman Haters Club

In his satirical and quite humorous article, “How to Write Unmaintainable Code,” Roedy Green raises an interesting question that really got me thinking about some things I had previously taken for granted with the the physical act of creating software.

 

 

 

“Why is programming still done primarily in a text editor?” 

 Before get rolling with my analysis of this question, please take a moment to read his conjecture on the topic. I’ll meet up with you again at the the other end.

Imagine having an accountant as a client who insisted on maintaining his general ledgers using a word processor. You would do you best to persuade him that his data should be structured. He needs validation with cross field checks. You would persuade him he could do so much more with that data when stored in a database, including controlled simultaneous update.

[Now] imagine taking on a software developer as a client. He insists on maintaining all his data with a text editor. He is not yet even exploiting the word processor’s colour, type size or fonts.

Think of what might happen if we started storing source code as structured data. We could view the same source code in many alternate ways, e.g.  as a decision table, a flow chart, a loop structure skeleton,  Java with various levels of detail or comments removed, as Java with highlights on the variables and method invocations of current interest, or as Java with generated comments about argument names and/or types.  You could see code with additional or fewer parentheses, (depending on how comfortable you feel with the precedence rules).

You could use the full colour abilities of the modern screen to give subliminal clues, e.g. by automatically assigning a portion of the spectrum to each package/class using a pastel shades as the backgrounds to any references to methods or variables of that class. You could bold face the definition of any identifier to make it stand out.

You could ask what methods/constructors will produce an object of type X? What methods will accept an object of type X as a parameter? What variables are accessible in this point in the code? By clicking on a method invocation or variable reference, you could see its definition, helping sort out which version of a given method will actually be invoked.You could do quite a bit of code writing by point and click.

He has some interesting points, doesn’t he? The comparison to an accountant using a word processor to create journal entries really struck me. I began to think about the Sisyphean labors of applying language/compiler specific markup in our code to satisfy our brutal task-master. It is easy to start thinking about programming IDE’s and compilers as a special class of software that is exempt from the normal  usability considerations because it is geared towards computer experts, but that just isn’t true.

Put Themselves in our Shoes

Consider the reaction of your users if you required them to enter data in your application using a big text-box using well-formed XML to provide context and meaning to each individual item. Sure they’d complain because they can’t remember the relevant attributes and always forget to close their tags, but that would be easily solved by updating the text box to have Intellisense!

Actually, no it wouldn’t. Clearly the intellisense solution addresses only the symptom, not the problem.

We Deserve Better

So why is it good enough for us?  Don’t programmers deserve good usability in our tools too?  Is it really necessary to spend valuable time matching up brackets/parens and chasing down missing line terminators?  Worse yet, if you frequently use different languages, the mental context switches force you into all sorts of mental gymnastics to keep it straight,  and incite a scolding from your compiler.

Why do we have to keep syntax ideosynchracies in our heads?

var myInteger //This is a Javascript comment
Dim myInteger as Int 'This is a VB comment
int myInteger  //This is a Java or C++ comment
myInteger=5 # This is a Ruby Comment
DECLARE @myInteger int --This is a t-SQL comment

Creating good visualizations for scoping, variable declaration and module interfaces is not an unsolvable problem. Also, the benefits of one more level of abstraction extend beyond increased productivity. For example, consider the portability that the increased normalization of code could produce.

Or Maybe Not…

At this point I am sure that many of you are still skeptical, and I am about to slam the door on the open mind of the remainder of you. What I am describing here is effectively form-based (a.k.a “drag and drop”) programming.

That Whooshing is the sound of this article entering free-fall from angry programmers down-voting it on Reddit.

This is exactly why we are still programming in glorified text-editors, because that is exactly what we want. And unlike programmers who write software for marketing people, the developers who create programming languages and IDE’s understand very well that other programmers emphatically prefer some additional complexity to even the remote possibility of democratizing software development.

Homework Assignment

As a mental exercise and preview of the finale of this article, consider this diagram showing the general sense of  superiority among programmers in the style of the famous Geek Hierarchy poster and ask yourself which direction the arrow would go in terms of general usability of the tools for each type of programmer.

Hierarchy of the World According To Programmers

Hierarchy of the World According To Programmers

 

To be Continued…

In part two of this series I discuss the psychological and frankly pragmatic reasons that our techno-brethren are so committed to keeping the field of programming as exclusive and specialized as possible.

Database Smells, Redneck style!

Your database might be a redneck if it…

JeffFoxworthyRedneck

  • relies on scheduled jobs to validate or fix referential integrity.
  • requires a regular expression to join tables.
  • uses foreign keys that decide at run-time what table or column they link to.
  • don’t need no steenking related tables. Not as long as text fields allow semi-colons! Array City, baby!
  • has primary keys that are updated more often than Twitter.
  • commits things to memory the same way I learned Spanish. Repetition, repetition, repetition.
  • thinks one-to-one relationships is this new black.
  • uses a date, bit or varchar(8000) as a primary key.
  • has an artificial seed value for identity fields to avoid PK conflicts with its twin database during replication.
  • contains both nulls and empty strings in the same column because they mean different things.
  • has more than one incrementing field per table.
  • thinks, “Constraints are for conformists, man!”
  • like your blog, has a public IP address. Unlike your blog, gets 1,000 hits a day from the Internet.
  • has an index on every column in every table.
  • has an index on one column in every database.
  • has tables with more columns than the parthenon.
  • gives the public role has more back-end access than Clay Aiken
  • contains a plain-text field named “password”, “Social security number”, or “launch codes.”
  • has stored procedures with more string manipulation than a kite flying tournament.
  • recycles identity values in a fruitless effort to save the planet by corrupting data.
  • has tables named by content AND function (EmployeeLookup)
  • may not have an identity column, but it has “identity rows” (OrderItem1,OrderItem2, OrderItem3,…)
  • likes nicknames more than GW. “Hi, I’m CustomerNumber, but my friends call me call me ClientID or CustNo.”
  • can’t let go of it’s old girlfriend, DOS and has fields named “CstInvID” just in case she ever comes back.
  • there’s a party in SA’s pants and everyone’s invited.
  • has gossipy tables that don’t mind their own business. “Harriet, let me tell you about Harriet. She lives on main street, her dad is retired, and she owes $8 in back invoices!”
  • has been sick with the Slammer virus since 2002, but doesn’t tell its partners.
  • forces you to repeat yourself. “Susie got married, can you change her last name to Murphy? Be sure to also update the name on her invoices and customer record.”
  • is full of tables with half of the columns populated 25% of the time because they are irrelevant for 75% of the rows.
  • has columns that only make sense when interpreted by their peers.

Have some to add?

Bring ’em on in the comments. I’ll add them to the article and even credit you with a link to your blog.

Why you need to know C++

I saw an interesting question on stack overflow wherein the questioner asks why everyone keeps telling him that he has to learn C++ to be a professional developer. Although it is true that most of these discussions are just launching pads for religious wars and I’m not alone in  my assertion that your language of choice is largely irrelevant , I do agree with the perhaps controversial notion that every working programmer should be at least conversant, if not fluent in C++.

Let me be entirely clear about this. I am not saying that C++ has to be your primary language or that you need to be a C++ guru to work in this business. However, it behooves you to be able to competently read and write basic code in this language.  That is, you should at the very least be able to implement the FizzBuzz program and the cliched example of a Zoo program to demonstrate polymorphism without lifting someone else’s code.

Before you dismiss me as a platform snob, consider the following six reasons that you need to know C++

1.      You should learn C++ for the same reasons that people study dead languages like Latin. The syntax and other memes in higher level languages were inspired by or directly stolen from C++. This is blatantly obvious for Java, C#, and JavaScript. Learning C++ is a quick way to grok the syntax for a several languages at once and better understand their origins.

2.      Understanding the details that are abstracted away from you in other languages (pointers, basic data structures, garbage collection, etc.) will ultimately make you more adept at solving problems when the abstractions innevitably leak or the stalwart underlying technology doesn’t live up to its stalwartitude.

Consider the real world example of anti-lock breaks, which allow you to maintain control of a car during an emergency stop. ABS has a completely idiot proof interface –Jam foot on pedal– that works just whether or not you understand that it will electronically pump the breaks to prevent the wheels from locking. However, during an attempt to thwart an impending moose collision it is certainly comforting to understand why the car is stopping in jerks instead of a smooth motion despite steady pressure on the pedal.

Also, this knowledge might have prevented my wife from listening to a shady mechanic who conned her into thinking that she had to pay his exorbitant rate to fix her ABS system since the truck wasn’t safe to drive without it to another mechanic for a second opinion.

3.      Like it or not, C++ is the lingua franca of programming. Some would argue for Java, or even JavaScript, which may appear true in much the same way that the ocean appears to contain water because of all the wet swimmers who frequent it. It’s best just to accept this, but if you need reinforcement all you need to know is that Software Jesus (Joel Spolskyagrees with me * on this, case closed.  

4.     Programmers and programming books that need to address programmers who speak an unknown or different language often use C++ for the same reasons noted in point #3. Learning this language will open access to more learning material and help you communicate with the programming community more easily.

5.     One day you are going to need sully yourself and interface directly to a C API from the lofty perch of your high level language. In my experience a lot of tool developers avoid the hassle of maintaining separate COM, .NET, and Java API’s by just providing a a C API that will appease all the peasants who are not as privileged as they to “work in a real language.”

6.     C++ on your resume can often lead to Salary++. I’ve found this is true even if the job doesn’t require any C++ programming.  It is just respected as a more hardcore language and the credibility rubs off on those who use it. That credibility makes you more valuable.

 * Update: As noted by a commenter (BTW: Thanks for the correction), Joel did make a definite distinction that new programmers should be learning C and not necessarily C++.  Apologies for misstating his position. While it is true that for just many of the points I made, you could easily substitute C for C++ and get the same bang for your buck, I contend that C++ is a stronger foundational language for the points detailed in 1,3, and 4.  Using a language that can’t talk about objects (C) to communicate with other programmers in an overwhelmingly OOP world isn’t gonna cut it.