Tough Lessons in Software

2008-08-07

I was reading through some interview questions the other day and one of them was “What was the toughest lesson you’ve learned in your job?”

The answer came immediately: the hardest problems in your job are human problems and cannot be solved with software. As a software engineer facing a problem, I immediately consider if and how I might solve this problem using the chest of tools I know and understand: languages, parsers, data structures and algorithms.  However there are a class of problems which simply cannot be solved by throwing code at it.  Project management is one of those thorny issues which is ever-present in every company in the world.  Predicting when a product will be delivered, estimating task length, resource allocation, all of these have a theoretical basis in operations research but only when building something physical.  Software is knowledge work and knowledge work is mental: it is very difficult to create a manifest of parts, a design blueprint, or a predictable development schedule when building software.

I wonder if part of this problem doesn’t have roots in our own evolution.  We are sensory-based creatures.  What we see, hear, touch, we appreciate, understand and can build.  Software is all in the head though.  It’s a very difficult to build and usually takes months or years to build a decent sized system.  I think our brains can handle a T1 of sensory input but when comes to pure cognititive thought, even the brightest of us is stuck with an old modem.  We can’t build a system in our head at once; we need to piece it together like an old hobo’s jacket over months.

Even the best software project management processes rely on developers to write down and estimate time for all the tasks required to build a system.  This is project management state of the art: force each developer to plan what they are going to build.  We try to use that old modem to gloss over the upcoming work and get a feel for its timespan based on a brief mental judgment of the work involved.  It’s never a perfect estimate, but with our brain’s limited bandwidth for cognition, it’s the best we’ve currently got.