Pro-testing Against Bad Software Quality

Standards and Best Practices in Testing

This is my response to two closely related texts written by James Christie. The first one is James' article Do standards keep testers in the kindergarten? and the second one is James' post titled ISO 29119 the new international software testing standard - what about it? at the Software Testing Club forums. Due to the texts being so closely related, I won't be commenting them in any particular order or referring to the exact source of each quote in detail.

Let's begin with two quotes, one from each text:

"Even if the standard's creators envisage a set of useful guidelines, from which people can select according to their need, the reality is that other people will interpret them as mandatory statements of “best practice”."

and

"I like for words to mean something. If it isn't really best, let's not call it best."

As a vapid cliche, Voltaire wrote: "The best is the enemy of the good".

One of the problems here, as so well put by Markus Ahnve (@mahnve on Twitter) at Turku Agile Day 2012 in Finland, is that "'best practices' are only guaranteed mediocrity". In order for something to qualify as a best practice it must, necessarily, forfeit context due to the vast diversity of software projects out there. In my view, that, on its own, is grounds enough for it to lose relevance and if it's not (fully) relevant in my context how can it be best practice for me? Either I'm missing out on something that matters, or I'm getting extra weight I really don't want or need.

Note that I'm intentionally linking 'best practices' with standards here since I really don't see much of a difference in between. Both seem, to me, as one-size-fits-all sets of rules that tell me how I should do things without even asking me what I'm actually doing.

As James hinted in the texts quoted above, standards, especially in software testing, would not be so much of a problem if people didn't take them - or expect them to be taken - like gospel. I believe the problem might, at least partially, boil down to the fact that it's easier to simplify things to vague generalizations than to try and account for all contexts (which would probably be impossible anyway).

People tend to be lazy so it should come as no surprise that there are those who would prefer to just resort to a standard rather than spend time thinking about what's best for the context at hand. With luck (from that person's perspective), this approach might even be seen as beneficial. People striving towards standards-compliance are obviously just being responsible and doing their very best to drive the project to optimal results, right?

Another potential problem with standards is, in my opinion, extremely well put in one of my favorite online comics: http://xkcd.com/927/

I don't know how to better put that to words than that.

"Obviously the key difference is that beginners do need some kind of structural scaffold or support; but I think we often fail to acknowledge that the nature of that early support can seriously constrain the possibilities apparent to a beginner, and restrict their later development."

I completely agree with James here. Maybe a stupid analogy but you don't buckle toddlers up to braces when they're trying to learn how to walk. You encourage them, you give them a taste of what walking is like by supporting them for a while and then leaving them to their own devices to build up the muscles, balance and coordination required. You give them something to reach out for - quite literally - and let them work out a way to get there.

In the same spirit, I wouldn't want to restrain the learning of a beginning tester by dictating rules, requirements and restrictions. I share my own experiences, I point them to various authors, books, forums and blogs and let them work things out from there, giving advice when they ask for it or are obviously lost.

"the real problem is neither the testers who want to introduce standards to our profession, nor the standards themselves. The problem lies with the way that they are applied."

I wouldn't leave the people wanting to introduce standards to software testing out of the picture since demand drives supply (or vice versa if your marketing is good enough). The problem here, as I see it, is the way how a lot people just wait to be commanded and when they receive recommendations they interpret them as absolutes ie. commandments instead. I believe this is strongly cultural and psychological.

Unfortunately, in my experience, this seems to have nothing to do with a person's position in a company or in the society in general. These people strive to memorize and follow the rules to the letter instead of trying to understand and apply them in a way that fits the current context. Ever heard of anyone going "by the book"? Yeah, me too, and I find the amount of such people disconcerting.

I'm going to stray from the subject for a bit but, since this is closely related to the interpretations, I think it's relevant and called for so bear with me. I've personally been involved in a number of projects where the so-called agile project model has actually been waterfall from start to finish and for no other reason than certain key people's strict adherence to the "rules" of agile. When those people are in a position of authority that by-the-book approach can wreak havoc all across the company while appearing beneficial at a quick glance. Being standards-compliant can never be a bad thing, right? CMMI level 5 or bust and all that.

I'll give you a hint: there will be no prince(ss) at the end of the dungeon but by the time you get there you will have created your own end-of-level boss to fight.

For me, the very first thing I ever learned about agile was: "adjust it to your needs". Incorporate the parts that serve your purpose and context and leave out the ones that are a hindrance. It's called agile for a reason so think for yourself because it's your context. The obvious problem here, of course, is the fact that you should have a solid grasp of the underlying ideas and principles or the advice is likely to backfire.

I do believe standards can be a good thing - even in software testing - if used constructively, as a support, instead of being taken as compulsory mandates that disallow any and all deviation from the norm. The problem here, as James mentioned, is the fact that people easily take the word "standard" as a synonym of "must".

Testing is a creative mental process closer to art than mechanical engineering. It most certainly isn't a repetitive conveyor belt performance as some people would like others to think (mostly out of pure ignorance, I'm guessing). If painting or composing were governed by strict standards the end results would probably be pretty darn boring and bland.