Pro-testing Against Bad Software Quality

8Jun/129

Standards and Best Practices in Testing

This is my response to two closely related texts written by James Christie. The first one is James' article Do standards keep testers in the kindergarten? and the second one is James' post titled ISO 29119 the new international software testing standard - what about it? at the Software Testing Club forums. Due to the texts being so closely related, I won't be commenting them in any particular order or referring to the exact source of each quote in detail.

Let's begin with two quotes, one from each text:

"Even if the standard's creators envisage a set of useful guidelines, from which people can select according to their need, the reality is that other people will interpret them as mandatory statements of “best practice”."

and

"I like for words to mean something. If it isn't really best, let's not call it best."

As a vapid cliche, Voltaire wrote: "The best is the enemy of the good".

One of the problems here, as so well put by Markus Ahnve (@mahnve on Twitter) at Turku Agile Day 2012 in Finland, is that "'best practices' are only guaranteed mediocrity". In order for something to qualify as a best practice it must, necessarily, forfeit context due to the vast diversity of software projects out there. In my view, that, on its own, is grounds enough for it to lose relevance and if it's not (fully) relevant in my context how can it be best practice for me? Either I'm missing out on something that matters, or I'm getting extra weight I really don't want or need.

Note that I'm intentionally linking 'best practices' with standards here since I really don't see much of a difference in between. Both seem, to me, as one-size-fits-all sets of rules that tell me how I should do things without even asking me what I'm actually doing.

As James hinted in the texts quoted above, standards, especially in software testing, would not be so much of a problem if people didn't take them - or expect them to be taken - like gospel. I believe the problem might, at least partially, boil down to the fact that it's easier to simplify things to vague generalizations than to try and account for all contexts (which would probably be impossible anyway).

People tend to be lazy so it should come as no surprise that there are those who would prefer to just resort to a standard rather than spend time thinking about what's best for the context at hand. With luck (from that person's perspective), this approach might even be seen as beneficial. People striving towards standards-compliance are obviously just being responsible and doing their very best to drive the project to optimal results, right?

Another potential problem with standards is, in my opinion, extremely well put in one of my favorite online comics: http://xkcd.com/927/

I don't know how to better put that to words than that.

"Obviously the key difference is that beginners do need some kind of structural scaffold or support; but I think we often fail to acknowledge that the nature of that early support can seriously constrain the possibilities apparent to a beginner, and restrict their later development."

I completely agree with James here. Maybe a stupid analogy but you don't buckle toddlers up to braces when they're trying to learn how to walk. You encourage them, you give them a taste of what walking is like by supporting them for a while and then leaving them to their own devices to build up the muscles, balance and coordination required. You give them something to reach out for - quite literally - and let them work out a way to get there.

In the same spirit, I wouldn't want to restrain the learning of a beginning tester by dictating rules, requirements and restrictions. I share my own experiences, I point them to various authors, books, forums and blogs and let them work things out from there, giving advice when they ask for it or are obviously lost.

"the real problem is neither the testers who want to introduce standards to our profession, nor the standards themselves. The problem lies with the way that they are applied."

I wouldn't leave the people wanting to introduce standards to software testing out of the picture since demand drives supply (or vice versa if your marketing is good enough). The problem here, as I see it, is the way how a lot people just wait to be commanded and when they receive recommendations they interpret them as absolutes ie. commandments instead. I believe this is strongly cultural and psychological.

Unfortunately, in my experience, this seems to have nothing to do with a person's position in a company or in the society in general. These people strive to memorize and follow the rules to the letter instead of trying to understand and apply them in a way that fits the current context. Ever heard of anyone going "by the book"? Yeah, me too, and I find the amount of such people disconcerting.

I'm going to stray from the subject for a bit but, since this is closely related to the interpretations, I think it's relevant and called for so bear with me. I've personally been involved in a number of projects where the so-called agile project model has actually been waterfall from start to finish and for no other reason than certain key people's strict adherence to the "rules" of agile. When those people are in a position of authority that by-the-book approach can wreak havoc all across the company while appearing beneficial at a quick glance. Being standards-compliant can never be a bad thing, right? CMMI level 5 or bust and all that.

I'll give you a hint: there will be no prince(ss) at the end of the dungeon but by the time you get there you will have created your own end-of-level boss to fight.

For me, the very first thing I ever learned about agile was: "adjust it to your needs". Incorporate the parts that serve your purpose and context and leave out the ones that are a hindrance. It's called agile for a reason so think for yourself because it's your context. The obvious problem here, of course, is the fact that you should have a solid grasp of the underlying ideas and principles or the advice is likely to backfire.

I do believe standards can be a good thing - even in software testing - if used constructively, as a support, instead of being taken as compulsory mandates that disallow any and all deviation from the norm. The problem here, as James mentioned, is the fact that people easily take the word "standard" as a synonym of "must".

Testing is a creative mental process closer to art than mechanical engineering. It most certainly isn't a repetitive conveyor belt performance as some people would like others to think (mostly out of pure ignorance, I'm guessing). If painting or composing were governed by strict standards the end results would probably be pretty darn boring and bland.

Comments (9) Trackbacks (1)
  1. Hello, Petteri!

    The problem here is that I see the world of “best-practices” quite similarly as you. The topic made my ears puff smoke, but I like the way you have tackled it. I do however try to challenge you, if I can. 😉

    I couldn’t agree more on the part about losing relevance in due to context. How would you handle a situation, where the practice is the current best in that context? Is that then the best practice?

    I will not go to the gospel-commandment-thingy-area as I might begin to rant. If you feel the best-practices aren’t good, is a vague generalization any good in making people to veer away from standards? If they are lazy, how will they react to “vagueness”? To answer the question about people striving towards the standard-compliance, my opinion is twofold. People will strive towards the optimal results no matter the compliance in their effort, or they won’t strive at all. The compliance might be the framework in which they… well do their striving. 😀 But do that and be responsible? What do you mean by responsible? Is it like “I will do drive towards the end of this project and document my way using a standard-driven method in order to make me more credible”?

    About the xkcd: Isn’t that context-driven thinking if your context needs the all-covering standard? How is the 15th standard wrong in any sense? I agree with giving free reign, but is there a risk of them wondering off the map into the “bad-practice” area? We need structure. Standards might just be the structure they need. Am I right?

    Who are “these people”? If they are of no particular position or class in society, how will we recognize the? Am I one of those people? Can I change, if I am? How? If I see the benefits in a standards-driven way, why should I try to change good behavior?

    But if I didn’t completely miss the point of the end part, you seem to say that the problem is not in the standards or the best-practices but in the way people use them by the letter. By thinking the “best-practice” as a vague generalization, you might get more out of it than trying to highlight the “bestness” of it. Is the problem the standard folk’s or the anti-standard folk’s?

    I hope you have as much a blast as I did while trying to challenge your opinions. 😀 I really liked the post overall and your stuff in general (but vaguely as it’s not suitable to my all liking-contexts). Looking forward to hearing your replies!

    BR, Peksi

    • Hi Peksi!

      A few quick comments.

      “How would you handle a situation, where the practice is the current best in that context? Is that then the best practice?”

      How would you know that is the best pratice? For whom would it be the best practice? What if the context keeps changing?

      “About the xkcd: Isn’t that context-driven thinking if your context needs the all-covering standard? How is the 15th standard wrong in any sense? I agree with giving free reign, but is there a risk of them wondering off the map into the “bad-practice” area? We need structure. Standards might just be the structure they need. Am I right?”

      It’s not about the context needing to have a standard to cover all other standards. It’s about replacing them with one. It’s not about right or wrong. It’s about not being able to generalize many standards to a single one.

      Ok and then back to business. 🙂

      Best regards,
      Jari

      • Cool! A conversation! In deed, how would I know that it’s the best practice. I would say, “it is the best for me in the context that I’m aware of. Am I aware of the context change? Am I aware of the different interpretations? I don’t know. It’s the best for me as far as I concern.

        So I might have misinterpreted the comic, as far as you see. In my context it was interpreted that way, thus I making it a valid comment. In my context it is exactly about that: I need to make it right and make it general. Standards might be the way to go, or am I wrong? Does this make my context-dependency invalid? If the standard is the correct way to achieve the goal in my context, is it a bad thing?

    • Pekka,

      Challenge accepted.

      “How would you handle a situation, where the practice is the current best in that context? Is that then the best practice?”

      I would use it, and abuse it, until it no longer was a best practice (as deemed best by the team) and then I’d find a new best practice – in context. And yes, it would then be the best practice for me and/or my team. In other words: I criticize the meaningless way the term ‘best practice’ is thrown around implying absolutes. I do not believe in universal “best” but I do believe in “best in context”.

      “If they are lazy, how will they react to “vagueness”?”

      Your guess is as good as any. People are individuals so one lazy person might react in a completely different manner than the lazy person next to him/her.

      “What do you mean by responsible? Is it like “I will do drive towards the end of this project and document my way using a standard-driven method in order to make me more credible”?”

      I was being sarcastic there. I meant that as a justification a person might make if his/her motives for resorting to a standard were questioned.

      As to what you wrote just before the quoted text, there’s also a third kind: those who just don’t care. The ones who are “just working here”. They might strive towards standards-compliance because they were told to do so and for that reason alone since they really don’t care how things happen. At least as long as the paychecks keep coming in time. I’ve met a few people who would never question a decision; they’d just take what was dictated to them and get to work. Is that responsible? From an employer perspective it might be since these guys always did their jobs. Personally, I’m not so sure. The last paragraph in this reply tells you why.

      “We need structure. Standards might just be the structure they need. Am I right?”

      Sure, standards may give structure, I’ll give you that, but do they give good structure and what do they lead up to? The Chinese foot binding custom gave girls’ feet structure. Was that a good thing?

      “Who are “these people”? If they are of no particular position or class in society, how will we recognize the? Am I one of those people? Can I change, if I am? How? If I see the benefits in a standards-driven way, why should I try to change good behavior?”

      “These people” refers to people who don’t (dare) question what they perceive as mandates but, instead, follow those perceived orders to the letter. Note that this has little to do with the manner or purpose in which these “orders” are given, and everything to do with the way these people interpret, and act upon, them.

      If you want a simplification, “these people” refers to people who don’t exercise critical thinking – for whatever reason. Based on that, I have a very strong feeling you’re not one of them. Simply because you do question.

      In the “how?” part I think you’re trying to set up a trap since it’s your context and I can’t really get inside your context in this case (my psychic skills are still somewhat limited). Telling you how to improve without knowing your context would make me the “best practice” guy and I don’t want to be that person so you’re just going to have to come up with your own solution there. Sorry.

      As a whole, this particular subject is way too broad to cover in a single blog comment and I can not claim expertise in either cognitive or behavioral psychology so take this all with a grain of salt.

      If you’re aware of the pros and cons of the approach and you still see benefits in a standards-driven way in software testing then, by all means, go for it. However, if you really are in the know why limit your potential by a standard? Much like “best practices” a standard in software testing does not guarantee good results, it only tries to guarantee a baseline (as set by the people behind the standard).

      A tweet by Iain McCowatt put it quite well, imo: “If you need ISO29119 you probably couldn’t make good use of it. If you could make good use of it, you probably don’t need it. #testing”

      “By thinking the “best-practice” as a vague generalization, you might get more out of it than trying to highlight the “bestness” of it. Is the problem the standard folk’s or the anti-standard folk’s?”

      I think the problem is in what I wrote above about “these people”. Lack of critical thinking in an environment where it’s, well, critical. I’m confident it applies in both camps.

      • Hi, Petteri. Thanks for responding to my rant. You fared quite good as you didn’t buckle under the challenge but you thrived and made your answers more to the point. By stressing that it is all about the context, you didn’t undermine your own writings. You accepted that sometimes a context might require a use of “bad practices” but which are decent in another context.

        It is all about critical thinking and some people don’t possess that skill. It is however a skill that can be learned. By challenging even the “good practices” a person can practice the skill and learn the skills required for constructive critical thinking. I was being a Devil’s advocate, and I really like the way you make sense into the certified/standardized world. Kudos!

        – Peksi

  2. Hi Petteri

    Thanks for your enthusiastic post. I fear for our craft if compliance with ISO 29119 suddenly becomes a compulsory requirement in testing projects.

    And as you wonderfully cited Voltaire, regarding best practices, the best is the enemy of the good.

    Best
    Ilari

  3. Thanks for following up my articles with this piece and extending the argument I was making. Of course I agree with everything you’ve written, except for one small thing! I don’t think Voltaire’s quote “the best is the enemy of the good” applies to “best practices”.

    Voltaire meant that the pursuit of perfection can prevent us from getting something that is good enough for our needs. That is very relevant to software development.

    Once I led a development to develop an insurance management information feeder system. Another team was developing another feed in parallel. The business needed both feeds to be ready in time for the annual financial planning exercise. My team did what we needed to deliver the feed in time, tailoring the scope and also the quality so the business got what it needed.

    The other team was led by someone who believed passionately that rigorous data analysis was essential to produce high quality applications. The team performed a perfect data analysis exercise and missed their target. The feed wouldn’t been needed for another year, so the developers were assigned to more urgent work.

    The following year there were no free people to complete the work and it was deferred again. It was never delivered, and all the work was wasted. My team did a good job, and the business got what it needed. The other team tried to achieve perfection, and delivered nothing.

    I don’t think “best practices” deserve to be regarded as perfection. I think the idea is fundamentally misconceived. There are good practices, that work well and should be used in particular contexts, and there are bad practices that do not work in that context. The pretence that some practices are universally “best” is often used to promote bad practices by people who don’t understand software development. These practices, however, are usually convenient from a contractual perspective. They can provide an objective means of assessing compliance with best practice. Yes, the circularity of argument is quite deliberate!

    I should also mention that two of the quotes you attribute to me came from other people.

    It was Ben Simo (@QualityFrog on Twitter) who said “I like for words to mean something. If it isn’t really best, let’s not call it best”.

    Also, it was Chris Atherton (@finiteattention) who provided the quote “beginners do need some kind of structural scaffold…”.

    Obviously I agree with both of them, so I don’t mind the quotes being attributed to me. However, I must give credit where it’s due, to Ben and Chris, who were both very helpful to me when I wrote my article.

    • James,

      Good point about the Voltaire quote. It seems I should have explained myself better there.

      I guess the point I was trying to make could be written as an adaptation of the Voltaire quote: A ‘best practice’ is the enemy of a ‘good practice’. This, of course, does not remove the misconception as you put it but maybe it makes my point a little clearer. I wasn’t really thinking that far ahead when using the quote so your elaboration is a valuable addition there.

      Also good thing you pointed out the original sources for the quotes and my apologies for misusing them. I guess I should’ve just referred to those two texts of yours, instead of attributing the quotes to you directly.

      By the way, for some reason, the circularity you used reminds me of the arguments ISTQB makes when selling their certification. I guess they are – or at least would like to be seen as – a standard in their own particular area.

  4. I’d like to argue that the reason a standard can do more harm than good is by looking at history. In most instances, standards destroy innovation. Look at the FCC’s attempts to create a closed Internet. Most people know that it will benefit a few corporations who will then have complete control, basically inviting a corrupted system.

    In a very young occupation such as software testing, innovation is constantly needed to improve the practice. How testers test are what I like to call a tester’s theory of what is a good approach for a given circumstance. It’s theory, so it can be proven wrong and then built into a better approach in the future. As for the new ISO standard, it has not been proven to help industry and to improve quality. It actually destroys quality because it destroys the creative thought process and critical thinking required to test anything. It also allows people who are not qualified to be seen as so because they just met the requirements of the standard.

    What bothers me the most about it is that it feeds into a system of corruption that is not to help the testing practice, but rather to fill a few companies’ pockets with gold.


Leave a comment