Pro-testing Against Bad Software Quality


An Object Is What the Object Does – Or Is It?

It's been a while since my last blog post and earlier today I came across a post that started a thought process I just had to write down. This post is based on the blog entry titled "“Testing vs Checking” : ‘Tomato’ vs ‘Tomato’" by Mario Gonzalez of Ortask.

As a quick recap, the gist of Mario's blog post is to criticize the differentiation James Bach, Michael Bolton and, generally, the people aligned with the context-driven approach to testing make between the terms "testing" and "checking". He basically goes on to claim the whole differentiation is a moot point or even potentially harmful to the testing community, and its future, as a whole (in terms of adding confusion to a field already swamped with definitions). Simplified, but that is my interpretation of the text.

[UPDATE April 8th]: As pointed out by @JariLaakso in Twitter, not all context-driven testers necessarily make the distinction between testing and checking, so I can't really say "generally, the people aligned with..." as I am not aware of the exact number of people who actually do. I may have simply just communicated with people who do make the distinction so reformat that one sentence in your mind to your liking. [END UPDATE]

However, I am not going to go any deeper into that or any of the other problems I perceived while reading through the post, at least not for now - I'm sure other people will. Instead, I will concentrate on the one thing that bothered me the most at the time of reading it: Mario's definition of a test. So I decided to dissect it. Here is the definition he offered:

a test attempts to prove something about a system

First of all, that is not even a definition of a test - it is a definition of the purpose of a test. Let me clarify with an analogy. Consider this for a definition of a car:

A car is used to transport people and goods from one place to another

Now, based on that "definition" alone, answer me the following questions:

  1. What does a car look like?
  2. What principles and mechanisms does a car operate on?
  3. Under what conditions and circumstances can a car be used to transport people and goods from one place to another?

You can't, can you? That's because I haven't given you a definition of a car - only what it's typically used for. In other words:

Defining what an object does does not define what the object is.

While still lacking and incomplete, a definition of a car could be something like: "A car is a typically four-wheeled, land-based vehicle that usually operates on the principle of an internal combustion engine turning the energy contained within a liquid fuel into mechanical movement through a series of controlled explosions the pressure of which cause a crankshaft to rotate and apply torque to the vehicle's drive wheels".

While the non sequitur I pointed out above would be reason enough to stop going any further, I want to go through the definition (of the purpose of a test) for the sake of completeness:

  • "A test" - Now what is that? In this context the question is impossible to answer - Mario hasn't told us!
  • "attempts" - How does "a test" attempt anything? It's not a sentient being. It would seem to me it is the tester who is the one to make the attempt through performing a test, making observations, analyzing and interpreting data, behavior and results before, during and after performing a test.
  • "to prove" - What, exactly, constitutes proof? Here are some definitions of the term:
    • evidence or argument establishing a fact or the truth of a statement (Oxford dictionaries)
    • the cogency of evidence that compels acceptance by the mind of a truth or a fact (Merriam-Webster)
    • sufficient evidence or an argument for the truth of a proposition (Wikipedia)

Now, how does one arrive at "proof", based on the above definitions? An obvious problem that immediately comes to mind is that, in many cases, "truths" or "facts" are relative and dependent on a number of factors. Don't believe me? Well, this is the last sentence in this blog entry. True at the time of writing, but false only seconds later when I kept going.

Also, if you want to pursue the scientific angle, I don't think anyone would take the result of a single experiment as any kind of proof of anything. You would need to repeat the experiment multiple times (and, ideally, have an independent peer group do the same) in order for it to gain any kind of credibility but therein lies a problem: the conditions would need to be exactly the same every time and that is virtually impossible to achieve in software testing. The date and time change, the amount of available CPU power, RAM and hard disk space varies, there might be network congestion or packet loss that you can not possibly predict, another application might suddenly misbehave and distort your test results or any number of other, unknown, factors that can affect the outcome of a test could manifest themselves.

It would seem to me that "proof" is much too strong a word to use in this context. Testing may suggest behavior that appears consistent but it can not prove that any more than a turkey being generously fed on a daily basis can predict that on the 180th day of its life the farmer comes out with an axe instead of seeds and takes the turkey's life instead of feeding it.

On with the definition:

  • "something" - Anything? Well, Mario did elaborate on this in the following paragraph so I'll just leave it at that.
  • "about a system" - Now there's another interesting word: "system". Various definitions to follow:
    • 1 a set of things working together as parts of a mechanism or an interconnecting network; a complex whole, or
    • 2 a set of principles or procedures according to which something is done; an organized scheme or method (Oxford dictionaries)
    • a group of devices or artificial objects or an organization forming a network especially for distributing something or serving a common purpose (Merriam-Webster)
    • a set of interacting or interdependent components forming an integrated whole or a set of elements (often called 'components') and relationships which are different from relationships of the set or its elements to other elements or sets (Wikipedia)

Complexity. On multiple levels. Especially when talking about computers and software. What if there is a fault in the hardware such that it exhibits certain behavior at one time but a different behavior at other times? Maybe a capacitor is nearing the end of its life and causes a test to give different results based on the ambient temperature of the environment in which the system resides. How many are prepared to, or even capable of, anticipating and accounting for something like that?

I'm not even trying to be funny here - my stereo amplifier does this and for exactly that reason!

Based on all of the above, I'm afraid I can only arrive at the conclusion that this particular definition of a test is fundamentally flawed (even starting from the fact that the claim of what is being defined is unrelated with the actual definition presented) and, in my view, would warrant serious reconsideration and refining.


Physics, Testing and Academia

The Introduction

I just read the Science Daily article: Quantum Physicists Show a Small Amount of Randomness Can Be Amplified Without Limit and, while reading, it occurred to me that quantum physics seems a bit like the context-driven approach to testing. Consider, for example, the following claims:

  • there is an element of uncertainty present at all times - we can not know all affecting variables
  • there is an element of randomness, beyond our control as observers, present in any system
  • it is unlikely we could ever have complete knowledge of the state of a system being observed

Sound familiar? Similarly, I find it relatively easy to (loosely) link classical physics with the more scholarly approaches to testing. Again, consider the following:

  • all effects are predictable and can be accurately represented in numbers and formulas
  • the state of a system being observed can be accurately measured at any point in time
  • standardization is valuable, and necessary

Exact, deterministic and absolute. Unfortunately, that leads me into the rant part.

The Rant

As I recently discussed with a PhD-in-physics friend of mine, the biggest problem I have with the way many people of science approach, well anything, is that the approach is often so absolute and unswerving. "There is no 'maybe' or 'what if', the formula shows how it is. Who are you to question Newton or Einstein?" To me, this seems hierarchical, authoritative, stagnatory, unimaginative, stubborn, cynical, naïve even.

When I was a kid I always thought the purpose of a scientist was to explore new possibilities and alternatives, not to dismiss them as irrelevant or stupid without a moment's thought if they disagreed with the leading views. I used to think that the purpose of a scientist was to constantly question everything in order to learn and to improve, not to meekly accept the Word of Truth(TM) handed down by higher authority figures. Unfortunately, based on approximately 20 years of observing people of academia, that does not seem to be the case very often.

I find it highly frustrating when a casual conversation first turns into an interesting debate and then rapidly degrades to almost religious raving when the opponent flatly refuses to even consider the possibility that there might be some level of uncertainty involved or that maybe the current facts of science related to the subject are actually no facts at all. Maybe they're just best guesses that were made based on our current level of understanding and limited by the sensitivity of the equipment used to measure. 3000 years ago scientists knew the Earth is flat. 100 years ago scientists knew that the speed of sound can not be exceeded. I don't think many people would share those notions now.

The Finale

What I'm trying to say here is that I, personally, would immensely enjoy seeing (more of) the people of hard sciences go into the critical questioning mode a little easier and just a tad more often. You don't lose your face or professional credibility for stating "we might not have perfected this yet", or "I wonder if it would be possible we've gotten this all wrong".


PS. Do note that I'm in no way belittling or disparaging natural sciences themselves. I wouldn't be here writing this rant without the practical applications and solutions that mathematics, chemistry and physics have brought along. I only criticize the overly fixed mindset of (some of) the people.


Schools of (pro)testing

I have been discussing Cem Kaner's announcement and the separation between the founders of the context-driven school of testing with some people, most of whom proclaim themselves as context-driven (myself included). This post was inspired by Cem's response to the responses (confusing, eh?) he got for the original announcement.

I must say I like Cem's way of thinking here as it would appear to me to be humane, and non-exclusive. I like that because it's really close to my own world view and way of thinking. Here are some comments to what Cem wrote:

"An analogy to religion often carries baggage: Divine sources of knowledge; Knowledge of The Truth; Public disagreement with The Truth is Heresy; An attitude that alternative views are irrelevant; An attitude that alternative views are morally wrong."

This is why I align myself rather closely with nontheistic Buddhist views and why Dalai lama is my idol. This is also something I wrote about in our discussions with the above-mentioned group of people. Here's what Dalai lama has to say about religions:

"I always believe that it is much better to have a variety of religions, a variety of philosophies, rather than one single religion or philosophy. This is necessary because of the different mental dispositions of each human being. Each religion has certain unique ideas or techniques, and learning about them can only enrich one's own faith."
--His Holiness the 14th Dalai lama

As I wrote in our discussion, I don't see any difficulty applying this more generally even within a single school - be it religion, philosophy or software testing. As long as there are opposing views and we keep our own thinking critical - especially concerning our own views and opinions - they can only help us become better and stronger by, for example, teaching us how many different angles there can be to viewing the same concepts/thoughts/ideas/practices/whatever.

You can't credibly claim your favorite color is blue if you haven't experienced red, green and yellow as well.

This is exactly why I am curious about the people who genuinely, for example, consider ISTQB certification a good idea. I wish to learn about their motives, their way of thinking and their reasoning, rather than just outright disregard their views as stupid, ignorant or irrelevant. That would be arrogant, inhumane and unfair. I don't need to agree with a view to be able to acknowledge the value of enthusiasm and sincerity (even if unfounded or misguided). Note that I'm talking about the people, not the certification, of which I have less than favorable opinions.

As Cem wrote, controversy is a good thing. It's just that there are constructive ways of dealing with controversy and then there are destructive ways of dealing with it and probably any number of ways that are somewhere in between. The Buddhist approach, in my view, is constructive. The purpose is to have your own view and then refine it by learning from others while accepting their right to differing views and different paths to learning. This approach can help you uncover faults in your own thinking you might never have realized had you not been dealing with people whose views disagree with your own.

This, by the way, is perfectly in line with People's Assertive Rights (that I feel everyone should know about and would benefit from embracing) but I won't go deeper into that here.

"In my view, there are legitimate differences in the testing community. I think that each of the major factions in the testing community has some very smart people, of high integrity, who are worth paying attention to. I’ve learned a lot from people who would never associate themselves with context-driven testing."

I pretty much already covered this above but as an addition I would take the analogy of the spectrum of colors:

Think of the spectrum as comprising the entire testing community, everyone in it. Now, think of that spectrum as being arbitrarily split into smaller sections, or "schools" of testing. Undoubtedly, there are people who would like to over-simplify things by assigning a single color to each of these sections. "This is the blue section", "this is the green section" and so on. Considering the generally acknowledged "schools" of testing that would give us what, 5-7 different colors? I don't know about anyone else but I think that would make a rather poor representation of the beauty of the spectrum (is that a straw man? I would like to think it's not since I don't mean to attack anyone but, rather, just clarify my own views on the subject). Kind of like the shadow at the back of a cave as a representation of complex three-dimensional objects in Plato's Allegory of the Cave.

The reality is that even if the spectrum is arbitrarily split into smaller sections, the color slide of the spectrum does not stop within any one of those sections. What this means in practice is that people within a single school will still have different views even if they generally adhere to a similar way of thinking on a broad scale. Fuchsia is still red even if it isn't scarlet (though some people might argue that fuchsia is closer to blue and not completely unjustly so). There are those who would be considered analytical by the people in the context-driven school but context-driven in the analytical school.

My point here is that it would be unrealistic, naive even, to think that every proponent or representative of a specific school of testing would unilaterally agree on everything. In my view, it would be much wiser and better for the community as a whole to embrace the variety (and controversy) than try to force people into a single mold.

"One of the profound ideas in those courses was a rejection of the law of the excluded middle. According to that law, if A is a proposition, then A must be true or Not-A must be true (but not both). Some of the Indian texts rejected that. They demanded that the reader consider {A and Not-A} and {neither A nor Not-A}."

I love this. This, I believe, is exactly the kind of thinking that is referred to when talking about "thinking outside the box". I challenge, question and protest against any artificial boundaries and restrictions. Especially in software testing.

By the way, when I do that, the most typical response I get is: "nobody would ever do that!"

Sound familiar?