Pro-testing Against Bad Software Quality

6Apr/134

An Object Is What the Object Does – Or Is It?

It's been a while since my last blog post and earlier today I came across a post that started a thought process I just had to write down. This post is based on the blog entry titled "“Testing vs Checking” : ‘Tomato’ vs ‘Tomato’" by Mario Gonzalez of Ortask.

As a quick recap, the gist of Mario's blog post is to criticize the differentiation James Bach, Michael Bolton and, generally, the people aligned with the context-driven approach to testing make between the terms "testing" and "checking". He basically goes on to claim the whole differentiation is a moot point or even potentially harmful to the testing community, and its future, as a whole (in terms of adding confusion to a field already swamped with definitions). Simplified, but that is my interpretation of the text.

[UPDATE April 8th]: As pointed out by @JariLaakso in Twitter, not all context-driven testers necessarily make the distinction between testing and checking, so I can't really say "generally, the people aligned with..." as I am not aware of the exact number of people who actually do. I may have simply just communicated with people who do make the distinction so reformat that one sentence in your mind to your liking. [END UPDATE]

However, I am not going to go any deeper into that or any of the other problems I perceived while reading through the post, at least not for now - I'm sure other people will. Instead, I will concentrate on the one thing that bothered me the most at the time of reading it: Mario's definition of a test. So I decided to dissect it. Here is the definition he offered:

a test attempts to prove something about a system

First of all, that is not even a definition of a test - it is a definition of the purpose of a test. Let me clarify with an analogy. Consider this for a definition of a car:

A car is used to transport people and goods from one place to another

Now, based on that "definition" alone, answer me the following questions:

  1. What does a car look like?
  2. What principles and mechanisms does a car operate on?
  3. Under what conditions and circumstances can a car be used to transport people and goods from one place to another?

You can't, can you? That's because I haven't given you a definition of a car - only what it's typically used for. In other words:

Defining what an object does does not define what the object is.

While still lacking and incomplete, a definition of a car could be something like: "A car is a typically four-wheeled, land-based vehicle that usually operates on the principle of an internal combustion engine turning the energy contained within a liquid fuel into mechanical movement through a series of controlled explosions the pressure of which cause a crankshaft to rotate and apply torque to the vehicle's drive wheels".

While the non sequitur I pointed out above would be reason enough to stop going any further, I want to go through the definition (of the purpose of a test) for the sake of completeness:

  • "A test" - Now what is that? In this context the question is impossible to answer - Mario hasn't told us!
  • "attempts" - How does "a test" attempt anything? It's not a sentient being. It would seem to me it is the tester who is the one to make the attempt through performing a test, making observations, analyzing and interpreting data, behavior and results before, during and after performing a test.
  • "to prove" - What, exactly, constitutes proof? Here are some definitions of the term:
    • evidence or argument establishing a fact or the truth of a statement (Oxford dictionaries)
    • the cogency of evidence that compels acceptance by the mind of a truth or a fact (Merriam-Webster)
    • sufficient evidence or an argument for the truth of a proposition (Wikipedia)

Now, how does one arrive at "proof", based on the above definitions? An obvious problem that immediately comes to mind is that, in many cases, "truths" or "facts" are relative and dependent on a number of factors. Don't believe me? Well, this is the last sentence in this blog entry. True at the time of writing, but false only seconds later when I kept going.

Also, if you want to pursue the scientific angle, I don't think anyone would take the result of a single experiment as any kind of proof of anything. You would need to repeat the experiment multiple times (and, ideally, have an independent peer group do the same) in order for it to gain any kind of credibility but therein lies a problem: the conditions would need to be exactly the same every time and that is virtually impossible to achieve in software testing. The date and time change, the amount of available CPU power, RAM and hard disk space varies, there might be network congestion or packet loss that you can not possibly predict, another application might suddenly misbehave and distort your test results or any number of other, unknown, factors that can affect the outcome of a test could manifest themselves.

It would seem to me that "proof" is much too strong a word to use in this context. Testing may suggest behavior that appears consistent but it can not prove that any more than a turkey being generously fed on a daily basis can predict that on the 180th day of its life the farmer comes out with an axe instead of seeds and takes the turkey's life instead of feeding it.

On with the definition:

  • "something" - Anything? Well, Mario did elaborate on this in the following paragraph so I'll just leave it at that.
  • "about a system" - Now there's another interesting word: "system". Various definitions to follow:
    • 1 a set of things working together as parts of a mechanism or an interconnecting network; a complex whole, or
    • 2 a set of principles or procedures according to which something is done; an organized scheme or method (Oxford dictionaries)
    • a group of devices or artificial objects or an organization forming a network especially for distributing something or serving a common purpose (Merriam-Webster)
    • a set of interacting or interdependent components forming an integrated whole or a set of elements (often called 'components') and relationships which are different from relationships of the set or its elements to other elements or sets (Wikipedia)

Complexity. On multiple levels. Especially when talking about computers and software. What if there is a fault in the hardware such that it exhibits certain behavior at one time but a different behavior at other times? Maybe a capacitor is nearing the end of its life and causes a test to give different results based on the ambient temperature of the environment in which the system resides. How many are prepared to, or even capable of, anticipating and accounting for something like that?

I'm not even trying to be funny here - my stereo amplifier does this and for exactly that reason!

Based on all of the above, I'm afraid I can only arrive at the conclusion that this particular definition of a test is fundamentally flawed (even starting from the fact that the claim of what is being defined is unrelated with the actual definition presented) and, in my view, would warrant serious reconsideration and refining.

Comments (4) Trackbacks (0)
  1. Hi Petteri,

    I enjoyed reading your post. I’m glad this has caused an intelligent, constructive discussion in our industry.

    However, I’d like to point you to the reply from one of the champions and definers of the whole “testing vs checking” issue:

    http://ortask.com/testing-vs-checking-tomato-vs-tomato/#comment-1415

    Unlike your response, there appears to have been no thought put into the “refutation.” Judge for yourself.

    • Hi Mario,

      Glad you liked it. I welcome open discussion, especially when the views are opposing, because I’ve found that to help me refine my own views and improve my skills in defending them.

      I can’t (and wouldn’t want to either) comment on behalf of James but I am slightly confused about your comment: I fail to see the relevance in the context of my blog post, considering I specifically didn’t take a stand on “testing vs checking” in it (except for a very brief summary of my understanding of your blog post in the beginning).

      That is not to say I would be neutral about the subject – just that I didn’t feel it to be my place to comment on that in this case.

      Also, I would still be curious about an actual definition of a test as you see it.

      • Petteri,

        My comment about Bach’s reply was simply to point out that, if the definer of the distinction cannot give a rational refutation, then it shows that the “testing vs checking” division is merely a re-labeling of the same old ideas. That’s all.

        On to your next question, which I am glad you asked. 🙂

        I agree with you that the definition I used is inaccurate. I could attempt to “refine” the definition and say: “we (testers) attempt to prove something about our SUT, and we do so via a test.” But the question still remains: what *exactly* is a test?

        The inaccuracy stems from the the fact that the word “test,” in English, can be both a verb and a noun. Further, “test” and “test case” are used as synonyms in the industry. When you compound those issues with the inherent ambiguity of natural language grammar (English, in this case), it becomes clear that we need a more rigorous treatment.

        Such is the limitation of definitions based on natural languages. That’s why the natural language definitions for “test” (from the IEEE 829, to the definition I used above, to Kaner’s own definition) are wordy, confusing, and, in the end, inaccurate.

        So, I’ll share with you the mathematical definition of a test that I’ve been using for my research.

        A test is a 3-tuple (a, s, o) where

        – a (greek lower-case alpha) is a finite set called the alpha-set of a test and contains the pre-conditions.
        – s (greek lower-case sigma) is a finite sequence called the sigma-sequence of a test and contains the steps of a test, in order.
        – o (greek lower-case omega) is a finite set called the omega-set of a test and contains the post-conditions.

        That 3-tuple form, (a, s, o), is called the *canonical form* for a test (or test case). There is evidence (much like the Church-Turing thesis for computation) that shows that every test has a canonical form, and I’ve been actively working on the proof.

        Granted, this definition might seem supremely simple and abstract at first glance. However, we can derive all the known and useful aspects of a “test” (and how it interacts with its associated SUT) from it in the form of models, cores, traces, abstract states, the execution operator, and so on. They just follow from this simple definition and they’re all precisely defined.

        Now, when I say “every test,” I really mean any kind of test: automated, manual, exploratory, written, undocumented, et cetera. Another way to explain it is by analogy with another mathematical structure: whether or not it is written down, with red ink or in the air, a matrix is still a matrix.

        Also, from this definition of a test, we are able to precisely define many useful things like test coverage, test equivalence (i.e. duplicates), along with many practical applications like test scheduling and inference.

        I encourage you to watch the demos for the inference and scheduling engines (easily located in the ortask website), as they are two of the many exciting and practical results that follow from the mathematical definition.

        I also invite you to check out the current series of posts about test equivalence and to comment on them. Cross-pollination if ideas has always been the driver of progress for all of the sciences, and this is much needed in our own.

        Thanks Petteri.

        Mario Gonzalez

        • By the way, I plan to explain all these things in future blog posts (from the 3-tuple definition, to abstract states, the step relation, et cetera). So stay tuned.


Leave a comment

No trackbacks yet.