Pro-testing Against Bad Software Quality

6Apr/134

An Object Is What the Object Does – Or Is It?

It's been a while since my last blog post and earlier today I came across a post that started a thought process I just had to write down. This post is based on the blog entry titled "“Testing vs Checking” : ‘Tomato’ vs ‘Tomato’" by Mario Gonzalez of Ortask.

As a quick recap, the gist of Mario's blog post is to criticize the differentiation James Bach, Michael Bolton and, generally, the people aligned with the context-driven approach to testing make between the terms "testing" and "checking". He basically goes on to claim the whole differentiation is a moot point or even potentially harmful to the testing community, and its future, as a whole (in terms of adding confusion to a field already swamped with definitions). Simplified, but that is my interpretation of the text.

[UPDATE April 8th]: As pointed out by @JariLaakso in Twitter, not all context-driven testers necessarily make the distinction between testing and checking, so I can't really say "generally, the people aligned with..." as I am not aware of the exact number of people who actually do. I may have simply just communicated with people who do make the distinction so reformat that one sentence in your mind to your liking. [END UPDATE]

However, I am not going to go any deeper into that or any of the other problems I perceived while reading through the post, at least not for now - I'm sure other people will. Instead, I will concentrate on the one thing that bothered me the most at the time of reading it: Mario's definition of a test. So I decided to dissect it. Here is the definition he offered:

a test attempts to prove something about a system

First of all, that is not even a definition of a test - it is a definition of the purpose of a test. Let me clarify with an analogy. Consider this for a definition of a car:

A car is used to transport people and goods from one place to another

Now, based on that "definition" alone, answer me the following questions:

  1. What does a car look like?
  2. What principles and mechanisms does a car operate on?
  3. Under what conditions and circumstances can a car be used to transport people and goods from one place to another?

You can't, can you? That's because I haven't given you a definition of a car - only what it's typically used for. In other words:

Defining what an object does does not define what the object is.

While still lacking and incomplete, a definition of a car could be something like: "A car is a typically four-wheeled, land-based vehicle that usually operates on the principle of an internal combustion engine turning the energy contained within a liquid fuel into mechanical movement through a series of controlled explosions the pressure of which cause a crankshaft to rotate and apply torque to the vehicle's drive wheels".

While the non sequitur I pointed out above would be reason enough to stop going any further, I want to go through the definition (of the purpose of a test) for the sake of completeness:

  • "A test" - Now what is that? In this context the question is impossible to answer - Mario hasn't told us!
  • "attempts" - How does "a test" attempt anything? It's not a sentient being. It would seem to me it is the tester who is the one to make the attempt through performing a test, making observations, analyzing and interpreting data, behavior and results before, during and after performing a test.
  • "to prove" - What, exactly, constitutes proof? Here are some definitions of the term:
    • evidence or argument establishing a fact or the truth of a statement (Oxford dictionaries)
    • the cogency of evidence that compels acceptance by the mind of a truth or a fact (Merriam-Webster)
    • sufficient evidence or an argument for the truth of a proposition (Wikipedia)

Now, how does one arrive at "proof", based on the above definitions? An obvious problem that immediately comes to mind is that, in many cases, "truths" or "facts" are relative and dependent on a number of factors. Don't believe me? Well, this is the last sentence in this blog entry. True at the time of writing, but false only seconds later when I kept going.

Also, if you want to pursue the scientific angle, I don't think anyone would take the result of a single experiment as any kind of proof of anything. You would need to repeat the experiment multiple times (and, ideally, have an independent peer group do the same) in order for it to gain any kind of credibility but therein lies a problem: the conditions would need to be exactly the same every time and that is virtually impossible to achieve in software testing. The date and time change, the amount of available CPU power, RAM and hard disk space varies, there might be network congestion or packet loss that you can not possibly predict, another application might suddenly misbehave and distort your test results or any number of other, unknown, factors that can affect the outcome of a test could manifest themselves.

It would seem to me that "proof" is much too strong a word to use in this context. Testing may suggest behavior that appears consistent but it can not prove that any more than a turkey being generously fed on a daily basis can predict that on the 180th day of its life the farmer comes out with an axe instead of seeds and takes the turkey's life instead of feeding it.

On with the definition:

  • "something" - Anything? Well, Mario did elaborate on this in the following paragraph so I'll just leave it at that.
  • "about a system" - Now there's another interesting word: "system". Various definitions to follow:
    • 1 a set of things working together as parts of a mechanism or an interconnecting network; a complex whole, or
    • 2 a set of principles or procedures according to which something is done; an organized scheme or method (Oxford dictionaries)
    • a group of devices or artificial objects or an organization forming a network especially for distributing something or serving a common purpose (Merriam-Webster)
    • a set of interacting or interdependent components forming an integrated whole or a set of elements (often called 'components') and relationships which are different from relationships of the set or its elements to other elements or sets (Wikipedia)

Complexity. On multiple levels. Especially when talking about computers and software. What if there is a fault in the hardware such that it exhibits certain behavior at one time but a different behavior at other times? Maybe a capacitor is nearing the end of its life and causes a test to give different results based on the ambient temperature of the environment in which the system resides. How many are prepared to, or even capable of, anticipating and accounting for something like that?

I'm not even trying to be funny here - my stereo amplifier does this and for exactly that reason!

Based on all of the above, I'm afraid I can only arrive at the conclusion that this particular definition of a test is fundamentally flawed (even starting from the fact that the claim of what is being defined is unrelated with the actual definition presented) and, in my view, would warrant serious reconsideration and refining.

2Aug/100

Because You’re Worth It…

This post should probably have been named as "Brothers in Arms, Part 3" as it was inspired by the blog post Two "Scoops" of "Bugs" by James Bach but I felt like doing things a little differently this time. Just to keep things fresh. If that's even a valid expression as most of this post is about something I wrote back in 2001 (I seem to have been rather productive back then).

As comments to what James Bach wrote, an obvious problem with the loose use of language is the communication gaps that it opens up between individuals. Simply put: more room for interpretation, higher chance of misunderstandings (pretty obvious, eh). Then again, as he mentioned in his post, it also makes communication smoother when you don't drill down to exact specifics of every little nuance and detail right away as going too deep too fast could lead to losing the big picture which would not be very likely to be a good thing. Balancing these two (high-level communication versus gory, but necessary, details) is an art form on its own, imo - a tester needs to speak manager in addition to tester, with maybe a little added marketing accent, to get things straight.

Anyway, a little something from the past:

There can be no single logical mappings between words and concepts - not even within a single language - simply because words are nothing more than words; generally-agreed symbols one after another - creating a generally-agreed "meaningful" combination. In actuality, words are just empty placeholders or pointers, if you want to put it that way - without any intrinsic or implicit content or value. It is us humans, as unique individuals, who fill those placeholders/pointers with subjective meanings that are unique to each and everyone and are entirely based on emotions.

Since there are and can never be two individuals exactly alike, there is no way a single word could have the exact same content for any two persons, unless explicitly agreed on, between those individuals. Thus, dictionary words aren't "true" or "right", they are just meanings intersubjectively agreed on by a group of people. Then again, the purpose of a dictionary is not to enforce meanings to words but, to explain - reflect, if you want - generally agreed meanings of those words at the time.

Media sells products with advertisements filled with images "giving meaning" to those empty words (words criticized by Richard Rorty) as if those words in themselves meant something and, you had to buy the product in order to become what the images are trying to associate you with. "Buy product X and you will become beautiful. Naturally, because you're worth it". See how it goes? Is it really me who is worth the product? Is it really owning the product that makes me worth something? At least, that is what many advertisements want to imply. What these advertisements also imply (but leave unsaid) is the image of what happens if you don't buy the product advertised; from that viewpoint the advertisement reads: Not buying this product makes you less worthy, less beautiful and less desirable.

No wonder people are confused and anxious.

Come to think of it, the Most Questionable Title for a Profession that I can think of would go somewhere along the lines of: Marketing Psychologist. O' fear ye mortal ones.

While, again, this is not directly testing-related, it's very strongly people-related, which does make it relevant. The single biggest reason for failed software projects in my experience is the lack of proper communication. One way or another. Which makes things interesting as one of my goals in testing is to try and help close down those communication gaps by pointing them out to people. Is that quality assistance?

6Jul/109

Brothers in Arms, Part 1

The art of making enemies?

I'm pretty sure it's not exactly the best approach around to start your life in the blogosphere (or anywhere, for that matter) by criticizing people, but hey, this blog is about protesting against poor software quality and quality always starts with the people involved (feel free to contradict me on that, I'd love to see someone explain to me why I'm wrong) - their attitudes included - so if a big part of the problem is in the attitudes and mental approaches of the people instead of the methods or techniques used, then that's what I will be protesting against today.

Starting the series...

As stated before, I will be posting a lot about my thoughts and views on what my colleagues in the testing world write in their own blogs and webpages or what I've been discussing with them through one medium or another.

The "honor" for the first post in this series goes to Rob Lambert for his excellent post Don't be a follower, be a tester, which woke a multitude of emotions and thoughts in me. Actually, the post irritated me. Not because I would disagree, but because he's so damn right in what he says. Briefly put: the irritating essence of the post is that the testing world is full of sheep bleating trendy catchphrases that make things sound really nice while actually meaning absolutely nothing (a favorite subject of criticism by Richard Rorty).

What really caught my attention were these three conclusions Rob made, based on the comments to one of his earlier posts:

We should not challenge the best practices of testing
We should not challenge the experts in testing
We should not talk about testing publicly
unless we are an expert or we know the experts.

My immediate first thought, after reading this was: What, exactly, constitutes an expert in testing? How does one become an expert in testing? Added bonus question: How, exactly, would knowing an expert in testing lend more credibility to what I am saying? I'm not the expert in question, am I?

So, how does this becoming-an-expert-in-testing happen? by not challenging the current authority figures? By not trying to come up with something new and/or improved? By not tackling the problems you see and just serenely accepting them (*baah*, said the sheep and continued munching on grass) as a necessary - and unchangeable - part of the world you're living in?

Let me answer that for you: It happens by stirring the hornet's nest, by challenging the consensus. It happens by telling people that they're doing things in an ineffective way, when they are, and then proposing an alternate way. It happens by not just accepting everything that's being force-fed to you without questioning. It happens by making mistakes, learning from them and then coming back stronger and better equipped than before. It happens by acknowledging the fact your thoughts and ideas could be shot down and torn to shreds in public, and still having the courage to present them despite the risk.

In brief: One does not become an expert (in testing) by being an idle spectator. You do it by taking a stand and making it your very own, personal, responsibility to change the way things are done. And that sure as hell won't happen if you don't throw your ideas out there for people to see and have a taste of. Of course, this is just my personal view - your mileage may vary.

Dramatic, I know, but sometimes being an opinionated tester is like the IT-world equivalent of being John Rambo...

As a sidenote: seems I just discovered a defect in WordPress while writing this post. 😛
Sidenote #2: seems I just discovered another defect in WordPress when trying to add more tags to this post (sorry Rob, WordPress refuses to store your name as a tag with Capital First Letters, I'm not trying to be disrespectful here...)