Pro-testing Against Bad Software Quality

22May/1210

Physics, Testing and Academia

The Introduction

I just read the Science Daily article: Quantum Physicists Show a Small Amount of Randomness Can Be Amplified Without Limit and, while reading, it occurred to me that quantum physics seems a bit like the context-driven approach to testing. Consider, for example, the following claims:

  • there is an element of uncertainty present at all times - we can not know all affecting variables
  • there is an element of randomness, beyond our control as observers, present in any system
  • it is unlikely we could ever have complete knowledge of the state of a system being observed

Sound familiar? Similarly, I find it relatively easy to (loosely) link classical physics with the more scholarly approaches to testing. Again, consider the following:

  • all effects are predictable and can be accurately represented in numbers and formulas
  • the state of a system being observed can be accurately measured at any point in time
  • standardization is valuable, and necessary

Exact, deterministic and absolute. Unfortunately, that leads me into the rant part.

The Rant

As I recently discussed with a PhD-in-physics friend of mine, the biggest problem I have with the way many people of science approach, well anything, is that the approach is often so absolute and unswerving. "There is no 'maybe' or 'what if', the formula shows how it is. Who are you to question Newton or Einstein?" To me, this seems hierarchical, authoritative, stagnatory, unimaginative, stubborn, cynical, naïve even.

When I was a kid I always thought the purpose of a scientist was to explore new possibilities and alternatives, not to dismiss them as irrelevant or stupid without a moment's thought if they disagreed with the leading views. I used to think that the purpose of a scientist was to constantly question everything in order to learn and to improve, not to meekly accept the Word of Truth(TM) handed down by higher authority figures. Unfortunately, based on approximately 20 years of observing people of academia, that does not seem to be the case very often.

I find it highly frustrating when a casual conversation first turns into an interesting debate and then rapidly degrades to almost religious raving when the opponent flatly refuses to even consider the possibility that there might be some level of uncertainty involved or that maybe the current facts of science related to the subject are actually no facts at all. Maybe they're just best guesses that were made based on our current level of understanding and limited by the sensitivity of the equipment used to measure. 3000 years ago scientists knew the Earth is flat. 100 years ago scientists knew that the speed of sound can not be exceeded. I don't think many people would share those notions now.

The Finale

What I'm trying to say here is that I, personally, would immensely enjoy seeing (more of) the people of hard sciences go into the critical questioning mode a little easier and just a tad more often. You don't lose your face or professional credibility for stating "we might not have perfected this yet", or "I wonder if it would be possible we've gotten this all wrong".

Done.

PS. Do note that I'm in no way belittling or disparaging natural sciences themselves. I wouldn't be here writing this rant without the practical applications and solutions that mathematics, chemistry and physics have brought along. I only criticize the overly fixed mindset of (some of) the people.

Comments (10) Trackbacks (1)
  1. I had the same thoughts a few years ago reading a lot about Niels Bohr while at the same time starting to embrace context driven testing systematically. And I agree completely: Overly fixed mindsets are a danger to our craft – and to all of software engineering, IMO.

    Quantum physics and especially the philosophy behind it certainly has a lot in common with context driven testing. I wrote this blog post about it back in 2010. I think you may find it interesting:
    blog.asym.dk/2010/12/21/bohr-on-testing/

    Fundamentally, it’s about embracing uncertainty and working with it as a premise, not a problem.

    Best,
    Anders

    • Hi Anders,

      Thank you for the comment. I’ll be sure to read your blog post because this subject has really been bugging me recently and I’m trying to think of new ways to break through the seemingly impenetrable stubbornness. I don’t want to just give up because these are intelligent people plus I want to learn to understand better what makes a person fall back to this kind of defense.

      • Hi Petteri

        Thanks for your reply and for your interest in my blog.

        I think it’s a difficult challenge you’re taking up there, but you’re not alone, and I honestly think we’re on the right path. Maybe I’m just over optimistic…

        The interesting thing about Niels Bohr is that he realised the philosophical potential of his theories already back in the 1920’s – and we’re still only beginning to understand what he understood then. He was a genius! – and I’m a big fan 😉

        But even geniuses can have hard times getting this: Einstein never accepted it. His words “God doesn’t play dice with the world” are still famous. The fact is that he was right about this, but that’s another story.

        /Anders

  2. ◦all effects are predictable and can be accurately represented in numbers and formulas
    ◦the state of a system being observed can be accurately measured at any point in time

    I dont know anyone working in testing who believes this.

    • Unfortunately, I do. Maybe not in these exact terms but the principles apply. Not surprisingly, I dare say, these same people are also pro-certification.

  3. Hi, Petteri!

    Once again I sat in the train thinking “what to go with the time given to me by this monopolistic railroad company”. So I read as many blogposts as I could and commented them all.

    I love the Agile flavor here: Fail, Analyze, Learn, Try, Fail again -> http://how-do-i-test.blogspot.com/2012/04/in-face-of-failure-part-2.html

    Why shouldn’t the scientists be wrong? I would love the attitude that they are wrong! That way more people could be making science, innovate and challenge the existing “truth”.

    “Who are you to question Newton or Einstein?” I’m a tester. They would have loved if I’d question them. They would have loved to make their theories even better! Or so I think. 😉

    BR, Peksi

    • I sure hope so too. I would love to help a scientist simply by asking questions – especially if the said scientist was willing to have his/her ideas and theories questioned by a “layman” (in the sense that no, I do not have a doctorate in physics or any other subject but that doesn’t mean I can’t think).

  4. Thanks Petteri. I found this very interesting. It not only ties in with my article about testing and standards, it is also very relevant to an article that I have just written about the way that language shapes our thoughts, and how we should be careful about the language we use when we are analysing problems and reporting on them. I was writing about E-prime, a stylised form of English that forces us to realise that our verdicts are provisional and subjective, rather than final and absolute. I believe that it is consistent with the scientific method, and with good testing. The article should be in the July issue of Testing Planet.

    I thought that scientists were comfortable with uncertainty and the provisional nature of their work. Experiments confirm hypotheses, but this is provisional until they come up with a fuller or better explanation.

    Last week I was speaking to a school teacher of physics here in Scotland. We were discussing the new Scottish schools curriculum (which is a completely different system from England). The teacher has a PhD in physics. He was talking approvingly about the way that the new curriculum tries to make pupils think for themselves and understand probability, uncertainty and error, instead of simply memorising formulae.

  5. I find that point of view surprising. When I worked on my diploma thesis I was constantly reminded to look out for experiments & results that would render existing models wrong. That way I do see a parallel between software development & science:
    Both deal with the real world and models of it. The ‘real world’ of science being, well, the real world, the reality of SW development the ‘real requirements’ or expectations (whether written down in some way or not).
    The ‘model’ in SW development is the software itself; the experiments are the tests.

    Whether in science or software testing, part of the job is to find out when/where the model doesn’t match reality anymore.

    That said, I con’t (completely) follow the analogy with classical & quantum mechanics: There are plenty of systems that behave as classical physical systems, yet cannot be predicted (whether forecast, climate changes, ocean currents, essentially any non-linear system), while some quantum systems can in fact be predicted precisely (if only given a large enough number of measurements) (the Mößbauer effect is a good example in my opinion) … and in fact modern computers: Driven by quantum physics effects, they’re pretty predictable. Oh well, to the extend that we can predict, or else we wouldn’t need software testers. 😉

    And there are hard problems in SW development: The traveling salesman problem & friends…

    • Stephan,

      Thank you for the comment. I welcome the critique with open arms – especially when it’s well-argumented.

      Talking about the points you make, I do believe I understand what you’re getting at but I must disagree with what you say about the (non-)predictability of various systems. The problem, in my view, lies in the limitations of the observations, measurements if you prefer, we (can) make. I’ll use weather forecasts as an example.

      Predicting the weather may seem like a (relatively simple) classical physical system but there are really a limitless number of variables affecting it and therein lies the problem. In part, this goes closer to philosophy, so I’ll try to keep it brief (slim chance of that, though, I’m afraid).

      Basically, every time you move or breathe or even when your heart pounds, you’re causing a seemingly insignificant change in the local environment around you. The very same miniscule effect happens around everything that lives and breathes and moves voluntarily (more about that in a bit). Now, I’m a firm believer in the chaos theory and I’m going to base this on the premise that everything is related to everything. That might be a false premise but it is the one I’ll go with here (do feel free to contradict me).

      According to chaos theory, even a just barely measurable change can have a huge impact in the long-term. Take sharpshooting as an example: the further away the target gets from the shooter, the more pronounced the effect of a minor shift in the shooter’s aim becomes and the less confidence there is about the shooter hitting the bullseye.

      This exact same principle applies to predicting the weather – the system involved is just immensely more complex. We can make relatively accurate (though sometimes completely false – it was supposed to be sunny today and it rained half the day) predictions in the short-term but the further ahead we go, the less accurate the predictions become until, at some arbitrary point in time, we have nothing more than wild guesses.

      Now, in my view, the challenge with making the predictions lies in the fact that there are so many variables at play we simply can not take all of them into account within any humanly reasonable time or amount of effort. So we only use certain major variables that we are aware of (to some degree, at the time) and feed them into our prediction model in order to come up with a (relatively reliable) forecast.

      However, as anyone who’s ever been caught in rain after a weather forecast promised sunshine all day long knows, sometimes the outcome is just all wrong. To me, this implies a much larger level of uncertainty than the meteorologists would like to admit. For the most part, however, that’s fully okay – the (short-term) forecasts are accurate enough most of the time but I wouldn’t accept the current weather prediction model as a standard.

      Now, the philosophical part of this lies in the fact (as based on the premise mentioned earlier) that in order for you to be able to make fully accurate predictions you would have to know, exactly, every single move every living creature makes. That would imply determinism which would, necessarily, force you to give up the notion of free will. If my movements have an effect in your model you must know exactly when and how I move without me being able to affect that.

      Of course, all of the above supports what you said but, then again, the part that I mostly disagree with is just the premise of weather forecasts (or the movement of ocean currents) being (solely) governed by classical physics. I would add a strong quantum effect in the equation – with the word ‘quantum’ being analogous to ‘random’ in this case due to our inability to account for, or even know, all affecting variables.

      I must confess I’m not familiar with the Mössbauer effect so I can’t really comment on that but I will find out more about it, now that you mentioned it.


Leave a comment