S12-402: Why Tests Don’t Pass or Fail

Track: Test Strategy and Design

Most testers think of tests as either passing or failing. Either they found a bug or they didn’t. Unfortunately, experience repeatedly shows us that passing a test doesn’t really mean there is no bug. It is quite possible for an error to surface in a test but not be noticed at the time. It is also possible for bugs to exist in the feature being tested in spite of the test of that capability. “Pass” really means that we didn’t notice anything unexpected.

Likewise, failing a test is no guarantee that a bug is present. There could be a bug in the test itself, a configuration problem, corrupted data, or a host of other explainable reasons that do not mean there is anything wrong with the software being tested. “Fail” really only means that something that was noticed warrants further investigation.

This session explains these ideas further, explores some of the implications, and suggests some ways to benefit from this new way of thinking about test outcomes. It concludes with examination of how to use this viewpoint to better prepare tests and report results.

Participants will walk away knowing:

  • Why tests don’t absolutely pass or fail
  • What that means in testing software
  • Some of the implications of tests that don’t pass or fail
  • A model for understanding outside influences and hidden outcomes during testing
Download Presentation
Please Note: The presentations are intended for attendees only. The presentations page is password protected – contact info@softwaretestpro.com for verification of attendance and the password to access the presentation.

Session Speaker:

Doug HoffmanDoug Hoffman – Consultant, Software Quality Methods
Doug Hoffman is a management consultant for Software Quality Methods, LLC and has over 30 years experience as a trainer in strategies and tactics for quality assurance. His technical focus is on test automation and test oracles. His management focus is on evaluating, recommending, and leading implementation of quality improvement programs. He is the President of the Association for Software Testing (AST); a Fellow of the ASQ; and holds MBA, MSEE, and BACS degrees. He is certified by ASQ as a Software Quality Engineer and as a Manager of Quality/Organizational Excellence. He has spoken at dozens of conferences and chaired several international conferences on software quality.