Short blog this time as it mainly just references Troy's work.

At one of my engagements a lot of the work the teams had to track was serial work with a (untrustworthy) third party, where the team effort is in testing results from the third party, waiting for the result, and then testing again. In these situations it is sometimes hard to determine when we have done something mainly because, as we all know, the more we test the more bugs we find. How do we feel better about knowing when we can stop?

The problem is that while we can find / fix defects and use things like rate of discovery / fix to look at trends, it really doesn’t tell us anything about end quality from the end user perspective.

I found this article by Troy Magennis - Latent Defect Estimation - where he talks about a way to estimate the amount of total defects we have based on collecting defect data in a slightly different way (basically set up two groups to look at defects in area of interest, and record all duplicates). The approach gives you an estimate for how many undiscovered defects you have which then can be used to improve decision making (eg do we release now?).

It seems to me that the approach could be applied in many situations, especially where there is manual or serial testing involved such as many regression test situations, final UAT in projects and so on.

Note that, like Troy, I feel compelled to say “do everything you can to not introduce defects into your system in the first place”. This approach is useful in situations where you know there are defects and need to understand what is likely to happen.

I was thinking that this would be a good experiment to run and expect I I'll do so.