Hi, Thanks a lot for the free book, it is very useful! I'd like to point out an error I see in Section 11.6.3.2, after Quick Quiz 11.10. It says the following: Suppose that a given test fails about once every hour, but after a bug fix, a 24-hour test run fails only twice. What is the probability of this being due to random chance, in other words, what is the probability that the fix had no statistical effect? In other parts of the book, care has been taken to say something like "confidence level" to make the probability statements correct. Not here. The only thing that we know about the probability of failing after the experiment is that "the probability of the test failing now is not zero". To know the "probability of this being due to random chance", or "probability that the fix had no effect", requires the knowledge of the behaviour of the bug itself (or, at least, the prior probability that the perhaps buggy system has a certain behaviour), and it is not specified in the question. E.g., in an extreme world that the test can only fail at a rate of once every hour or never at all, we must admit that we are very very lucky during this particular test. The probability being calculated needs to be specified like "the probability of this happening *in case* the fix has no effect". That "the fix has no effect" needs to be a *prerequisite*, and cannot itself be the probability to be computed. The complement, i.e., 1 - 1.2e-8, is "the confidence level that the probability of failing is less than the original". I'd suggest using the "confidence level" wording here, but explain what it is earlier in the book to tell the less mathematically apt readers understand the wordings. Regards, Isaac -- To unsubscribe from this list: send the line "unsubscribe perfbook" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
