On 13/03/2017 18:11, Junio C Hamano wrote:
Vegard Nossum <vegard.nossum@xxxxxxxxxx> writes:
However, I think it's more useful to think of these testcases not as
"binary test that nobody knows what they are doing", but as "(sometimes
invalid) packfiles which tickle interesting code paths in the packfile
parser".
With this perspective it becomes clearer that while they were generated
from the code, they also in a sense describe the packfile format itself.
I do agree with these two paragraphs (that is why I said that
continuously running fuzzer tests on the codebase would have value),
and I really appreciate the effort.
I did a few experiments in changing the code of the packfile reader in
various small ways (e.g. deleting a check, reordering some code) to see
the effects of the testcases found by fuzzing, and I have to admit it
was fairly disappointing. The testcases I added did not catch a single
buggy change, whereas the other testcases did catch many of them.
In short, the summary of the above three paragraphs is that we still
do believe the general approach of using fuzzer has value, but your
experiment indicates that data presented in the patch in this thread
weren't particularly good examples to demonstrate the merit?
Correct.
I thought a priori that the testcases found by AFL would work well as a
regression suite in the face of buggy code changes, but this turned out
not to be the case in practice when I tried introducing bugs on purpose.
The testcases would still have value for the following purposes:
- as a seed for continued fuzzing (as the fuzzing effort would not have
to start over from scratch)
- as a way to quickly find an input that reaches a specific line of code
without having to manually poke at a packfile
- as a basis for writing new testcases with specific expected results
Vegard