Vegard Nossum <vegard.nossum@xxxxxxxxxx> writes: > However, I think it's more useful to think of these testcases not as > "binary test that nobody knows what they are doing", but as "(sometimes > invalid) packfiles which tickle interesting code paths in the packfile > parser". > > With this perspective it becomes clearer that while they were generated > from the code, they also in a sense describe the packfile format itself. I do agree with these two paragraphs (that is why I said that continuously running fuzzer tests on the codebase would have value), and I really appreciate the effort. > I did a few experiments in changing the code of the packfile reader in > various small ways (e.g. deleting a check, reordering some code) to see > the effects of the testcases found by fuzzing, and I have to admit it > was fairly disappointing. The testcases I added did not catch a single > buggy change, whereas the other testcases did catch many of them. In short, the summary of the above three paragraphs is that we still do believe the general approach of using fuzzer has value, but your experiment indicates that data presented in the patch in this thread weren't particularly good examples to demonstrate the merit?