On Sat, 24 Sept 2022 at 00:07, Jeff King <peff@xxxxxxxx> wrote: > > I wondered if we might have caught this in a more automatic way. The > output of: > > cd Documentation > ./doc-diff cc75e556a9^ cc75e556a9 > > makes the problem apparent, but I don't fault reviewers for not running > it. I rarely remember to do so. And in general you need a human looking > at doc-diff output to know if it was the intended change or not. > > I wondered if it might be worth running > > ./doc-diff v2.37.0 v2.38.0-rc1 > > near a release to scan over all of the changes. But the diff is over > 8000 lines, and I admit my eyes glazed over before I got to the > problematic hunks (even though I knew I was looking for them!). I know how you felt when you looked at that doc-diff... I was lucky enough to first look at ./doc-diff v2.37.0 v2.38.0-rc0 so that when I then looked at ./doc-diff v2.38.0-rc0 v2.38.0-rc1 this stood out quite well. > So I dunno. I think doc-diff is a potentially useful tool, but I'm not > sure how to point the human attention at the right spot to find a bug. I try to do some such doc-diffing every now and then, and in particular around rc time. It has caught a few buglets, usually nothing major. I've also done ./doc-diff origin/next origin/seen at times to catch such things a lot earlier, but it's not often that I find the time to do so / think about doing it. > Maybe "given enough eyeballs, all bugs are shallow" is our best bet > here. After all, it did find this bug before the release. :) Yeah, and luck never hurts -- I think it's fair to assume that I would have missed this bug if it hadn't come in after rc0. Martin
