On Wed, Jun 14, 2023 at 02:27:46PM +0200, Dmitry Vyukov wrote: > On Wed, 14 Jun 2023 at 04:04, Darrick J. Wong <djwong@xxxxxxxxxx> wrote: > > On Tue, Jun 13, 2023 at 08:49:38AM +0200, Dmitry Vyukov wrote: > > > On Mon, 12 Jun 2023 at 18:16, Jan Kara <jack@xxxxxxx> wrote: > > > CONFIG_INSECURE description can say something along the lines of "this > > > kernel includes subsystems with known bugs that may cause security and > > > data integrity issues". When a subsystem adds "depends on INSECURE", > > > the commit should list some of the known issues. > > > > > > Then I see how trading disabling things on syzbot in exchange for > > > "depends on INSECURE" becomes reasonable and satisfies all parties to > > > some degree. > > > > Well in that case, post a patchset adding "depends on INSECURE" for > > every subsystem that syzbot files bugs against, if the maintainers do > > not immediately drop what they're doing to resolve the bug. > > Hi Darrick, > > Open unfixed bugs are fine (for some definition of fine). > What's discussed here is different. It's not having any filed bugs at > all due to not testing a thing and then not having any visibility into > the state of things. Just because syzbot doesn't test something, it does not mean the code is not tested, nor does it mean the developers who are responsible for the code have no visibility into the state of their code. The reason they want to avoid this sort of corruption injection testing in syzbot is that it *does not provide a net benefit* to anyone. The number (and value) of real bugs it might find are vastly outweighed by the cost of filtering out the many, many false positives the testing methodology raises. Keep in mind that syzbot does not provide useful unit and functional test coverage. We have to run tests suites like fstests to provide this sort of functionality and visibility into the *correct operation of the code*. However, alongside all the unit/functional tests in fstests, we also have non-deterministic stress and fuzzer tests that are similar in nature to syzbot. They often flush out weird integration level bugs before we even get to merging the code. These non-deterministic stress tests in fstests have found *hundreds* of bugs over the *couple of decades* we have been running them, and they also have a history of uncovering entire new classes of bugs we've had to address. At this point, syzbot is yet to do prove it is more than a one-trick pony - it typically only finds a single class of filesystem bug. That is, it only finds bugs that are related to undetected physical structure corruption of the filesystem that result in macro level failures (crash, warn, hang). Syzbot does nothing to ensure correct behaviour is occuring, that data integrity is maintained by the filesystem, that crash recovery after failures works correctly, etc. These things are *by far* the most important things we have to ensure during filesystem development. IOWs, the sorts of problems that syzbot finds in filesystems are way down the list of important things we need to validate. Yes, structural validation testing is something we should be running, and it's clear that is does get run (both from fstests and syzbot). Hence the claim that "because syzbot doesn't run we don't have visibility of code bugs" is naive, conceited, incredibly narcissistic and demonstratable false. It also indicates a very poor understanding of where syzbot actually fits into the overall engineering processes. -Dave. -- Dave Chinner david@xxxxxxxxxxxxx
