Hi Ted, On Tue, Aug 20, 2019 at 11:56:23AM -0400, Theodore Y. Ts'o wrote: > On Tue, Aug 20, 2019 at 10:24:11AM +0800, Chao Yu wrote: > > Out of curiosity, it looks like every mainstream filesystem has its own > > fuzz/injection tool in their tool-set, if it's really such a generic > > requirement, why shouldn't there be a common tool to handle that, let specified > > filesystem fill the tool's callback to seek a node/block and supported fields > > can be fuzzed in inode. It can help to avoid redundant work whenever Linux > > welcomes a new filesystem.... > > The reason why there needs to be at least some file system specific > code for fuzz testing is because for efficiency's sake, you don't want > to fuzz every single bit in the file system, but just the ones which > are most interesting (e.g., the metadata blocks). For file systems > which use checksum to protect against accidental corruption, the file > system fuzzer needs to also fix up the checksums (since you can be > sure malicious attackers will do this). I personally agree with your idea, some fs specfic code needed at least. > > What you *can* do is to make the file system specific portion of the > work as small as possible. Great work in this area is Professor Kim's > Janus[1][2] and Hydra[2] work. (Hydra is about to be published at SOSP 19, > and was partially funded from a Google Faculty Research Work.) For EROFS, it's a special case since it is a RO fs, and erofs mkfs will generate reproducable images (which means, for one dir trees, it only generates exact one result except for build time). For this reason (read only), the fastest way to do fault injection for EROFS is to modify mkfs and inject some faults when writing each metadata to the generated images, that is all what I'm done that night. I think it has similar effect since EROFS is an RO fs for some given dirs (except for it cannot modify a specfic field from a given image, but from its source dir). Actually I found 6 issues related with corrupted compressed images, and it can now run smoothly for about an hour. https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git/commit/?h=experimental-fuzzer What we will do next is to develop an independent fuzzer, which need the ability of parsing the filesystem image (which share common code with erofs-fuse to parse the filesystem) and then it can inject some faults to the original image. > > [1] https://taesoo.kim/pubs/2019/xu:janus.pdf > [2] https://github.com/sslab-gatech/janus > [3] https://github.com/sslab-gatech/hydra > > > > Personally speaking, debugging tool is way more important than a running > > > kernel module/fuse. > > > It's human trying to write the code, most of time is spent educating > > > code readers, thus debugging tool is way more important than dead cold code. > > I personally find that having a tool like e2fsprogs' debugfs program > to be really handy. It's useful for creating regression test images; > it's useful for debugging results from fuzzing systems like Janus; > it's useful for examining broken file systems extracted from busted > Android handsets during dogfood to root cause bugs which escaped > xfstests testing; etc. Yes, what I want to say is that these fses (ext4, xfs, btrfs, ...) do awesome work by many people for many time. But I personally have 2 hand, 24 hours (including work hours) a day only. I will refer to these debugfs / xfs_db / btrfs ins dump-tree great work to get your experience and develop our EROFS tools. I learn a lot from this topic. Thanks, Gao Xiang > > Cheers, > > - Ted
