Re: fuzzing bcachefs with dm-flakey

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Mon, May 29, 2023 at 04:59:40PM -0400, Mikulas Patocka wrote:
> Hi
> 
> I improved the dm-flakey device mapper target, so that it can do random 
> corruption of read and write bios - I uploaded it here: 
> https://people.redhat.com/~mpatocka/testcases/bcachefs/dm-flakey.c
> 
> I set up dm-flakey, so that it corrupts 10% of read bios and 10% of write 
> bios with this command:
> dmsetup create flakey --table "0 `blockdev --getsize /dev/ram0` flakey /dev/ram0 0 0 1 4 random_write_corrupt 100000000 random_read_corrupt 100000000"

I've got some existing ktest tests for error injection:
https://evilpiepirate.org/git/ktest.git/tree/tests/bcachefs/single_device.ktest#n200
https://evilpiepirate.org/git/ktest.git/tree/tests/bcachefs/replication.ktest#n491

I haven't looked at dm-flakey before, I take it you're silently
corrupting data instead of just failing the IOs like these tests do?

Let's add what you're doing to ktest, and see if we can merge it with
the existing tests.

> I created a bcachefs volume on a single disk (metadata and data checksums 
> were turned off) and mounted it on dm-flakey. I got:
> 
> crash: https://people.redhat.com/~mpatocka/testcases/bcachefs/crash1.txt
> deadlock: https://people.redhat.com/~mpatocka/testcases/bcachefs/crash2.txt
> infinite loop: https://people.redhat.com/~mpatocka/testcases/bcachefs/crash3.txt

Fun!

> Here I uploaded an image that causes infnite loop when we run bcachefs 
> fsck on it or when we attempt mount it:
> https://people.redhat.com/~mpatocka/testcases/bcachefs/inf-loop.gz
> 
> 
> I tried to run bcachefs on two block devices and fuzzing just one of them 
> (checksums and replication were turned on - so bcachefs shold correct the 
> corrupted data) - in this scenario, bcachefs doesn't return invalid data, 
> but it sometimes returns errors and sometimes crashes.
> 
> This script will trigger an oops on unmount:
> 	https://people.redhat.com/~mpatocka/testcases/bcachefs/crash4.txt
> or nonsensical errors returned to userspace:
> 	rm: cannot remove '/mnt/test/test/cmd_migrate.c': Unknown error 2206
> or I/O errors returned to userspace:
> 	diff: /mnt/test/test/rust-src/target/release/.fingerprint/bch_bindgen-f0bad16858ff0019/lib-bch_bindgen.json: Input/output error
> 
> #!/bin/sh -ex
> umount /mnt/test || true
> dmsetup remove_all || true
> rmmod brd || true
> SRC=/usr/src/git/bcachefs-tools
> while true; do
>         modprobe brd rd_size=1048576
>         bcachefs format --replicas=2 /dev/ram0 /dev/ram1
>         dmsetup create flakey --table "0 `blockdev --getsize /dev/ram0` linear /dev/ram0 0"
>         mount -t bcachefs /dev/mapper/flakey:/dev/ram1 /mnt/test
>         dmsetup load flakey --table "0 `blockdev --getsize /dev/ram0` flakey /dev/ram0 0 0 1 4 random_write_corrupt 100000000 random_read_corrupt 100000000"
>         dmsetup suspend flakey
>         dmsetup resume flakey
>         cp -a "$SRC" /mnt/test/test
>         diff -r "$SRC" /mnt/test/test
>         echo 3 >/proc/sys/vm/drop_caches
>         diff -r "$SRC" /mnt/test/test
>         echo 3 >/proc/sys/vm/drop_caches
>         diff -r "$SRC" /mnt/test/test
>         echo 3 >/proc/sys/vm/drop_caches
>         rm -rf /mnt/test/test
>         echo 3 >/proc/sys/vm/drop_caches
>         cp -a "$SRC" /mnt/test/test
>         echo 3 >/proc/sys/vm/drop_caches
>         diff -r "$SRC" /mnt/test/test
>         umount /mnt/test
>         dmsetup remove flakey
>         rmmod brd
> done
> 
> The oops happens in set_btree_iter_dontneed and it is caused by the fact 
> that iter->path is NULL. The code in try_alloc_bucket is buggy because it 
> sets "struct btree_iter iter = { NULL };" and then jumps to the "err" 
> label that tries to dereference values in "iter".

Good catches on all of them. Darrick's been on me to get fuzz testing
going, looks like it's definitely needed :)

However, there's two things I want in place first before I put much
effort into fuzz testing:

 - Code coverage analysis. ktest used to have integrated code coverage
   analysis, where you'd tell it a subdirectory of the kernel tree
   (doing code coverage analysis for the entire kernel is impossibly
   slow) and it would run tests and then give you the lcov output.

   However, several years ago something about kbuild changed, and the
   method ktest was using for passing in build flags for a specific
   subdir on the command line stopped working. I would like to track
   down someone who understands kbuild and get this working again.

 - Fault injection

   Years and years ago, when I was still at Google and this was just
   bcache, we had fault injection that worked like dynamic debug: you
   could call dynamic_fault("type of fault") anywhere in your code,
   and it returned a bool indicating whether that fault had been enabled
   - and faults were controllable at runtime via debugfs, we had tests
   that iterated over e.g. faults in the initialization path, or memory
   allocation failures, and flipped them on one by one and ran
   $test_workload.

   The memory allocation profiling stuff that Suren and I have been
   working on includes code tagging, which is for (among other things) a
   new and simplified implementation of dynamic fault injection, which
   I'm going to push forward again once the memory allocation profiling
   stuff gets merged.

The reason I want this stuff is because fuzz testing tends to be a
heavyweight, scattershot approach.

I want to be able to look at the code coverage analysis first to e.g.
work on a chunk of code at a time and make sure it's tested thoroughly,
instead of jumping around in the code at random depending on what fuzz
testing finds, and when we are fuzz testing I want to be able to add
fault injection points and write unit tests so that we can have much
more targeted, quicker to run tests going forward.

Can I get you interested in either of those things? I'd really love to
find someone to hand off or collaborate with on the fault injection
stuff in particular.

> Bcachefs gives not much usefull error messages, like "Fatal error: Unknown 
> error 2184" or "Error in recovery: cannot allocate memory" or "mount(2) 
> system call failed: Unknown error 2186." or "rm: cannot remove 
> '/mnt/test/xfstests-dev/tools/fs-walk': Unknown error 2206".

Those are mostly missing bch2_err_str()/bch2_err_class() calls:
 - bch2_err_str(), to print error string for our private error code
 - bch2_err_class(), to convert private error code to standard error
   code before returning it to outside bcachefs code

except error in recovery, cannot allocate memory - that's ancient code
that still squashes to -ENOMEM



[Index of Archives]     [Linux Ext4 Filesystem]     [Union Filesystem]     [Filesystem Testing]     [Ceph Users]     [Ecryptfs]     [NTFS 3]     [AutoFS]     [Kernel Newbies]     [Share Photos]     [Security]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux Cachefs]     [Reiser Filesystem]     [Linux RAID]     [NTFS 3]     [Samba]     [Device Mapper]     [CEPH Development]

  Powered by Linux