On Fri, Nov 04, 2016 at 05:18:00PM -0700, Darrick J. Wong wrote:
> Previously, our XFS fuzzing efforts were limited to using the xfs_db
> blocktrash command to scribble garbage all over a block. This is
> pretty easy to discover; it would be far more interesting if we could
> fuzz individual fields looking for unhandled corner cases. Since we
> now have an online scrub tool, use it to check for our targeted
> corruptions prior to the usual steps of writing to the FS, taking it
> offline, repairing, and re-checking.
>
> These tests use the new xfs_db 'fuzz' command to test corner case
> handling of every field. The 'print' command tells us which fields
> are available, and the fuzz command can write zeroes or ones to the
> field; set the high, middle, or low bit; add or subtract numbers; or
> randomize the field. We loop through all fields and all fuzz verbs to
> see if we can trip up the kernel.
>
> Signed-off-by: Darrick J. Wong <darrick.wong@xxxxxxxxxx>
The first test gave me a kernel crash :) xfs/1300 crashed your kernel
djwong-devel branch. I appended the console log at the end of this mail
if you have interest to see it.
And another xfs/1300 run gave me this failure message:
+/mnt/testarea/scratch: Kernel lacks GETFSMAP; scrub will be less efficient. (xfs.c line 661)
+/mnt/testarea/scratch: Kernel cannot help scrub metadata; scrub will be incomplete. (xfs.c line 661)
+/mnt/testarea/scratch: Kernel cannot help scrub inodes; scrub will be incomplete. (xfs.c line 661)
+/mnt/testarea/scratch: Kernel cannot help scrub extent map; scrub will be less efficient. (xfs.c line 661)
Is this known issue or something should be filtered out in the test?
And ext4/1300 generated large .out.bad file (51M), containing something
like:
+/mnt/testarea/scratch/test/68/S_IFREG.FMT_ETREE: extent (1101381632/2469888/4096) ends past end of filesystem at 31457280. (generic.c line 272)
+/mnt/testarea/scratch/test/68/S_IFREG.FMT_ETREE: extent (1101389824/2478080/4096) starts past end of filesystem at 31457280. (generic.c line 264)
+/mnt/testarea/scratch/test/68/S_IFREG.FMT_ETREE: extent (1101389824/2478080/4096) ends past end of filesystem at 31457280. (generic.c line 272)
+/mnt/testarea/scratch/test/68/S_IFREG.FMT_ETREE: extent (1101398016/2486272/4096) starts past end of filesystem at 31457280. (generic.c line 264)
+/mnt/testarea/scratch/test/68/S_IFREG.FMT_ETREE: extent (1101398016/2486272/4096) ends past end of filesystem at 31457280. (generic.c line 272)
+/mnt/testarea/scratch/test/68/S_IFREG.FMT_ETREE: extent (1101406208/2494464/4096) starts past end of filesystem at 31457280. (generic.c line 264)
+/mnt/testarea/scratch/test/68/S_IFREG.FMT_ETREE: extent (1101406208/2494464/4096) ends past end of filesystem at 31457280. (generic.c line 272)
+/mnt/testarea/scratch/test/68/S_IFREG.FMT_ETREE: extent (1101414400/2502656/4096) starts past end of filesystem at 31457280. (generic.c line 264)
+/mnt/testarea/scratch/test/68/S_IFREG.FMT_ETREE: extent (1101414400/2502656/4096) ends past end of filesystem at 31457280. (generic.c line 272)
Seems like scrub found something wrong (real problems) and became very
noisy?
More comments inline below.
> ---
> common/fuzzy | 49 ++++++++++++++++++++++++++++++------
> common/populate | 10 ++++---
> common/rc | 15 +++++++++++
> tests/ext4/1300 | 60 ++++++++++++++++++++++++++++++++++++++++++++
[snip]
>
> diff --git a/common/fuzzy b/common/fuzzy
> index 6af47f1..dbff744 100644
> --- a/common/fuzzy
> +++ b/common/fuzzy
> @@ -85,32 +85,47 @@ _scratch_scrub() {
> # Filter the xfs_db print command's field debug information
> # into field name and type.
> __filter_xfs_db_print_fields() {
> + filter="$1"
> + if [ -z "${filter}" ] || [ "${filter}" = "nofilter" ]; then
> + filter='^'
> + fi
> grep ' = ' | while read key equals value; do
> - fuzzkey="$(echo "${key}" | sed -e 's/\([a-zA-Z0-9_]*\)\[\([0-9]*\)-[0-9]*\]/\1[\2]/g')"
> - if [[ "${value}" == "["* ]]; then
> + # Filter out any keys with an array index >= 10, and
> + # collapse any array range ("[1-195]") to the first item.
> + fuzzkey="$(echo "${key}" | sed -e '/\([a-z]*\)\[\([0-9][0-9]\+\)\].*/d' -e 's/\([a-zA-Z0-9_]*\)\[\([0-9]*\)-[0-9]*\]/\1[\2]/g')"
> + if [ -z "${fuzzkey}" ]; then
> + continue
> + elif [[ "${value}" == "["* ]]; then
> echo "${value}" | sed -e 's/^.//g' -e 's/.$//g' -e 's/,/\n/g' | while read subfield; do
> echo "${fuzzkey}.${subfield}"
> done
> else
> echo "${fuzzkey}"
> fi
> - done
> + done | egrep "${filter}"
> }
>
> # Navigate to some part of the filesystem and print the field info.
> +# The first argument is an egrep filter for the fields
> +# The rest of the arguments are xfs_db commands to locate the metadata.
> _scratch_xfs_list_metadata_fields() {
> + filter="$1"
> + shift
> if [ -n "${SCRATCH_XFS_LIST_METADATA_FIELDS}" ]; then
> - echo "${SCRATCH_XFS_LIST_METADATA_FIELDS}" | sed -e 's/ /\n/g'
> + echo "${SCRATCH_XFS_LIST_METADATA_FIELDS}" | \
> + sed -e 's/ /\n/g' | __filter_xfs_db_print_fields "${filter}"
> return;
> fi
>
> (for arg in "$@"; do
> echo "${arg}"
> done
> - echo "print") | _scratch_xfs_db | __filter_xfs_db_print_fields
> + echo "print") | _scratch_xfs_db | __filter_xfs_db_print_fields "${filter}"
> }
>
> # Get a metadata field
> +# The first arg is the field name
> +# The rest of the arguments are xfs_db commands to find the metadata.
> _scratch_xfs_get_metadata_field() {
> key="$1"
> shift
> @@ -124,6 +139,9 @@ _scratch_xfs_get_metadata_field() {
> }
>
> # Set a metadata field
> +# The first arg is the field name
> +# The second arg is the new value
> +# The rest of the arguments are xfs_db commands to find the metadata.
> _scratch_xfs_set_metadata_field() {
> key="$1"
> value="$2"
> @@ -136,6 +154,9 @@ _scratch_xfs_set_metadata_field() {
> }
>
> # Fuzz a metadata field
> +# The first arg is the field name
> +# The second arg is the xfs_db fuzz verb
> +# The rest of the arguments are xfs_db commands to find the metadata.
> _scratch_xfs_fuzz_metadata_field() {
> key="$1"
> value="$2"
> @@ -263,12 +284,24 @@ _scratch_xfs_list_fuzz_verbs() {
> sed -e 's/[,.]//g' -e 's/Verbs: //g' -e 's/ /\n/g'
> }
>
> -# Fuzz the fields of some piece of metadata
> -_scratch_xfs_fuzz_fields() {
> - _scratch_xfs_list_metadata_fields "$@" | while read field; do
> +# Fuzz some of the fields of some piece of metadata
> +# The first argument is an egrep filter
> +# The rest of the arguments are xfs_db commands to locate the metadata.
> +_scratch_xfs_fuzz_some_fields() {
> + filter="$1"
> + shift
> + echo "Fields we propose to fuzz: $@"
> + _scratch_xfs_list_metadata_fields "${filter}" "$@"
> + _scratch_xfs_list_metadata_fields "${filter}" "$@" | while read field; do
> _scratch_xfs_list_fuzz_verbs | while read fuzzverb; do
> __scratch_xfs_fuzz_mdrestore
> __scratch_xfs_fuzz_field_test "${field}" "${fuzzverb}" "$@"
> done
> done
> }
> +
> +# Fuzz all of the fields of some piece of metadata
> +# All arguments are xfs_db commands to locate the metadata.
> +_scratch_xfs_fuzz_fields() {
> + _scratch_xfs_fuzz_some_fields '' "$@"
> +}
I think all the fuzz update here should be folded to patch 7/9.
> diff --git a/common/populate b/common/populate
> index 15d68fc..7d103f0 100644
> --- a/common/populate
> +++ b/common/populate
> @@ -180,13 +180,13 @@ _scratch_xfs_populate() {
> # FMT_EXTENTS with a remote less-than-a-block value
> echo "+ attr extents with a remote less-than-a-block value"
> touch "${SCRATCH_MNT}/ATTR.FMT_EXTENTS_REMOTE3K"
> - $XFS_IO_PROG -f -c "pwrite -S 0x43 0 3k" "${SCRATCH_MNT}/attrvalfile" > /dev/null
> + $XFS_IO_PROG -f -c "pwrite -S 0x43 0 $((blksz - 300))" "${SCRATCH_MNT}/attrvalfile" > /dev/null
> attr -q -s user.remotebtreeattrname "${SCRATCH_MNT}/ATTR.FMT_EXTENTS_REMOTE3K" < "${SCRATCH_MNT}/attrvalfile"
>
> # FMT_EXTENTS with a remote block-size value
> echo "+ attr extents with a remote one-block value"
> touch "${SCRATCH_MNT}/ATTR.FMT_EXTENTS_REMOTE4K"
> - $XFS_IO_PROG -f -c "pwrite -S 0x44 0 4k" "${SCRATCH_MNT}/attrvalfile" > /dev/null
> + $XFS_IO_PROG -f -c "pwrite -S 0x44 0 ${blksz}" "${SCRATCH_MNT}/attrvalfile" > /dev/null
> attr -q -s user.remotebtreeattrname "${SCRATCH_MNT}/ATTR.FMT_EXTENTS_REMOTE4K" < "${SCRATCH_MNT}/attrvalfile"
> rm -rf "${SCRATCH_MNT}/attrvalfile"
>
> @@ -482,8 +482,8 @@ _scratch_xfs_populate_check() {
> __populate_check_xfs_aformat "${btree_attr}" "btree"
> __populate_check_xfs_agbtree_height "bno"
> __populate_check_xfs_agbtree_height "cnt"
> - test -n $is_rmapbt && __populate_check_xfs_agbtree_height "rmap"
> - test -n $is_reflink && __populate_check_xfs_agbtree_height "refcnt"
> + test $is_rmapbt -ne 0 && __populate_check_xfs_agbtree_height "rmap"
> + test $is_reflink -ne 0 && __populate_check_xfs_agbtree_height "refcnt"
> }
And these folded to patch 1/9?
>
> # Check data fork format of ext4 file
> @@ -609,7 +609,7 @@ _scratch_populate_cached() {
> rm -rf "$(find "${POPULATE_METADUMP}" -mtime +2 2>/dev/null)"
>
> # Throw away cached image if it doesn't match our spec.
> - meta_descr="FSTYP ${FSTYP} MKFS_OPTIONS ${MKFS_OPTIONS} ARGS $@"
> + meta_descr="FSTYP ${FSTYP} MKFS_OPTIONS $(_scratch_mkfs_options) ARGS $@"
> cmp -s "${POPULATE_METADUMP_DESCR}" <(echo "${meta_descr}") || rm -rf "${POPULATE_METADUMP}"
>
> # Do we have a cached image?
This to patch 6/9?
Because we usually don't introduce something in patch 1 and fix them in
patch 2, I think :)
> diff --git a/common/rc b/common/rc
> index d904582..ec1f5de 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -1870,6 +1870,21 @@ _require_xfs_finobt()
> _scratch_unmount
> }
>
> +# Do we have a fre
> +_require_scratch_finobt()
> +{
> + _require_scratch
> +
> + if [ $FSTYP != "xfs" ]; then
> + _notrun "finobt not supported by scratch filesystem type: $FSTYP"
> + return
> + fi
> + _scratch_mkfs > /dev/null
> + _scratch_mount
> + xfs_info $SCRATCH_MNT | grep -q 'finobt=1' || _notrun "finobt not supported by scratch filesystem type: $FSTYP"
> + _scratch_unmount
> +}
> +
> # this test requires xfs sysfs attribute support
> #
> _require_xfs_sysfs()
> diff --git a/tests/ext4/1300 b/tests/ext4/1300
> new file mode 100755
> index 0000000..3f8135e
> --- /dev/null
> +++ b/tests/ext4/1300
[all the tests look fine to me, snip]
> --- a/tests/xfs/group
> +++ b/tests/xfs/group
> @@ -333,3 +333,34 @@
> 345 auto quick clone
> 346 auto quick clone
> 347 auto quick clone
> +1300 dangerous_fuzzers scrub
ext4/1300 is "auto quick scrub", I think xfs/1300 should be in auto
group too?
Thanks,
Eryu
P.S. console log of xfs/1300 crash
[165877.766244] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038
[165877.774197] IP: [<ffffffffa0680c13>] xfs_scrub_get_inode+0xc3/0x1c0 [xfs]
[165877.781162] PGD 179c1b067 [165877.783784] PUD 14d994067
PMD 0 [165877.787130]
[165877.788722] Oops: 0000 [#1] SMP
[165877.791951] Modules linked in: dm_delay dm_zero btrfs xor raid6_pq dm_thin_pool dm_persistent_data dm_bio_prison dm_snapshot dm_bufio loop dm_flakey xfs libcrc32c binfmt_misc ip6t_rpfilter ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw iTCO_wdt gf128mul glue_helper ipmi_devintf cdc_ether iTCO_vendor_support ablk_helper cryptd usbnet i2c_i801 lpc_ich mii pcspkr i2c_smbus sg i7core_edac mfd_core ipmi_si edac_core ipmi_msghandler shpchp ioatdma dca acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 jbd2 mbcache sr_mod sd_mod cdrom mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_generic pata_acpi drm ata_piix libata crc32c_intel megaraid_sas serio_raw i2c_core bnx2 dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_debug]
[165877.898708] CPU: 5 PID: 26242 Comm: xfs_scrub Tainted: G W 4.9.0-rc3.djwong+ #16
[165877.907308] Hardware name: IBM System x3550 M3 -[7944OEJ]-/90Y4784 , BIOS -[D6E150CUS-1.11]- 02/08/2011
[165877.917124] task: ffff88017a286a40 task.stack: ffffc9000cca0000
[165877.923126] RIP: 0010:[<ffffffffa0680c13>] [<ffffffffa0680c13>] xfs_scrub_get_inode+0xc3/0x1c0 [xfs]
[165877.932503] RSP: 0018:ffffc9000cca3ad0 EFLAGS: 00010246
[165877.937898] RAX: 0000000000000000 RBX: fffffffffffffffe RCX: 0000000000000017
[165877.945111] RDX: 0000000000000000 RSI: ffffc9000cca3ce8 RDI: 0000000000001123
[165877.952328] RBP: ffffc9000cca3b20 R08: 0000000000000003 R09: 0000000000000014
[165877.959544] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880278f59680
[165877.966759] R13: ffffc9000cca3ba8 R14: ffffc9000cca3ce8 R15: 0000000000000000
[165877.973976] FS: 00007f3d099f9700(0000) GS:ffff88017bb00000(0000) knlGS:0000000000000000
[165877.982143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[165877.987969] CR2: 0000000000000038 CR3: 000000016ad2d000 CR4: 00000000000006e0
[165877.995185] Stack:
[165877.997286] ffff880278f59680 0000000000000000 ffff880220cc5c00 ffff880265190780
[165878.004840] ffff880101a67800 ffffc9000cca3ba8 ffff880278f59680 ffff88025a9e6000
[165878.012396] ffffc9000cca3ce8 0000000000000000 ffffc9000cca3b58 ffffffffa0681a61
[165878.019950] Call Trace:
[165878.022551] [<ffffffffa0681a61>] __xfs_scrub_setup_inode.isra.63+0x81/0x280 [xfs]
[165878.030236] [<ffffffffa0681c70>] xfs_scrub_setup_inode+0x10/0x20 [xfs]
[165878.036963] [<ffffffffa068f57f>] xfs_scrub_metadata+0x2ff/0x450 [xfs]
[165878.043607] [<ffffffffa066aa1d>] xfs_ioc_scrub_metadata+0x4d/0x80 [xfs]
[165878.050424] [<ffffffffa066d029>] xfs_file_ioctl+0x9c9/0xb10 [xfs]
[165878.056689] [<ffffffff8110692f>] ? get_futex_key+0x1df/0x360
[165878.062516] [<ffffffff81106b31>] ? futex_wake+0x81/0x150
[165878.068003] [<ffffffff812189c6>] do_vfs_ioctl+0x96/0x5b0
[165878.073482] [<ffffffff81218f59>] SyS_ioctl+0x79/0x90
[165878.078621] [<ffffffff81003997>] do_syscall_64+0x67/0x180
[165878.084191] [<ffffffff816a6c2b>] entry_SYSCALL64_slow_path+0x25/0x25
[165878.090714] Code: 8b 14 24 49 8b 75 00 85 c0 41 89 c7 44 0f b6 82 93 00 00 00 44 0f b6 8a 94 00 00 00 0f b6 8a 53 02 00 00 49 8b 55 08 48 8b 7e 08 <4c> 8b 62 38 75 38 48 8b 7d d0 8b 46 10 39 87 98 03 00 00 74 21
[165878.110930] RIP [<ffffffffa0680c13>] xfs_scrub_get_inode+0xc3/0x1c0 [xfs]
[165878.117938] RSP <ffffc9000cca3ad0>
[165878.121512] CR2: 0000000000000038
[165878.129219] ---[ end trace d23e56c58f53ccb9 ]---
--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
