On Tue, Mar 21, 2017 at 6:50 AM, Qu Wenruo <quwenruo@xxxxxxxxxxxxxx> wrote:
> Hi Filipe,
>
> At 02/15/2017 04:35 AM, fdmanana@xxxxxxxxxx wrote:
>>
>> From: Filipe Manana <fdmanana@xxxxxxxx>
>>
>> Test that both a full and incremental btrfs send operation preserves file
>> holes.
>
>
> I found the test case fails with compress=lzo mount option.
I had seen it too but had no time so far to investigate it.
>
> In fact, it turns out to be 2 bugs:
>
> [Inline-then-regular layout]
> And further more, it seems that it can create inline-then-regular file
> extents layout with compress=lzo.
That's not good.
I'll take a look at it.
Thanks.
>
> It does not always create such inline-then-regular, but the possibility
> seems quite high in my test box, near 80%.
>
> Btrfs dump tree for snap2 (258):
> item 9 key (257 INODE_REF 256) itemoff 15736 itemsize 13
> inode ref index 2 namelen 3 name: foo
> item 10 key (257 EXTENT_DATA 0) itemoff 15663 itemsize 73
> generation 7 type 0 (inline)
> inline extent data size 52 ram_bytes 4096 compression 2
> (lzo)
> item 11 key (257 EXTENT_DATA 4096) itemoff 15610 itemsize 53
> generation 8 type 1 (regular)
> extent data disk byte 0 nr 0
> extent data offset 0 nr 1048576 ram 1052672
> extent compression 0 (none)
> item 12 key (257 EXTENT_DATA 1052672) itemoff 15557 itemsize 53
> generation 8 type 1 (regular)
> extent data disk byte 12582912 nr 4096
> extent data offset 0 nr 4096 ram 4096
> extent compression 0 (none)
>
> [Send-stream without hole]
> This bug is 100% reproducible, the send stream of snap2 contains no hole,
> just the full file contents.
>
> Thanks,
> Qu
>
>
>>
>> This used to fail when the filesystem had the NO_HOLES feature enabled,
>> that is, when the test is run with MKFS_OPTIONS="-O no-holes".
>>
>> This is fixed by the following patch for the linux kernel:
>>
>> "Btrfs: incremental send, fix unnecessary hole writes for sparse files"
>>
>> Signed-off-by: Filipe Manana <fdmanana@xxxxxxxx>
>> ---
>> tests/btrfs/137 | 141
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++
>> tests/btrfs/137.out | 63 +++++++++++++++++++++++
>> tests/btrfs/group | 1 +
>> 3 files changed, 205 insertions(+)
>> create mode 100755 tests/btrfs/137
>> create mode 100644 tests/btrfs/137.out
>>
>> diff --git a/tests/btrfs/137 b/tests/btrfs/137
>> new file mode 100755
>> index 0000000..3ff2c6b
>> --- /dev/null
>> +++ b/tests/btrfs/137
>> @@ -0,0 +1,141 @@
>> +#! /bin/bash
>> +# FS QA Test No. btrfs/137
>> +#
>> +# Test that both incremental and full send operations preserve file
>> holes.
>> +#
>> +#-----------------------------------------------------------------------
>> +#
>> +# Copyright (C) 2017 SUSE Linux Products GmbH. All Rights Reserved.
>> +# Author: Filipe Manana <fdmanana@xxxxxxxx>
>> +#
>> +# This program is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU General Public License as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it would be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write the Free Software Foundation,
>> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
>> +#-----------------------------------------------------------------------
>> +#
>> +
>> +seq=`basename $0`
>> +seqres=$RESULT_DIR/$seq
>> +echo "QA output created by $seq"
>> +
>> +tmp=/tmp/$$
>> +status=1 # failure is the default!
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +_cleanup()
>> +{
>> + cd /
>> + rm -fr $send_files_dir
>> + rm -f $tmp.*
>> +}
>> +
>> +# get standard environment, filters and checks
>> +. ./common/rc
>> +. ./common/filter
>> +. ./common/punch
>> +
>> +# real QA test starts here
>> +_supported_fs btrfs
>> +_supported_os Linux
>> +_require_test
>> +_require_scratch
>> +_require_xfs_io_command "fiemap"
>> +
>> +send_files_dir=$TEST_DIR/btrfs-test-$seq
>> +
>> +rm -f $seqres.full
>> +rm -fr $send_files_dir
>> +mkdir $send_files_dir
>> +
>> +_scratch_mkfs >>$seqres.full 2>&1
>> +_scratch_mount
>> +
>> +# Create the first test file.
>> +$XFS_IO_PROG -f -c "pwrite -S 0xaa 0 4K" $SCRATCH_MNT/foo |
>> _filter_xfs_io
>> +
>> +# Create a second test file with a 1Mb hole.
>> +$XFS_IO_PROG -f \
>> + -c "pwrite -S 0xaa 0 4K" \
>> + -c "pwrite -S 0xbb 1028K 4K" \
>> + $SCRATCH_MNT/bar | _filter_xfs_io
>> +
>> +$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \
>> + $SCRATCH_MNT/snap1 >/dev/null
>> +
>> +# Now add one new extent to our first test file, increasing its size and
>> leaving
>> +# a 1Mb hole between the first extent and this new extent.
>> +$XFS_IO_PROG -c "pwrite -S 0xbb 1028K 4K" $SCRATCH_MNT/foo |
>> _filter_xfs_io
>> +
>> +# Now overwrite the last extent of our second test file.
>> +$XFS_IO_PROG -c "pwrite -S 0xcc 1028K 4K" $SCRATCH_MNT/bar |
>> _filter_xfs_io
>> +
>> +$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \
>> + $SCRATCH_MNT/snap2 >/dev/null
>> +
>> +echo
>> +echo "File digests in the original filesystem:"
>> +md5sum $SCRATCH_MNT/snap1/foo | _filter_scratch
>> +md5sum $SCRATCH_MNT/snap1/bar | _filter_scratch
>> +md5sum $SCRATCH_MNT/snap2/foo | _filter_scratch
>> +md5sum $SCRATCH_MNT/snap2/bar | _filter_scratch
>> +
>> +echo
>> +echo "File snap1/foo fiemap results in the original filesystem:"
>> +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/foo | _filter_fiemap
>> +echo
>> +echo "File snap1/bar fiemap results in the original filesystem:"
>> +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/bar | _filter_fiemap
>> +echo
>> +echo "File snap2/foo fiemap results in the original filesystem:"
>> +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/foo | _filter_fiemap
>> +echo
>> +echo "File snap2/bar fiemap results in the original filesystem:"
>> +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/bar | _filter_fiemap
>> +echo
>> +
>> +# Create the send streams to apply later on a new filesystem.
>> +$BTRFS_UTIL_PROG send $SCRATCH_MNT/snap1 -f $send_files_dir/1.snap 2>&1 \
>> + | _filter_scratch
>> +$BTRFS_UTIL_PROG send -p $SCRATCH_MNT/snap1 $SCRATCH_MNT/snap2 \
>> + -f $send_files_dir/2.snap 2>&1 | _filter_scratch
>> +
>> +# Create a new filesystem, receive the send streams and verify that the
>> file
>> +# contents are the same as in the original filesystem and that the file
>> holes
>> +# exists in both snapshots.
>> +_scratch_unmount
>> +_scratch_mkfs >>$seqres.full 2>&1
>> +_scratch_mount
>> +$BTRFS_UTIL_PROG receive $SCRATCH_MNT -f $send_files_dir/1.snap
>> >/dev/null
>> +$BTRFS_UTIL_PROG receive $SCRATCH_MNT -f $send_files_dir/2.snap
>> >/dev/null
>> +
>> +echo
>> +echo "File digests in the new filesystem:"
>> +md5sum $SCRATCH_MNT/snap1/foo | _filter_scratch
>> +md5sum $SCRATCH_MNT/snap1/bar | _filter_scratch
>> +md5sum $SCRATCH_MNT/snap2/foo | _filter_scratch
>> +md5sum $SCRATCH_MNT/snap2/bar | _filter_scratch
>> +
>> +echo
>> +echo "File snap1/foo fiemap results in the new filesystem:"
>> +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/foo | _filter_fiemap
>> +echo
>> +echo "File snap1/bar fiemap results in the new filesystem:"
>> +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/bar | _filter_fiemap
>> +echo
>> +echo "File snap2/foo fiemap results in the new filesystem:"
>> +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/foo | _filter_fiemap
>> +echo
>> +echo "File snap2/bar fiemap results in the new filesystem:"
>> +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/bar | _filter_fiemap
>> +
>> +status=0
>> +exit
>> diff --git a/tests/btrfs/137.out b/tests/btrfs/137.out
>> new file mode 100644
>> index 0000000..8554399
>> --- /dev/null
>> +++ b/tests/btrfs/137.out
>> @@ -0,0 +1,63 @@
>> +QA output created by 137
>> +wrote 4096/4096 bytes at offset 0
>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> +wrote 4096/4096 bytes at offset 0
>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> +wrote 4096/4096 bytes at offset 1052672
>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> +wrote 4096/4096 bytes at offset 1052672
>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> +wrote 4096/4096 bytes at offset 1052672
>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> +
>> +File digests in the original filesystem:
>> +3e4309c7cc81f23d45e260a8f13ca860 SCRATCH_MNT/snap1/foo
>> +f3934f0cf164e2efa1bab71f2f164990 SCRATCH_MNT/snap1/bar
>> +f3934f0cf164e2efa1bab71f2f164990 SCRATCH_MNT/snap2/foo
>> +d3dc847171f9081bd75d7a2d3b53d322 SCRATCH_MNT/snap2/bar
>> +
>> +File snap1/foo fiemap results in the original filesystem:
>> +0: [0..7]: data
>> +
>> +File snap1/bar fiemap results in the original filesystem:
>> +0: [0..7]: data
>> +1: [8..2055]: hole
>> +2: [2056..2063]: data
>> +
>> +File snap2/foo fiemap results in the original filesystem:
>> +0: [0..7]: data
>> +1: [8..2055]: hole
>> +2: [2056..2063]: data
>> +
>> +File snap2/bar fiemap results in the original filesystem:
>> +0: [0..7]: data
>> +1: [8..2055]: hole
>> +2: [2056..2063]: data
>> +
>> +At subvol SCRATCH_MNT/snap1
>> +At subvol SCRATCH_MNT/snap2
>> +At subvol snap1
>> +
>> +File digests in the new filesystem:
>> +3e4309c7cc81f23d45e260a8f13ca860 SCRATCH_MNT/snap1/foo
>> +f3934f0cf164e2efa1bab71f2f164990 SCRATCH_MNT/snap1/bar
>> +f3934f0cf164e2efa1bab71f2f164990 SCRATCH_MNT/snap2/foo
>> +d3dc847171f9081bd75d7a2d3b53d322 SCRATCH_MNT/snap2/bar
>> +
>> +File snap1/foo fiemap results in the new filesystem:
>> +0: [0..7]: data
>> +
>> +File snap1/bar fiemap results in the new filesystem:
>> +0: [0..7]: data
>> +1: [8..2055]: hole
>> +2: [2056..2063]: data
>> +
>> +File snap2/foo fiemap results in the new filesystem:
>> +0: [0..7]: data
>> +1: [8..2055]: hole
>> +2: [2056..2063]: data
>> +
>> +File snap2/bar fiemap results in the new filesystem:
>> +0: [0..7]: data
>> +1: [8..2055]: hole
>> +2: [2056..2063]: data
>> diff --git a/tests/btrfs/group b/tests/btrfs/group
>> index ea88ba4..009dea1 100644
>> --- a/tests/btrfs/group
>> +++ b/tests/btrfs/group
>> @@ -139,3 +139,4 @@
>> 134 auto quick send
>> 135 auto quick send
>> 136 auto convert
>> +137 auto quick send
>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
