I decided to deploy dm-cache with LVM on my workstation, but soon after
found myself nursing data corruption.
Once I narrowed it down I found it is easy to re-create: detach a
writeback cache during file copying, and re-attach it.
The commands I used are below. The summary is:
1) setup logical volume of writeback cachepool (SSD) plus spinning disk
2) rsync data to the volume
3) during rsync, detach the cache
4) stop rsync and fsck -- volume is OK at this point
5) re-attach the cache
6) run fsck -- volume is now bad
I think I hit this when, during deployment, I detached the cache to
enlarge volume.
According to cache_check the coherency of the cache is valid.
Is it just there is no protection at all to prevent attaching an
out-of-sync cache to a backing device?
I'd assume the user tools to bring the cache in sync on "lvconvert --type
cache" (probably by emptying it)
If the observed behaviour is intended, it should probably be warned of
loudly in the lvmcache(7) man page.
Or is this a sign of actual data corruption by dm-cache?
Many thanks
--
Mark
System info:
$ uname -a
Linux stax 3.19.5-mh #79 SMP PREEMPT Tue Apr 21 23:27:45 BST 2015 i686 Intel(R) Xeon(R) CPU E5410 @ 2.33GHz GenuineIntel GNU/Linux
$ lvm version
LVM version: 2.02.118(2) (2015-03-24)
Library version: 1.02.95 (2015-03-24)
Driver version: 4.29.0
$ cache_check --version
0.4.1
# Summary of steps to setup the cache device
$ lvcreate -n cache-meta -L 512M vg1 /dev/sda10
$ lvcreate -n cache -L 64G vg1 /dev/sda10
$ lvconvert --type cache-pool --poolmetadata vg1/cache-meta vg1/cache
$ lvconvert --type cache --cachepool vg1/cache vg1/origin
# mkfs.ext4 /dev/vg1/origin
$ mount /dev/vg1/origin /mnt/test
# Populate with data and check device
$ rsync /path/to/sources /mnt/test/test-data
$ umount [...]
$ fsck /dev/vg1/origin
<pass>
# Start an rsync
$ mount [...]
$ rsync /path/to/sources /mnt/test/test-data
# With rsync RUNNING, detach the cache
$ lvconvert --splitcache vg1/cache
Flushing cache for origin.
Logical volume vg1/origin is not cached and cache pool vg1/cache is unused.
# Shortly after, stop the rsync and fsck
$ fsck -f /dev/vg1/origin
<pass>
# Attach the cache and fsck again, it is now corrupt
$ lvconvert --type cache --cachepool vg1/cache vg1/origin
$ fsck -f /dev/vg1/origin
fsck from util-linux 2.21.2
e2fsck 1.42.8 (20-Jun-2013)
/dev/mapper/vg1-origin: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (455389520, counted=448177147).
Fix<y>? no
Free inodes count wrong (120759015, counted=120758982).
Fix<y>? no
/dev/mapper/vg1-origin: 40217/120799232 files (30.4% non-contiguous), 27795120/483184640 blocks
$ cache_check /dev/mapper/vg1-cache_cmeta
examining superblock
examining mapping array
examining hint array
examining discard bitset
$ cache_dump /dev/mapper/vg1-cache_cmeta | head
<superblock uuid="" block_size="128" nr_cache_blocks="524288" policy="cleaner" hint_width="4">
<mappings>
<mapping cache_block="484520" origin_block="1802240" dirty="false"/>
<mapping cache_block="484521" origin_block="1769472" dirty="false"/>
<mapping cache_block="484522" origin_block="1736704" dirty="false"/>
<mapping cache_block="484523" origin_block="1703936" dirty="false"/>
<mapping cache_block="484524" origin_block="1671168" dirty="false"/>
<mapping cache_block="484525" origin_block="1638400" dirty="false"/>
<mapping cache_block="484526" origin_block="1605632" dirty="false"/>
<mapping cache_block="484527" origin_block="1540096" dirty="false"/>
# Detach the cache and the backing volume on spinning disk is now corrupt
$ lvconvert --splitcache vg1/cache
Flushing cache for origin.
Logical volume vg1/origin is not cached and cache pool vg1/cache is unused.
$ fsck -f /dev/vg1/origin
fsck from util-linux 2.21.2
e2fsck 1.42.8 (20-Jun-2013)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (455389520, counted=448177147).
Fix<y>? no
Free inodes count wrong (120759015, counted=120758982).
Fix<y>? no
/dev/mapper/vg1-origin: 40250/120799232 files (30.4% non-contiguous), 27795120/483184640 blocks
# END
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
