On Mon, Sep 26, 2016 at 8:39 AM, Nikolay Borisov <kernel@xxxxxxxx> wrote: > > > On 09/22/2016 06:36 PM, Ilya Dryomov wrote: >> On Thu, Sep 15, 2016 at 3:18 PM, Ilya Dryomov <idryomov@xxxxxxxxx> wrote: >>> On Thu, Sep 15, 2016 at 2:43 PM, Nikolay Borisov <kernel@xxxxxxxx> wrote: >>>> >>>> [snipped] >>>> >>>> cat /sys/bus/rbd/devices/47/client_id >>>> client157729 >>>> cat /sys/bus/rbd/devices/1/client_id >>>> client157729 >>>> >>>> Client client157729 is alxc13, based on correlation by the ip address >>>> shown by the rados -p ... command. So it's the only client where the rbd >>>> images are mapped. >>> >>> Well, the watches are there, but cookie numbers indicate that they may >>> have been re-established, so that's inconclusive. >>> >>> My suggestion would be to repeat the test and do repeated freezes to >>> see if snapshot continues to follow HEAD. >>> >>> Further, to rule out a missed snap context update, repeat the test, but >>> stick >>> >>> # echo 1 >/sys/bus/rbd/devices/<ID_OF_THE_ORIG_DEVICE>/refresh >>> >>> after "rbd snap create" (for the today's test, ID_OF_THE_ORIG_DEVICE >>> would be 47). >> >> Hi Nikolay, >> >> Any news on this? > > Hello, > > I was on holiday hence the radio silence. Here is the latest set of > tests that were run: > > Results: > > c11579 (100GB - used: 83GB): > root@alxc13:~# rbd showmapped |grep c11579 > 47 rbd c11579 - /dev/rbd47 > root@alxc13:~# fsfreeze -f /var/lxc/c11579 > root@alxc13:~# md5sum <(dd if=/dev/rbd47 iflag=direct bs=8M) > 12800+0 records in > 12800+0 records out > 107374182400 bytes (107 GB) copied, 686.382 s, 156 MB/s > f2edb5abb100de30c1301b0856e595aa /dev/fd/63 > root@alxc13:~# rbd snap create rbd/c11579@snap_test > root@alxc13:~# rbd map c11579@snap_test > /dev/rbd1 > root@alxc13:~# echo 1 >/sys/bus/rbd/devices/47/refresh > root@alxc13:~# md5sum <(dd if=/dev/rbd47 iflag=direct bs=8M) > 12800+0 records in > 12800+0 records out > 107374182400 bytes (107 GB) copied, 915.225 s, 117 MB/s > f2edb5abb100de30c1301b0856e595aa /dev/fd/63 > root@alxc13:~# md5sum <(dd if=/dev/rbd1 iflag=direct bs=8M) > 12800+0 records in > 12800+0 records out > 107374182400 bytes (107 GB) copied, 863.464 s, 143 MB/s > f2edb5abb100de30c1301b0856e595aa /dev/fd/63 > root@alxc13:~# file -s /dev/rbd1 > /dev/rbd1: Linux rev 1.0 ext4 filesystem data (extents) (large files) > (huge files) > root@alxc13:~# fsfreeze -u /var/lxc/c11579 > root@alxc13:~# md5sum <(dd if=/dev/rbd47 iflag=direct bs=8M) > 12800+0 records in > 12800+0 records out > 107374182400 bytes (107 GB) copied, 730.243 s, 147 MB/s > 65294ce9eae5694a56054ec4af011264 /dev/fd/63 > root@alxc13:~# md5sum <(dd if=/dev/rbd1 iflag=direct bs=8M) > 12800+0 records in > 12800+0 records out > 107374182400 bytes (107 GB) copied, 649.373 s, 165 MB/s > f2edb5abb100de30c1301b0856e595aa /dev/fd/63 > > 30min later: > root@alxc13:~# md5sum <(dd if=/dev/rbd1 iflag=direct bs=8M) > 12800+0 records in > 12800+0 records out > 107374182400 bytes (107 GB) copied, 648.328 s, 166 MB/s > f2edb5abb100de30c1301b0856e595aa /dev/fd/63 > > > > c12607 (30GB - used: 4GB): > root@alxc13:~# rbd showmapped |grep c12607 > 39 rbd c12607 - /dev/rbd39 > root@alxc13:~# fsfreeze -f /var/lxc/c12607 > root@alxc13:~# md5sum <(dd if=/dev/rbd39 iflag=direct bs=8M) > 3840+0 records in > 3840+0 records out > 32212254720 bytes (32 GB) copied, 228.2 s, 141 MB/s > e6ce3ea688a778b9c732041164b4638c /dev/fd/63 > root@alxc13:~# rbd snap create rbd/c12607@snap_test > root@alxc13:~# rbd map c12607@snap_test > /dev/rbd21 > root@alxc13:~# rbd snap protect rbd/c12607@snap_test > root@alxc13:~# echo 1 >/sys/bus/rbd/devices/39/refresh > root@alxc13:~# md5sum <(dd if=/dev/rbd39 iflag=direct bs=8M) > 3840+0 records in > 3840+0 records out > 32212254720 bytes (32 GB) copied, 217.138 s, 148 MB/s > e6ce3ea688a778b9c732041164b4638c /dev/fd/63 > root@alxc13:~# md5sum <(dd if=/dev/rbd21 iflag=direct bs=8M) > 3840+0 records in > 3840+0 records out > 32212254720 bytes (32 GB) copied, 212.254 s, 152 MB/s > e6ce3ea688a778b9c732041164b4638c /dev/fd/63 > root@alxc13:~# file -s /dev/rbd21 > /dev/rbd21: Linux rev 1.0 ext4 filesystem data (extents) (large files) > (huge files) > root@alxc13:~# fsfreeze -u /var/lxc/c12607 > root@alxc13:~# md5sum <(dd if=/dev/rbd39 iflag=direct bs=8M) > 3840+0 records in > 3840+0 records out > 32212254720 bytes (32 GB) copied, 322.964 s, 99.7 MB/s > 71c5efc24162452473cda50155cd4399 /dev/fd/63 > root@alxc13:~# md5sum <(dd if=/dev/rbd21 iflag=direct bs=8M) > 3840+0 records in > 3840+0 records out > 32212254720 bytes (32 GB) copied, 326.273 s, 98.7 MB/s > e6ce3ea688a778b9c732041164b4638c /dev/fd/63 > root@alxc13:~# file -s /dev/rbd21 > /dev/rbd21: Linux rev 1.0 ext4 filesystem data (extents) (large files) > (huge files) > root@alxc13:~# > > 30min later: > root@alxc13:~# md5sum <(dd if=/dev/rbd21 iflag=direct bs=8M) > 3840+0 records in > 3840+0 records out > 32212254720 bytes (32 GB) copied, 359.917 s, 89.5 MB/s > e6ce3ea688a778b9c732041164b4638c /dev/fd/63 > > Everything seems consistent, but when an rsync was initiated from the > snapshot it again failed. Unfortunately I deem those results rather > unstable because they now contradict the ones which I showed you earlier > with the differing checksums. Could you try running a few of these instead of just one? I'm interested in the following data points: - whether echo 1 >refresh consistently makes a difference (even if it's just md5sum and not a successful rsync) - if it does, that's a krbd issue which will need to be looked at regardless, so try with and w/o manual refresh - how does dd iflag=direct a freezed HEAD (/dev/rbd47) into a file on another FS + losetup -r + mount + rsync behave - add a cp before "rbd snap create" and do the same with the loop image (md5sum, file -s, rsync, etc) Thanks, Ilya _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com