Hello list, I have the following cluster: ceph status cluster a2fba9c1-4ca2-46d8-8717-a8e42db14bb0 health HEALTH_OK monmap e2: 5 mons at {alxc10=xxxxx:6789/0,alxc11=xxxxx:6789/0,alxc5=xxxxx:6789/0,alxc6=xxxx:6789/0,alxc7=xxxxx:6789/0} election epoch 196, quorum 0,1,2,3,4 alxc10,alxc5,alxc6,alxc7,alxc11 mdsmap e797: 1/1/1 up {0=alxc11.xxxx=up:active}, 2 up:standby osdmap e11243: 50 osds: 50 up, 50 in pgmap v3563774: 8192 pgs, 3 pools, 1954 GB data, 972 kobjects 4323 GB used, 85071 GB / 89424 GB avail 8192 active+clean client io 168 MB/s rd, 11629 kB/s wr, 3447 op/s It's running ceph version 0.94.7 (d56bdf93ced6b80b07397d57e3fa68fe68304432) and kernel 4.4.14 I have multiple rbd devices which are used as the root for lxc-based containers and have ext4. At some point I want to create a an rbd snapshot, for this the sequence of operations I do is thus: 1. freezefs -f /path/to/where/ext4-ontop-of-rbd-is-mounted 2. rbd snap create "${CEPH_POOL_NAME}/${name-of-blockdev}@${name-of-snapshot} 3. freezefs -u /path/to/where/ext4-ontop-of-rbd-is-mounted <= At this point normal container operation continues => 4. Mount the newly created snapshot to a 2nd location as read-only and rsync the files from it to a remote server. However as I start rsyncing stuff to the remote server then certain files in the snapshot are reported as corrupted. freezefs implies filesystem syncing I also tested with manually doing sync/syncfs on the fs which is being snapshot. Before and after the freezefs and the corruption is still present. So it's unlikely there are dirty buffers in the page cache. I'm using the kernel rbd driver for the clients. The theory currently is there are some caches which are not being flushed, other than the linux page cache. Reading the doc implies that only librbd is using separate caching but I'm not using librbd. Any ideas would be much appreciated. Regards, Nikolay _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com