On Wed, Jun 15, 2016 at 10:21 PM, Jason Gress <jgress@xxxxxxxxxxxxx> wrote: > While trying to use CephFS as a clustered filesystem, we stumbled upon a > reproducible bug that is unfortunately pretty serious, as it leads to data > loss. Here is the situation: > > We have two systems, named ftp01 and ftp02. They are both running CentOS > 7.2, with this kernel release and ceph packages: > > kernel-3.10.0-327.18.2.el7.x86_64 That is an old-ish kernel to be using with cephfs. It may well be the source of your issues. > [root@ftp01 cron]# rpm -qa | grep ceph > ceph-base-10.2.1-0.el7.x86_64 > ceph-deploy-1.5.33-0.noarch > ceph-mon-10.2.1-0.el7.x86_64 > libcephfs1-10.2.1-0.el7.x86_64 > ceph-selinux-10.2.1-0.el7.x86_64 > ceph-mds-10.2.1-0.el7.x86_64 > ceph-common-10.2.1-0.el7.x86_64 > ceph-10.2.1-0.el7.x86_64 > python-cephfs-10.2.1-0.el7.x86_64 > ceph-osd-10.2.1-0.el7.x86_64 > > Mounted like so: > XX.XX.XX.XX:/ftp/cron /var/spool/cron ceph > _netdev,relatime,name=ftp01,secretfile=/etc/ceph/ftp01.secret 0 0 > And: > XX.XX.XX.XX:/ftp/cron /var/spool/cron ceph > _netdev,relatime,name=ftp02,secretfile=/etc/ceph/ftp02.secret 0 0 > > This filesystem has 234GB worth of data on it, and I created another > subdirectory and mounted it, NFS style. > > Here were the steps to reproduce: > > First, I created a file (I was mounting /var/spool/cron on two systems) on > ftp01: > (crond is not running right now on either system to keep the variables down) > > [root@ftp01 cron]# cp /tmp/root . > > Shows up on both fine: > [root@ftp01 cron]# ls -la > total 2 > drwx------ 1 root root 0 Jun 15 15:50 . > drwxr-xr-x. 10 root root 104 May 19 09:34 .. > -rw------- 1 root root 2043 Jun 15 15:50 root > [root@ftp01 cron]# md5sum root > 0636c8deaeadfea7b9ddaa29652b43ae root > > [root@ftp02 cron]# ls -la > total 2 > drwx------ 1 root root 2043 Jun 15 15:50 . > drwxr-xr-x. 10 root root 104 May 19 09:34 .. > -rw------- 1 root root 2043 Jun 15 15:50 root > [root@ftp02 cron]# md5sum root > 0636c8deaeadfea7b9ddaa29652b43ae root > > Now, I vim the file on one of them: > [root@ftp01 cron]# vim root > [root@ftp01 cron]# ls -la > total 2 > drwx------ 1 root root 0 Jun 15 15:51 . > drwxr-xr-x. 10 root root 104 May 19 09:34 .. > -rw------- 1 root root 2044 Jun 15 15:50 root > [root@ftp01 cron]# md5sum root > 7a0c346bbd2b61c5fe990bb277c00917 root > > [root@ftp02 cron]# md5sum root > 7a0c346bbd2b61c5fe990bb277c00917 root > > So far so good, right? Then, a few seconds later: > > [root@ftp02 cron]# ls -la > total 0 > drwx------ 1 root root 0 Jun 15 15:51 . > drwxr-xr-x. 10 root root 104 May 19 09:34 .. > -rw------- 1 root root 0 Jun 15 15:50 root > [root@ftp02 cron]# cat root > [root@ftp02 cron]# md5sum root > d41d8cd98f00b204e9800998ecf8427e root > > And on ftp01: > > [root@ftp01 cron]# ls -la > total 2 > drwx------ 1 root root 0 Jun 15 15:51 . > drwxr-xr-x. 10 root root 104 May 19 09:34 .. > -rw------- 1 root root 2044 Jun 15 15:50 root > [root@ftp01 cron]# md5sum root > 7a0c346bbd2b61c5fe990bb277c00917 root > > I later create a 'root2' on ftp02 and cause a similar issue. The end > results are two non-matching files: > > [root@ftp01 cron]# ls -la > total 2 > drwx------ 1 root root 0 Jun 15 15:53 . > drwxr-xr-x. 10 root root 104 May 19 09:34 .. > -rw------- 1 root root 2044 Jun 15 15:50 root > -rw-r--r-- 1 root root 0 Jun 15 15:53 root2 > > [root@ftp02 cron]# ls -la > total 2 > drwx------ 1 root root 0 Jun 15 15:53 . > drwxr-xr-x. 10 root root 104 May 19 09:34 .. > -rw------- 1 root root 0 Jun 15 15:50 root > -rw-r--r-- 1 root root 1503 Jun 15 15:53 root2 > > We were able to reproduce this on two other systems with the same cephfs > filesystem. I have also seen cases where the file would just blank out on > both as well. > > We could not reproduce it with our dev/test cluster running the development > ceph version: > > ceph-10.2.2-1.g502540f.el7.x86_64 Strange. In that cluster, was the same 3.x kernel in use? There aren't a whole lot of changes on the server side in v10.2.2 that I could imagine affecting this case. The best thing to do right now is to try using ceph-fuse in your production environment, to check that it is not exhibiting the same behaviour as the old kernel client. Once you confirm that, I would recommend upgrading your kernel to the most recent 4.x that you are comfortable with, and confirm that that also does not exhibit the bad behaviour. John > Is this a known bug with the current production Jewel release? If so, will > it be patched in the next release? > > Thank you very much, > > Jason Gress > > "This message and any attachments may contain confidential information. If > you > have received this message in error, any use or distribution is prohibited. > Please notify us by reply e-mail if you have mistakenly received this > message, > and immediately and permanently delete it and any attachments. Thank you." > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |