Thanks Greg ---------------------------------------- > Date: Thu, 24 Oct 2013 13:30:02 -0700 > From: greg@xxxxxxxxxxx > To: wateringcan@xxxxxxxxx; git-harry@xxxxxxxxxx > CC: ceph-users@xxxxxxxxxxxxxx > Subject: Re: PG repair failing when object missing > > I also created a ticket to try and handle this particular instance of bad behavior: > http://tracker.ceph.com/issues/6629 > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com > > On October 24, 2013 at 1:22:54 PM, Greg Farnum (gregory.farnum@xxxxxxxxxxx) wrote: >> >>I was also able to reproduce this, guys, but I believe it’s specific to the mode of testing rather than to anything being wrong with the OSD. In particular, after restarting the OSD whose file I removed and running repair, it did so successfully. >>The OSD has an “fd cacher” which caches open file handles, and we believe this is what causes the observed behavior: if the removed object is among the most recent objects touched, the FileStore (an OSD subsystem) has an open fd cached, so when manually deleting the file the FileStore now has a deleted file open. When the repair happens, it finds that open file descriptor and applies the repair to it — which of course doesn’t help put it back into place! >>-Greg >>Software Engineer #42 @ http://inktank.com | http://ceph.com >> >>On October 24, 2013 at 2:52:54 AM, Matt Thompson (wateringcan@xxxxxxxxx) wrote: >>> >>>Hi Harry, >>> >>>I was able to replicate this. >>> >>>What does appear to work (for me) is to do an osd scrub followed by a pg >>>repair. I've tried this 2x now and in each case the deleted file gets >>>copied over to the OSD from where it was removed. However, I've tried a >>>few pg scrub / pg repairs after manually deleting a file and have yet to >>>see the file get copied back to the OSD on which it was deleted. Like you >>>said, the pg repair sets the health of the PG back to active+clean, but >>>then re-running the pg scrub detects the file as missing again and sets it >>>back to active+clean+inconsistent. >>> >>>Regards, >>>Matt >>> >>> >>>On Wed, Oct 23, 2013 at 3:45 PM, Harry Harrington wrote: >>> >>>> Hi, >>>> >>>> I've been taking a look at the repair functionality in ceph. As I >>>> understand it the osds should try to copy an object from another member of >>>> the pg if it is missing. I have been attempting to test this by manually >>>> removing a file from one of the osds however each time the repair >>>> completes the the file has not been restored. If I run another scrub on the >>>> pg it gets flagged as inconsistent. See below for the output from my >>>> testing. I assume I'm missing something obvious, any insight into this >>>> process would be greatly appreciated. >>>> >>>> Thanks, >>>> Harry >>>> >>>> # ceph --version >>>> ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) >>>> # ceph status >>>> cluster a4e417fe-0386-46a5-4475-ca7e10294273 >>>> health HEALTH_OK >>>> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum >>>> 0 ceph1 >>>> osdmap e13: 3 osds: 3 up, 3 in >>>> pgmap v232: 192 pgs: 192 active+clean; 44 bytes data, 15465 MB used, >>>> 164 GB / 179 GB avail >>>> mdsmap e1: 0/0/1 up >>>> >>>> file removed from osd.2 >>>> >>>> # ceph pg scrub 0.b >>>> instructing pg 0.b on osd.1 to scrub >>>> >>>> # ceph status >>>> cluster a4e417fe-0386-46a5-4475-ca7e10294273 >>>> health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors >>>> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum >>>> 0 ceph1 >>>> osdmap e13: 3 osds: 3 up, 3 in >>>> pgmap v233: 192 pgs: 191 active+clean, 1 active+clean+inconsistent; 44 >>>> bytes data, 15465 MB used, 164 GB / 179 GB avail >>>> mdsmap e1: 0/0/1 up >>>> >>>> # ceph pg repair 0.b >>>> instructing pg 0.b on osd.1 to repair >>>> >>>> # ceph status >>>> cluster a4e417fe-0386-46a5-4475-ca7e10294273 >>>> health HEALTH_OK >>>> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum >>>> 0 ceph1 >>>> osdmap e13: 3 osds: 3 up, 3 in >>>> pgmap v234: 192 pgs: 192 active+clean; 44 bytes data, 15465 MB used, >>>> 164 GB / 179 GB avail >>>> mdsmap e1: 0/0/1 up >>>> >>>> # ceph pg scrub 0.b >>>> instructing pg 0.b on osd.1 to scrub >>>> >>>> # ceph status >>>> cluster a4e417fe-0386-46a5-4475-ca7e10294273 >>>> health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors >>>> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum >>>> 0 ceph1 >>>> osdmap e13: 3 osds: 3 up, 3 in >>>> pgmap v236: 192 pgs: 191 active+clean, 1 active+clean+inconsistent; 44 >>>> bytes data, 15465 MB used, 164 GB / 179 GB avail >>>> mdsmap e1: 0/0/1 up >>>> >>>> >>>> >>>> The logs from osd.1: >>>> 2013-10-23 14:12:31.188281 7f02a5161700 0 log [ERR] : 0.b osd.2 missing >>>> 3a643fcb/testfile1/head//0 >>>> 2013-10-23 14:12:31.188312 7f02a5161700 0 log [ERR] : 0.b scrub 1 >>>> missing, 0 inconsistent objects >>>> 2013-10-23 14:12:31.188319 7f02a5161700 0 log [ERR] : 0.b scrub 1 errors >>>> 2013-10-23 14:13:03.197802 7f02a5161700 0 log [ERR] : 0.b osd.2 missing >>>> 3a643fcb/testfile1/head//0 >>>> 2013-10-23 14:13:03.197837 7f02a5161700 0 log [ERR] : 0.b repair 1 >>>> missing, 0 inconsistent objects >>>> 2013-10-23 14:13:03.197850 7f02a5161700 0 log [ERR] : 0.b repair 1 >>>> errors, 1 fixed >>>> 2013-10-23 14:14:47.232953 7f02a5161700 0 log [ERR] : 0.b osd.2 missing >>>> 3a643fcb/testfile1/head//0 >>>> 2013-10-23 14:14:47.232985 7f02a5161700 0 log [ERR] : 0.b scrub 1 >>>> missing, 0 inconsistent objects >>>> 2013-10-23 14:14:47.232991 7f02a5161700 0 log [ERR] : 0.b scrub 1 errors >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>>_______________________________________________ >>>ceph-users mailing list >>>ceph-users@xxxxxxxxxxxxxx >>>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |