I was also able to reproduce this, guys, but I believe it’s specific to the mode of testing rather than to anything being wrong with the OSD. In particular, after restarting the OSD whose file I removed and running repair, it did so successfully. The OSD has an “fd cacher” which caches open file handles, and we believe this is what causes the observed behavior: if the removed object is among the most recent <n> objects touched, the FileStore (an OSD subsystem) has an open fd cached, so when manually deleting the file the FileStore now has a deleted file open. When the repair happens, it finds that open file descriptor and applies the repair to it — which of course doesn’t help put it back into place! -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On October 24, 2013 at 2:52:54 AM, Matt Thompson (wateringcan@xxxxxxxxx) wrote: > >Hi Harry, > >I was able to replicate this. > >What does appear to work (for me) is to do an osd scrub followed by a pg >repair. I've tried this 2x now and in each case the deleted file gets >copied over to the OSD from where it was removed. However, I've tried a >few pg scrub / pg repairs after manually deleting a file and have yet to >see the file get copied back to the OSD on which it was deleted. Like you >said, the pg repair sets the health of the PG back to active+clean, but >then re-running the pg scrub detects the file as missing again and sets it >back to active+clean+inconsistent. > >Regards, >Matt > > >On Wed, Oct 23, 2013 at 3:45 PM, Harry Harrington wrote: > >> Hi, >> >> I've been taking a look at the repair functionality in ceph. As I >> understand it the osds should try to copy an object from another member of >> the pg if it is missing. I have been attempting to test this by manually >> removing a file from one of the osds however each time the repair >> completes the the file has not been restored. If I run another scrub on the >> pg it gets flagged as inconsistent. See below for the output from my >> testing. I assume I'm missing something obvious, any insight into this >> process would be greatly appreciated. >> >> Thanks, >> Harry >> >> # ceph --version >> ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) >> # ceph status >> cluster a4e417fe-0386-46a5-4475-ca7e10294273 >> health HEALTH_OK >> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum >> 0 ceph1 >> osdmap e13: 3 osds: 3 up, 3 in >> pgmap v232: 192 pgs: 192 active+clean; 44 bytes data, 15465 MB used, >> 164 GB / 179 GB avail >> mdsmap e1: 0/0/1 up >> >> file removed from osd.2 >> >> # ceph pg scrub 0.b >> instructing pg 0.b on osd.1 to scrub >> >> # ceph status >> cluster a4e417fe-0386-46a5-4475-ca7e10294273 >> health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors >> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum >> 0 ceph1 >> osdmap e13: 3 osds: 3 up, 3 in >> pgmap v233: 192 pgs: 191 active+clean, 1 active+clean+inconsistent; 44 >> bytes data, 15465 MB used, 164 GB / 179 GB avail >> mdsmap e1: 0/0/1 up >> >> # ceph pg repair 0.b >> instructing pg 0.b on osd.1 to repair >> >> # ceph status >> cluster a4e417fe-0386-46a5-4475-ca7e10294273 >> health HEALTH_OK >> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum >> 0 ceph1 >> osdmap e13: 3 osds: 3 up, 3 in >> pgmap v234: 192 pgs: 192 active+clean; 44 bytes data, 15465 MB used, >> 164 GB / 179 GB avail >> mdsmap e1: 0/0/1 up >> >> # ceph pg scrub 0.b >> instructing pg 0.b on osd.1 to scrub >> >> # ceph status >> cluster a4e417fe-0386-46a5-4475-ca7e10294273 >> health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors >> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum >> 0 ceph1 >> osdmap e13: 3 osds: 3 up, 3 in >> pgmap v236: 192 pgs: 191 active+clean, 1 active+clean+inconsistent; 44 >> bytes data, 15465 MB used, 164 GB / 179 GB avail >> mdsmap e1: 0/0/1 up >> >> >> >> The logs from osd.1: >> 2013-10-23 14:12:31.188281 7f02a5161700 0 log [ERR] : 0.b osd.2 missing >> 3a643fcb/testfile1/head//0 >> 2013-10-23 14:12:31.188312 7f02a5161700 0 log [ERR] : 0.b scrub 1 >> missing, 0 inconsistent objects >> 2013-10-23 14:12:31.188319 7f02a5161700 0 log [ERR] : 0.b scrub 1 errors >> 2013-10-23 14:13:03.197802 7f02a5161700 0 log [ERR] : 0.b osd.2 missing >> 3a643fcb/testfile1/head//0 >> 2013-10-23 14:13:03.197837 7f02a5161700 0 log [ERR] : 0.b repair 1 >> missing, 0 inconsistent objects >> 2013-10-23 14:13:03.197850 7f02a5161700 0 log [ERR] : 0.b repair 1 >> errors, 1 fixed >> 2013-10-23 14:14:47.232953 7f02a5161700 0 log [ERR] : 0.b osd.2 missing >> 3a643fcb/testfile1/head//0 >> 2013-10-23 14:14:47.232985 7f02a5161700 0 log [ERR] : 0.b scrub 1 >> missing, 0 inconsistent objects >> 2013-10-23 14:14:47.232991 7f02a5161700 0 log [ERR] : 0.b scrub 1 errors >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >_______________________________________________ >ceph-users mailing list >ceph-users@xxxxxxxxxxxxxx >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |