To add -- I thought I was running 0.67.4 on my test cluster (fc 19), but I appear to be running 0.69. Not sure how that happened as my yum config is still pointing to dumpling. :)
On Thu, Oct 24, 2013 at 10:52 AM, Matt Thompson <wateringcan@xxxxxxxxx> wrote:
Hi Harry,I was able to replicate this.What does appear to work (for me) is to do an osd scrub followed by a pg repair. I've tried this 2x now and in each case the deleted file gets copied over to the OSD from where it was removed. However, I've tried a few pg scrub / pg repairs after manually deleting a file and have yet to see the file get copied back to the OSD on which it was deleted. Like you said, the pg repair sets the health of the PG back to active+clean, but then re-running the pg scrub detects the file as missing again and sets it back to active+clean+inconsistent.Regards,MattOn Wed, Oct 23, 2013 at 3:45 PM, Harry Harrington <git-harry@xxxxxxxxxx> wrote:
Hi,
I've been taking a look at the repair functionality in ceph. As I understand it the osds should try to copy an object from another member of the pg if it is missing. I have been attempting to test this by manually removing a file from one of the osds however each time the repair completes the the file has not been restored. If I run another scrub on the pg it gets flagged as inconsistent. See below for the output from my testing. I assume I'm missing something obvious, any insight into this process would be greatly appreciated.
Thanks,
Harry
# ceph --version
ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
# ceph status
cluster a4e417fe-0386-46a5-4475-ca7e10294273
health HEALTH_OK
monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum 0 ceph1
osdmap e13: 3 osds: 3 up, 3 in
pgmap v232: 192 pgs: 192 active+clean; 44 bytes data, 15465 MB used, 164 GB / 179 GB avail
mdsmap e1: 0/0/1 up
file removed from osd.2
# ceph pg scrub 0.b
instructing pg 0.b on osd.1 to scrub
# ceph status
cluster a4e417fe-0386-46a5-4475-ca7e10294273
health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors
monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum 0 ceph1
osdmap e13: 3 osds: 3 up, 3 in
pgmap v233: 192 pgs: 191 active+clean, 1 active+clean+inconsistent; 44 bytes data, 15465 MB used, 164 GB / 179 GB avail
mdsmap e1: 0/0/1 up
# ceph pg repair 0.b
instructing pg 0.b on osd.1 to repair
# ceph status
cluster a4e417fe-0386-46a5-4475-ca7e10294273
health HEALTH_OK
monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum 0 ceph1
osdmap e13: 3 osds: 3 up, 3 in
pgmap v234: 192 pgs: 192 active+clean; 44 bytes data, 15465 MB used, 164 GB / 179 GB avail
mdsmap e1: 0/0/1 up
# ceph pg scrub 0.b
instructing pg 0.b on osd.1 to scrub
# ceph status
cluster a4e417fe-0386-46a5-4475-ca7e10294273
health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors
monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum 0 ceph1
osdmap e13: 3 osds: 3 up, 3 in
pgmap v236: 192 pgs: 191 active+clean, 1 active+clean+inconsistent; 44 bytes data, 15465 MB used, 164 GB / 179 GB avail
mdsmap e1: 0/0/1 up
The logs from osd.1:
2013-10-23 14:12:31.188281 7f02a5161700 0 log [ERR] : 0.b osd.2 missing 3a643fcb/testfile1/head//0
2013-10-23 14:12:31.188312 7f02a5161700 0 log [ERR] : 0.b scrub 1 missing, 0 inconsistent objects
2013-10-23 14:12:31.188319 7f02a5161700 0 log [ERR] : 0.b scrub 1 errors
2013-10-23 14:13:03.197802 7f02a5161700 0 log [ERR] : 0.b osd.2 missing 3a643fcb/testfile1/head//0
2013-10-23 14:13:03.197837 7f02a5161700 0 log [ERR] : 0.b repair 1 missing, 0 inconsistent objects
2013-10-23 14:13:03.197850 7f02a5161700 0 log [ERR] : 0.b repair 1 errors, 1 fixed
2013-10-23 14:14:47.232953 7f02a5161700 0 log [ERR] : 0.b osd.2 missing 3a643fcb/testfile1/head//0
2013-10-23 14:14:47.232985 7f02a5161700 0 log [ERR] : 0.b scrub 1 missing, 0 inconsistent objects
2013-10-23 14:14:47.232991 7f02a5161700 0 log [ERR] : 0.b scrub 1 errors
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
