1) find all OSDs hosting the PG
ceph pg map 8.e82
2) find the directory with the object on the OSDs
should be something like /var/lib/ceph/ceph-XX/current/8.e82_head/
3) look in this directory for files named like what you see in logs (rb.0.bfcb12.238e1f29.0000002acd39)
there are _head_ objects that contain the original data, and then objects named with the snapshot id instead (_1fc8ce82_ instead of _head_)
4) compare what files are there on the OSDs
5a) you are lucky and one of the OSDs has them - in that case you could either copy them to the others (don't forget xattrs!) or rebuild them via backfills from the good OSD
5b) you are not that lucky and the files are not there - I'm not that sure what to do then
You could in theory just copy the _head_ object contents to the missing objects and then drop the image.
Or you could maybe just delete the _head_ objects (since you don't need that image anymore), but I don't know whether there's some info stored (in leveldb, or somewhere else) about the rbd image or if all the info is in the objects themselves.
I think others here will help you more in that case.
I'm not sure if there's an option to "delete rbd image, ignore missing files, call it a day" - that one would be handy for situations like this.
Jan
Is there some way to manually correct this error while this bug is still needing review? I have one PG that is stuck inconsistent with the same error. I already created a new RBD image and migrated the data to it. The original RBD image was "rb.0.ac3386.238e1f29". The new image is "rb.0.bfcb12.238e1f29".
2015-10-20 19:18:07.686783 7f50e4c1d700 0 log_channel(cluster) log [INF] : 8.e82 repair starts
2015-10-20 19:18:40.300721 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : repair 8.e82 1fc8ce82/rb.0.ac3386.238e1f29.00000008776e/snapdir//8 missing clones
2015-10-20 19:18:40.301094 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : repair 8.e82 9cc8ce82/rb.0.bfcb12.238e1f29.0000002acd39/head//8 expected clone 1fc8ce82/rb.0.ac3386.238e1f29.00000008776e/44//8
2015-10-20 19:18:40.301124 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : repair 8.e82 fb78ce82/rb.0.bfcb12.238e1f29.0000000e69a3/head//8 expected clone 9cc8ce82/rb.0.bfcb12.238e1f29.0000002acd39/44//8
2015-10-20 19:18:40.301140 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : repair 8.e82 8038ce82/rb.0.bfcb12.238e1f29.0000002b7781/head//8 expected clone fb78ce82/rb.0.bfcb12.238e1f29.0000000e69a3/44//8
2015-10-20 19:18:40.301155 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : repair 8.e82 c8b7ce82/rb.0.bfcb12.238e1f29.000000059252/head//8 expected clone 8038ce82/rb.0.bfcb12.238e1f29.0000002b7781/44//8
2015-10-20 19:18:40.301170 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : repair 8.e82 9d26ce82/rb.0.bfcb12.238e1f29.0000000cd86d/head//8 expected clone c8b7ce82/rb.0.bfcb12.238e1f29.000000059252/44//8
2015-10-20 19:18:40.301185 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : repair 8.e82 c006ce82/rb.0.bfcb12.238e1f29.0000000c53d6/head//8 expected clone 9d26ce82/rb.0.bfcb12.238e1f29.0000000cd86d/44//8
2015-10-20 19:18:40.301200 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : repair 8.e82 3434ce82/rb.0.bfcb12.238e1f29.0000002cb957/head//8 expected clone c006ce82/rb.0.bfcb12.238e1f29.0000000c53d6/44//8
2015-10-20 19:18:47.724047 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : 8.e82 repair 8 errors, 0 fixed
2
Thanks,
Chris
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxxhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
