The cluster version is 0.94.3.
I have one placement group that is stuck inconsistent.
$ ceph health detail
HEALTH_ERR 1 pgs inconsistent; 1 scrub errors
pg 8.e82 is active+clean+inconsistent, acting [15,43]
1 scrub errors
I tried to run "ceph pg repair 8.e82" but it will not repair it. In the OSD log with debugging turned up to 20 I find this:
2015-10-16 23:28:17.693819 7f3241102700 20 osd.15 pg_epoch: 257666 pg[8.e82( v 257666'39827 (257219'36804,257666'39827] local-les=257620 n=1778 ec=250794 les/c 257620/257666 257314/257619/257619) [15,43] r=0 lpr=257619 crt=257666'39824 lcod 257666'39826 mlcod 257666'39826 active+clean+scrubbing+deep+inconsistent] deep-scrub 1fc8ce82/rb.0.ac3386.238e1f29.00000008776e/head//8 1fc8ce82/rb.0.ac3386.238e1f29.00000008776e/head//8(254220'28664 client.11563455.0:339667580 wrlock_by=unknown.0.0:0 dirty|omap_digest s 4194304 uv 28664 od ffffffff)
2015-10-16 23:28:17.693861 7f3241102700 20 osd.15 pg_epoch: 257666 pg[8.e82( v 257666'39827 (257219'36804,257666'39827] local-les=257620 n=1778 ec=250794 les/c 257620/257666 257314/257619/257619) [15,43] r=0 lpr=257619 crt=257666'39824 lcod 257666'39826 mlcod 257666'39826 active+clean+scrubbing+deep+inconsistent] deep-scrub 1fc8ce82/rb.0.ac3386.238e1f29.00000008776e/68//8 1fc8ce82/rb.0.ac3386.238e1f29.00000008776e/68//8(254220'28658 osd.33.0:2528615 [68] dirty|data_digest|omap_digest s 4194304 uv 9406 dd 5fa9a617 od ffffffff)
2015-10-16 23:28:17.693893 7f3241102700 -1 log_channel(cluster) log [ERR] : deep-scrub 8.e82 1fc8ce82/rb.0.ac3386.238e1f29.00000008776e/head//8 missing clones
2015-10-16 23:28:17.693899 7f3241102700 20 osd.15 pg_epoch: 257666 pg[8.e82( v 257666'39827 (257219'36804,257666'39827] local-les=257620 n=1778 ec=250794 les/c 257620/257666 257314/257619/257619) [15,43] r=0 lpr=257619 crt=257666'39824 lcod 257666'39826 mlcod 257666'39826 active+clean+scrubbing+deep+inconsistent] snapset 68=[68]:[68]+head
I verified the object "rb.0.ac3386.238e1f29.00000008776e" exists on both OSDs. MD5 hashes are the same on both files. I also compared the xattr attributes with "getfattr -d rb.0.ac3386.238e1f29.00000008776e*" on both OSDs.
I also tried removing one of the objects and repairing the PG according to this:
http://www.sebastien-han.fr/blog/2015/04/27/ceph-manually-repair-object/
I've been digging but I can not find anything about "missing clones". Any help would be appreciated.
Thanks,
Chris
