Yes, that suggestion worked for us, although we hit this when we've upgraded to 10.2.10 from 10.2.7. I guess this was fixed via http://tracker.ceph.com/issues/21440 and http://tracker.ceph.com/issues/19404 Thanks, -Pavan. On 7/16/18, 5:07 AM, "ceph-users on behalf of Matthew Vernon" <ceph-users-bounces@xxxxxxxxxxxxxx on behalf of mv3@xxxxxxxxxxxx> wrote: Hi, Our cluster is running 10.2.9 (from Ubuntu; on 16.04 LTS), and we have a pg that's stuck inconsistent; if I repair it, it logs "failed to pick suitable auth object" (repair log attached, to try and stop my MUA mangling it). We then deep-scrubbed that pg, at which point rados list-inconsistent-obj 67.2e --format=json-pretty produces a bit of output (also attached), which includes that all 3 osds have a zero-sized object e.g. "osd": 1937, "errors": [ "omap_digest_mismatch_oi" ], "size": 0, "omap_digest": "0x45773901", "data_digest": "0xffffffff" All 3 osds have different omap_digest, but all have 0 size. Indeed, looking on the OSD disks directly, each object is 0 size (i.e. they are identical). This looks similar to one of the failure modes in http://tracker.ceph.com/issues/21388 where the is a suggestion (comment 19 from David Zafman) to do: rados -p default.rgw.buckets.index setomapval .dir.861ae926-7ff0-48c5-86d6-a6ba8d0a7a14.7130858.6 temporary-key anything [deep-scrub] rados -p default.rgw.buckets.index rmomapkey .dir.861ae926-7ff0-48c5-86d6-a6ba8d0a7a14.7130858.6 temporary-key Is this likely to be the correct approach here, to? And is there an underlying bug in ceph that still needs fixing? :) Thanks, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |