> 在 2021年10月4日，00:53，Michael Thomas <wart@xxxxxxxxxxx> 写道：
>
> I recently started getting inconsistent PGs in my Octopus (15.2.14) ceph cluster. I was able to determine that they are all coming from the same OSD: osd.143. This host recently suffered from an unplanned power loss, so I'm not surprised that there may be some corruption. This PG is part of a EC 8+2 pool.
>
> The OSD logs from the PG's primary OSD show this and similar errors from the PG's most recent deep scrub:
>
> 2021-10-03T03:25:25.969-0500 7f6e6801f700 -1 log_channel(cluster) log [ERR] : 23.1fa shard 143(1) soid 23:5f8c3d4e:::10000179969.00000168:head : candidate had a read error
>
> In attempting to fix it, I first ran 'ceph pg repair 23.1fa' on the PG. This accomplished nothing. Next I ran a shallow fsck on the OSD:
I expect this ‘ceph pg repair’ command could handle this kind of errors. After issuing this command, the pg should enter a state like “active+clean+scrubbing+deep+inconsistent+repair”, then you wait for the repair to finish (this can take hours), and you should be able to recover from the inconsistent state. What do you mean by “This accomplished nothing”?
> # ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-143
> fsck success
>
> I estimated that a deep fsck will take ~24 hours to run on this mostly full 16TB HDD. Before doing that, I wanted to see if I could simply remove the offending object and let ceph recover itself. Unfortunately, ceph-objectstore-tool core dumps when I try to remove this object:
>
> # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-143 --pgid 23.1fa '{"oid":"10000179969.00000168","key":"","snapid":-2,"hash":1924936186,"max":0,"pool":23,"namespace":"","shard_id":1,"max":0}' remove
> *** Caught signal (Segmentation fault) **
> in thread 7fdc491a88c0 thread_name:ceph-objectstor
> ceph version 15.2.14 (cd3bb7e87a2f62c1b862ff3fd8b1eec13391a5be) octopus (stable)
> 1: (()+0xf630) [0x7fdc3e62a630]
> 2: (__pthread_rwlock_rdlock()+0xb) [0x7fdc3e62614b]
> 3: (BlueStore::collection_bits(boost::intrusive_ptr<ObjectStore::CollectionImpl>&)+0x148) [0x5583c8fa7878]
> 4: (main()+0x4b50) [0x5583c8a85270]
> 5: (__libc_start_main()+0xf5) [0x7fdc3cfe7555]
> 6: (()+0x39d3a0) [0x5583c8ab03a0]
> Segmentation fault (core dumped)
>
> As a last resort, I know that I can map this OID back to the cephfs file and simply remove/restore the offending file to fix the object. But before I do that, I'm running a deep fsck to see if that can fix this and the other inconsistent objects. In the meantime, I wondered if there was anything else I could do to clean up this inconsistent PG?
>
> --Mike
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
