On Thu, Mar 07, 2019 at 01:37:55PM -0300, Herbert Alexander Faleiros wrote:
> Hi,
>
> # ceph health detail
> HEALTH_ERR 3 scrub errors; Possible data damage: 1 pg inconsistent
> OSD_SCRUB_ERRORS 3 scrub errors
> PG_DAMAGED Possible data damage: 1 pg inconsistent
> pg 2.2bb is active+clean+inconsistent, acting [36,12,80]
>
> # ceph pg repair 2.2bb
> instructing pg 2.2bb on osd.36 to repair
>
> But:
>
> 2019-03-07 13:23:38.636881 [ERR] Health check update: Possible data damage: 1 pg inconsistent, 1 pg repair (PG_DAMAGED)
> 2019-03-07 13:20:38.373431 [ERR] 2.2bb deep-scrub 3 errors
> 2019-03-07 13:20:38.373426 [ERR] 2.2bb deep-scrub 0 missing, 1 inconsistent objects
> 2019-03-07 13:20:43.486860 [ERR] Health check update: 3 scrub errors (OSD_SCRUB_ERRORS)
> 2019-03-07 13:19:17.741350 [ERR] deep-scrub 2.2bb 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : is an unexpected clone
> 2019-03-07 13:19:17.523042 [ERR] 2.2bb shard 36 soid 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : data_digest 0xffffffff != data_digest 0xfc6b9538 from shard 12, size 0 != size 4194304 from auth oi 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986(482757'14986708 client.112595650.0:344888465 dirty|omap_digest s 4194304 uv 14974021 od ffffffff alloc_hint [0 0 0]), size 0 != size 4194304 from shard 12
> 2019-03-07 13:19:17.523038 [ERR] 2.2bb shard 36 soid 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : candidate size 0 info size 4194304 mismatch
> 2019-03-07 13:16:48.542673 [ERR] 2.2bb repair 2 errors, 1 fixed
> 2019-03-07 13:16:48.542656 [ERR] 2.2bb repair 1 missing, 0 inconsistent objects
> 2019-03-07 13:16:53.774956 [ERR] Health check update: Possible data damage: 1 pg inconsistent (PG_DAMAGED)
> 2019-03-07 13:16:53.774916 [ERR] Health check update: 2 scrub errors (OSD_SCRUB_ERRORS)
> 2019-03-07 13:15:16.986872 [ERR] repair 2.2bb 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : is an unexpected clone
> 2019-03-07 13:15:16.986817 [ERR] 2.2bb shard 36 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : missing
> 2019-03-07 13:12:18.517442 [ERR] Health check update: Possible data damage: 1 pg inconsistent, 1 pg repair (PG_DAMAGED)
>
> Also tried deep-scrub and scrub, same results.
>
> Also set noscrub,nodeep-scrub, kicked currently active scrubs one at
> a time using 'ceph osd down <id>'. After the last scrub was kicked,
> forced scrub ran immediately then 'ceph pg repair', no luck.
>
> Finally tryed the manual aproach:
>
> - stop osd.36
> - flush-journal
> - rm rbd\udata.dfd5e2235befd0.000000000001c299__4f986_CBDE52BB__2
> - start osd.36
> - ceph pg repair 2.2bb
>
> Also no luck...
>
> rbd\udata.dfd5e2235befd0.000000000001c299__4f986_CBDE52BB__2 at osd.36
> is empty (0 size). At osd.80 4.0M, osd.2 is bluestore (can't find it).
>
> Ceph is 12.2.10, I'm currently migrating all my OSDs to bluestore.
>
> Is there anything else I can do?
Should I do something like this? (below, after stop osd.36)
# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-36/ --journal-path /dev/sdc1 rbd_data.dfd5e2235befd0.000000000001c299 remove-clone-metadata 326022
I'm no sure about rbd_data.$RBD and $CLONEID (took from rados
list-inconsistent-obj, also below).
> # rados list-inconsistent-obj 2.2bb | jq
> {
> "epoch": 484655,
> "inconsistents": [
> {
> "object": {
> "name": "rbd_data.dfd5e2235befd0.000000000001c299",
> "nspace": "",
> "locator": "",
> "snap": 326022,
> "version": 14974021
> },
> "errors": [
> "data_digest_mismatch",
> "size_mismatch"
> ],
> "union_shard_errors": [
> "size_mismatch_info",
> "obj_size_info_mismatch"
> ],
> "selected_object_info": {
> "oid": {
> "oid": "rbd_data.dfd5e2235befd0.000000000001c299",
> "key": "",
> "snapid": 326022,
> "hash": 3420345019,
> "max": 0,
> "pool": 2,
> "namespace": ""
> },
> "version": "482757'14986708",
> "prior_version": "482697'14980304",
> "last_reqid": "client.112595650.0:344888465",
> "user_version": 14974021,
> "size": 4194304,
> "mtime": "2019-03-02 22:30:23.812849",
> "local_mtime": "2019-03-02 22:30:23.813281",
> "lost": 0,
> "flags": [
> "dirty",
> "omap_digest"
> ],
> "legacy_snaps": [],
> "truncate_seq": 0,
> "truncate_size": 0,
> "data_digest": "0xffffffff",
> "omap_digest": "0xffffffff",
> "expected_object_size": 0,
> "expected_write_size": 0,
> "alloc_hint_flags": 0,
> "manifest": {
> "type": 0,
> "redirect_target": {
> "oid": "",
> "key": "",
> "snapid": 0,
> "hash": 0,
> "max": 0,
> "pool": -9223372036854776000,
> "namespace": ""
> }
> },
> "watchers": {}
> },
> "shards": [
> {
> "osd": 12,
> "primary": false,
> "errors": [],
> "size": 4194304,
> "omap_digest": "0xffffffff",
> "data_digest": "0xfc6b9538"
> },
> {
> "osd": 36,
> "primary": true,
> "errors": [
> "size_mismatch_info",
> "obj_size_info_mismatch"
> ],
> "size": 0,
> "omap_digest": "0xffffffff",
> "data_digest": "0xffffffff",
> "object_info": {
> "oid": {
> "oid": "rbd_data.dfd5e2235befd0.000000000001c299",
> "key": "",
> "snapid": 326022,
> "hash": 3420345019,
> "max": 0,
> "pool": 2,
> "namespace": ""
> },
> "version": "482757'14986708",
> "prior_version": "482697'14980304",
> "last_reqid": "client.112595650.0:344888465",
> "user_version": 14974021,
> "size": 4194304,
> "mtime": "2019-03-02 22:30:23.812849",
> "local_mtime": "2019-03-02 22:30:23.813281",
> "lost": 0,
> "flags": [
> "dirty",
> "omap_digest"
> ],
> "legacy_snaps": [],
> "truncate_seq": 0,
> "truncate_size": 0,
> "data_digest": "0xffffffff",
> "omap_digest": "0xffffffff",
> "expected_object_size": 0,
> "expected_write_size": 0,
> "alloc_hint_flags": 0,
> "manifest": {
> "type": 0,
> "redirect_target": {
> "oid": "",
> "key": "",
> "snapid": 0,
> "hash": 0,
> "max": 0,
> "pool": -9223372036854776000,
> "namespace": ""
> }
> },
> "watchers": {}
> }
> },
> {
> "osd": 80,
> "primary": false,
> "errors": [],
> "size": 4194304,
> "omap_digest": "0xffffffff",
> "data_digest": "0xfc6b9538"
> }
> ]
> }
> ]
> }
