Hi Brad
Thank you very much for your attention.
On 07.03.2018 23:46, Brad Hubbard wrote:
On Thu, Mar 8, 2018 at 1:22 AM, Harald Staub <harald.staub@xxxxxxxxx> wrote:
"ceph pg repair" leads to:
5.7bd repair 2 errors, 0 fixed
Only an empty list from:
rados list-inconsistent-obj 5.7bd --format=json-pretty
Inspired by http://tracker.ceph.com/issues/12577 , I tried again with more
verbose logging and searched the osd logs e.g. for "!=", "mismatch", could
not find anything interesting. Oh well, these are several millions of lines
...
Any hint what I could look for?
Try searching for "scrub_compare_maps" and looking for "5.7bd" in that context.
These lines (from the primary OSD) may be interesting:
2018-03-07 14:20:31.405120 7f42497c4700 10 osd.340 pg_epoch: 505959
pg[5.7bd( v 505959'35722945 (505688'35721366,505959'35722945]
local-lis/les=505083/505086 n=16133 ec=859/859 lis/c 505083/505083
les/c/f 505086/505086/0 505083/505083/505083) [340,491,442] r=0
lpr=505083 crt=505959'35722945 lcod 505959'35722944 mlcod
505959'35722944 active+clean+scrubbing+deep+inconsistent+repair
snaptrimq=[3565b~18,35674~2]] be_select_auth_object: error(s) osd 442
for obj 5:bde7a84d:::rbd_data.d393823accce24.0000000000010214:336d7,
object_info_inconsistency
2018-03-07 14:20:31.405134 7f42497c4700 10 osd.340 pg_epoch: 505959
pg[5.7bd( v 505959'35722945 (505688'35721366,505959'35722945]
local-lis/les=505083/505086 n=16133 ec=859/859 lis/c 505083/505083
les/c/f 505086/505086/0 505083/505083/505083) [340,491,442] r=0
lpr=505083 crt=505959'35722945 lcod 505959'35722944 mlcod
505959'35722944 active+clean+scrubbing+deep+inconsistent+repair
snaptrimq=[3565b~18,35674~2]] be_select_auth_object: selecting osd 340
for obj 5:bde7a84d:::rbd_data.d393823accce24.0000000000010214:336d7 with
oi
5:bde7a84d:::rbd_data.d393823accce24.0000000000010214:336d7(505072'35716889
osd.340.0:258067 dirty|data_digest|omap_digest s 4194304 uv 35452964 dd
68383c60 od ffffffff alloc_hint [0 0 0])
2018-03-07 14:20:31.405172 7f42497c4700 10 osd.340 pg_epoch: 505959
pg[5.7bd( v 505959'35722945 (505688'35721366,505959'35722945]
local-lis/les=505083/505086 n=16133 ec=859/859 lis/c 505083/505083
les/c/f 505086/505086/0 505083/505083/505083) [340,491,442] r=0
lpr=505083 crt=505959'35722945 lcod 505959'35722944 mlcod
505959'35722944 active+clean+scrubbing+deep+inconsistent+repair
snaptrimq=[3565b~18,35674~2]] be_select_auth_object: error(s) osd 442
for obj 5:bde7a84d:::rbd_data.d393823accce24.0000000000010214:head,
snapset_inconsistency object_info_inconsistency
2018-03-07 14:20:31.405404 7f42497c4700 10 osd.340 pg_epoch: 505959
pg[5.7bd( v 505959'35722945 (505688'35721366,505959'35722945]
local-lis/les=505083/505086 n=16133 ec=859/859 lis/c 505083/505083
les/c/f 505086/505086/0 505083/505083/505083) [340,491,442] r=0
lpr=505083 crt=505959'35722945 lcod 505959'35722944 mlcod
505959'35722944 active+clean+scrubbing+deep+inconsistent+repair
snaptrimq=[3565b~18,35674~2]] scrub_snapshot_metadata (repair) finish
2018-03-07 14:20:31.405413 7f42497c4700 10 osd.340 pg_epoch: 505959
pg[5.7bd( v 505959'35722945 (505688'35721366,505959'35722945]
local-lis/les=505083/505086 n=16133 ec=859/859 lis/c 505083/505083
les/c/f 505086/505086/0 505083/505083/505083) [340,491,442] r=0
lpr=505083 crt=505959'35722945 lcod 505959'35722944 mlcod
505959'35722944 active+clean+scrubbing+deep+inconsistent+repair
snaptrimq=[3565b~18,35674~2]] scrub_compare_maps: discarding scrub results
Then I had another idea. The inconsistency errors were triggered by a
scrub, not a deep scrub. So I triggered another scrub:
ceph pg scrub 5.7bd
And the problem got fixed.
Cheers
Harry
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
