Hi *,
after updating our CEPH cluster from 14.2.9 to 14.2.10 it accumulates
scrub errors on multiple osds:
[cephmon1] /root # ceph health detail
HEALTH_ERR 6 scrub errors; Possible data damage: 6 pgs inconsistent
OSD_SCRUB_ERRORS 6 scrub errors
PG_DAMAGED Possible data damage: 6 pgs inconsistent
pg 3.69 is active+clean+inconsistent, acting [59,65,61]
pg 3.73 is active+clean+inconsistent, acting [73,88,25]
pg 12.29 is active+clean+inconsistent, acting [55,92,42]
pg 12.38 is active+clean+inconsistent, acting [150,42,13]
pg 12.46 is active+clean+inconsistent, acting [55,18,84]
pg 12.75 is active+clean+inconsistent, acting [55,155,49]
They all can easily get repaired (ceph pg repair $pg) - but I wonder
what could be the source of the problem. The cluster started with
Luminous some years ago, was updated to Mimic, then Nautilus. Never
seen this before!
OSDs are a mixture of HDD/SSD, both are affected. All on Bluestore.
Any idea? Was there maybe a code change between 14.2.9 & 14.2.10 that
could explain this? Errors in syslog look like this:
Aug 5 19:21:21 krake08 ceph-osd: 2020-08-05 19:21:21.831 7fb6b2b9d700 -1 log_channel(cluster) log [ERR] : 12.38 scrub : stat mismatch, got 74/74 objects, 20/20 clones, 74/74 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 182904850/172877842 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.
Aug 5 19:21:21 krake08 ceph-osd: 2020-08-05 19:21:21.831 7fb6b2b9d700 -1 log_channel(cluster) log [ERR] : 12.38 scrub 1 errors
Aug 6 08:28:44 krake08 ceph-osd: 2020-08-06 08:28:44.477 7fb6b2b9d700 -1 log_channel(cluster) log [ERR] : 12.38 repair : stat mismatch, got 76/76 objects, 22/22 clones, 76/76 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 183166994/173139986 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.
Aug 6 08:28:44 krake08 ceph-osd: 2020-08-06 08:28:44.477 7fb6b2b9d700 -1 log_channel(cluster) log [ERR] : 12.38 repair 1 errors, 1 fixed
Thanks in advance,
Andreas
--
| Andreas Haupt | E-Mail: andreas.haupt@xxxxxxx
| DESY Zeuthen | WWW: http://www-zeuthen.desy.de/~ahaupt
| Platanenallee 6 | Phone: +49/33762/7-7359
| D-15738 Zeuthen | Fax: +49/33762/7-7216
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
