Note that in this case the inconsistencies are not coming from object reads, but from comparing the omap digests of an rgw index shard. This seems to be a result of a replication issue sometime in the past on this cluster. On Fri, Dec 4, 2020 at 10:32 AM Eugen Block <eblock@xxxxxx> wrote: > > Hi, > > this is not necessarily but most likely a hint to a (slowly) failing > disk. Check all OSDs for this PG for disk errors in dmesg and smartctl. > > Regards, > Eugen > > > Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@xxxxxxxxx>: > > > Hi, > > > > Not sure is it related to my 15.2.7 update, but today I got many > > time this issue: > > > > 2020-12-04T15:14:23.910799+0700 osd.40 (osd.40) 11 : cluster [DBG] > > 11.2 deep-scrub starts > > 2020-12-04T15:14:23.947255+0700 osd.40 (osd.40) 12 : cluster [ERR] > > 11.2 soid > > 11:434f049b:::.dir.75333f99-93d0-4238-91a4-ba833a0edd24.1744118.372.1:head : > > omap_digest 0x48532c00 != omap_digest 0x8a18f5d7 from shard 40 > > 2020-12-04T15:14:23.977138+0700 mgr.hk-cephmon-2s02 (mgr.2120884) > > 4330 : cluster [DBG] pgmap v4338: 209 pgs: 209 active+clean; 2.8 GiB > > data, 21 TiB used, 513 TiB / 534 TiB avail; 32 KiB/s rd, 32 op/s > > 2020-12-04T15:14:24.030888+0700 osd.40 (osd.40) 13 : cluster [ERR] > > 11.2 soid > > 11:4b86603b:::.dir.75333f99-93d0-4238-91a4-ba833a0edd24.1744118.197.3:head : > > omap_digest 0xcb62779b != omap_digest 0xefef7471 from shard 40 > > 2020-12-04T15:14:24.229000+0700 osd.40 (osd.40) 14 : cluster [ERR] > > 11.2 deep-scrub 0 missing, 2 inconsistent objects > > 2020-12-04T15:14:24.229003+0700 osd.40 (osd.40) 15 : cluster [ERR] > > 11.2 deep-scrub 2 errors > > 2020-12-04T15:14:25.978189+0700 mgr.hk-cephmon-2s02 (mgr.2120884) > > 4331 : cluster [DBG] pgmap v4339: 209 pgs: 1 > > active+clean+scrubbing+deep, 208 active+clean; 2.8 GiB data, 21 TiB > > used, 513 TiB / 534 TiB avail; 55 KiB/s rd, 0 B/s wr, 61 op/s > > 2020-12-04T15:14:27.978588+0700 mgr.hk-cephmon-2s02 (mgr.2120884) > > 4332 : cluster [DBG] pgmap v4340: 209 pgs: 1 > > active+clean+scrubbing+deep, 208 active+clean; 2.8 GiB data, 21 TiB > > used, 513 TiB / 534 TiB avail; 43 KiB/s rd, 0 B/s wr, 49 op/s > > 2020-12-04T15:14:30.293180+0700 mon.hk-cephmon-2s01 (mon.0) 4475 : > > cluster [ERR] Health check failed: 2 scrub errors (OSD_SCRUB_ERRORS) > > 2020-12-04T15:14:30.293196+0700 mon.hk-cephmon-2s01 (mon.0) 4476 : > > cluster [ERR] Health check failed: Possible data damage: 1 pg > > inconsistent (PG_DAMAGED) > > > > I had to repair pg and it worked fine, but not sure where this come > > from. I have this in the log only :/ > > > > Thank you. > > > > ________________________________ > > This message is confidential and is for the sole use of the intended > > recipient(s). It may also be privileged or otherwise protected by > > copyright or other legal rules. If you have received it by mistake > > please let us know by reply email and delete it from your system. It > > is prohibited to copy this message or disclose its content to > > anyone. Any confidentiality or privilege is not waived or lost by > > any mistaken delivery or unauthorized disclosure of the message. All > > messages sent to and from Agoda may be monitored to ensure > > compliance with company policies, to protect the company's interests > > and to remove potential malware. Electronic messages may be > > intercepted, amended, lost or deleted, or contain viruses. > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |