Neat, hadn't seen that command before. Here's the fsck log from the primary OSD: https://pastebin.com/nZ0H5ag3 Looks like the OSD's bluestore "filesystem" itself has some underlying errors, though I'm not sure what to do about them. -----Original Message----- From: Brad Hubbard [mailto:bhubbard@xxxxxxxxxx] Sent: Monday, August 21, 2017 7:05 PM To: Edward R Huyer <erhvks@xxxxxxx> Cc: ceph-users@xxxxxxxxxxxxxx Subject: Re: PG reported as inconsistent in status, but no inconsistencies visible to rados Could you provide the output of 'ceph-bluestore-tool fsck' for one of these OSDs? On Tue, Aug 22, 2017 at 2:53 AM, Edward R Huyer <erhvks@xxxxxxx> wrote: > This is an odd one. My cluster is reporting an inconsistent pg in > ceph status and ceph health detail. However, rados > list-inconsistent-obj and rados list-inconsistent-snapset both report > no inconsistencies. Scrubbing the pg results in these errors in the osd logs: > > > > OSD 63 (primary): > > 2017-08-21 12:41:03.580068 7f0b36629700 -1 > bluestore(/var/lib/ceph/osd/ceph-63) _verify_csum bad crc32c/0x1000 > checksum at blob offset 0x0, got 0x6b6b9184, expected 0x6706be76, > device location [0x23f39d0000~1000], logical extent 0x0~1000, object > #9:55bf7cc6:::rbd_data.33992ae8944a.000000000000200f:e# > > 2017-08-21 12:41:03.961945 7f0b36629700 -1 log_channel(cluster) log [ERR] : > 9.aa soid 9:55bf7cc6:::rbd_data.33992ae8944a.000000000000200f:e: > failed to pick suitable object info > > 2017-08-21 12:41:15.357484 7f0b36629700 -1 log_channel(cluster) log [ERR] : > 9.aa deep-scrub 3 errors > > > > OSD 50: > > 2017-08-21 12:41:03.592918 7f264be6d700 -1 > bluestore(/var/lib/ceph/osd/ceph-50) _verify_csum bad crc32c/0x1000 > checksum at blob offset 0x0, got 0x64a1e2b1, expected 0x6706be76, > device location [0x3418830000~1000], logical extent 0x0~1000, object > #9:55bf7cc6:::rbd_data.33992ae8944a.000000000000200f:e# > > > > OSD 46: > > 2017-08-21 12:41:03.531394 7fb396b1f700 -1 > bluestore(/var/lib/ceph/osd/ceph-46) _verify_csum bad crc32c/0x1000 > checksum at blob offset 0x0, got 0x7aa05c01, expected 0x6706be76, > device location [0x1d6e1e0000~1000], logical extent 0x0~1000, object > #9:55bf7cc6:::rbd_data.33992ae8944a.000000000000200f:e# > > > > This is on Ceph 12.1.4 (previously 12.1.1). > > > > Thoughts? > > > > ----- > > Edward Huyer > > School of Interactive Games and Media > > Rochester Institute of Technology > > Golisano 70-2373 > > 152 Lomb Memorial Drive > > Rochester, NY 14623 > > 585-475-6651 > > erhvks@xxxxxxx > > > > Obligatory Legalese: > > The information transmitted, including attachments, is intended only > for the > person(s) or entity to which it is addressed and may contain > confidential and/or privileged material. Any review, retransmission, > dissemination or other use of, or taking of any action in reliance > upon this information by persons or entities other than the intended > recipient is prohibited. If you received this in error, please contact > the sender and destroy any copies of this information. > > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Cheers, Brad _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |