On Sun, Feb 10, 2019 at 1:56 AM Ruben Rodriguez <ruben@xxxxxxx> wrote: > > Hi there, > > Running 12.2.11-1xenial on a machine with 6 SSD OSD with bluestore. > > Today we had two disks fail out of the controller, and after a reboot > they both seemed to come back fine but ceph-osd was only able to start > in one of them. The other one gets this: > > 2019-02-08 18:53:00.703376 7f64f948ce00 -1 > bluestore(/var/lib/ceph/osd/ceph-3) _verify_csum bad crc32c/0x1000 > checksum at blob offset 0x0, got 0x95104dfc, expected 0xb9e3e26d, device > location [0x4000~1000], logical extent 0x0~1000, object > #-1:7b3f43c4:::osd_superblock:0# > 2019-02-08 18:53:00.703406 7f64f948ce00 -1 osd.3 0 OSD::init() : unable > to read osd superblock > > Note that there are no actual IO errors being shown by the controller in > dmesg, and that the disk is readable. The metadata FS is mounted and > looks normal. > > I tried running "ceph-bluestore-tool repair --path > /var/lib/ceph/osd/ceph-3 --deep 1" and that gets many instances of: Running this with debug_bluestore=30 might give more information on the nature of the IO error. > > 2019-02-08 19:00:31.783815 7fa35bd0df80 -1 > bluestore(/var/lib/ceph/osd/ceph-3) _verify_csum bad crc32c/0x1000 > checksum at blob offset 0x0, got 0x95104dfc, expected 0xb9e3e26d, device > location [0x4000~1000], logical extent 0x0~1000, object > #-1:7b3f43c4:::osd_superblock:0# > 2019-02-08 19:00:31.783866 7fa35bd0df80 -1 > bluestore(/var/lib/ceph/osd/ceph-3) fsck error: > #-1:7b3f43c4:::osd_superblock:0# error during read: (5) Input/output error > > ...which is the same error. Due to a host being down for unrelated > reasons, this is preventing some PG's from activating, keeping one pool > inaccessible. There is no critical data in it, but I'm more interested > in solving the issue for reliability. > > Any advice? What does bad crc indicate in this context? Should I send > this to the bug tracker instead? > -- > Ruben Rodriguez | Chief Technology Officer, Free Software Foundation > GPG Key: 05EF 1D2F FE61 747D 1FC8 27C3 7FAC 7D26 472F 4409 > https://fsf.org | https://gnu.org > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Cheers, Brad _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |