cannot open /dev/xvdb: Input/output error

[Massimiliano Cuttini <max@xxxxxxxxxxxxx>]

After 4 months of test we decided to go live and store real VDI in production.
However just the same day something went suddenly wrong.

The last copy of the VDI in Ceph was corrupted.
Trying to fix the filesystem works for open it, but mysqld never went online even after reinstalling but only forcing fixing of innoDB.
To many database corrupted. Finally the last reboot VDI became inaccessible and do not start.

At the sametime other 3 VDI stuck and becomes inaccessible, try to access to them get this error:
cannot open /dev/xvdb: Input/output error

If I run from the Dom(0) the ceph -s it seems allright:

    cluster 33693979-3177-44f4-bcb8-1b3bbc658352
     health HEALTH_OK
     monmap e2: 2 mons at {ceph-node1=xxxxxx:6789/0,ceph-node2=yyyyyy:6789/0}
            election epoch 218, quorum 0,1 ceph-node1,ceph-node2
     osdmap e2560: 8 osds: 8 up, 8 in
            flags sortbitwise,require_jewel_osds
      pgmap v8311677: 484 pgs, 3 pools, 940 GB data, 444 kobjects
            1883 GB used, 5525 GB / 7408 GB avail
                 484 active+clean
  client io 482 kB/s wr, 0 op/s rd, 5 op/s wr

Everything seems ok.
I can see pool and rbd:

NAME                                                                                  SIZE PARENT FMT PROT LOCK
VHD-1649fde4-6637-43d8-b815-656e4080887d                                           102400M          2
VHD-1bcd9d72-d8fe-4fc2-ad6a-ce84b18e4340                                           102400M          2
VHD-239ea931-5ddd-4aaf-bc89-b192641f6dcf                                              200G          2
VHD-3e16395d-7dad-4680-a7ad-7f398da7fd9e                                              200G          2
VHD-41a76fe7-c9ff-4082-adb4-43f3120a9106                                           102400M          2
VHD-41a76fe7-c9ff-4082-adb4-43f3120a9106@SNAP-346d01f4-cbd4-4e8a-af8f-473c2de3c60d 102400M          2 yes
VHD-48fdb12d-110b-419c-9330-0f05827fe41e                                           102400M          2
VHD-602b05be-395d-442e-bd68-7742deaf97bd                                              200G          2
VHD-691e4fb4-5f7b-4bc1-af7b-98780d799067                                           102400M          2
VHD-6da2154e-06fd-4063-8af5-ae86ae61df50                                            10240M          2      excl
VHD-8631ab86-c85c-407b-9e15-bd86e830ba74                                              200G          2
VHD-97cbc3d2-519c-4d11-b795-7a71179e193c                                           102400M          2
VHD-97cbc3d2-519c-4d11-b795-7a71179e193c@SNAP-ad9ab028-57d9-48e4-9d13-a9a02b06e68a 102400M          2 yes
VHD-acb2a9b0-e98d-474e-aa42-ed4e5534ddbe                                           102400M          2      excl
VHD-adaa639e-a7a4-4723-9c18-0c2b3ab9d99f                                           102400M          2
VHD-bb0f40c7-206b-4023-b56f-4a70d02d5a58                                              200G          2
VHD-bb0f40c7-206b-4023-b56f-4a70d02d5a58@SNAP-f8b1cc40-4a9b-4550-8bfa-7fad467e726b    200G          2 yes
VHD-c8aca7bd-1e37-4af4-b642-f267602e210f                                           102400M          2
VHD-c8aca7bd-1e37-4af4-b642-f267602e210f@SNAP-0e830e68-c7b4-49a9-8210-4c0b4ecca762 102400M          2 yes
VHD-cf2139ac-b1c4-404d-87da-db8f992a3e72                                           102400M          2
VHD-cf2139ac-b1c4-404d-87da-db8f992a3e72@SNAP-4fb59ee5-3256-4e75-9d48-facdd818d754 102400M          2 yes

however they are unbootable and unreadable even as secondary driver.
Hypervisor refure to load them in the VMs.

What Can I do?

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com