On Thu, Aug 19, 2021 at 2:52 PM Ignacio García <igarcia@xxxxxxxxxxxxxxxxx> wrote: > > Hi, > > In a production ceph system (with pending tasks as you'll see) we have > had a disaster: the server's boot disk where the only monitor was > running has failed, containing as well the monitor daemon data. We will > appreciate any help you can offer before we break anything that could be > recoverable trying non expert solutions. > > Following are the details: > > > * system overview: > > - 2 commodity servers, 4 HD each, 6 HDs for ceph osds > - replica size 2; 1 only monitor > - server 1: 1 mon, 1 mgr, 1 mds, 3 osds > - server 2: 1 mgr, 1 mds, 3 osds > - ceph octopus 15.2.11 containerized docker daemons; cephadm deployed > - used for libvirt VMs rbd images, and 1 cephfs > > > * hard disk structure details: > > - server 1: running 1 mon, 1 mgr, 1 mds, 3 osds > > /dev/sda 2TB --> server 1 boot disk, root, and ceph daemons data > (/var/lib/ceph, etc) --> FAILED > /dev/sdc 8TB --> Osd.2 > /dev/sdb 8TB --> Osd.1 > /dev/sdd 2TB --> Osd.0 > > - server 2: running 1 mgr, 1 mds, 3 osds > > /dev/sda 240GB (SSD) --> server 2 boot disk, root, and ceph > daemons data (/var/lib/ceph, etc) > /dev/sdb 8T --> Osd.3 > /dev/sdc 8T --> Osd.4 > /dev/sdd 2T --> Osd.5 > > > * the problems: > > --> server 1 /dev/sda HD failed, then server 1 is down: no monitors, > server 2 osds unable to start, ceph down > --> client.admin keyring lost > > > Is there any solution to recover the system?? Thank you very much in > advance. https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#recovery-using-osds Note the caveats about needing to re-create the filesystem metadata separately. I think that's described somewhere in the mailing list archive, but I've not done it myself. -Greg > > Miguel Garcia > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |