Hi list, We have a lot of failed instances when reboot host after upgrade cpeh client to pacific. On 160 instances failed represent 15-20% We have been impacted by this treads too when upgrade cluster: https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/C7FIBFYP32KSWJXL2XSJJKXNMASKMAQ4/ I'm working with beard lionel and for now we have stopped to do bluestore_fsck_quick_fix_on_mount on OSD with osd.95 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics message. Because the process failed 50% of time and we can't start ever OSD Exemple all keys are prefixed by IO. with other rbd volume header keys are prefixed by 'X.' root@ceph-mon4:/usr/local/sbin# rados -p openstack2_vms listomapvals rbd_header.db979141b71efb key (25 bytes): 00000000 00 00 00 00 00 0a 49 4f 2e 63 72 65 61 74 65 5f |......IO.create_| 00000010 74 69 6d 65 73 74 61 6d 70 |timestamp| 00000019 value (8 bytes) : 00000000 5c 22 87 5e 2d de 61 2c |\".^-.a,| 00000008 key (17 bytes): 00000000 00 00 00 00 00 0a 49 4f 2e 66 65 61 74 75 72 65 |......IO.feature| 00000010 73 |s| 00000011 To repair our VM I writed a little script it delete rbd_header obekct and re-create one with correct keys name and content from backuped header. Any advise are welcome thanks Ronan Lanore ________________________________ Ce message et toutes les pièces jointes (ci-après le "message") sont établis à l'intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur ou s'il ne vous est pas destiné, merci de le détruire ainsi que toute copie de votre système et d'en avertir immédiatement l'expéditeur. Toute lecture non autorisée, toute utilisation de ce message qui n'est pas conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer l'intégrité de ce message électronique susceptible d'altération, l'expéditeur (et ses filiales) décline(nt) toute responsabilité au titre de ce message dans l'hypothèse où il aurait été modifié ou falsifié. This message and any attachments (the "message") is intended solely for the intended recipient(s) and is confidential. If you receive this message in error, or are not the intended recipient(s), please delete it and any copies from your systems and immediately notify the sender. Any unauthorized view, use that does not comply with its purpose, dissemination or disclosure, either whole or partial, is prohibited. Since the internet cannot guarantee the integrity of this message which may not be reliable, the sender (and its subsidiaries) shall not be liable for the message if modified or falsified. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |