In the middle of a normal cephadm upgrade from 16.2.5 to 16.2.6, after the mgrs had successfully upgraded, 2/5 mons didn’t come back up (and the upgrade stopped at that point). Attempting to manually restart the crashed mons resulted in **all** of the other mons crashing too, usually with: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::malformed_input' what(): void FSMap::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version v < 7: Malformed input After some messing around with the monmaps to try and get the few working mons back in a quorum, we’re now in a state where one mon can run fine (but not reach a quorum, obviously), but as soon as a second comes up it crashes instantly. I also can’t start any mon with a monmap containing only one mon – same output as above. The rest of the cluster is working as expected (with the obvious exception of new connections failing). Anyone seen this or have ideas? Happy to provide more info from the cluster, just wasn’t sure what would actually be helpful… -- _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |