Hi Patrick, Yes we had been successfully running on Pacific v16.2.5 Thanks for the pointer to the bug, we eventually ended up taking eveything down and rebuilding the monstore using monstore-tool. Perhaps a longer and less pleasant path than necessary but it was effective. -Jon On Thu, Oct 07, 2021 at 09:11:21PM -0400, Patrick Donnelly wrote: :Hello Jonathan, : :On Tue, Oct 5, 2021 at 9:13 AM Jonathan D. Proulx <jon@xxxxxxxxxxxxx> wrote: :> :> In the middle of a normal cephadm upgrade from 16.2.5 to 16.2.6, after the mgrs had successfully upgraded, 2/5 mons didn’t come back up (and the upgrade stopped at that point). Attempting to manually restart the crashed mons resulted in **all** of the other mons crashing too, usually with: :> :> terminate called after throwing an instance of 'ceph::buffer::v15_2_0::malformed_input' what(): void FSMap::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version v < 7: Malformed input : :You upgraded from v16.2.5 and not Octopus? I would expect your cluster :to crash when upgrading to any version of Pacific: : :https://tracker.ceph.com/issues/51673 : :Only the crash error has changed from an assertion to an exception. : :-- :Patrick Donnelly, Ph.D. :He / Him / His :Principal Software Engineer :Red Hat Sunnyvale, CA :GPG: 19F28A586F808C2402351B93C3301A3E258DD79D : -- _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |