Thanks! Upgrading through Octopus first seems to work, at least in my test scenario. I tested it by adding a new Nautilus monitor, then stopping it (and removing it from the cluster). I used monmaptool to remove the real production monitors so it would run as a single isolated node. I verified I could not directly update to Pacific but I could step through an Octopus -> Pacific process just fine. Does that seem like a valid test? I tried updating a production monitor to Octopus first this weekend and couldn’t update to Pacific – did it fail because I didn’t update all of the monitors to Octopus before trying Pacific? I’m concerned about the other issues you’re seeing though. Are you having to restart your monitors constantly to control the data size, or was that a one-time thing? Is it common for scrubs to stop working or is that a rare problem? The ticket doesn’t seem to be getting much attention, I’m not sure how to interpret that. -- Sam Clippinger From: André Cruz <acruz@xxxxxxxxxxxxxx> Sent: Monday, March 21, 2022 8:20 AM To: Clippinger, Sam <Sam.Clippinger@xxxxxxxxxx> Cc: ceph-users@xxxxxxx Subject: Re: HELP! Upgrading monitors from 14.2.22 to 16.2.7 immediately crashes in FSMap::decode() Hey Sam. I recently upgraded a cluster from Nautilus to Pacific, coincidentally the cluster was setup with Hammer as well back in 2015. I had to first upgrade monitors to Octopus and then Pacific, even though we’re supposed to be able to skip one major version (https://github.com/ceph/ceph/pull/42349#issuecomment-1022599322<https://urldefense.com/v3/__https:/github.com/ceph/ceph/pull/42349*issuecomment-1022599322__;Iw!!EJc4YC3iFmQ!GpmETghM-E5XayK1UMAonWb27kN0xdDb_AGGgXIdSHRW7QFWIxa9rrvqwoPhPF7Tq3E$>). After this extra step I had to disable the crush location hook. I used a script and, again, it crashed the monitor. It worked when I specified the crush location directly in the config. Then the monitors were able to start correctly. However, after all the components were upgraded, I ran into some other issues. Namely: - monitor data size kept increasing. This was solved by restarting all the monitors again. - scrubs don’t work: seems related to https://tracker.ceph.com/issues/54172<https://urldefense.com/v3/__https:/tracker.ceph.com/issues/54172__;!!EJc4YC3iFmQ!GpmETghM-E5XayK1UMAonWb27kN0xdDb_AGGgXIdSHRW7QFWIxa9rrvqwoPhqLjV0KY$> and I also see a lot of messages on the OSD logs such as "handle_scrub_reserve_grant: received unsolicited reservation grant””. This is still not solved. I would strongly advise you to stay in Octopus for now. Best regards, André On 21 Mar 2022, at 11:26, Clippinger, Sam <Sam.Clippinger@xxxxxxxxxx<mailto:Sam.Clippinger@xxxxxxxxxx>> wrote: I get this output from "ceph mon dump", it shows all monitors are Nautilus and msgrv2 is in use: # ceph mon dump epoch 25 fsid a7fcde57-88df-4f14-a290-d170f0bedb25 last_changed 2022-03-19 20:44:22.775653 created 2015-10-19 15:28:40.133957 min_mon_release 14 (nautilus) 0: [v2:10.5.131.202:3300/0,v1:10.5.131.202:6789/0] mon.olaxps-cephmon20 1: [v2:10.5.131.203:3300/0,v1:10.5.131.203:6789/0] mon.olaxps-cephmon21 2: [v2:10.5.131.204:3300/0,v1:10.5.131.204:6789/0] mon.olaxps-cephmon22 dumped monmap epoch 25 This cluster was originally installed in 2015 with the Hammer release. It's been upgraded a number of times since then. -- Sam Clippinger -----Original Message----- From: Tyler Stachecki <stachecki.tyler@xxxxxxxxx<mailto:stachecki.tyler@xxxxxxxxx>> Sent: Sunday, March 20, 2022 9:40 PM To: Clippinger, Sam <Sam.Clippinger@xxxxxxxxxx<mailto:Sam.Clippinger@xxxxxxxxxx>> Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> Subject: Re: HELP! Upgrading monitors from 14.2.22 to 16.2.7 immediately crashes in FSMap::decode() CAUTION - EXTERNAL EMAIL: Do not click any links or open any attachments unless you trust the sender and know the content is safe. What does 'ceph mon dump | grep min_mon_release' say? You're running msgrv2 and all Ceph daemons are talking on v2, since you're on Nautilus, right? Was the cluster conceived on Nautilus, or something earlier? Tyler On Sun, Mar 20, 2022 at 10:30 PM Clippinger, Sam <Sam.Clippinger@xxxxxxxxxx<mailto:Sam.Clippinger@xxxxxxxxxx>> wrote: Hello! I need some help. I'm trying to upgrade from Ceph Nautilus 14.2.22 cluster to Pacific (manually, not using cephadm). I've only tried upgrading one monitor so far and I've hit several snags. I've tried to troubleshooting the issue without losing the cluster (of course it's a production cluster, the test cluster upgraded just fine). This cluster has 3 monitor/manager VMs with 4 CPUs and 16 GB RAM, running CentOS 7. It has 5 storage servers with 48 CPUs and 196 GB RAM, running Rocky Linux 8. All of the Ceph daemons run in Docker containers built from Rocky Linux 8, the Ceph binaries are installed from the RPMs on download.ceph.com<https://urldefense.com/v3/__http:/download.ceph.com__;!!EJc4YC3iFmQ!GpmETghM-E5XayK1UMAonWb27kN0xdDb_AGGgXIdSHRW7QFWIxa9rrvqwoPhyT3MndE$>. This cluster was originally installed with Hammer (IIRC) and upgraded through a number of versions (messenger v2 is enabled). This cluster is only used for OpenStack RBD volumes, not CephFS or S3. Upgrading a monitor to Octopus 15.2.16 works fine, it starts up and rejoins the quorum. When I upgrade to Pacific 16.2.5 or 16.2.7, it immediately crashes. Upgrading to Pacific directly from Nautilus does the same thing. Adding "mon_mds_skip_sanity = true" to ceph.conf doesn't change anything. I've tried compacting and rebuilding the monitor store, it doesn't help. I can add new Nautilus 14.2.22 monitors to the cluster, they start and join in a few seconds but updating them also crashes immediately. I can post the entire crash output if it would help, but I think these are the relevant lines from 16.2.5: ---------------------------------------------------------------------- ---------- 2022-03-19T14:05:36.549-0500 7ffb78025700 0 starting mon.olaxps-ceph90 rank 3 at public addrs [v2:10.5.240.81:3300/0,v1:10.5.240.81:6789/0] at bind addrs [v2:10.5.240.81:3300/0,v1:10.5.240.81:6789/0] mon_data /var/lib/ceph/mon/ceph-olaxps-ceph90 fsid a7fcde57-88df-4f14-a290-d170f0bedb25 2022-03-19T14:05:36.550-0500 7ffb78025700 1 mon.olaxps-ceph90@-1(???)<mailto:mon.olaxps-ceph90@-1(???)> e24 preinit fsid a7fcde57-88df-4f14-a290-d170f0bedb25 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_A RCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/r elease/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/mds/FSMap.cc<https://urldefense.com/v3/__http:/FSMap.cc__;!!EJc4YC3iFmQ!GpmETghM-E5XayK1UMAonWb27kN0xdDb_AGGgXIdSHRW7QFWIxa9rrvqwoPhP7CU2EY$>: In function 'void FSMap::decode(ceph::buffer::v15_2_0::list::const_iterator&)' thread 7ffb78025700 time 2022-03-19T14:05:36.552097-0500 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_A RCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/r elease/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/mds/FSMap.cc<https://urldefense.com/v3/__http:/FSMap.cc__;!!EJc4YC3iFmQ!GpmETghM-E5XayK1UMAonWb27kN0xdDb_AGGgXIdSHRW7QFWIxa9rrvqwoPhP7CU2EY$>: 648: ceph_abort_msg("abort() called") ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7ffb6f1b3264] 2: (FSMap::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0xc7 3) [0x7ffb6f6fa003] 3: (MDSMonitor::update_from_paxos(bool*)+0x18a) [0x563c5606697a] 4: (PaxosService::refresh(bool*)+0x10e) [0x563c55f87c7e] 5: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x563c55e39eac] 6: (Monitor::init_paxos()+0x10c) [0x563c55e3a1bc] 7: (Monitor::preinit()+0xd30) [0x563c55e67660] 8: main() 9: __libc_start_main() 10: _start() 2022-03-19T14:05:36.551-0500 7ffb78025700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_A RCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/r elease/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/mds/FSMap.cc<https://urldefense.com/v3/__http:/FSMap.cc__;!!EJc4YC3iFmQ!GpmETghM-E5XayK1UMAonWb27kN0xdDb_AGGgXIdSHRW7QFWIxa9rrvqwoPhP7CU2EY$>: In function 'void FSMap::decode(ceph::buffer::v15_2_0::list::const_iterator&)' thread 7ffb78025700 time 2022-03-19T14:05:36.552097-0500 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_A RCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/r elease/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/mds/FSMap.cc<https://urldefense.com/v3/__http:/FSMap.cc__;!!EJc4YC3iFmQ!GpmETghM-E5XayK1UMAonWb27kN0xdDb_AGGgXIdSHRW7QFWIxa9rrvqwoPhP7CU2EY$>: 648: ceph_abort_msg("abort() called") ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7ffb6f1b3264] 2: (FSMap::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0xc7 3) [0x7ffb6f6fa003] 3: (MDSMonitor::update_from_paxos(bool*)+0x18a) [0x563c5606697a] 4: (PaxosService::refresh(bool*)+0x10e) [0x563c55f87c7e] 5: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x563c55e39eac] 6: (Monitor::init_paxos()+0x10c) [0x563c55e3a1bc] 7: (Monitor::preinit()+0xd30) [0x563c55e67660] 8: main() 9: __libc_start_main() 10: _start() *** Caught signal (Aborted) ** in thread 7ffb78025700 thread_name:ceph-mon ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 1: /lib64/libpthread.so.0(+0x12c20) [0x7ffb6cca9c20] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x7ffb6f1b3335] 5: (FSMap::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0xc7 3) [0x7ffb6f6fa003] 6: (MDSMonitor::update_from_paxos(bool*)+0x18a) [0x563c5606697a] 7: (PaxosService::refresh(bool*)+0x10e) [0x563c55f87c7e] 8: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x563c55e39eac] 9: (Monitor::init_paxos()+0x10c) [0x563c55e3a1bc] 10: (Monitor::preinit()+0xd30) [0x563c55e67660] 11: main() 12: __libc_start_main() 13: _start() 2022-03-19T14:05:36.553-0500 7ffb78025700 -1 *** Caught signal (Aborted) ** in thread 7ffb78025700 thread_name:ceph-mon ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 1: /lib64/libpthread.so.0(+0x12c20) [0x7ffb6cca9c20] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x7ffb6f1b3335] 5: (FSMap::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0xc7 3) [0x7ffb6f6fa003] 6: (MDSMonitor::update_from_paxos(bool*)+0x18a) [0x563c5606697a] 7: (PaxosService::refresh(bool*)+0x10e) [0x563c55f87c7e] 8: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x563c55e39eac] 9: (Monitor::init_paxos()+0x10c) [0x563c55e3a1bc] 10: (Monitor::preinit()+0xd30) [0x563c55e67660] 11: main() 12: __libc_start_main() 13: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. ---------------------------------------------------------------------- ---------- And from 16.2.7: ---------------------------------------------------------------------- ---------- 2022-03-19T14:09:48.739-0500 7ff5f1209700 0 starting mon.olaxps-ceph90 rank 3 at public addrs [v2:10.5.240.81:3300/0,v1:10.5.240.81:6789/0] at bind addrs [v2:10.5.240.81:3300/0,v1:10.5.240.81:6789/0] mon_data /var/lib/ceph/mon/ceph-olaxps-ceph90 fsid a7fcde57-88df-4f14-a290-d170f0bedb25 2022-03-19T14:09:48.741-0500 7ff5f1209700 1 mon.olaxps-ceph90@-1(???)<mailto:mon.olaxps-ceph90@-1(???)> e24 preinit fsid a7fcde57-88df-4f14-a290-d170f0bedb25 2022-03-19T14:09:48.741-0500 7ff5f1209700 -1 mon.olaxps-ceph90@-1(???).mds<mailto:mon.olaxps-ceph90@-1(???).mds> e0 unable to decode FSMap: void FSMap::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version v < 7: Malformed input terminate called after throwing an instance of 'ceph::buffer::v15_2_0::malformed_input' what(): void FSMap::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version v < 7: Malformed input *** Caught signal (Aborted) ** in thread 7ff5f1209700 thread_name:ceph-mon ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) 1: /lib64/libpthread.so.0(+0x12c20) [0x7ff5e60c1c20] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7ff5e56d809b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7ff5e56de53c] 6: /lib64/libstdc++.so.6(+0x96597) [0x7ff5e56de597] 7: __cxa_rethrow() 8: /usr/bin/ceph-mon(+0x23256a) [0x55fa726a356a] 9: (PaxosService::refresh(bool*)+0x10e) [0x55fa7286e29e] 10: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fa7271f2dc] 11: (Monitor::init_paxos()+0x10c) [0x55fa7271f5ec] 12: (Monitor::preinit()+0xd30) [0x55fa7274caa0] 13: main() 14: __libc_start_main() 15: _start() 2022-03-19T14:09:48.742-0500 7ff5f1209700 -1 *** Caught signal (Aborted) ** in thread 7ff5f1209700 thread_name:ceph-mon ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) 1: /lib64/libpthread.so.0(+0x12c20) [0x7ff5e60c1c20] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7ff5e56d809b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7ff5e56de53c] 6: /lib64/libstdc++.so.6(+0x96597) [0x7ff5e56de597] 7: __cxa_rethrow() 8: /usr/bin/ceph-mon(+0x23256a) [0x55fa726a356a] 9: (PaxosService::refresh(bool*)+0x10e) [0x55fa7286e29e] 10: (Monitor::refresh_from_paxos(bool*)+0x18c) [0x55fa7271f2dc] 11: (Monitor::init_paxos()+0x10c) [0x55fa7271f5ec] 12: (Monitor::preinit()+0xd30) [0x55fa7274caa0] 13: main() 14: __libc_start_main() 15: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. ---------------------------------------------------------------------- ---------- Both versions seem to crash in FSMap::decode(), though the message from 16.2.7 is a little more verbose. The stack trace looks different from https://urldefense.com/v3/__https://tracker.ceph.com/issues/52820__;!!EJc4YC3iFmQ!DqTiMk9mw0j_5m0wykqyZeUf9XAmPgMfLFiYAdufg-l7802jCe3CqvVAy-m8tmiEZMw$<https://urldefense.com/v3/__https:/tracker.ceph.com/issues/52820__;!!EJc4YC3iFmQ!DqTiMk9mw0j_5m0wykqyZeUf9XAmPgMfLFiYAdufg-l7802jCe3CqvVAy-m8tmiEZMw$> , though the "malformed input" message is the same. I found the recent reports of the sanity checking bug in 16.2.7 (https://urldefense.com/v3/__https://tracker.ceph.com/issues/54161__;!!EJc4YC3iFmQ!DqTiMk9mw0j_5m0wykqyZeUf9XAmPgMfLFiYAdufg-l7802jCe3CqvVAy-m8TTsrbKU$<https://urldefense.com/v3/__https:/tracker.ceph.com/issues/54161__;!!EJc4YC3iFmQ!DqTiMk9mw0j_5m0wykqyZeUf9XAmPgMfLFiYAdufg-l7802jCe3CqvVAy-m8TTsrbKU$> and https://urldefense.com/v3/__https://github.com/ceph/ceph/pull/44910__;!!EJc4YC3iFmQ!DqTiMk9mw0j_5m0wykqyZeUf9XAmPgMfLFiYAdufg-l7802jCe3CqvVAy-m8HXt5Big$<https://urldefense.com/v3/__https:/github.com/ceph/ceph/pull/44910__;!!EJc4YC3iFmQ!DqTiMk9mw0j_5m0wykqyZeUf9XAmPgMfLFiYAdufg-l7802jCe3CqvVAy-m8HXt5Big$> ) but this looks like a different problem. Just to be sure, I recompiled 16.2.7 from the SRPM with the patches from that PR applied. They didn't help, it still crashes with the same error. This may be unrelated, but I've also tried adding a new monitor to the cluster running Octopus or Pacific -- I figured replacing the existing monitors would be just as good as upgrading. I have tried Octopus 15.2.16, Pacific 16.2.5 and Pacific 16.2.7 without success. Each version produces the same behavior: the existing monitors start using between 80%-350% CPU (they run on 4 CPU VMs) and their memory usage climbs out of control until they crash (their containers are limited to 12 GB RAM, they normally use less than 1 GB). While this is happening, the cluster basically freezes -- clients cannot connect, "ceph status" times out, etc. The logs from the existing monitors are filled with tens of millions of lines like these: ---------------------------------------------------------------------- ---------- 2022-03-19 16:05:19.854 7f426c214700 1 mon.olaxps-cephmon22@2(peon)<mailto:mon.olaxps-cephmon22@2(peon)> e17 adding peer [v2:10.5.240.81:3300/0,v1:10.5.240.81:6789/0] to list of hints 2022-03-19 16:05:19.854 7f426c214700 1 mon.olaxps-cephmon22@2(peon)<mailto:mon.olaxps-cephmon22@2(peon)> e17 adding peer [v2:10.5.240.81:3300/0,v1:10.5.240.81:6789/0] to list of hints 2022-03-19 16:05:19.854 7f426c214700 1 mon.olaxps-cephmon22@2(peon)<mailto:mon.olaxps-cephmon22@2(peon)> e17 adding peer [v2:10.5.240.81:3300/0,v1:10.5.240.81:6789/0] to list of hints 2022-03-19 16:05:19.854 7f426c214700 1 mon.olaxps-cephmon22@2(peon)<mailto:mon.olaxps-cephmon22@2(peon)> e17 adding peer [v2:10.5.240.81:3300/0,v1:10.5.240.81:6789/0] to list of hints 2022-03-19 16:05:19.854 7f426c214700 1 mon.olaxps-cephmon22@2(peon)<mailto:mon.olaxps-cephmon22@2(peon)> e17 adding peer [v2:10.5.240.81:3300/0,v1:10.5.240.81:6789/0] to list of hints ---------------------------------------------------------------------- ---------- The new monitor also uses high CPU and memory but doesn't spam its logs. It never joins the cluster and doesn't write much to disk, even after waiting almost an hour. After reading https://urldefense.com/v3/__https://www.mail-archive.com/ceph-users@xxxxxxx/msg12031.html__;!!EJc4YC3iFmQ!DqTiMk9mw0j_5m0wykqyZeUf9XAmPgMfLFiYAdufg-l7802jCe3CqvVAy-m8MQ9ESLs$<https://urldefense.com/v3/__https:/www.mail-archive.com/ceph-users@xxxxxxx/msg12031.html__;!!EJc4YC3iFmQ!DqTiMk9mw0j_5m0wykqyZeUf9XAmPgMfLFiYAdufg-l7802jCe3CqvVAy-m8MQ9ESLs$> , I added the option "mon_sync_max_payload_size = 4096" to ceph.conf on all monitors (and restarted), it didn't help. Killing the new monitor unfreezes the cluster and returns the existing monitors to their typical CPU usage. They don't release their excess memory without being restarted. I was able to update a similar (but newer) test cluster to Pacific, so this smells like something specific to the data in this cluster. What else can I do to troubleshoot? I can provide more output and config files if those would help; I didn't want to post a bunch of huge files if they aren't relevant. Any suggestions? -- Sam Clippinger ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are for the sole use of the intended recipient(s) and contain information that may be Garmin confidential and/or Garmin legally privileged. If you have received this email in error, please notify the sender by reply email and delete the message. Any disclosure, copying, distribution or use of this communication (including attachments) by someone other than the intended recipient is prohibited. Thank you. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |