Off thread sent my mgr database / core dumps to ceph devs. For documentation purposes / other peoples help, I could mitigate the issue by destroying my journal, here are the I used commands: # Create an backup copy of my old mgr pool ceph osd pool create mgr-backup-2022-08-19 rados cppool .mgr mgr-backup-2022-08-19 ceph osd pool application enable mgr-backup-2022-08-19 mgr ceph osd pool application enable mgr-backup-2022-08-19 mgr_devicehealth --yes-i-really-mean-it # Delete the journal objects rados -p .mgr ls --all rados --pool=.mgr --namespace devicehealth rm main.db-journal.0000000000000000 rados --pool=.mgr --namespace devicehealth rm main.db-journal.0000000000000001 # Managers now start successfully. On Wed, Aug 17, 2022 at 3:30 AM Patrick Donnelly <pdonnell@xxxxxxxxxx> wrote: > Thank you, that's helpful. I have created a ticket with my findings so far: > > https://tracker.ceph.com/issues/57152 > > Please follow there for updates. > > On Mon, Aug 15, 2022 at 4:12 PM Daniel Williams <danielwoz@xxxxxxxxx> > wrote: > > > > ceph-post-file: a9802e30-0096-410e-b5c0-f2e6d83acfd6 > > > > On Tue, Aug 16, 2022 at 3:13 AM Patrick Donnelly <pdonnell@xxxxxxxxxx> > wrote: > >> > >> On Mon, Aug 15, 2022 at 11:39 AM Daniel Williams <danielwoz@xxxxxxxxx> > wrote: > >> > > >> > Using ubuntu with apt repository from ceph. > >> > > >> > Ok that helped me figure out that it's .mgr not mgr. > >> > # ceph -v > >> > ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy > (stable) > >> > # export CEPH_CONF='/etc/ceph/ceph.conf' > >> > # export CEPH_KEYRING='/etc/ceph/ceph.client.admin.keyring' > >> > # export CEPH_ARGS='--log_to_file true --log-file ceph-sqlite.log > --debug_cephsqlite 20 --debug_ms 1' > >> > # sqlite3 > >> > SQLite version 3.31.1 2020-01-27 19:55:54 > >> > Enter ".help" for usage hints. > >> > sqlite> .load libcephsqlite.so > >> > sqlite> .open file:///.mgr:devicehealth/main.db?vfs=ceph > >> > sqlite> .tables > >> > Segmentation fault (core dumped) > >> > > >> > # dpkg -l | grep ceph | grep sqlite > >> > ii libsqlite3-mod-ceph 17.2.3-1focal > amd64 SQLite3 VFS for Ceph > >> > > >> > Attached ceph-sqlite.log > >> > >> No real good hint in the log unfortunately. I will need the core dump > >> to see where things went wrong. Can you upload it with > >> > >> https://docs.ceph.com/en/quincy/man/8/ceph-post-file/ > >> > >> ? > >> > >> -- > >> Patrick Donnelly, Ph.D. > >> He / Him / His > >> Principal Software Engineer > >> Red Hat, Inc. > >> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > >> > > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Principal Software Engineer > Red Hat, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |