Eugen, i had to restart all five monitors from scratch, so now i have a valid quorum 5/5 and all mons are v17.2.3. This was really strange. The main problem seemingly was: mon.ceph1n011@0(electing) e46 handle_auth_request failed to assign global_id and so forth... The one Mon with levelb i did reinstall with correct kv_backend Thanks for Helping Hand, Christoph Am Mi., 28. Sept. 2022 um 13:13 Uhr schrieb Ackermann, Christoph < c.ackermann@xxxxxxxxxxxx>: > Eugen, > > thank you very much. This (ceph1n021) is indeed the one using leveldb (in > kv_backend) > > Other mons have kv_backend "rocksdb" but unfortunately after reinstalling > ceph-mon@ceph1n021 we get no ceph status anymore and our mon logs get > filed with: > > 2022-09-28T13:10:54.822+0200 7fbc6a863700 0 log_channel(cluster) log > [INF] : mon.ceph1n011 calling monitor election > 2022-09-28T13:10:54.824+0200 7fbc6a863700 1 > paxos.0).electionLogic(789785) init, last seen epoch 789785, mid-election, > bumping > 2022-09-28T13:10:54.827+0200 7fbc6a863700 1 mon.ceph1n011@0(electing) > e46 collect_metadata sda: no unique device id for sda: fallback method has > model 'Virtual disk ' but no serial' > > > 2022-09-28T07:11:37.380-0400 7efc2d6ec700 1 mon.ceph1n020@2(electing) > e46 collect_metadata md125: no unique device id for md125: fallback method > has no model nor serial > 2022-09-28T07:11:37.384-0400 7efc2d6ec700 1 mon.ceph1n020@2(electing) > e46 collect_metadata md125: no unique device id for md125: fallback method > has no model nor serial > 2022-09-28T07:11:37.389-0400 7efc2d6ec700 1 mon.ceph1n020@2(electing) > e46 collect_metadata md125: no unique device id for md125: fallback method > has no model nor serial > 2022-09-28T07:11:37.393-0400 7efc2d6ec700 1 mon.ceph1n020@2(electing) > e46 collect_metadata md125: no unique device id for md125: fallback method > has no model nor serial > [root@ceph1n020 ~]# > > It seems there is a problem with the /var device or some ... > > Now we have an urgent state of our produktion cluster. :-( > > Do you have some hints for us? > > Best regards, > Christoph > > > > > Am Mi., 28. Sept. 2022 um 12:21 Uhr schrieb Eugen Block <eblock@xxxxxx>: > >> Hi, >> >> there was a thread about deprecating leveldb [1], but I didn't get the >> impression that it already has been deprecated. But the thread >> mentions that it's not tested anymore, so that might explain it. To >> confirm that you use leveldb you can run: >> >> cat /var/lib/ceph/mon/ceph-<MON>/kv_backend >> >> So you already have successfully upgraded other MONs, what kv_backend >> do they use? If this is the last one with leveldb you can probably >> move the old store content and recreate an empty MON. >> >> [1] >> >> https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/K4OSAA4AJS2V7FQI6GNCKCK3IRQDBQRS/ >> >> Zitat von "Ackermann, Christoph" <c.ackermann@xxxxxxxxxxxx>: >> >> > Hello List, >> > >> > i'm on the way to upgrade our "non cephadm" from Octopus to Quiny. It >> > fails/stuck on third ceph-mon ceph1n021 with an strange error: >> > >> > 2022-09-28T11:04:27.691+0200 7f8681543880 -1 _open error initializing >> > leveldb db back storage in /var/lib/ceph/mon/ceph-ceph1n021/store.db >> > >> > This monitor contains a lot of ldb files, wondering if we do not use >> > leveldb anymore.. >> > >> > [root@ceph1n021 ~]# ls /var/lib/ceph/mon/ceph-ceph1n021/store.db/ >> > 3251327.ldb 5720254.ldb 6568574.ldb 6652800.ldb 6726397.ldb >> > 6726468.ldb 6726623.ldb 6726631.ldb 6726638.ldb 6726646.ldb >> > 6726653.ldb IDENTITY >> > 3251520.ldb 6497196.ldb 6575398.ldb 6654280.ldb 6726398.ldb >> > 6726469.ldb 6726624.ldb 6726632.ldb 6726639.ldb 6726647.ldb >> > 6726654.ldb LOCK >> > 3251566.ldb 6517010.ldb 6595757.ldb 6680000.ldb 6726399.ldb >> > 6726588.ldb 6726627.ldb 6726634.ldb 6726642.ldb 6726648.ldb >> > 6726655.ldb MANIFEST-5682438 >> > 3251572.ldb 6523701.ldb 6601653.ldb 6699521.ldb 6726400.ldb >> > 6726608.ldb 6726628.ldb 6726635.ldb 6726643.ldb 6726649.ldb >> > 6726656.ldb OPTIONS-000005 >> > 3251583.ldb 6543819.ldb 6624261.ldb 6706116.ldb 6726401.ldb >> > 6726618.log 6726629.ldb 6726636.ldb 6726644.ldb 6726650.ldb >> > 6726657.ldb >> > 3251584.ldb 6549696.ldb 6627961.ldb 6725307.ldb 6726467.ldb >> > 6726622.ldb 6726630.ldb 6726637.ldb 6726645.ldb 6726651.ldb >> CURRENT >> > >> > All other ceph-mon "store.db" folder consists only expected files like: >> > >> > [root@ceph1n020 ~]# ls -l /var/lib/ceph/mon/ceph-ceph1n020/store.db/ >> > total 153252 >> > -rw-------. 1 ceph ceph 11230512 Sep 28 05:13 1040392.log >> > -rw-------. 1 ceph ceph 67281589 Sep 28 05:11 1040394.sst >> > -rw-------. 1 ceph ceph 40121324 Sep 28 05:11 1040395.sst >> > -rw-------. 1 ceph ceph 16 Aug 19 06:29 CURRENT >> > -rw-------. 1 ceph ceph 37 Feb 21 2022 IDENTITY >> > -rw-r--r--. 1 ceph ceph 0 Feb 21 2022 LOCK >> > -rw-------. 1 ceph ceph 8465618 Sep 28 05:11 MANIFEST-898389 >> > -rw-------. 1 ceph ceph 4946 Aug 19 04:51 OPTIONS-898078 >> > -rw-------. 1 ceph ceph 4946 Aug 19 06:29 OPTIONS-898392 >> > >> > >> > "mon": { >> > "ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) >> > octopus (stable)": 3, >> > "ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) >> > quincy (stable)": (ceph-mon@ceph1n011 and ceph-mon@ceph1n012) >> > >> > Is it safe to go forward restarting the rest of these monitors >> (ceph1n019 >> > and ceph1n020) and what can we do to fix errors on ceph-mon@ceph1n021 >> ? >> > >> > Best regards, >> > Christoph >> > >> > >> > >> > Christoph Ackermann | System Engineer >> > INFOSERVE GmbH | Am Felsbrunnen 15 | D-66119 Saarbrücken >> > Fon +49 (0)681 88008-59 | Fax +49 (0)681 88008-33 | >> c.ackermann@xxxxxxxxxxxx >> > | www.infoserve.de >> > INFOSERVE Datenschutzhinweise: www.infoserve.de/datenschutz >> > Handelsregister: Amtsgericht Saarbrücken, HRB 11001 | Erfüllungsort: >> > Saarbrücken >> > Geschäftsführer: Dr. Stefan Leinenbach | Ust-IdNr.: DE168970599 >> > >> > <https://facebook.com/infoserve.de> >> > <https://www.xing.com/companies/infoservegmbh> >> > <https://www.youtube.com/channel/UCUj8C3TGGhQZPVvxu4woXmQ> >> > <https://www.linkedin.com/company-beta/10095540> >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@xxxxxxx >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |