On 2/19/20 8:49 AM, Sean Matheny wrote: > Thanks, > >> If the OSDs have a newer epoch of the OSDMap than the MON it won't work. > > How can I verify this? (i.e the epoch of the monitor vs the epoch of the > osd(s)) > Check the status of the OSDs: $ ceph daemon osd.X status This should tell the newest map it has. Then check on the mons: $ ceph osd dump|head -n 10 Or using ceph-monstore-tool to see what the latest map is the MON has. Wido > Cheers, > Sean > > >> On 19/02/2020, at 7:25 PM, Wido den Hollander <wido@xxxxxxxx >> <mailto:wido@xxxxxxxx>> wrote: >> >> >> >> On 2/19/20 5:45 AM, Sean Matheny wrote: >>> I wanted to add a specific question to the previous post, in the >>> hopes it’s easier to answer. >>> >>> We have a Luminous monitor restored from the OSDs using >>> ceph-object-tool, which seems like the best chance of any success. We >>> followed this rough process: >>> >>> https://tracker.ceph.com/issues/24419 >>> >>> The monitor has come up (as a single monitor cluster), but it’s >>> reporting wildly inaccurate info, such as the number of osds that are >>> down (157 but all 223 are down), and hosts (1, but all 14 are down). >>> >> >> Have you verified that the MON's database has the same epoch of the >> OSDMap (or newer) as all the other OSDs? >> >> If the OSDs have a newer epoch of the OSDMap than the MON it won't work. >> >>> The OSD Daemons are still off, but I’m not sure if starting them back >>> up with this monitor will make things worse. The fact that this mon >>> daemon can’t even see how many OSDs are correctly down makes me think >>> that nothing good will come from turning the OSDs back on. >>> >>> Do I run risk of further corruption (i.e. on the Ceph side, not >>> client data as the cluster is paused) if I proceed and turn on the >>> osd daemons? Or is it worth a shot? >>> >>> Are these Ceph health metrics commonly inaccurate until it can talk >>> to the daemons? >> >> The PG stats will be inaccurate indeed and the number of OSDs can vary >> as long as they aren't able to peer with each other and the MONs. >> >>> >>> (Also other commands like `ceph osd tree` agree with the below `ceph >>> -s` so far) >>> >>> Many thanks for any wisdom… I just don’t want to make things >>> (unnecessarily) much worse. >>> >>> Cheers, >>> Sean >>> >>> >>> root@ntr-mon01:/var/log/ceph# ceph -s >>> cluster: >>> id: ababdd7f-1040-431b-962c-c45bea5424aa >>> health: HEALTH_WARN >>> pauserd,pausewr,noout,norecover,noscrub,nodeep-scrub >>> flag(s) set >>> 157 osds down >>> 1 host (15 osds) down >>> Reduced data availability: 12225 pgs inactive, 885 pgs >>> down, 673 pgs peering >>> Degraded data redundancy: 14829054/35961087 objects >>> degraded (41.236%), 2869 pgs degraded, 2995 pgs undersized services: >>> mon: 1 daemons, quorum ntr-mon01 >>> mgr: ntr-mon01(active) >>> osd: 223 osds: 66 up, 223 in >>> flags pauserd,pausewr,noout,norecover,noscrub,nodeep-scrub data: >>> pools: 14 pools, 15220 pgs >>> objects: 10.58M objects, 40.1TiB >>> usage: 43.0TiB used, 121TiB / 164TiB avail >>> pgs: 70.085% pgs unknown >>> 10.237% pgs not active >>> 14829054/35961087 objects degraded (41.236%) >>> 10667 unknown >>> 2869 active+undersized+degraded >>> 885 down >>> 673 peering >>> 126 active+undersized >>> >>> >>> On 19/02/2020, at 10:18 AM, Sean Matheny <s.matheny@xxxxxxxxxxxxxx >>> <mailto:s.matheny@xxxxxxxxxxxxxx><mailto:s.matheny@xxxxxxxxxxxxxx>> >>> wrote: >>> >>> Hi folks, >>> >>> Our entire cluster is down at the moment. >>> >>> We started upgrading from 12.2.13 to 14.2.7 with the monitors. The >>> first monitor we upgraded crashed. We reverted to luminous on this >>> one and tried another, and it was fine. We upgraded the rest, and >>> they all worked. >>> >>> Then we upgraded the first one again, and after it became the leader, >>> it died. Then the second one became the leader, and it died. Then the >>> third became the leader, and it died, leaving mon 4 and 5 unable to >>> form a quorum. >>> >>> We tried creating a single monitor cluster by editing the monmap of >>> mon05, and it died in the same way, just without the paxos >>> negotiation first. >>> >>> We have tried to revert to a luminous (12.2.12) monitor backup taken >>> a few hours before the crash. The mon daemon will start, but is >>> flooded with blocked requests and unknown pgs after a while. For >>> better or worse we removed the “noout” flag and 144 of 232 OSDs are >>> now showing as down: >>> >>> cluster: >>> id: ababdd7f-1040-431b-962c-c45bea5424aa >>> health: HEALTH_ERR >>> noout,nobackfill,norecover flag(s) set >>> 101 osds down >>> 9 hosts (143 osds) down >>> 1 auth entities have invalid capabilities >>> Long heartbeat ping times on back interface seen, longest >>> is 15424.178 msec >>> Long heartbeat ping times on front interface seen, longest >>> is 14763.145 msec >>> Reduced data availability: 521 pgs inactive, 48 pgs stale >>> 274 slow requests are blocked > 32 sec >>> 88 stuck requests are blocked > 4096 sec >>> 1303 slow ops, oldest one blocked for 174 sec, >>> mon.ntr-mon01 has slow ops >>> too many PGs per OSD (299 > max 250) services: >>> mon: 1 daemons, quorum ntr-mon01 (age 3m) >>> mgr: ntr-mon01(active, since 30m) >>> mds: cephfs:1 {0=akld2e18u42=up:active(laggy or crashed)} >>> osd: 223 osds: 66 up, 167 in >>> flags noout,nobackfill,norecover >>> rgw: 2 daemons active (ntr-rgw01, ntr-rgw02) data: >>> pools: 14 pools, 15220 pgs >>> objects: 35.26M objects, 134 TiB >>> usage: 379 TiB used, 1014 TiB / 1.4 PiB avail >>> pgs: 3.423% pgs unknown >>> 14651 active+clean >>> 521 unknown >>> 48 stale+active+clean io: >>> client: 20 KiB/s rd, 439 KiB/s wr, 7 op/s rd, 54 op/s wr >>> >>> These luminous OSD daemons are not down, but are all in fact running. >>> They just have no comms with the monitor: >>> >>> 2020-02-19 10:12:33.565680 7ff222e24700 1 osd.0 pg_epoch: 305104 >>> pg[100.37as3( v 129516'2 (0'0,129516'2] local-lis/les=297268/297269 >>> n=0 ec=129502/129502 lis/c 297268/297268 les/c/f 297269/297358/0 >>> 297268/297268/161526) [41,192,216,0,160,117]p41(0) r=3 lpr=305101 >>> crt=129516'2 lcod 0'0 unknown NOTIFY mbc={}] state<Start>: >>> transitioning to Stray >>> 2020-02-19 10:12:33.565861 7ff222e24700 1 osd.0 pg_epoch: 305104 >>> pg[4.53c( v 305046'1933429 (304777'1931907,305046'1933429] >>> local-lis/les=298009/298010 n=7350 ec=768/768 lis/c 298009/298009 >>> les/c/f 298010/298010/0 297268/298009/298009) [0,61,103] r=0 >>> lpr=305101 crt=305046'1933429 lcod 0'0 mlcod 0'0 unknown mbc={}] >>> state<Start>: transitioning to Primary >>> 2020-02-19 10:12:33.566742 7ff222e24700 1 osd.0 pg_epoch: 305104 >>> pg[100.des4( v 129516'1 (0'0,129516'1] local-lis/les=292010/292011 >>> n=1 ec=129502/129502 lis/c 292010/292010 les/c/f 292011/292417/0 >>> 292010/292010/280955) [149,62,209,187,0,98]p149(0) r=4 lpr=305072 >>> crt=129516'1 lcod 0'0 unknown NOTIFY mbc={}] state<Start>: >>> transitioning to Stray >>> 2020-02-19 10:12:33.566896 7ff23ccd9e00 0 osd.0 305104 done with >>> init, starting boot process >>> 2020-02-19 10:12:33.566956 7ff23ccd9e00 1 osd.0 305104 start_boot >>> >>> One oddity in our deployment is that there was a test mds instance, >>> and it is running mimic. I shut it down, as the monitor trace has an >>> MDS call in it, but the nautilus monitors still die the same way. >>> >>> "mds": { >>> "ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) >>> mimic (stable)": 1 >>> }, >>> >>> >>> ... >>> -11> 2020-02-18 09:50:00.800 7fd164a1a700 5 >>> mon.ntr-mon02@1(leader).paxos(paxos recovering c 85448935..85449502) >>> is_readable = 0 - now=2020-02-18 09:50:00.804429 >>> lease_expire=0.000000 has v0 lc 85449502 >>> -10> 2020-02-18 09:50:00.800 7fd164a1a700 5 >>> mon.ntr-mon02@1(leader).paxos(paxos recovering c 85448935..85449502) >>> is_readable = 0 - now=2020-02-18 09:50:00.804446 >>> lease_expire=0.000000 has v0 lc 85449502 >>> -9> 2020-02-18 09:50:00.800 7fd164a1a700 5 >>> mon.ntr-mon02@1(leader).paxos(paxos recovering c 85448935..85449502) >>> is_readable = 0 - now=2020-02-18 09:50:00.804460 >>> lease_expire=0.000000 has v0 lc 85449502 >>> -8> 2020-02-18 09:50:00.800 7fd164a1a700 4 set_mon_vals no >>> callback set >>> -7> 2020-02-18 09:50:00.800 7fd164a1a700 4 mgrc handle_mgr_map Got >>> map version 2301191 >>> -6> 2020-02-18 09:50:00.804 7fd164a1a700 4 mgrc handle_mgr_map >>> Active mgr is now v1:10.31.88.17:6801/2924412 >>> -5> 2020-02-18 09:50:00.804 7fd164a1a700 0 log_channel(cluster) >>> log [DBG] : monmap e25: 5 mons at >>> {ntr-mon01=v1:10.31.88.14:6789/0,ntr-mon02=v1:10.31.88.15:6789/0,ntr-mon03=v1:10.31.88.16:6789/0,ntr-mon04=v1:10.31.88.17:6789/0,ntr-mon05=v1:10.31.88.18:6789/0} >>> -4> 2020-02-18 09:50:00.804 7fd164a1a700 10 log_client _send_to_mon >>> log to self >>> -3> 2020-02-18 09:50:00.804 7fd164a1a700 10 log_client log_queue >>> is 3 last_log 3 sent 2 num 3 unsent 1 sending 1 >>> -2> 2020-02-18 09:50:00.804 7fd164a1a700 10 log_client will send >>> 2020-02-18 09:50:00.806845 mon.ntr-mon02 (mon.1) 3 : cluster [DBG] >>> monmap e25: 5 mons at >>> {ntr-mon01=v1:10.31.88.14:6789/0,ntr-mon02=v1:10.31.88.15:6789/0,ntr-mon03=v1:10.31.88.16:6789/0,ntr-mon04=v1:10.31.88.17:6789/0,ntr-mon05=v1:10.31.88.18:6789/0} >>> -1> 2020-02-18 09:50:00.804 7fd164a1a700 5 >>> mon.ntr-mon02@1(leader).paxos(paxos active c 85448935..85449502) >>> is_readable = 1 - now=2020-02-18 09:50:00.806920 >>> lease_expire=2020-02-18 09:50:05.804479 has v0 lc 85449502 >>> 0> 2020-02-18 09:50:00.812 7fd164a1a700 -1 *** Caught signal >>> (Aborted) ** >>> in thread 7fd164a1a700 thread_name:ms_dispatch >>> >>> ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) >>> nautilus (stable) >>> 1: (()+0x11390) [0x7fd171e98390] >>> 2: (gsignal()+0x38) [0x7fd1715e5428] >>> 3: (abort()+0x16a) [0x7fd1715e702a] >>> 4: (__gnu_cxx::__verbose_terminate_handler()+0x135) [0x7fd173673bf5] >>> 5: (__cxxabiv1::__terminate(void (*)())+0x6) [0x7fd173667bd6] >>> 6: (()+0x8b6c21) [0x7fd173667c21] >>> 7: (()+0x8c2e34) [0x7fd173673e34] >>> 8: (std::__throw_out_of_range(char const*)+0x3f) [0x7fd17367f55f] >>> 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xcf0) [0x79ae00] >>> 10: (MDSMonitor::tick()+0xc9) [0x79c669] >>> 11: (MDSMonitor::on_active()+0x28) [0x785e88] >>> 12: (PaxosService::_active()+0xdd) [0x6d4b2d] >>> 13: (Context::complete(int)+0x9) [0x600789] >>> 14: (void finish_contexts<std::__cxx11::list<Context*, >>> std::allocator<Context*> > >(CephContext*, >>> std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xa8) >>> [0x6299a8] >>> 15: (Paxos::finish_round()+0x76) [0x6cb276] >>> 16: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0xbff) >>> [0x6cc47f] >>> 17: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x24b) >>> [0x6ccf2b] >>> 18: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x15c5) >>> [0x5fa6f5] >>> 19: (Monitor::_ms_dispatch(Message*)+0x4d2) [0x5fad42] >>> 20: (Monitor::ms_dispatch(Message*)+0x26) [0x62b046] >>> 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> >>> const&)+0x26) [0x6270b6] >>> 22: (DispatchQueue::entry()+0x1219) [0x7fd1732b7e59] >>> 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fd17336836d] >>> 24: (()+0x76ba) [0x7fd171e8e6ba] >>> 25: (clone()+0x6d) [0x7fd1716b741d] >>> ... >>> >>> Ceph versions output >>> >>> { >>> "mon": { >>> "ceph version 12.2.13 >>> (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 1, >>> "ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) >>> nautilus (stable)": 4 >>> }, >>> "mgr": { >>> "ceph version 12.2.12 >>> (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 1, >>> "ceph version 12.2.13 >>> (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 1, >>> "ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) >>> nautilus (stable)": 2 >>> }, >>> "osd": { >>> "ceph version 12.2.11 >>> (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)": 175, >>> "ceph version 12.2.12 >>> (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 32, >>> "ceph version 12.2.13 >>> (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 16 >>> }, >>> "mds": { >>> "ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) >>> mimic (stable)": 1 >>> }, >>> "rgw": { >>> "ceph version 12.2.12 >>> (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 2 >>> }, >>> "overall": { >>> "ceph version 12.2.11 >>> (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)": 175, >>> "ceph version 12.2.12 >>> (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 35, >>> "ceph version 12.2.13 >>> (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 18, >>> "ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) >>> mimic (stable)": 1, >>> "ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) >>> nautilus (stable)": 6 >>> } >>> } >>> >>> We’ve filed a bug report with the actions of the actual cascading >>> crash described above (when we upgraded mon01 and it became the leader): >>> https://tracker.ceph.com/issues/44185 (parts here copied from that >>> report) >>> >>> Right now we’re not sure what the best path to some sort of recovery >>> would be. All OSD Daemons are still on Luminous, so AFAICT, we could >>> build the monitor db from the OSDs with >>> https://github.com/ceph/ceph/blob/luminous/doc/rados/troubleshooting/troubleshooting-mon.rst#recovery-using-osds >>> which describes using this script: >>> >>> >>> #!/bin/bash >>> hosts="ntr-sto01 ntr-sto02" >>> ms=/tmp/mon-store/ >>> mkdir $ms >>> # collect the cluster map from OSDs >>> for host in $hosts; do >>> echo $host >>> rsync -avz $ms root@$host:$ms >>> rm -rf $ms >>> ssh root@$host <<EOF >>> for osd in /var/lib/ceph/osd/ceph-*; do >>> ceph-objectstore-tool --data-path \$osd --op update-mon-db >>> --mon-store-path $ms >>> done >>> EOF >>> rsync -avz root@$host:$ms $ms >>> done >>> >>> If this is our best idea to try, should we try the mon store from the >>> above script on a luminous or nautilus mon daemon? Any other ideas to >>> try at this dark hour? : \ >>> >>> Cheers, >>> Sean >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@xxxxxxx >>> <mailto:ceph-users@xxxxxxx><mailto:ceph-users@xxxxxxx> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>> <mailto:ceph-users-leave@xxxxxxx><mailto:ceph-users-leave@xxxxxxx> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>> <mailto:ceph-users-leave@xxxxxxx> > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |