Hi Dan and Gregory, the MONs went a bit too wild and started voting every 1-2 minutes. The logs were clean, so I started with the usual method of restarting one by one. Seems they got the message and the frenzy has stopped. We won't be able to diagnose this further, I'm afraid. Thanks for your hints. Best regards. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@xxxxxx> Sent: 09 February 2023 17:29:49 To: Gregory Farnum; Dan van der Ster Cc: ceph-users@xxxxxxx Subject: Re: Frequent calling monitor election Hi Dan and Gregory, thanks! These are good pointers. Will look into that tomorrow. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Gregory Farnum <gfarnum@xxxxxxxxxx> Sent: 09 February 2023 17:12:23 To: Dan van der Ster Cc: Frank Schilder; ceph-users@xxxxxxx Subject: Re: Re: Frequent calling monitor election Also, that the current leader (ceph-01) is one of the monitors proposing an election each time suggests the problem is with getting commit acks back from one of its followers. On Thu, Feb 9, 2023 at 8:09 AM Dan van der Ster <dvanders@xxxxxxxxx> wrote: > > Hi Frank, > > Check the mon logs with some increased debug levels to find out what > the leader is busy with. > We have a similar issue (though, daily) and it turned out to be > related to the mon leader timing out doing a SMART check. > See https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftracker.ceph.com%2Fissues%2F54313&data=05%7C01%7Cfrans%40dtu.dk%7C9e88ce19bd4b4cbde9d308db0abb1b1b%7Cf251f123c9ce448e927734bb285911d9%7C0%7C0%7C638115570946651737%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ft7rAlQ3N4yLrU%2BDqXKqTOW6oW7holvmpIgegFYBlxc%3D&reserved=0 for how I debugged that. > > Cheers, Dan > > On Thu, Feb 9, 2023 at 7:56 AM Frank Schilder <frans@xxxxxx> wrote: > > > > Hi all, > > > > our monitors have enjoyed democracy since the beginning. However, I don't share a sudden excitement about voting: > > > > 2/9/23 4:42:30 PM[INF]overall HEALTH_OK > > 2/9/23 4:42:30 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) > > 2/9/23 4:42:26 PM[INF]mon.ceph-01 calling monitor election > > 2/9/23 4:42:26 PM[INF]mon.ceph-26 calling monitor election > > 2/9/23 4:42:26 PM[INF]mon.ceph-25 calling monitor election > > 2/9/23 4:42:26 PM[INF]mon.ceph-02 calling monitor election > > 2/9/23 4:40:00 PM[INF]overall HEALTH_OK > > 2/9/23 4:30:00 PM[INF]overall HEALTH_OK > > 2/9/23 4:24:34 PM[INF]overall HEALTH_OK > > 2/9/23 4:24:34 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) > > 2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election > > 2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election > > 2/9/23 4:24:29 PM[INF]mon.ceph-03 calling monitor election > > 2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election > > 2/9/23 4:24:29 PM[INF]mon.ceph-26 calling monitor election > > 2/9/23 4:24:29 PM[INF]mon.ceph-25 calling monitor election > > 2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election > > 2/9/23 4:24:04 PM[INF]overall HEALTH_OK > > 2/9/23 4:24:03 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) > > 2/9/23 4:23:59 PM[INF]mon.ceph-01 calling monitor election > > 2/9/23 4:23:59 PM[INF]mon.ceph-02 calling monitor election > > 2/9/23 4:20:00 PM[INF]overall HEALTH_OK > > 2/9/23 4:10:00 PM[INF]overall HEALTH_OK > > 2/9/23 4:00:00 PM[INF]overall HEALTH_OK > > 2/9/23 3:50:00 PM[INF]overall HEALTH_OK > > 2/9/23 3:43:13 PM[INF]overall HEALTH_OK > > 2/9/23 3:43:13 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) > > 2/9/23 3:43:08 PM[INF]mon.ceph-01 calling monitor election > > 2/9/23 3:43:08 PM[INF]mon.ceph-26 calling monitor election > > 2/9/23 3:43:08 PM[INF]mon.ceph-25 calling monitor election > > > > We moved a switch from one rack to another and after the switch came beck up, the monitors frequently bitch about who is the alpha. How do I get them to focus more on their daily duties again? > > > > Thanks for any help! > > ================= > > Frank Schilder > > AIT Risø Campus > > Bygning 109, rum S14 > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |