Hi Dan, Thanks for your response, It is not happening daily or around the same time. It is happening randomly in different times of the day. I checked logs and there is no ScrubResult log message in our logs. I also checked our monitoring for CPU load and there isn't any unusual load on the server's CPU at those times. On Sun, Nov 7, 2021 at 12:45 PM Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote: > Hi, > > If it's happening daily, around the same time, then it's possibly due to > the mon scrub, which we recently found to be CPU intensive on some clusters > and can cause mon elections. > Do you have ScrubResult log messages in ceph log? You can also check > previous days and see how long the mon scrub is taking to complete. (Time > from first to last entry) > > Cheers, Dan > > > > On Sun, Nov 7, 2021, 9:50 AM mahnoosh shahidi <mahnooosh.shd@xxxxxxxxx> > wrote: > >> Hi, >> >> We have a ceph cluster with 3 mon nodes in octopus 15.2.12. Recently, one >> of our monitor nodes randomly gets out of quorum and rejoins again. The >> rocksdb compaction queue of the monitor has 5 entries most of the time and >> rocksdb submit sync latency is about 1 second. There isn't any problem >> with >> mon disks. Restarting mon and mgr daemons does not help either and there >> is >> not any special log in mon logs. >> Anybody have any idea what the problem is? >> >> Regards >> Mahnoosh >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |