Maybe the debugging steps in that insights tracker can be helpful anyway: https://tracker.ceph.com/issues/39955 -- dan On Thu, Feb 25, 2021 at 9:27 PM Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx> wrote: > > Thanks for the tip, but I do not have degraded PGs and the module is already disabled. > > > On 25. Feb 2021, at 21:17, Seena Fallah <seenafallah@xxxxxxxxx> wrote: > > I had the same problem in my cluster and it was because of insights mgr module that was storing lots of data to the RocksDB because mu cluster was degraded. > If you have degraded pgs try to disable insights module. > > On Thu, Feb 25, 2021 at 11:40 PM Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote: >> >> > "source": "osd.104... >> >> What's happening on that osd? Is it something new which corresponds to when >> your mon started growing? Are other OSDs also flooding the mons with logs? >> >> I'm mobile so can't check... Are those logging configs the defaults? If not >> .... revert to default... >> >> BTW do your mons have stable quorum or are they flapping with this load? >> >> .. dan >> >> >> >> On Thu, Feb 25, 2021, 8:58 PM Janek Bevendorff < >> janek.bevendorff@xxxxxxxxxxxxx> wrote: >> >> > Thanks, Dan. >> > >> > On the first MON, the command doesn’t even return, but I was able to get a >> > dump from the one I restarted most recently. The oldest ops look like this: >> > >> > { >> > "description": "log(1000 entries from seq 17876238 at >> > 2021-02-25T15:13:20.306487+0100)", >> > "initiated_at": "2021-02-25T20:40:34.698932+0100", >> > "age": 183.762551121, >> > "duration": 183.762599201, >> > "type_data": { >> > "events": [ >> > { >> > "time": "2021-02-25T20:40:34.698932+0100", >> > "event": "initiated" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.698636+0100", >> > "event": "throttled" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.698932+0100", >> > "event": "header_read" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.701407+0100", >> > "event": "all_read" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.701455+0100", >> > "event": "dispatched" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.701458+0100", >> > "event": "mon:_ms_dispatch" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.701459+0100", >> > "event": "mon:dispatch_op" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.701459+0100", >> > "event": "psvc:dispatch" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.701490+0100", >> > "event": "logm:wait_for_readable" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.701491+0100", >> > "event": "logm:wait_for_readable/paxos" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.701496+0100", >> > "event": "paxos:wait_for_readable" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.989198+0100", >> > "event": "callback finished" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.989199+0100", >> > "event": "psvc:dispatch" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.989208+0100", >> > "event": "logm:preprocess_query" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.989208+0100", >> > "event": "logm:preprocess_log" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.989278+0100", >> > "event": "forward_request_leader" >> > }, >> > { >> > "time": "2021-02-25T20:40:34.989344+0100", >> > "event": "forwarded" >> > }, >> > { >> > "time": "2021-02-25T20:41:58.658022+0100", >> > "event": "resend forwarded message to leader" >> > }, >> > { >> > "time": "2021-02-25T20:42:27.735449+0100", >> > "event": "resend forwarded message to leader" >> > } >> > ], >> > "info": { >> > "seq": 41550, >> > "src_is_mon": false, >> > "source": "osd.104 v2:XXX:6864/16579", >> > "forwarded_to_leader": true >> > } >> > >> > >> > Any idea what that might be about? Almost looks like this: >> > https://tracker.ceph.com/issues/24180 >> > I set debug_mon to 0, but I keep getting a lot of log spill in journals. >> > It’s about 1-2 messages per second, mostly RocksDB stuff, but nothing that >> > actually looks serious or even log-worthy. I noticed that before that >> > despite logging being set to warning level, the cluster log keeps being >> > written to the MON log. But it shouldn’t cause such massive stability >> > issues, should it? The date on the log op is also weird. 15:13+0100 was >> > hours ago. >> > >> > Here’s my log config: >> > >> > global advanced clog_to_syslog_level >> > warning >> > global basic err_to_syslog >> > true >> > global basic log_to_file >> > false >> > global basic log_to_stderr >> > false >> > global basic log_to_syslog >> > true >> > global advanced mon_cluster_log_file_level >> > error >> > global advanced mon_cluster_log_to_file >> > false >> > global advanced mon_cluster_log_to_stderr >> > false >> > global advanced mon_cluster_log_to_syslog >> > false >> > global advanced >> > mon_cluster_log_to_syslog_level warning >> > >> > >> > >> > Ceph version is 15.2.8. >> > >> > Janek >> > >> > >> > On 25. Feb 2021, at 20:33, Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote: >> > >> > ceph daemon mon.`hostname -s` ops >> > >> > That should show you the accumulating ops. >> > >> > .. dan >> > >> > >> > On Thu, Feb 25, 2021, 8:23 PM Janek Bevendorff < >> > janek.bevendorff@xxxxxxxxxxxxx> wrote: >> > >> >> Hi, >> >> >> >> All of a sudden, we are experiencing very concerning MON behaviour. We >> >> have five MONs and all of them have thousands up to tens of thousands of >> >> slow ops, the oldest one blocking basically indefinitely (at least the >> >> timer keeps creeping up). Additionally, the MON stores keep inflating >> >> heavily. Under normal circumstances we have about 450-550MB there. Right >> >> now its 27GB and growing (rapidly). >> >> >> >> I tried restarting all MONs, I disabled auto-scaling (just in case) and >> >> checked the system load and hardware. I also restarted the MGR and MDS >> >> daemons, but to no avail. >> >> >> >> Is there any way I can debug this properly? I can’t seem to find how I >> >> can actually view what ops are causing this and what client (if any) may be >> >> responsible for it. >> >> >> >> Thanks >> >> Janek >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users@xxxxxxx >> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> >> > >> > >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |