Hi Ricardo, I just had a similar issue recently. I did a dump of the monitor store (i.e., something like "ceph-monstore-tool /var/lib/ceph/mon/mon-a/ dump-keys") and most messages were of type 'logm'. For me I think it was a lot of log messages coming from an oddly behaving OSD. I've seen folks advise disabling the Ceph mgr insights module if you have it running and there are degraded PGs, to see if that helps. What finally solved it for me was doing a rolling restart of my nodes, but I started from all PGs active+clean. --Lincoln ________________________________ From: ricardo.re.azevedo@xxxxxxxxx <ricardo.re.azevedo@xxxxxxxxx> Sent: Wednesday, March 10, 2021 5:59 PM To: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Subject: mon db growing. over 500Gb Hi all, I have a fairly pressing issue. I had a monitor fall out of quorum because it ran out of disk space during rebalancing from switching to upmap. I noticed all my monitor store.db started taking up nearly all disk space so I set noout, nobackfill and norecover and shutdown all the monitor daemons. Each store.db was at: mon.a 89GB (the one that firt dropped out) mon.a 400GB mon.c 400GB I tried setting mon_compact_on_start. This brought mon.a down to 1GB. Cool. However, when I try it on the other monitors it increased the db size ~1Gb/10s so I shut them down again. Any idea what is going on? Or how can I shrik back down the db? _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |