Hi,
our cluster is on Octopus 15.2.4. We noticed that our MON all ran out of
space yesterday because the store.db folder kept growing until it filled
up the filesystem. We added more space to the MON nodes but store.db
keeps growing.
Right now it's ~220GiB on the two MON nodes that are active. We shut
down on MON node when it hit ~98GiB and it seems that it trimmed its
local store.db down to 102MiB and now also keeps growing again.
Checking the keys in store.db while the MON is offline shows a lot of
"logm" and "osdmap" keys:
ceph-monstore-tool <path> dump-keys|awk '{print $1}'|uniq -c
86 auth
2 config
11 health
275929 logm
55 mds_health
1 mds_metadata
602 mdsmap
599 mgr
1 mgr_command_descs
3 mgr_metadata
209 mgrstat
461 mon_config_key
1 mon_sync
7 monitor
1 monitor_store
7 monmap
454 osd_metadata
1 osd_pg_creating
4804 osd_snap
138366 osdmap
538 paxos
5 pgmap
I already tried compacting it with "ceph tell ..." and
"ceph-monstore-tool <path> compact" but it stayed the same size. Also
copying it with "ceph-monstore-tool <path> store-copy <new-path>" just
created a copy of the same size.
Out cluster is currently in WARN status because we are low on space and
several OSDs are in a backfill_full state. Could this be related?
Regards,
Michael
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
