So, I thought I’d post with what I learned re: what to do with this problem. This system is a 3-node Proxmox cluster, and each node had: 1 x 1TB NVMe 2 x 512GB HDD I had maybe 100GB of data in this system total. Then I added: 2 x 256GB SSD 1 x 1TB HDD To each system, and let it start rebalancing. When it started the management interface showed the storage as being out of order in various ways, but it was clear that Ceph was rebalancing PGs across the 3 nodes and the “broken” part of the graphic display was shrinking as it spread data across the added OSDs. In the process, however, the monitors racked up ENORMOUS amounts of files. On one machine, the boot drive only has 64GB of space total so the partition where /var/lib/ceph/somethingsomething.db lived was only 27GB. This filled up very, very fast, and eventually killed the monitor on that node. I figured out you can `ceph-monstore-tool compact` or `ceph-kvstore-tool rocksdb /path compact` to get the system to truncate the files in there, but even when I scheduled those jobs to run on each monitor every minute the amount of space being taken up by those rocksdb files grew and grew until they threatened to kill the monitors on the nodes with larger amounts of space too. Other, dumber measures I tried taking to give the system more space for these files ended up screwing up my Proxmox system, so now I have to reinstall. What can be done about this problem so that I don’t have this issue when I try to implement again? Thanks! _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |