If someone is interested: we've found a workaround in this mailing list: https://www.spinics.net/lists/ceph-users/msg47963.html It looks like an old bug. We fixed the issue by restarting all ceph-mon services one by one. Mon's store uses ~500Mb now and osd's removed old osd maps: ~# find /var/lib/ceph/osd/ceph-224/current/meta/ | wc -l 1839 New osd's have only 1.35GiB used space after first start with no weight. 08.10.2018, 22:31, "Aleksei Zakharov" <zakharov.a.g@xxxxxxxxx>: > As i can see, all pg's are active+clean: > > ~# ceph -s > cluster: > id: d168189f-6105-4223-b244-f59842404076 > health: HEALTH_WARN > noout,nodeep-scrub flag(s) set > mons 1,2,3,4,5 are using a lot of disk space > > services: > mon: 5 daemons, quorum 1,2,3,4,5 > mgr: api1(active), standbys: api2 > osd: 832 osds: 791 up, 790 in > flags noout,nodeep-scrub > > data: > pools: 10 pools, 52336 pgs > objects: 47.78M objects, 238TiB > usage: 854TiB used, 1.28PiB / 2.12PiB avail > pgs: 52336 active+clean > > io: > client: 929MiB/s rd, 1.16GiB/s wr, 31.85kop/s rd, 36.19kop/s wr > > 08.10.2018, 22:11, "Wido den Hollander" <wido@xxxxxxxx>: >> On 10/08/2018 05:04 PM, Aleksei Zakharov wrote: >>> Hi all, >>> >>> We've upgraded our cluster from jewel to luminous and re-created monitors using rocksdb. >>> Now we see, that mon's are using a lot of disk space and used space only grows. It is about 17GB for now. It was ~13GB when we used leveldb and jewel release. >>> >>> When we added new osd's we saw that it downloads from monitors a lot of data. It was ~15GiB few days ago and it is ~18GiB today. >>> One of the osd's we created uses filestore and it looks like old osd maps are not removed: >>> >>> ~# find /var/lib/ceph/osd/ceph-224/current/meta/ | wc -l >>> 73590 >>> >>> I've tried to run manual compaction (ceph tell mon.NUM compact) but it doesn't help. >>> >>> So, how to stop this growth of data on monitors? >> >> What is the status of Ceph? Can you post the output of: >> >> $ ceph -s >> >> MONs do not trim their database if one or more PGs aren't active+clean. >> >> Wido >> >>> -- >>> Regards, >>> Aleksei Zakharov >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -- > Regards, > Aleksei Zakharov > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Regards, Aleksei Zakharov _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |