Hi Eugene! I have a case, where PG with millions of objects, like this ``` root@host# ./show_osd_pool_pg_usage.sh <pool> | less | head id used_mbytes used_objects omap_used_mbytes omap_used_keys -- ----------- ------------ ---------------- -------------- 17.c91 1213.2482748031616 2539152 0 0 17.9ae 1213.3145303726196 2539025 0 0 17.1a4 1213.432228088379 2539752 0 0 17.8f4 1213.4958791732788 2539831 0 0 17.f9 1213.5339193344116 2539837 0 0 17.c9d 1213.564414024353 2540014 0 0 17.89 1213.6339054107666 2540183 0 0 17.412 1213.6393299102783 2539797 0 0 ``` And OSD was very small, like 1TB with RocksDB ~150-200GB. Actually currently you see splitted PG. So one OSD was serve 64PG * 4M = 256,000,000 of objects... Main problem was - to remove something, you need to move something. While the move is in progress, nothing is deleted Also, deleting is slower than writing. So one task for all operations was impossible. I do it manually for a 9 moths. After the splitting of the some PG was completed, I took other PG away from the most crowded (from the operator’s point of view, problematic) OSD. The pgremapper [1] helped me with this. As far as I remember, in this way I got from 2048 to 3000 PG, then I was able to set 4096 PG, after which it became possible to move to 4TV NVME Your case doesn't look that scary. Firstly, your 85% means that you have hundreds of free gigabytes (8TB's). If new data does not arrive, the reservation mechanism is sufficient and after some time the process will end. On the other hand, I had a replica, so compared to the EC - my case is a simpler In any case, it’s worth trying and using the maximum capabilities of the upmap Good luck, k [1] https://github.com/digitalocean/pgremapper > On 9 Apr 2024, at 11:39, Eugen Block <eblock@xxxxxx> wrote: > > I'm trying to estimate the possible impact when large PGs are splitted. Here's one example of such a PG: > > PG_STAT OBJECTS BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG UP > 86.3ff 277708 414403098409 0 0 3092 3092 [187,166,122,226,171,234,177,163,155,34,81,239,101,13,117,8,57,111] > > Their main application is RGW on EC (currently 1024 PGs on 240 OSDs), 8TB HDDs backed by SSDs. There are 6 RGWs running behind HAProxies. It took me a while to convince them to do a PG split and now they're trying to assess how big the impact could be. The fullest OSD is already at 85% usage, the least filled one at 59%, so there is definitely room for a better balancing which, will be necessary until the new hardware arrives. The current distribution is around 100 PGs per OSD which usually would be fine, but since the PGs are that large only a few PGs difference have a huge impact on the OSD utilization. > > I'm targeting 2048 PGs for that pool for now, probably do another split when the new hardware has been integrated. > Any comments are appreciated! _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |