Thanks you for your suggestions Josh, it is really appreciated. Pgremapper looks interesting and definitely something I will look into. I know the balancer will reach a well balanced PG landscape eventually, but I am not sure that it will prioritise backfill after “most available location” first. Then I might end up in the same situation, where some of the old (but not retired) OSD starts getting full. Then there is the “undo-upmaps” script left or maybe even the script that I propose in combination with “cancel-backfill”, as it just moves what Ceph was planing to move anyway, just in a prioritised manner. Have you tried the pgremapper youself Josh? Is it safe to use? And does the Ceph developers vouch for this methode? Status now is ~1,600,000,000 objects are now move, which is about half of all of the planned backfills. I have been reweighing OSD down, as they get to close to maximum usage, which works to some extend. Monitors on the other hand are now complaining about using a lot of disk space, due to the long time backfilling. There is still plenty of disk space on the mons, but I feel that the backfill is getting slower and slower, although still the same amount of PGs are backfilling. Can large disk usage on mons slow down backfill and other operations? Is it dangerous? Best, Jesper -------------------------- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Universitetsbyen 81 8000 Aarhus C E-mail: jelka@xxxxxxxxx Tlf: +45 50906203 > On 28 Jul 2022, at 22.26, Josh Baergen <jbaergen@xxxxxxxxxxxxxxxx> wrote: > > I don't have many comments on your proposed approach, but just wanted > to note that how I would have approached this, assuming that you have > the same number of old hosts, would be to: > 1. Swap-bucket the hosts. > 2. Downweight the OSDs on the old hosts to 0.001. (Marking them out > (i.e. weight 0) prevents maps from being applied.) > 3. Add the old hosts back to the CRUSH map in their old racks or whatever. > 4. Use https://github.com/digitalocean/pgremapper#cancel-backfill. > 5. Then run https://github.com/digitalocean/pgremapper#undo-upmaps in > a loop to drain the old OSDs. > > This gives you the maximum concurrency and efficiency of movement, but > doesn't necessarily solve your balance issue if it's the new OSDs that > are getting full (that wasn't clear to me). It's still possible to > apply steps 2, 4, and 5 if the new hosts are in place. If you're not > in a rush could actually use the balancer instead of undo-upmaps in > step 5 to perform the rest of the data migration and then you wouldn't > have full OSDs. > > Josh > > On Fri, Jul 22, 2022 at 1:57 AM Jesper Lykkegaard Karlsen > <jelka@xxxxxxxxx> wrote: >> >> It seems like a low hanging fruit to fix? >> There must be a reason why the developers have not made a prioritized order of backfilling PGs. >> Or maybe the prioritization is something else than available space? >> >> The answer remains unanswered, as well as if my suggested approach/script would work or not? >> >> Summer vacation? >> >> Best, >> Jesper >> >> -------------------------- >> Jesper Lykkegaard Karlsen >> Scientific Computing >> Centre for Structural Biology >> Department of Molecular Biology and Genetics >> Aarhus University >> Universitetsbyen 81 >> 8000 Aarhus C >> >> E-mail: jelka@xxxxxxxxx >> Tlf: +45 50906203 >> >> ________________________________ >> Fra: Janne Johansson <icepic.dz@xxxxxxxxx> >> Sendt: 20. juli 2022 19:39 >> Til: Jesper Lykkegaard Karlsen <jelka@xxxxxxxxx> >> Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx> >> Emne: Re: replacing OSD nodes >> >> Den ons 20 juli 2022 kl 11:22 skrev Jesper Lykkegaard Karlsen <jelka@xxxxxxxxx>: >>> Thanks for you answer Janne. >>> Yes, I am also running "ceph osd reweight" on the "nearfull" osds, once they get too close for comfort. >>> >>> But I just though a continuous prioritization of rebalancing PGs, could make this process more smooth, with less/no need for handheld operations. >> >> You are absolutely right there, just wanted to chip in with my >> experiences of "it nags at me but it will still work out" so other >> people finding these mails later on can feel a bit relieved at knowing >> that a few toofull warnings aren't a major disaster and that it >> sometimes happens, because ceph looks for all possible moves, even >> those who will run late in the rebalancing. >> >> -- >> May the most significant bit of your life be positive. >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
