On 1/17/23 08:39, Iztok Gregori wrote:
Thank for your response and advice.
On 16/01/23 15:17, Boris Behrens wrote:
Hmm.. I ran into some similar issue.
IMHO there are two ways to work around the problem until the new disk
in place:
1. change the backfill full threshold (I use these commands:
https://www.suse.com/support/kb/doc/?id=000019724
<https://www.suse.com/support/kb/doc/?id=000019724>)
If I understand correctly the "backfillfull_ratio" is a ratio above
which a warning is triggered and the cluster will deny backfilling to
the OSD in question. But my OSD (87.53% ) is not above the ratio (90%).
Granted, it is possible that "after" the 3 PGs are moved to that OSD the
ratio will be crossed, but right now we are bellow.
I have checked the actual calculations, but Ceph might calculate the
amount of space left on the OSD when the mapped backfills would have
finished. And because of that trigger the warning.
My final goal is to empty the "damaged" OSD/disk, replace it and fill it
up again. But I can't do that because it has still 3 PGs on it.
2. reweight the backfill full OSDs just a little bit, so they move
data to disks that are free enough (i.e. `ceph osd reweight osd.60
0.9`) - if you have enough capacity in the cluster (577+ OSDs should
be able to take that :) )
You mean first reweight to (something like) .9, let the OSD free up
space and then put it back to 1 so the PGs can be backfilled?
Pretty sure that's what was meant.
Idally I would like just to manually set the new "location" of the PGs
away from the nearfull OSD.60. I see there are some commands called
"ceph osd pg-upmap" and "ceph osd pg-upmap-items" which could be the
right tool for what I want to achieve. But I didn't found a lot of
information about it, sombody knows something more, are those tools
"safe" to run in my case?
Yes. A couple of tools rely heavily on "upmap". Including the ceph
balancer itself. Instead of using "weights", which was the knob to use
_before_ luminous, upmaps are the "new" (since Luminous) way of
balancing clusters. Precisely for what you want to achieve, have a way
to map PGs to OSDs.
If you want to use a tool, you can look at [1]. It's made and used by
Digital Ocean. You can also put upmaps yourself, i.e. ceph osd
pg-upmap-items $pg-id $osd-id
If possible (when you have new hardware and more space available) start
with using upmaps to achieve better balancing. Newer tools, Ceph
balancer as well, will use upmaps more and more. Not only to obtain more
evenly distributed space utilization, but also (read) performance. Of
you need more persuasion, you might want to watch this talk [2].
Gr. Stefan
[1]: https://github.com/digitalocean/pgremapper/
[2]: https://www.youtube.com/watch?v=niFNZN5EKvE
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
