Am 27.09.21 um 22:38 schrieb Josh Baergen: >> I have a question regarding the last step. It seems to me that the ceph balancer is not able to remove the upmaps >> created by pgremapper, but instead creates new upmaps to balance the pgs among osds. > The balancer will prefer to remove existing upmaps[1], but it's not > guaranteed. The upmap has to exist between the source and target OSD > already decided on by the balancer in order for this to happen. The > reality, though, is that the upmap balancer will need to create many > upmap exception table entries to balance any sizable system. > > Like you, I would prefer to have as few upmap exception table entries > as possible in a system (fewer surprises when an OSD fails), but we > regularly run systems that have thousands of entries without any > discernible impact and haven't had any major operational issues that > result from it, except for on really old systems that are just awful > to work with in the first place. Hi Josh, thanks for the update. With the pgremapper run first and then the upmap balancer enabled I ended up with an upmap entry for almost every placement group in a pool. So I decided to write a tool which analizes the upmap entries and put them in a digraph. When I check for circles I found that running the upmap balancer alone never seems to create any kind of circle in the graph (i will have to check the code you mentioned [1] if there is any mechanism to avoid that). Running pgremapper + balancer created circles with sometimes several dozen nodes. I would update the docs of the pgremapper to warn about this fact and guide the users to use undo-upmap to slowly remove the upmaps create by cancel-backfill. It might be a nice addition to pgremapper to add an option to optimze the upmap table. When searching for circles you might want to limit the depth of the DFS otherwise the runtime will be crazy. Thanks, Peter _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |