Re: replacing OSD nodes

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Thanks for you answer Janne.

Yes, I am also running "ceph osd reweight" on the "nearfull" osds, once they get too close for comfort.

But I just though a continuous prioritization of rebalancing PGs, could make this process more smooth, with less/no need for handheld operations.

Best,
Jesper

--------------------------
Jesper Lykkegaard Karlsen
Scientific Computing
Centre for Structural Biology
Department of Molecular Biology and Genetics
Aarhus University
Universitetsbyen 81
8000 Aarhus C

E-mail: jelka@xxxxxxxxx
Tlf:    +45 50906203

________________________________
Fra: Janne Johansson <icepic.dz@xxxxxxxxx>
Sendt: 20. juli 2022 10:47
Til: Jesper Lykkegaard Karlsen <jelka@xxxxxxxxx>
Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx>
Emne: Re:  replacing OSD nodes

Den tis 19 juli 2022 kl 13:09 skrev Jesper Lykkegaard Karlsen <jelka@xxxxxxxxx>:
>
> Hi all,
> Setup: Octopus - erasure 8-3
> I had gotten to the point where I had some rather old OSD nodes, that I wanted to replace with new ones.
> The procedure was planned like this:
>
>   *   add new replacement OSD nodes
>   *   set all OSDs on the retiring nodes to out.
>   *   wait for everything to rebalance
>   *   remove retiring nodes

> After around 50% misplaced objects remaining, the OSDs started to complain about backfillfull OSDs and nearfull OSDs.
> A bit of a surprise to me, as RAW size is only 47% used.
> It seems that rebalancing does not happen in a prioritized manner, where planed backfill starts with the OSD with most space available space, but "alphabetically" according to pg-name.
> Is this really true?

I don't know if it does it in any particular order, just that it
certainly doesn't fire off requests to the least filled OSD to receive
data first, so when I have gotten into similar situations, it just
tried to run as many moves as possible given max_backfill and all
that, then some/most might get stuck in toofull, but as the rest of
the slots progress, space gets available and at some point those
toofull ones get handled. It delays the completion but hasn't caused
me any other specific problems.

Though I will admit I have used "ceph osd reweight osd.123
<less-than-1.0>" at times to force emptying of some OSDs, but that was
more my impatience than anything else.


--
May the most significant bit of your life be positive.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux