Hi Bryan, How does the norebalance procedure work? You set the flag, increase the weight, then I expect the PGs to stay in remapped unless they're degraded ... why would a PG be degraded just because of a weight change? And then what happens when you unset norebalance? Cheers, Dan On Mon, Aug 7, 2017 at 6:07 PM, Bryan Stillwell <bstillwell@xxxxxxxxxxx> wrote: > Dan, > > We recently went through an expansion of an RGW cluster and found that we needed 'norebalance' set whenever making CRUSH weight changes to avoid slow requests. We were also increasing the CRUSH weight by 1.0 each time which seemed to reduce the extra data movement we were seeing with smaller weight increases. Maybe something to try out next time? > > Bryan > > From: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> on behalf of Dan van der Ster <dan@xxxxxxxxxxxxxx> > Date: Friday, August 4, 2017 at 1:59 AM > To: Laszlo Budai <laszlo@xxxxxxxxxxxxxxxx> > Cc: ceph-users <ceph-users@xxxxxxxxxxxxxx> > Subject: Re: expanding cluster with minimal impact > > Hi Laszlo, > > The script defaults are what we used to do a large intervention (the > default delta weight is 0.01). For our clusters going any faster > becomes disruptive, but this really depends on your cluster size and > activity. > > BTW, in case it wasn't clear, to use this script for adding capacity > you need to create the new OSDs to your cluster with initial crush > weight = 0.0 > > osd crush initial weight = 0 > osd crush update on start = true > > -- Dan > > > > On Thu, Aug 3, 2017 at 8:12 PM, Laszlo Budai <laszlo@xxxxxxxxxxxxxxxx> wrote: > Dear all, > > I need to expand a ceph cluster with minimal impact. Reading previous > threads on this topic from the list I've found the ceph-gentle-reweight > script > (https://github.com/cernceph/ceph-scripts/blob/master/tools/ceph-gentle-reweight) > created by Dan van der Ster (Thank you Dan for sharing the script with us!). > > I've done some experiments, and it looks promising, but it is needed to > properly set the parameters. Did any of you tested this script before? what > is the recommended delta_weight to be used? From the default parameters of > the script I can see that the default delta weight is .5% of the target > weight that means 200 reweighting cycles. I have experimented with a > reweight ratio of 5% while running a fio test on a client. The results were > OK (I mean no slow requests), but my test cluster was a very small one. > > If any of you has done some larger experiments with this script I would be > really interested to read about your results. > > Thank you! > Laszlo > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |