Hi John,
We recently added a lot of nodes to our ceph clusters. To mitigate lot of problems (we are using tree algorithm) we added an empty node first to the crushmap and then added OSDs with zero weight, made sure the ceph health is OK and then started ramping up each OSD. I created a script to do it dynamically, which will check CPU of the new host with OSDs that is being added, max backfills, and degradation values from ceph -s command and depending on those values it will ramp up more OSDs to their full weight. This made sure the cluster performance isn't impacted too much.
The other values which I kept changing so as not to cause any issue for client io are,
ceph tell osd.* injectargs '--osd-max-backfills 4'
ceph tell osd.* injectargs '--osd-recovery-max-active 6'
ceph tell osd.* injectargs '--osd-recovery-threads 5'
ceph tell osd.* injectargs '--osd-recovery-op-priority 20'
--Pardhiv Karri
On Thu, Jun 7, 2018 at 2:23 PM, Paul Emmerich <paul.emmerich@xxxxxxxx> wrote:
Hi,the "osd_recovery_sleep_hdd/ssd" options are way better to fine-tune the impact of a backfill operation in this case.Paul--2018-06-07 20:55 GMT+02:00 David Turner <drakonstein@xxxxxxxxx>:A recommendation for adding disks with minimal impact is to add them with a crush weight of 0 (configurable in the ceph.conf file and then increasing their weight in small increments until you get to the desired OSD weight. That way you're never moving too much data at once and can stop at any time.If you don't want to be quite this paranoid, you can just manage the osd_max_backfill settings and call it a day while letting the OSDs add to their full weight from the start. It all depends on your client IO needs, how much data you have, speed of disks/network, etc.On Wed, Jun 6, 2018 at 3:09 AM John Molefe <John.Molefe@xxxxxxxxx> wrote:______________________________Hi everyoneWe have completed all phases and the only remaining part is just adding the disks to the current cluster but i am afraid of impacting performance as it is on production.Any guides and advices on how this can be achieved with least impact on production??Thanks in advanceJohn
Vrywaringsklousule / Disclaimer: http://www.nwu.ac.za/it/gov-ma
n/disclaimer.html _________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
Pardhiv Karri
"Rise and Rise again until LAMBS become LIONS"
"Rise and Rise again until LAMBS become LIONS"
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
