I've been working on automating a lot of our ceph admin tasks lately and am
pretty pleased with how the puppet-ceph module has worked for installing
packages, managing ceph.conf, and creating the mon nodes. However, I don't
like the idea of puppet managing the OSDs. Since we also use ansible in my
group, I took a look at ceph-ansible to see how it might be used to
complete
this task. I see examples for doing a rolling update and for doing an os
migration, but nothing for adding a node or multiple nodes at once. I
don't
have a problem doing this work, but wanted to check with the community if
any one has experience using ceph-ansible for this?
After a lot of trial and error I found the following process works well
when
using ceph-deploy, but it's a lot of steps and can be error prone
(especially if you have old cephx keys that haven't been removed yet):
# Disable backfilling and scrubbing to prevent too many performance
# impacting tasks from happening at the same time. Maybe adding norecover
# to this list might be a good idea so only peering happens at first.
ceph osd set nobackfill
ceph osd set noscrub
ceph osd set nodeep-scrub
# Zap the disks to start from a clean slate
ceph-deploy disk zap dnvrco01-cephosd-025:sd{b..y}
# Prepare the disks. I found sleeping between adding each disk can help
# prevent performance problems.
ceph-deploy osd prepare dnvrco01-cephosd-025:sdh:/dev/sdb; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdi:/dev/sdb; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdj:/dev/sdb; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdk:/dev/sdc; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdl:/dev/sdc; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdm:/dev/sdc; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdn:/dev/sdd; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdo:/dev/sdd; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdp:/dev/sdd; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdq:/dev/sde; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdr:/dev/sde; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sds:/dev/sde; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdt:/dev/sdf; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdu:/dev/sdf; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdv:/dev/sdf; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdw:/dev/sdg; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdx:/dev/sdg; sleep 15
ceph-deploy osd prepare dnvrco01-cephosd-025:sdy:/dev/sdg; sleep 15
# Weight in the new OSDs. We set 'osd_crush_initial_weight = 0' to prevent
# them from being added in during the prepare step. Maybe a longer weight
# in the last step would make this step unncessary.
ceph osd crush reweight osd.450 1.09; sleep 60
ceph osd crush reweight osd.451 1.09; sleep 60
ceph osd crush reweight osd.452 1.09; sleep 60
ceph osd crush reweight osd.453 1.09; sleep 60
ceph osd crush reweight osd.454 1.09; sleep 60
ceph osd crush reweight osd.455 1.09; sleep 60
ceph osd crush reweight osd.456 1.09; sleep 60
ceph osd crush reweight osd.457 1.09; sleep 60
ceph osd crush reweight osd.458 1.09; sleep 60
ceph osd crush reweight osd.459 1.09; sleep 60
ceph osd crush reweight osd.460 1.09; sleep 60
ceph osd crush reweight osd.461 1.09; sleep 60
ceph osd crush reweight osd.462 1.09; sleep 60
ceph osd crush reweight osd.463 1.09; sleep 60
ceph osd crush reweight osd.464 1.09; sleep 60
ceph osd crush reweight osd.465 1.09; sleep 60
ceph osd crush reweight osd.466 1.09; sleep 60
ceph osd crush reweight osd.467 1.09; sleep 60
# Once all the OSDs are added to the cluster, allow the backfill process to
# begin.
ceph osd unset nobackfill
# Then once cluster is healthy again, re-enable scrubbing
ceph osd unset noscrub
ceph osd unset nodeep-scrub
This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
