On Jan 3, 2014, at 4:43 PM, Dane Elwell <dane.elwell@xxxxxxxxx> wrote: > I was wondering if there are any procedures for rebooting a node? Presumably when a node is rebooted, Ceph will lose contact with the OSDs and begin moving data around. I’ve not actually had to reboot a node of my cluster yet, but may need to do so in the near future. Does Ceph handle reboots gracefully or will it immediately begin moving data around? There's a delay. By default I think it is 5 minutes. You can also run "ceph osd set noout" beforehand to prevent OSD's from being marked 'out' no matter how long they may have been 'down'. After your maintenance don't forget to run "ceph osd unset noout" to put things back to normal. > I ask because due to the hardware we’re running, we have a fair few OSDs per node (between 32-40, I know this isn’t ideal). We recently had a node die on us briefly and it took about 2 hours to get back to HEALTH_OK once the node was back online after being down for around 15 minutes. This is with about 16TB of data (of 588TB total), so I’m worried about how a reboot (or another node failure) will affect us when we have more data on there. I normally set the "noout" flag as above, then reboot a single node and wait for all the OSDs to come back online and for peering, etc. to finish. I like to run "ceph osd tree" and "ceph pg stat" while waiting to see how things are going. Only once the cluster is happy and stable after the first reboot will I start a second. This all presumes that your crush map has multiple replicas and stores them on different hosts, of course. JN _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |