Hi Sage, Our clusters are slightly different but no the monitors start just fine. On our test and rgw clusters we run monitors co-located with our OSDs. The monitors start just fine. My understanding is that when booting the hosts detect a disk hot-plug event in udev via the /lib/udev/rules.d/95-ceph-osd.rules file. Running the '/usr/sbin/ceph-disk-activate /dev/vdb1' command for example in our test cluster does the right thing # /usr/sbin/ceph-disk-activate /dev/vdb1 === osd.2 === create-or-move updated item name 'osd.2' weight 0.02 at location {host=ceph02,root=default} to crush map Starting Ceph osd.2 on ceph02... starting osd.2 at :/0 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal Our best guess so far is that this line is not matching the underlying disk that is getting hotplugged (95-ceph-osd.rules). Is ID_PART_ENTRY_TYPE just the partition UUID or are we not understanding identifier correctly? ENV{ID_PART_ENTRY_TYPE}=="4fbd7e29-9d25-41b8-afd0-062c0ceff05d", Thanks, derek On 1/27/14, 11:29 AM, Sage Weil wrote: > Hi Derek, > > Would like to get to the bottom of your problem. Is it that the monitors > don't start after a reboot? Is there an error in > /var/log/ceph/ceph-mon.`hostname`.log? > > sage > > On Mon, 27 Jan 2014, Derek Yarnell wrote: > >> Hi, >> >> Would I take this to understand that this may be a known issue with udev >> on RHEL then? We will for now add them to the fstab. >> >> Thanks, >> derek >> >> On 1/25/14, 9:23 PM, Michael J. Kidd wrote: >>> While clearly not optimal for long term flexibility, I've found that >>> adding my OSD's to fstab allows the OSDs to mount during boot, and they >>> start automatically when they're already mounted during boot. >>> >>> Hope this helps until a permanent fix is available. >>> >>> Michael J. Kidd >>> Sr. Storage Consultant >>> Inktank Professional Services >>> >>> >>> On Fri, Jan 24, 2014 at 9:08 PM, Derek Yarnell <derek@xxxxxxxxxxxxxx >>> <mailto:derek@xxxxxxxxxxxxxx>> wrote: >>> >>> So we have a test cluster, and two production clusters all running on >>> RHEL6.5. Two are running Emperor and one of them running Dumpling. On >>> all of them our OSDs do not start at boot it seems via the udev rules. >>> The OSDs were created with ceph-deploy and are all GPT. The OSDs are >>> visable with `ceph-disk list` and running `/usr/sbin/ceph-disk-activate >>> {device}` mounts and adds them. Running a `partprobe {device}` does not >>> seem to trigger the udev rule at all. >>> >>> I had found this issue[1] but we are definitely running code that was >>> released after this ticket was closed. Has there been anyone else that >>> has problems with udev on RHEL mounting their OSDs? >>> >>> [1] - http://tracker.ceph.com/issues/5194 >>> >>> Thanks, >>> derek >>> >>> -- >>> Derek T. Yarnell >>> University of Maryland >>> Institute for Advanced Computer Studies >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> >> >> -- >> Derek T. Yarnell >> University of Maryland >> Institute for Advanced Computer Studies >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com