hrm.... with cciss try adding rootdelay=90 to your boot options. I'm not sure if it will delay udev for non-root, but I've heard of other people needing the delay for the / mount because the cciss devices might not be ready immediately On Mon, Aug 11, 2014 at 7:54 AM, O'Reilly, Dan <Daniel.OReilly at dish.com> wrote: > More information: > > > > When the system is booted, for whatever reason udev doesn?t seem to find > the devices used for OSD. However, once the system comes up, I can perform > a ?udevadm trigger ?action=add? command and all the devices appear. > Perhaps some sort of race condition? I am using a 95-ceph-osd.rules file > for udev. > > > > *From:* ceph-users [mailto:ceph-users-bounces at lists.ceph.com] *On Behalf > Of *O'Reilly, Dan > *Sent:* Sunday, August 10, 2014 7:38 AM > *To:* 'Andrew Woodward' > *Cc:* ceph-users; ceph-community at lists.ceph.com > *Subject:* Re: [ceph-users] [Ceph-community] working ceph.conf file? > > > > Thanks for the information on the ceph.conf contents. I wasn?t clear if > that we necessary or not. > > > > Here's the issue: > > > > I set up JBOD OSD disks with separate journals on a set of mirrored disks > to provide some protection of the journals in case of a journal disk > failure. That works fine - until you reboot the system. The journal entry > in the OSD metadata points to /dev/disk/by-partuuid, which apparently isn?t > preserved after a reboot. > > > > # ll /var/lib/ceph/osd/ceph-0 > > total 56 > > -rw-r--r-- 1 root root 502 Aug 7 09:05 activate.monmap > > -rw-r--r-- 1 root root 3 Aug 7 09:05 active > > -rw-r--r-- 1 root root 37 Aug 7 09:05 ceph_fsid > > drwxr-xr-x 81 root root 8192 Aug 8 10:27 current > > -rw-r--r-- 1 root root 37 Aug 7 09:05 fsid > > *lrwxrwxrwx 1 root root 58 Aug 7 09:05 journal -> > /dev/disk/by-partuuid/12f2c3db-524d-4cc0-a759-afea4a083237* > > -rw-r--r-- 1 root root 37 Aug 7 09:05 journal_uuid > > -rw------- 1 root root 56 Aug 7 09:05 keyring > > -rw-r--r-- 1 root root 21 Aug 7 09:05 magic > > -rw-r--r-- 1 root root 6 Aug 7 09:05 ready > > -rw-r--r-- 1 root root 4 Aug 7 09:05 store_version > > -rw-r--r-- 1 root root 42 Aug 7 09:05 superblock > > -rw-r--r-- 1 root root 0 Aug 7 09:06 sysvinit > > -rw-r--r-- 1 root root 2 Aug 7 09:05 whoami > > [NEW:note: root at tm1cldosdl01 on parent: /root] > > # ll /dev/disk/by-partuuid/12f2c3db-524d-4cc0-a759-afea4a083237 > > *lrwxrwxrwx 1 root root 18 Aug 7 09:06 > /dev/disk/by-partuuid/12f2c3db-524d-4cc0-a759-afea4a083237 -> > ../../cciss/c0d1p1* > > > > Because the /dev/disk/by-partuuid directory no longer exists after a > reboot, the mount of the OSD disk fails after a reboot because the journal > partition can?t be determined. The fundamental problem is, then, how do I > preserve the /dev/disk/by-partuuid directory between reboots? This is RHEL > 6.5, by the way. > > > > Thanks! > > > > *From:* Andrew Woodward [mailto:xarses at gmail.com <xarses at gmail.com>] > *Sent:* Friday, August 08, 2014 3:57 PM > *To:* O'Reilly, Dan > *Cc:* ceph-community at lists.ceph.com; ceph-users > *Subject:* Re: [Ceph-community] working ceph.conf file? > > > > Dan, > > > > It is not necessary to specify the OSD data in ceph.conf anymore. Ceph has > two auto-start functions besides this method. > > > > udev rules: > > ceph uses a udev rule to scan and attempt to mount (and activate) > partitions with specific GUID set for the partition typecode > > sgdisk --typecode=<<partation number>>:<<GUID>> /dev/<<disk>> > > > > the exact GUID's to use can be found > https://github.com/ceph/ceph/blob/master/udev/95-ceph-osd.rules. These > are set automaticly using ceph-disk (or ceph-deploy) if it creates the > partition from an empty disk, in the case that it does not, you have to set > them by hand, all be it should probably do this, or at least tell you > should do this. > > > > ceph init script: > > the ceph init script will scan /var/lib/ceph/osd (or the otherwise > configured location) for <cluster name>-<id> (default cluster name is > ceph) folders and attempt to start the osd service for each of them if they > look correct > > > > lastly, and possibly the most annoying way is that you can configure each > OSD and path in ceph.conf, I don't have any good examples as the two prior > are more flexible / require less config. > > > > > > > > On Fri, Aug 8, 2014 at 8:53 AM, O'Reilly, Dan <Daniel.OReilly at dish.com> > wrote: > > Does anybody have a good sample ceph.conf file I can use for reference? > I?m having a problem where OSD?s won?t come back up after a sysem reboot. > > > > Dan O'Reilly > > UNIX Systems Administration > > [image: cid:638154011 at 09122011-048B] > > 9601 S. Meridian Blvd. > > Englewood, CO 80112 > > 720-514-6293 > > > > > > > _______________________________________________ > Ceph-community mailing list > Ceph-community at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-community-ceph.com > > > > > > -- > > Andrew > > Mirantis > > Ceph community > -- Andrew Mirantis Ceph community -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140811/201dbea0/attachment.htm> -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 1895 bytes Desc: not available URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140811/201dbea0/attachment.jpg>
 |