Hallo,apologies for reviving an old thread, but I just wasted again one full day as I had forgotten about this issue...
To recap, udev rules nowadays do not (at least in my case, I am using disks served via FiberChannel) create the links /dev/disk/by-partuuid that ceph-disk expects.
I see the "culprit" is this line in (am on CentOS, but Ubuntu has the same issue): /usr/lib/udev/rules.d/60-persistent-storage.rules
..... # skip rules for inappropriate block devicesKERNEL=="fd*|mtd*|nbd*|gnbd*|btibm*|dm-*|md*|zram*|mmcblk[0-9]*rpmb", GOTO="persistent_storage_end"
..... stating that multipath'ed devices (called dm-*) should be skipped. I can happily live with the file mentioned below, but was wondering: - is there any hope that newer kernels may handle multipath devices properly? - as an alternative, could it be possible to update ceph-disk such that symlinks for journal use some other /dev/disk/by-<whatever>? Thanks! Fulvio On 3/16/2017 5:59 AM, Gunwoo Gim wrote:
Thank you so much Peter. The 'udevadm trigger' after 'partprobe' triggered the udev rules and I've found out that even before the udev ruleset triggers the owner is already ceph:ceph.I've dug into ceph-disk a little more and found out that there is a symbolic link of /dev/disk/by-partuuid/120c536d-cb30-4cea-b607-dd347022a497 at [/dev/mapper/vg--hdd1-lv--hdd1p1(the_filestore_osd)]/journal and the source doesn't exist. though it exists in /dev/disk/by-parttypeuuid which has been populated by /lib/udev/rules.d/60-ceph-by-parttypeuuid.rulesSo I added this in /lib/udev/rules.d/60-ceph-by-parttypeuuid.rules:# when ceph-disk prepares a filestore osd it makes a symbolic link by disk/by-partuuid but LVM2 doesn't seem to populate /dev/disk/by-partuuid. ENV{ID_PART_ENTRY_SCHEME}=="gpt", ENV{ID_PART_ENTRY_TYPE}=="?*", ENV{ID_PART_ENTRY_UUID}=="?*", SYMLINK+="disk/by-partuuid/$env{ID_PART_ENTRY_UUID}"And finally got the osds all up and in. :DYeah, It wasn't actually a permission problem, but the link just wasn't existing.~ # ceph-disk -v activate /dev/mapper/vg--hdd1-lv--hdd1p1 ...mount: Mounting /dev/mapper/vg--hdd1-lv--hdd1p1 on /var/lib/ceph/tmp/mnt.ECAifr with options noatime,largeio,inode64,swalloc command_check_call: Running command: /bin/mount -t xfs -o noatime,largeio,inode64,swalloc -- /dev/mapper/vg--hdd1-lv--hdd1p1 /var/lib/ceph/tmp/mnt.ECAifrmount: DIGGIN ls -al /var/lib/ceph/tmp/mnt.ECAifr mount: DIGGIN total 36 drwxr-xr-x 3 ceph ceph 174 Mar 14 11:51 . drwxr-xr-x 6 ceph ceph 4096 Mar 16 11:30 .. -rw-r--r-- 1 root root 202 Mar 16 11:19 activate.monmap -rw-r--r-- 1 ceph ceph 37 Mar 14 11:45 ceph_fsid drwxr-xr-x 3 ceph ceph 39 Mar 14 11:51 current -rw-r--r-- 1 ceph ceph 37 Mar 14 11:45 fsidlrwxrwxrwx 1 ceph ceph 58 Mar 14 11:45 journal -> /dev/disk/by-partuuid/120c536d-cb30-4cea-b607-dd347022a497-rw-r--r-- 1 ceph ceph 37 Mar 14 11:45 journal_uuid -rw-r--r-- 1 ceph ceph 21 Mar 14 11:45 magic -rw-r--r-- 1 ceph ceph 4 Mar 14 11:51 store_version -rw-r--r-- 1 ceph ceph 53 Mar 14 11:51 superblock -rw-r--r-- 1 ceph ceph 2 Mar 14 11:51 whoami ...ceph_disk.main.Error: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', u'0', '--monmap', '/var/lib/ceph/tmp/mnt.ECAifr/activate.monmap', '- -osd-data', '/var/lib/ceph/tmp/mnt.ECAifr', '--osd-journal', '/var/lib/ceph/tmp/mnt.ECAifr/journal', '--osd-uuid', u'377c336b-278d-4caf-b2f5-592ac72cd9b6', '- -keyring', '/var/lib/ceph/tmp/mnt.ECAifr/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : 2017-03-16 11:30:05.238725 7f918fbc0a40 -1 filestore(/v ar/lib/ceph/tmp/mnt.ECAifr) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.ECAifr/journal: (13) Permission denied 2017-03-16 11:30:05.238756 7f918fbc0a40 -1 OSD::mkfs: ObjectStore::mkfs failed with error -13 2017-03-16 11:30:05.238833 7f918fbc0a40 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.ECAifr: (13) Permission denied~ # blkid /dev/mapper/vg--*lv-*p* | grep '120c536d-cb30-4cea-b607-dd347022a497' /dev/mapper/vg--ssd1-lv--ssd1p1: PARTLABEL="ceph journal" PARTUUID="120c536d-cb30-4cea-b607-dd347022a497"~ # ls -al /dev/disk/by-id | grep dm-22lrwxrwxrwx 1 root root 11 Mar 16 11:37 dm-name-vg--ssd1-lv--ssd1p1 -> ../../dm-22 lrwxrwxrwx 1 root root 11 Mar 16 11:37 dm-uuid-part1-LVM-n1SH1FvtfjgxJOMWN9aHurFvn2BpIsLZi89GWxA68hLmUQV6l5oyiEOPsFciRbKg -> ../../dm-22~ # ls -al /dev/disk/by-parttypeuuid | grep dm-22lrwxrwxrwx 1 root root 11 Mar 16 11:37 45b0969e-9b03-4f30-b4c6-b4b80ceff106.120c536d-cb30-4cea-b607-dd347022a497 -> ../../dm-22~ # ls -al /dev/disk/by-uuid | grep dm-22 ~ # ls -al /dev/disk/by-partuuid/ | grep dm-22 ~ # ls -al /dev/disk/by-path | grep dm-22 Best Regards, Nicholas Gim.On Wed, Mar 15, 2017 at 6:46 PM Peter Maloney <peter.maloney@xxxxxxxxxxxxxxxxxxxx <mailto:peter.maloney@xxxxxxxxxxxxxxxxxxxx>> wrote:On 03/15/17 08:43, Gunwoo Gim wrote:After a reboot, all the partitions of LVM don't show up in /dev/mapper -nor in the /dev/dm-<dm-num> or /proc/partitions- though the whole disks show up; I have to make the hosts run one 'partprobe' every time they boot so as to have the partitions all show up.Maybe you need this after partprobe: udevadm triggerI've found out that the udev rules have never triggered even when I removed the DEVTYPE checking part; checked with a udev line: RUN+="/bin/echo 'add /dev/$name' >> /root/log.txt" I've also tried chowning all the /dev/dm-<num> to ceph:disk in vain. Do I have to use the udev rules even if the /dev/dm-<num> s are already owned by ceph:ceph?No, I think you just need them owned by ceph:ceph. Test that with something like: sudo -u ceph hexdump -C /dev/dm-${number} | head (which reads, not writes...so not a full test, but close enough) And also make sure the files in /var/lib/ceph/{osd,mon,...} are owned by ceph:ceph too. Maybe you have a mix of root and ceph, which is easy to cause by running it as root when ceph owns some files. And FYI, I don't like udev, and did not use ceph-deploy or ceph-disk. I did it with a very simple init script instead:#!/bin/bash mkdir -p /var/run/ceph chown ceph:ceph /var/run/ceph chgrp -R ceph /var/log/ceph for d in /var/lib/ceph/osd/*/journal; do d=$(readlink -f "$d") chown ceph:ceph "$d" doneThis works on ubuntu 14.04 as is, a badly written init script, but I think centos will not accept it without the lsb tags. A side effect of doing it this way is you have to manually run the script again when replacing or adding disks, since it is not run after hot swap like udev is.Thank you very much for reading. Best Regards, Nicholas. On Wed, Mar 15, 2017 at 1:06 AM Gunwoo Gim <wind8702@xxxxxxxxx <mailto:wind8702@xxxxxxxxx>> wrote: Thank you very much, Peter. I'm sorry for not clarifying the version number; it's kraken and 11.2.0-1xenial. I guess the udev rules in this file are supposed to change them : /lib/udev/rules.d/95-ceph-osd.rules ...but the rules' filters don't seem to match the DEVTYPE part of the prepared partitions on the LVs I've got on the host. Would it have been the cause of trouble? I'd love to be informed of a good way to make it work with the logical volumes; should I fix the udev rule? ~ # cat /lib/udev/rules.d/95-ceph-osd.rules | head -n 19 # OSD_UUID ACTION=="add", SUBSYSTEM=="block", \ ENV{DEVTYPE}=="partition", \ ENV{ID_PART_ENTRY_TYPE}=="4fbd7e29-9d25-41b8-afd0-062c0ceff05d", \ OWNER:="ceph", GROUP:="ceph", MODE:="660", \ RUN+="/usr/sbin/ceph-disk --log-stdout -v trigger /dev/$name" ACTION=="change", SUBSYSTEM=="block", \ ENV{ID_PART_ENTRY_TYPE}=="4fbd7e29-9d25-41b8-afd0-062c0ceff05d", \ OWNER="ceph", GROUP="ceph", MODE="660" # JOURNAL_UUID ACTION=="add", SUBSYSTEM=="block", \ ENV{DEVTYPE}=="partition", \ ENV{ID_PART_ENTRY_TYPE}=="45b0969e-9b03-4f30-b4c6-b4b80ceff106", \ OWNER:="ceph", GROUP:="ceph", MODE:="660", \ RUN+="/usr/sbin/ceph-disk --log-stdout -v trigger /dev/$name" ACTION=="change", SUBSYSTEM=="block", \ ENV{ID_PART_ENTRY_TYPE}=="45b0969e-9b03-4f30-b4c6-b4b80ceff106", \ OWNER="ceph", GROUP="ceph", MODE="660" ~ # udevadm info /dev/mapper/vg--ssd1-lv--ssd1p1 | grep ID_PART_ENTRY_TYPE E: ID_PART_ENTRY_TYPE=45b0969e-9b03-4f30-b4c6-b4b80ceff106 ~ # udevadm info /dev/mapper/vg--ssd1-lv--ssd1p1 | grep DEVTYPE E: DEVTYPE=disk Best Regards, Nicholas.--You can find my PGP public key here: https://google.com/+DewrKim/about <http://google.com/+DewrKim/about>_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
