On 11/6/18 3:31 AM, Hayashida, Mami wrote: > 2018-11-05 12:47:01.075573 7f1f2775ae00 -1 bluestore(/var/lib/ceph/osd/ceph-60) _open_db add block device(/var/lib/ceph/osd/ceph-60/block.db) returned: (13) Permission denied Looks like the permissions on the block.db device are wrong. As far as I know ceph-volume is responsible for setting this at activation time. > I already ran the "ceph-volume lvm activate --all " command right after > I prepared (using "lvm prepare") those OSDs. Do I need to run the > "activate" command again? The activation is required on every boot to create the /var/lib/ceph/osd/* directory, but that should be automatically done by systemd units (since you didn't run it after the reboot and yet the directories exist, it seems to have worked). Can you ls -l the OSD directory (/var/lib/ceph/osd/ceph-60/) and also any devices symlinked to from there, to see the permissions? Then run the activate command again and list the permissions again to see if they have changed, and if they have, try to start the OSD again. I found one Ubuntu bug that suggests there may be a race condition: https://bugs.launchpad.net/bugs/1767087 I get the feeling the ceph-osd activation may be happening before the block.db device is ready, so when it gets created by LVM it's already too late and doesn't have the right permissions. You could fix it with a udev rule (like Ubuntu did) but if this is indeed your issue then it sounds like something that should be fixed in Ceph. Perhaps all you need is a systemd unit override to make sure ceph-volume@* services only start after LVM is ready. A usable udev rule could look like this (e.g. put it in /etc/udev/rules.d/90-lvm-permisions.rules): ACTION=="change", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", \ ENV{DM_LV_NAME}=="db*", ENV{DM_VG_NAME}=="ssd0", \ OWNER="ceph", GROUP="ceph", MODE="660" Reboot after that and see if the OSDs come up without further action. -- Hector Martin (hector@xxxxxxxxxxxxxx) Public Key: https://mrcn.st/pub _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |