On Wed, Jan 22, 2020 at 12:00:28PM -0500, Wesley Dillingham wrote: > After upgrading to Nautilus 14.2.6 from Luminous 12.2.12 we are seeing > the following behavior on OSDs which were created with "ceph-volume lvm > create --filestore --osd-id <osd> --data <device> --journal <journal>" > Upon restart of the server containing these OSDs they fail to start > with the following error in the logs: >2020-01-21 13:36:11.635 7fee633e8a80 -1 filestore(/var/lib/ceph/osd/ceph-199) mo >unt(1928): failed to open journal /var/lib/ceph/osd/ceph-199/journal: (13) Permi >ssion denied > > /var/lib/ceph/osd/ceph-199/journal symlinks to /dev/sdc5 in our case > and inspecting the ownership on /dev/sdc5 it is root:root, chowning > that to ceph:ceph causes the osd to start and come back up and in near > instantly. > As a note these OSDs we experience this with are OSDs which have > previously failed and been replaced using the above ceph-volume, longer > running OSDs in the same server created with ceph-disk or ceph-volume > simple (that have a corresponding .json in /etc/ceph/osd) start up fine > and get ceph:ceph on their journal partition. Bluestore OSDs also do > not have any issue. > My hope is that I can preemptively fix these OSDs before shutting them > down so that reboots happen seamlessly. Thanks for any insight. ceph-volume is supposed to take care of this via the ceph-volume@ systemd unit. This is a one shot unit, that should set things up and then start the osd. The unit name is a bit convoluted: ceph-volume@<osd-id>-<osd-uuid>, there should be symbolic link in /etc/systemd/system/multi-user.target.wants/ You can also check cat /var/log/ceph/ceph-volume-systemd.log for any errors. Feel free to open a tracker ticket on https://tracker.ceph.com/projects/ceph-volume > > Respectfully, > Wes Dillingham > [1]wes@xxxxxxxxxxxxxxxxx > [2]LinkedIn > >References > > 1. mailto:wes@xxxxxxxxxxxxxxxxx > 2. http://www.linkedin.com/in/wesleydillingham >_______________________________________________ >ceph-users mailing list -- ceph-users@xxxxxxx >To unsubscribe send an email to ceph-users-leave@xxxxxxx -- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |