On 02/02/2019 05:07, Stuart Longland wrote:
On 1/2/19 10:43 pm, Alfredo Deza wrote:
The tmpfs setup is expected. All persistent data for bluestore OSDs
setup with LVM are stored in LVM metadata. The LVM/udev handler for
bluestore volumes create these tmpfs filesystems on the fly and populate
them with the information from the metadata.
That is mostly what happens. There isn't a dependency on UDEV anymore
(yay), but the reason why files are mounted on tmpfs
is because *bluestore* spits them out on activation, this makes the
path fully ephemeral (a great thing!)
The step-by-step is documented in this summary section of 'activate'
http://docs.ceph.com/docs/master/ceph-volume/lvm/activate/#summary
Filestore doesn't have any of these capabilities and it is why it does
have an actual existing path (vs. tmpfs), and the files come from the
data partition that
gets mounted.
Well, for whatever reason, ceph-osd isn't calling the activate script
before it starts up.
It is worth noting that the systems I'm using do not use systemd out of
simplicity. I might need to write an init script to do that. It wasn't
clear last weekend what commands I needed to run to activate a BlueStore
OSD.
The way you do this on Gentoo is by writing the OSD FSID into
/etc/conf.d/ceph-osd.<id>. You need to make note of the ID when the OSD
is first deployed.
# echo "bluestore_osd_fsid=$(cat /var/lib/ceph/osd/ceph-0/fsid)" >
/etc/conf.d/ceph-osd.0
And then of course do the usual initscript symlink enable on Gentoo:
# ln -s ceph /etc/init.d/ceph-osd.0
# rc-update add ceph-osd.0 default
This will then call `ceph-volume lvm activate` for that OSD for you
before bringing it up, which will populate the tmpfs. It is the Gentoo
OpenRC equivalent of enabling the systemd unit for that osd-fsid on
systemd systems (but ceph-volume won't do it for you).
You may also want to add some dependencies for all OSDs depending on
your setup (e.g. I run single-host and the mon has the OSD dm-crypt
keys, so that has to come first):
# echo 'rc_need="ceph-mon.0"' > /etc/conf.d/ceph-osd
The Gentoo initscript setup for Ceph is unfortunately not very well
documented. I've been meaning to write a blogpost about this to try to
share what I've learned :-)
--
Hector Martin (hector@xxxxxxxxxxxxxx)
Public Key: https://marcan.st/marcan.asc
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
