Responding partially to my own query, I have decided on the following
structure, in order to have encrypted OSDs/Bluestore journals and not wait
for proper ceph-volume support.
1) SSD(s), fully encrypted, acting as PV(s) for VG(s) to store LVs for the
Block DBs. My current setup is 1 SSD for 4 Block DBs. The names of the VGs
or the LVs can be chosen freely. My VG name is "bluestore"
2) Each of my SATA disks gets encrypted fully by me, then I build a
PV/VG/LV for all the space.
3) For all these encrypted disks, I create a keyfile and auto-open them on
reboot.
4) I create the OSDs with a string such as:
ceph-volume lvm prepare --bluestore --block.db bluestore/ceph04bluf --data
ceph04vgf/ceph04lvf
(as mentioned, the VG/LV names are yours to choose)
This way proper LVM tags are placed on all block devices.
If for some reason the OSDs are not properly activated (as in my case) or
the /var/lib/ceph/osd/ceph-<id> directories not populated, use the
following script:
#!/bin/bash
# the assumption is that all of my OSD block VGs are in the form of
/dev/`hostname`vg<id>
#you can change this string to whatever you like
for mylv in $( ls /dev/`hostname`vg*/* ); do
# check whether we have ceph tags, as I might have created an LV but
not yet provisioned it
lvs -o lv_tags $mylv | grep ceph >/dev/null || continue
# I have decided to eval the whole lvm tags to variables prepended by
my, for future use
eval `lvs -o lv_tags $mylv |tail -n1|sed -e 's/^[ ]*//g;s/,/\n/g'|cut
-d '.' -f2-|sed -e 's/^/my/g'`
ceph-volume lvm activate --bluestore $myosd_id $myosd_fsid
done
On Fri, Dec 4, 2020 at 5:37 PM Panayiotis Gotsis <panos.gotsis@xxxxxxxxx>
wrote:
> Hello,
>
> I have made some tests with creating OSDs and I have found out that there
> are big issues with the ceph-volume functionality.
>
> 1) If using dmcrypt and separate data and db block devices, ceph-volume
> creates cryprodevs/PVs/VGs/LVs for both devices. This might seem as
> normal, until one considers the possibility that a single SSD will support
> multiple OSDs and the whole concept of being able to resize bluestore DB
> partitions gets very complicated like this; I can have a full SSD
> encrypted, create a PV/VG on top and provide the LVs myself without an
> extra layer of encryption.
>
> 2) If I try to circumvent this limitation by asking ceph-volume to use
> already encrypted devices for both the data and the bluestore db, the OSDs
> are not auto scanned at startup, ceph-volume simple scan complaints that
> they are not real devices and nothing starts.
>
> Am I missing something?
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
