Hi,
I'm helping someone setup their first Ceph cluster and I'm running into
issues when it comes to provision the OSDs. The cluster is made up of 5
servers with 20 OSD HDDs each and 2 NVME drives. The 2 NVME drives are
set in software raid 1 with a separate partition kept free to house a
journal for the 20 OSDs. Now, Cephadm will not provision the OSDs this
way as far as I can tell. We tried several things and had to settle on
preparing the drives one by one with ceph-volume and then activate them
with the ceph cephadm command:
ceph volume lvm prepare --no-systemd --bluestore --data /dev/sda
--block.db /dev/md2p1
[...]
ceph cephadm osd activate <hostname>
While this command technically works, when I activate all the OSDs at
once, I end up with only the last created OSD having access to the
journal and all the other OSD containers failing to boot up. I'm
guessing the last created OSD is locking the journal and none of the
other OSDs can access it. Is there a way I can make this work? I thought
about splitting the free space on the NVMEs into 20 different
partitions, but that feels ridiculous and overkill.
Additionally, will we really gain any kind of performance from the
journal being on NVMEs? I know the recommended value in this case would
be 6 HDDs per NVME so I feel like I'm just creating a chokepoint that'll
slow down the cluster instead of speeding it up, despite the major IOPS
difference between HDDs and NVMEs.
--
Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
