Hi We have 20 ceph node, each with 12 x 18Tb, 2 x nvme 1Tb I try this method to create osd ceph orch apply -i osd_spec.yaml with this conf osd_spec.yaml service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: rotational: 1 db_devices: paths: - /dev/nvme0n1 - /dev/nvme1n1 this created 6 osd with wal/db on /dev/nvme0n1 and 6 on /dev/nvme1n1 per node but when I do a lvs, I see only 6 x 80Go partitions on each nvme... I think this is dynamic sizing, but I'm not sure, I don't know how to check it... Our cluster will only host couple of files, a small one and a big one ~2GB for cephfs only use, and with only 8 users accessing datas I don't know if this is optimum, we are in testing process... ----- Mail original ----- > De: "Stefan Kooman" <stefan@xxxxxx> > À: "Jake Grimmett" <jog@xxxxxxxxxxxxxxxxx>, "Christian Wuerdig" <christian.wuerdig@xxxxxxxxx>, "Satish Patel" > <satish.txt@xxxxxxxxx> > Cc: "ceph-users" <ceph-users@xxxxxxx> > Envoyé: Lundi 20 Juin 2022 16:59:58 > Objet: Re: Suggestion to build ceph storage > On 6/20/22 16:47, Jake Grimmett wrote: >> Hi Stefan >> >> We use cephfs for our 7200CPU/224GPU HPC cluster, for our use-case >> (large-ish image files) it works well. >> >> We have 36 ceph nodes, each with 12 x 12TB HDD, 2 x 1.92TB NVMe, plus a >> 240GB System disk. Four dedicated nodes have NVMe for metadata pool, and >> provide mon,mgr and MDS service. >> >> I'm not sure you need 4% of OSD for wal/db, search this mailing list >> archive for a definitive answer, but my personal notes are as follows: >> >> "If you expect lots of small files: go for a DB that's > ~300 GB >> For mostly large files you are probably fine with a 60 GB DB. >> 266 GB is the same as 60 GB, due to the way the cache multiplies at each >> level, spills over during compaction." > > There is (experimental ...) support for dynamic sizing in Pacific [1]. > Not sure if it's stable yet in Quincy. > > Gr. Stefan > > [1]: > https://docs.ceph.com/en/quincy/rados/configuration/bluestore-config-ref/#sizing > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx -- Christophe BAILLON Mobile :: +336 16 400 522 Work :: https://eyona.com Twitter :: https://twitter.com/ctof _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx