Hi Darren, >> I'm going to deploy a new cluster soon based on 6.4TB NVME PCI-E Cards, I will have only 1 NVME card per node and 38 nodes. >> >> The use case is to offer cephfs volumes for a k8s platform, I plan to use an EC-POOL 8+3 for the cephfs_data pool. >> >> Do you have recommendations for the setup or mistakes to avoid? I use ceph-ansible to deploy all myclusters. As describe in another mail, the context is: 3 Intel Server 1U for MONs/MDSs/MGRs services + K8s daemons CPU : 2 x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (24c/48t) Memory : 64GB Disk OS : 2x Intel SSD DC S3520 240GB Network : SFP+ 10Gb/s 38 Dell C4140 1U for OSD node : CPU : 2 x Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz (28c/56t) Memory : 384GB GPU : 4 Nvidia V100 32GB NVLink Disk OS : M.2 240G NVME : Dell 6.4TB NVME PCI-E Drive (Samsung PM1725b), only 1 slot available Network : 2x SFP+ 10Gb/s (1 NIC for public network and 1 NIC for private Network) Each server is used in a k8s cluster to give access to GPUs and CPUs for X-learning labs. Ceph have to share the CPU and memory with the K8s cluster. I plan to have only 1 filesystem > So I would not be putting 1 x 6.4TB device in but multiple smaller devices in each node. Because I don't have place for more > What CPU are you thinking of using? > How many CPU's? If you have 1 PCIe card then it will only be connected to 1 CPU so will you be able to use all of the performance of multiple CPU's. We plan to use two Xeon Gold 6132. > What network are you thinking of? Wouldn't be doing less than 100G or multiple 25G connections. Unfortunately, we are not ready for 25Gb/s neither 100Gb/s yet, "only" 10Gbits/s fat tree network. > Where is you CephFS metadata being stored? What are you doing about CephFS metadata servers? The CephFS metadata will be on separate servers as describe above. > What about some faster storage for the WAL? Dell 6.4TB NVME PCI-E Drives (Samsung PM1725b) look fast for me, don't they? > What is your IO Profile? Read/Write split? IO Profile will be mixed cases, this cephfs will be used as scratch storage to compute processes in our K8s clusters. > It may be the case that EC is not the best fit for the workload you are trying to do. As usual, we have to choose between performance and usable space... Thanks, Best regards, -- Yoann Moulin EPFL IC-IT _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |