Hi folks, i am in the unfortunate situation that i do not have enough or big enough NVMe devices available to acommodate DB+WAL for my spinning rust-based OSDs. In a few words, i have three hosts with 20 5.5T HDDs each and a 1.6TB PCIe NVMe. When sizing this and putting together the BoM, i had the old-style level mechanism in mind which meant L0+L1 w/ default settings results in ~ 30G per OSD for DB (and WAL as far as i understand, we do not plan to have a separate WAL device). Thus, 32G per DB+WAL would result in 32*20 = 640G of capacity needed on the NVMe, should be fine. Our workload will almost exclusively be RBD, with one CephFS for ISO images. No RGW AFAIK. Now, i want to understand if we can still go with ~ 1% of data device capacity, or if we need to provision more. If we cannot provision more, what would be the impact in that case? Is RBD fine and CephFS would suffer a lot? 5.5T * 1% would result in ~ 60G * 20 (or later 24) = 1200G (1440G) usage on the NVMe for DB+WAL For all my calculations, i am putting my faith in the official Ceph documentation: "The general recommendation is to have block.db size in between 1% to 4% of block size. For RGW workloads, it is recommended that the block.db size isn’t smaller than 4% of block, because RGW heavily uses it to store metadata (omap keys). For example, if the block size is 1TB, then block.db shouldn’t be less than 40GB. For RBD workloads, 1% to 2% of block size is usually enough." However, Proxmox apparently recommends 10% of the data device size to provision for DB+WAL. Obviously, 4% and even more 10% would quadruple the cost. We don’t want to achieve stellar performance, just “good-enough” for ~ 200 VMs that currently reside on redundant Datacore FC storage backed by FC RAID (N*16G, 24x 7.2k). I expect commit latency for Ceph with QD=1 to be ~ 3x compared with the old solution, but all other parameters would probably be a lot better. Cheers, Gregor _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |