Thank you for your response. I was planning to use a 100GbE or 45GbE bond for this cluster. It was acceptable for our use case to lose sequential/larger I/O speed for it. Dual socket would be and option, but I do not want to touch numa, cgroups and the rest settings. Most of the time is just easier to add a higher clock CPU or more cores. The plan is currently for 2xosd per nvme device, but if testing shows that it’s better to use one. We will stick with one. Which RocksDB settings would you recommend tweaking? I haven’t had the chance to test them yet. Most of the clusters I have access to are using leveldb and are still running filestore. On Tue, Mar 5, 2019 at 5:35 PM Mark Nelson <mnelson@xxxxxxxxxx> wrote: > > Hi, > > > I've got a ryzen7 1700 box that I regularly run tests on along with the > upstream community performance test nodes that have Intel Xeon E5-2650v3 > processors in them. The Ryzen is 3.0GHz/3.7GHz turbo while the Xeons > are 2.3GHz/3.0GHz. The Xeons are quite a bit faster clock/clock in the > tests I've done with Ceph. Typically I see a single OSD using fewer > cores on the Xeon processors vs Ryzen to hit similar performance numbers > despite being clocked lower (though I haven't verified the turbo > frequencies of both under load). On the other hand, the Ryzen processor > is significantly cheaper per core. If you only looked at cores you'd > think something like Ryzen would be the way to go, but there are other > things to consider. The number of PCIE lanes, memory configuration, > cache configuration, and CPU interconnect (in multi-socket > configurations) all start becoming really important if you are targeting > multiple NVMe drives like what you are talking about below. The EPYC > processors give you more of all of that, but also costs a lot more than > Ryzen. Ultimately the CPU is only a small part of the price for nodes > like this so I wouldn't skimp if your goal is to maximize IOPS. > > > With 10 NVMe drives per node, I'm guessing that a single EPYC 7451 is > going to be CPU bound for small IO workloads (2.4c/4.8t per OSD), but > will be network bound for large IO workloads unless you are sticking > 2x100GbE in. You might want to consider jumping up to the 7601. That > would get you closer to where you want to be for 10 NVMe drives > (3.2c/6.4t per OSD). Another option might be dual 7351s in this chassis: > > https://www.supermicro.com/Aplus/system/1U/1123/AS-1123US-TN10RT.cfm > > > Figure that with sufficient client parallelism/load you'll get about > 3000-6000 read IOPS/core and about 1500-3000 write IOPS/core (before > replication) with OSDs typically topping out at a max of about 6-8 cores > each. Doubling up OSDs on each NVMe drive might improve or hurt > performance depending on what the limitations are (typically it seems to > help most when the kv sync thread is the primary bottleneck in > bluestore, which most likely happens with tons of slow cores and very > fast NVMe drives). Those are all very rough hand-wavy numbers and > depend on a huge variety of factors so take them with a grain of salt. > Doing things like disabling authentication, disabling logging, forcing > high level P/C states, tweaking RocksDB WAL and compaction settings, the > number of osd shards/threads, and the system numa configuration might > get you higher performance/core, though it's all pretty hard to predict > without outright testing it. > > > Though you didn't ask about it, probably the most important thing you > can spend money on with NVMe drives is getting high write endurance > (DWPD) if you expect even a moderately high write workload. > > > Mark > > > On 3/5/19 3:49 AM, Darius Kasparavičius wrote: > > Hello, > > > > > > I was thinking of using AMD based system for my new nvme based > > cluster. In particular I'm looking at > > https://www.supermicro.com/Aplus/system/1U/1113/AS-1113S-WN10RT.cfm > > and https://www.amd.com/en/products/cpu/amd-epyc-7451 CPU's. Have > > anyone tried running it on this particular hardware? > > > > General idea is 6 nodes with 10 nvme drives and 2 osds per nvme drive. > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |