On 3/5/19 10:20 AM, Darius Kasparavičius wrote:
Thank you for your response.
I was planning to use a 100GbE or 45GbE bond for this cluster. It was
acceptable for our use case to lose sequential/larger I/O speed for
it. Dual socket would be and option, but I do not want to touch numa,
cgroups and the rest settings. Most of the time is just easier to add
a higher clock CPU or more cores. The plan is currently for 2xosd per
nvme device, but if testing shows that it’s better to use one. We will
stick with one. Which RocksDB settings would you recommend tweaking? I
haven’t had the chance to test them yet. Most of the clusters I have
access to are using leveldb and are still running filestore.
Yeah, numa makes everything more complicated. I'd just consider jumping
up to the 7601 then if IOPS is a concern and know that you might still
be CPU bound (though it's also possible you could also hit some other
bottleneck before it becomes an issue). Given that the cores aren't
clocked super high it's possible that you might see a benefit to 2x
OSDs/device.
RocksDB is tough. Right now we are heavily tuned to favor reducing
write amplification but eat CPU to do it. That can help performance
when write throughput is a bottleneck and also reduces wear on the drive
(which is always good, but especially with low write endurance drives).
Reducing the size of the WAL buffers will (probably) reduce CPU usage
and also reduce the amount of memory used by the OSD, but we've observed
higher write-amplification on our test nodes. I suspect that might be a
worthwhile trade-off for nvdimms or optane, but I'm not sure it's a good
idea for typical NVMe drives.
Mark
On Tue, Mar 5, 2019 at 5:35 PM Mark Nelson <mnelson@xxxxxxxxxx> wrote:
Hi,
I've got a ryzen7 1700 box that I regularly run tests on along with the
upstream community performance test nodes that have Intel Xeon E5-2650v3
processors in them. The Ryzen is 3.0GHz/3.7GHz turbo while the Xeons
are 2.3GHz/3.0GHz. The Xeons are quite a bit faster clock/clock in the
tests I've done with Ceph. Typically I see a single OSD using fewer
cores on the Xeon processors vs Ryzen to hit similar performance numbers
despite being clocked lower (though I haven't verified the turbo
frequencies of both under load). On the other hand, the Ryzen processor
is significantly cheaper per core. If you only looked at cores you'd
think something like Ryzen would be the way to go, but there are other
things to consider. The number of PCIE lanes, memory configuration,
cache configuration, and CPU interconnect (in multi-socket
configurations) all start becoming really important if you are targeting
multiple NVMe drives like what you are talking about below. The EPYC
processors give you more of all of that, but also costs a lot more than
Ryzen. Ultimately the CPU is only a small part of the price for nodes
like this so I wouldn't skimp if your goal is to maximize IOPS.
With 10 NVMe drives per node, I'm guessing that a single EPYC 7451 is
going to be CPU bound for small IO workloads (2.4c/4.8t per OSD), but
will be network bound for large IO workloads unless you are sticking
2x100GbE in. You might want to consider jumping up to the 7601. That
would get you closer to where you want to be for 10 NVMe drives
(3.2c/6.4t per OSD). Another option might be dual 7351s in this chassis:
https://www.supermicro.com/Aplus/system/1U/1123/AS-1123US-TN10RT.cfm
Figure that with sufficient client parallelism/load you'll get about
3000-6000 read IOPS/core and about 1500-3000 write IOPS/core (before
replication) with OSDs typically topping out at a max of about 6-8 cores
each. Doubling up OSDs on each NVMe drive might improve or hurt
performance depending on what the limitations are (typically it seems to
help most when the kv sync thread is the primary bottleneck in
bluestore, which most likely happens with tons of slow cores and very
fast NVMe drives). Those are all very rough hand-wavy numbers and
depend on a huge variety of factors so take them with a grain of salt.
Doing things like disabling authentication, disabling logging, forcing
high level P/C states, tweaking RocksDB WAL and compaction settings, the
number of osd shards/threads, and the system numa configuration might
get you higher performance/core, though it's all pretty hard to predict
without outright testing it.
Though you didn't ask about it, probably the most important thing you
can spend money on with NVMe drives is getting high write endurance
(DWPD) if you expect even a moderately high write workload.
Mark
On 3/5/19 3:49 AM, Darius Kasparavičius wrote:
Hello,
I was thinking of using AMD based system for my new nvme based
cluster. In particular I'm looking at
https://www.supermicro.com/Aplus/system/1U/1113/AS-1113S-WN10RT.cfm
and https://www.amd.com/en/products/cpu/amd-epyc-7451 CPU's. Have
anyone tried running it on this particular hardware?
General idea is 6 nodes with 10 nvme drives and 2 osds per nvme drive.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
