On 11/21/22 11:34, Sven Kieske wrote:
On Fr, 2022-11-11 at 10:11 +0100, huxiaoyu@xxxxxxxxxxxx wrote:
Thanks a lot for your insightful blogs on Ceph performance. It is really very informative and interesting.
When i read Ceph OSD CPU Scaling, i am wondering by which way you scale CPU cores per OSD, in other words, how do you allocate a specified num of cores for OSDs? I am interested to repeat some of
your tests on hybrid NVMe/HDD environments.
I would like to second this question, as I wasn't able to find anything in the documentation so far.
did you just alter the ceph osd service files to taskset the osd process to specific nodes or are there configuration
switches, which I didn't yet find, to tell ceph how many cores per OSD should be used?
Hi Folks,
Sorry I got hit with a customer case and then was sick last week. I
used numactl to control which "processors" (as shown in /proc/cpuinfo)
the OSDs would be assigned by wrapping the osd processes. For example:
numactl --physcpubind=0-11 /usr/local/bin/ceph-osd
That would tell the osd process to run on the first 12 processors.
You can use the "P" field in top -H to see which processors threads are
running on for verification. In my case I'm also using a single socket
system which makes things a little bit easier, but if you are using a
dual socket system you might need to be careful about whether or not
memory is being used from the same numa node as the cores (might make
sense to manually bind memory to the same socket as the cores you are
using).
One point I will make is that I didn't make any specific effort to
ensure that "real" and HT processors on the same physical core were
paired. I.e. when utilizing HT processors it might make sense to do
something like:
numactl --physcpubind=0-1,64-65 /usr/local/bin/ceph-osd
numactl --physcpubind=2-3,66-67 /usr/local/bin/ceph-osd
etc.
I probably should have tried that as well to see if it made any
difference but didn't think of it at the time. I'm also not 100% sure
on how container environments handle CPU restrictions like this. The
approach I used may not be a perfect representation of what we see in
real life deployments, but I do suspect the general trends will look
pretty similar.
Oh, I'm not using any of the osd service files to start up OSDs so I'm
not sure what you might need to do on that side.
Mark
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
