On 2020-01-10 17:18, kefu chai wrote:
[skip]
First thing that catches my eye is that for small blocks there is no
big
difference at all, but as the block increases, crimsons iops starts to
that's also our findings. and it's expected. as async messenger uses
the same reactor model as seastar does. actually its original
implementation was adapted from seastar's socket stream
implementation.
Hm, regardless of model messenger should not be a bottleneck. Take
a look on the results of fio_ceph_messenger load (runs pure messenger),
I can squeeze IOPS=89.8k, BW=351MiB/s on 4k block size, iodepth=32.
(also good example https://github.com/ceph/ceph/pull/26932 , almost
~200k)
With PG layer (memstore_debug_omit_block_device_write=true option)
I can reach 40k iops max. Without PG layer (immediate completion
from the transport callback, osd_immediate_completions=true)
I get almost 60k.
Seems that here starts playing costs on client side and these costs
prevail.
decline. Can it be the transport issue? Can be tested as well.
because seastar's socket facility reads from the wire with 4K chunk
size, while classic OSD's async messenger reads the payload with the
size suggested by the header. so when it comes to larger block size,
it takes crimson-osd multiple syscalls and memcpy calls to read the
request from wire, that's why classic OSD wins in this case.
Do you plan to fix that?
have you tried to use multiple fio clients to saturate CPU capacity of
OSD nodes?
Not yet. But regarding CPU I have these numbers:
output of pidstat while rbd.fio is running, 4k block only:
legacy-osd
[roman@dell ~]$ pidstat 1 -p 109930
Linux 5.3.13-arch1-1 (dell) 01/09/2020 _x86_64_ (8 CPU)
03:51:49 PM UID PID %usr %system %guest %wait %CPU
CPU Command
03:51:51 PM 1000 109930 14.00 8.00 0.00 0.00 22.00
1 ceph-osd
03:51:52 PM 1000 109930 40.00 19.00 0.00 0.00 59.00
1 ceph-osd
03:51:53 PM 1000 109930 44.00 17.00 0.00 0.00 61.00
1 ceph-osd
03:51:54 PM 1000 109930 40.00 20.00 0.00 0.00 60.00
1 ceph-osd
03:51:55 PM 1000 109930 39.00 18.00 0.00 0.00 57.00
1 ceph-osd
03:51:56 PM 1000 109930 41.00 20.00 0.00 0.00 61.00
1 ceph-osd
03:51:57 PM 1000 109930 41.00 15.00 0.00 0.00 56.00
1 ceph-osd
03:51:58 PM 1000 109930 42.00 16.00 0.00 0.00 58.00
1 ceph-osd
03:51:59 PM 1000 109930 42.00 15.00 0.00 0.00 57.00
1 ceph-osd
03:52:00 PM 1000 109930 43.00 15.00 0.00 0.00 58.00
1 ceph-osd
03:52:01 PM 1000 109930 24.00 12.00 0.00 0.00 36.00
1 ceph-osd
crimson-osd
[roman@dell ~]$ pidstat 1 -p 108141
Linux 5.3.13-arch1-1 (dell) 01/09/2020 _x86_64_ (8 CPU)
03:47:50 PM UID PID %usr %system %guest %wait %CPU
CPU Command
03:47:55 PM 1000 108141 67.00 11.00 0.00 0.00 78.00
0 crimson-osd
03:47:56 PM 1000 108141 79.00 12.00 0.00 0.00 91.00
0 crimson-osd
03:47:57 PM 1000 108141 81.00 9.00 0.00 0.00 90.00
0 crimson-osd
03:47:58 PM 1000 108141 78.00 12.00 0.00 0.00 90.00
0 crimson-osd
03:47:59 PM 1000 108141 78.00 12.00 0.00 1.00 90.00
0 crimson-osd
03:48:00 PM 1000 108141 78.00 13.00 0.00 0.00 91.00
0 crimson-osd
03:48:01 PM 1000 108141 79.00 13.00 0.00 0.00 92.00
0 crimson-osd
03:48:02 PM 1000 108141 78.00 12.00 0.00 0.00 90.00
0 crimson-osd
03:48:03 PM 1000 108141 77.00 11.00 0.00 0.00 88.00
0 crimson-osd
03:48:04 PM 1000 108141 79.00 12.00 0.00 1.00 91.00
0 crimson-osd
Seems quite saturated, almost twice more than legacy-osd. Did you see
something
similar?
--
Roman
