Hi,
I would like to share some benchmarking numbers on xio messenger and
simple messenger
HW/SW configuration:
---------------------
. 1 32-core Xeon E5-2697V3 2.6G (Haswell) node, 64GB of memory.
. Hyperthreading is ON/enabled, 64 cores
. Mellanox ConnectX3-EN 40Gb/s HCAs, fw- 2.33.5000
. Mellanox SX1012 40Gb/s switch EN.
. Ubuntu 14.04 LTS stock kernel
. MLNX_OFED_LINUX-2.4-1.0.0 sw package
. Accelio master branch (tag v-1.3)
. Ceph master (Jan-29) + pr #3544 (xio spread portals).
. Use ramdisks and filestore backend
. Use fio_rbd on user rbd as client
1 OSD, 1 client node
-------------------------
a. 1 rbd image
xio messenger:
. ~9100 iops (4K random write, 6 cores used on osd node, numjobs=1,
iodepth=64)
. ~21k iops (4K random read, 4 cores used, numjobs=1, iodepth=32)
. ~121k iops (4K random read, 15 cores used, numjobs=8, iodepth=32)
. ~520MB/s (256K random write, 3 cores used, numjobs=1, iodepth=64)
. ~3140MB/s (256K random read, 4 cores used, numjobs=1, iodepth=32)
. ~4330MB/s (256K random read, 6 cores used, numjobs=8)
simple messenger:
. ~8500 iops (4K random write, 7 cores used)
. ~20k iops (4K random read, 5 cores used)
. ~105k iops (4K random read, 20 cores used, numjobs=8, iopdepth=32)
. ~450MB/s (256K random write, 3 cores used)
. ~1140MB/s (256K random read, 3 cores used)
. ~4330MB/s (256K random read, 8 cores used, numjobs=8)
b. 2 rbd images on two separated pools, 2 fio_rbd instances
xio messenger:
. ~9100 iops (4K random write, 6 cores used on osd node, each fio_rbd
instance has numjobs=1, iodepth=64)
. ~155k iops (4K random read, 19 cores used, each fio_rbd instance has
numjobs=8, iodepth=32)
. ~4225MB/s (256K random read, 6 cores used, each fio_rbd instance has
numjobs=1, iodepth=32)
. ~4330MB/s (256K random read, 8 cores used, each fio_rbd instance has
numjobs=8, iodepth=32)
simple messenger:
. ~7800 iops (4K random write, 7 cores used on osd node, each fio_rbd
instance has numjobs=1, iodepth=64)
. ~125k iops (4K random read, 25 cores used, each fio_rbd instance has
numjobs=8, iodepth=32)
. ~2068MB/s (256K random read, 4 cores used, each fio_rbd instance has
numjobs=1, iodepth=32)
. ~4330MB/s (256K random read, 11 cores used, each fio_rbd instance has
numjobs=8, iodepth=32)
2 OSDs, 1 client node, 4 rbd images on 4 separated pools
--------------------------------------------------------------------
4K random read: xio messenger max at ~272k iops, simple messenger max at
~170k iops
4 OSDs, 1 client node, 4 rbd images on 4 separated pools
---------------------------------------------------------------------
4K random read: xio messenger max at ~355k iops, simple messenger max at
~204k iops
8 OSDs, 1 client node, 4 rbd images on 4 separated pools
---------------------------------------------------------------------
4K random read: xio messenger max at ~355K iops, simple messenger max at
~225k iops
I attach here the ceph configuration files that I used.
Please note that I enable flow-control and turn off header_crc &
data_crc for both xio & simple.
Are the simple messenger numbers looking reasonable and in the ballpark?
Please share your numbers and configuration if you have higher numbers.
