Hi, I would like to share some benchmarking numbers on xio messenger and simple messenger HW/SW configuration: --------------------- . 1 32-core Xeon E5-2697V3 2.6G (Haswell) node, 64GB of memory. . Hyperthreading is ON/enabled, 64 cores . Mellanox ConnectX3-EN 40Gb/s HCAs, fw- 2.33.5000 . Mellanox SX1012 40Gb/s switch EN. . Ubuntu 14.04 LTS stock kernel . MLNX_OFED_LINUX-2.4-1.0.0 sw package . Accelio master branch (tag v-1.3) . Ceph master (Jan-29) + pr #3544 (xio spread portals). . Use ramdisks and filestore backend . Use fio_rbd on user rbd as client 1 OSD, 1 client node ------------------------- a. 1 rbd image xio messenger: . ~9100 iops (4K random write, 6 cores used on osd node, numjobs=1, iodepth=64) . ~21k iops (4K random read, 4 cores used, numjobs=1, iodepth=32) . ~121k iops (4K random read, 15 cores used, numjobs=8, iodepth=32) . ~520MB/s (256K random write, 3 cores used, numjobs=1, iodepth=64) . ~3140MB/s (256K random read, 4 cores used, numjobs=1, iodepth=32) . ~4330MB/s (256K random read, 6 cores used, numjobs=8) simple messenger: . ~8500 iops (4K random write, 7 cores used) . ~20k iops (4K random read, 5 cores used) . ~105k iops (4K random read, 20 cores used, numjobs=8, iopdepth=32) . ~450MB/s (256K random write, 3 cores used) . ~1140MB/s (256K random read, 3 cores used) . ~4330MB/s (256K random read, 8 cores used, numjobs=8) b. 2 rbd images on two separated pools, 2 fio_rbd instances xio messenger: . ~9100 iops (4K random write, 6 cores used on osd node, each fio_rbd instance has numjobs=1, iodepth=64) . ~155k iops (4K random read, 19 cores used, each fio_rbd instance has numjobs=8, iodepth=32) . ~4225MB/s (256K random read, 6 cores used, each fio_rbd instance has numjobs=1, iodepth=32) . ~4330MB/s (256K random read, 8 cores used, each fio_rbd instance has numjobs=8, iodepth=32) simple messenger: . ~7800 iops (4K random write, 7 cores used on osd node, each fio_rbd instance has numjobs=1, iodepth=64) . ~125k iops (4K random read, 25 cores used, each fio_rbd instance has numjobs=8, iodepth=32) . ~2068MB/s (256K random read, 4 cores used, each fio_rbd instance has numjobs=1, iodepth=32) . ~4330MB/s (256K random read, 11 cores used, each fio_rbd instance has numjobs=8, iodepth=32) 2 OSDs, 1 client node, 4 rbd images on 4 separated pools -------------------------------------------------------------------- 4K random read: xio messenger max at ~272k iops, simple messenger max at ~170k iops 4 OSDs, 1 client node, 4 rbd images on 4 separated pools --------------------------------------------------------------------- 4K random read: xio messenger max at ~355k iops, simple messenger max at ~204k iops 8 OSDs, 1 client node, 4 rbd images on 4 separated pools --------------------------------------------------------------------- 4K random read: xio messenger max at ~355K iops, simple messenger max at ~225k iops I attach here the ceph configuration files that I used. Please note that I enable flow-control and turn off header_crc & data_crc for both xio & simple. Are the simple messenger numbers looking reasonable and in the ballpark? Please share your numbers and configuration if you have higher numbers. thanks, -vu
Attachment:
ceph_perf_simple_1osd.conf
Description: ceph_perf_simple_1osd.conf
Attachment:
ceph_perf_xio_1osd.conf
Description: ceph_perf_xio_1osd.conf
