Hi Mark:
Base on 1f5d75f31aa1a7b4,
IOPS 4K RW 4KRR
Async 144450 612716
Simple 111187 414672
Async use the default value.
My cluster: 4 node, 16 osd(ssd + nvme(store rocksdb/wal). For test use fio+librbd.
But the results are opposite.
Thanks!
-----Original Message-----
From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Mark Nelson
Sent: Thursday, September 22, 2016 2:50 AM
To: ceph-devel <ceph-devel@xxxxxxxxxxxxxxx>
Subject: async messenger random read performance on NVMe
Recently in master we made async messenger default. After doing a bunch of bisection, it turns out that this caused a fairly dramatic decrease in bluestore random read performance. This is on a cluster with fairly fast NVMe cards, 16 OSDs across 4 OSD hosts. There are 8 fio client processes with 32 concurrent threads each.
Ceph master using bluestore
Parameters tweaked:
ms_async_send_inline
ms_async_op_threads
ms_async_max_op_threads
simple: 168K IOPS
send_inline: true
async 3/5 threads: 111K IOPS
async 4/8 threads: 125K IOPS
async 8/16 threads: 128K IOPS
async 16/32 threads: 128K IOPS
async 24/48 threads: 128K IOPS
async 25/50 threads: segfault
async 26/52 threads: segfault
async 32/64 threads: segfault
send_inline: false
async 3/5 threads: 153K IOPS
async 4/8 threads: 153K IOPS
async 8/16 threads: 152K IOPS
So definitely setting send_inline to false helps pretty dramatically, though we're still a little slower for small random reads than simple messenger. Haomai, regarding the segfaults, I took a quick look with gdb at the core file but didn't see anything immediately obvious. It might be worth seeing if you can reproduce.
On the performance front, I'll try to see if I can see anything obvious in perf.
Mark
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
