Several weeks ago, I was chatting with Mark Nelson and he mentioned that his testing showed that the kclient would plateau out at a much lower throughput than the hardware he was using was capable of. While I don't have any details about the bottleneck he was hitting, I noticed that libceph is heavily serialized on mutexes, the con->mutex, in particular. This patchset is is an attempt to get the con->mutex out of the outgoing message submission codepath. In addition to potentially helping performance, this may also help to close some potential races in the cephfs client. Right now, we have to drop some spinlocks in order to send cap messages, but this may allow us to avoid doing that. My main concern with the set is whether this might open up some races wrt changes to the con->state. I didn't hit any problems like that in testing, but it's possibly an issue. We should be able to work around that if it's a problem though, possibly by adding a new list_head to stage submissions before adding them to the out_queue. I've done some performance testing with the set, but the results were inconclusive. I suspect my home test rig is hitting other bottlenecks before con->mutex contention becomes an issue. It would be nice to do some testing on a setup that allows for higher throughput to see if it helps. Comments and suggestions welcome... Jeff Layton (2): libceph: defer clearing standby state to work function libceph: allow tasks to submit messages without taking con->mutex include/linux/ceph/messenger.h | 2 ++ net/ceph/messenger.c | 48 ++++++++++++++++++++-------------- net/ceph/messenger_v1.c | 35 +++++++++++++------------ net/ceph/messenger_v2.c | 5 ++++ 4 files changed, 54 insertions(+), 36 deletions(-) -- 2.31.1
