Previously, when clients' sustained offered write load exceeded the
sustained throughput of the OSDs, normal operation was that client
messages timed out while waiting to be processed by the OSDs. The
client response to this was to reset the connection to the OSD
handling a timed-out message.
That has at least two types of impact:
- the reset frequently happens while data is being sent, so data
that was successfully received must be discarded and resent.
- after several such connection resets, many sockets can remain open,
waiting for readers to be granted space by the policy throttler,
so that they can notice that the pipe has been shut down, and the
socket can be closed.
This patchset causes Ceph OSDs to send keepalives when waiting for
sufficient buffer space to receive a message from a client. There is
also a companion kernel client patch that causes clients to notice
the keepalives, and not reset a connection serving a timed-out
message if anything, particularly a keepalive, has been received
recently.
This patchset also has the operational impact of eliminating client log
messages about resetting OSDs under normal operation with a heavy write
load, which makes it easier to notice other issues in the client logs.
Jim Schutt (3):
common/Throttle: Remove unused return type on Throttle::get()
common/Throttle: Add timed_wait().
msgr: Send keepalive periodically when waiting in policy throttler
src/common/Throttle.h | 45 +++++++++++++++++++++++++++++++++++++++++--
src/common/config.cc | 1 +
src/common/config.h | 1 +
src/msg/SimpleMessenger.cc | 6 ++++-
4 files changed, 49 insertions(+), 4 deletions(-)
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
