On Mon, 2022-03-14 at 21:35 +0000, Chuck Lever III wrote: > Hey Trond- > > I've made some progress getting RPC-with-TLS working in > the Linux NFS client, but I recently hit an interesting > snag and could use a little advice. > > The software kTLS infrastructure uses do_tcp_sendpages() > under the covers, and that function is clearing > SOCKWQ_ASYNC_NOSPACE from under xs_nospace(). That > prevents xs_run_error_worker() from waking up xprt->sending, > stalling an RPC transport waiting for more socket write > space. I'm not sure how to address this, and I'm interested > in your opinion. > How is it achieving this? We only set SOCKWQ_ASYNC_NOSPACE after the call to xprt_sock_sendmsg(). > For example, why check that flag rather than just waking > up xprt->sending unconditionally? The socket code calls ->write_space() in all sorts of situations, so we need to distinguish between the cases where we are actually waiting for buffer memory, and the situations where we are not. Otherwise, we'd be calling xs_run_error_worker() all the time. > > Also just for my own understanding of how the write_space > mechanism is supposed to work for RPC, I instrumented the > code that bumps and decrements sk_write_pending, and found > that under normal workloads, the value of that field goes > negative and stays there. I'm not sure that's intended...? > It is not intended, no. Looks like there have been various refactorings that have mangled that code. From what I can see, the socket code assumes that sk_write_pending should always be bumped with the sock_lock() held. Let me write a fix for that... -- Trond Myklebust Linux NFS client maintainer, Hammerspace trond.myklebust@xxxxxxxxxxxxxxx
