This picks up a discussion we had at bakeathon - I'll try to summarize quickly. There's been new reports of TEST_STATEID storms where clients spend all of their cpu and network resources sending TEST_STATEID. In network captures, we see both SEQ4_STATUS_RECALLABLE_STATE_REVOKED and SEQ4_STATUS_CB_PATH_DOWN. Now we can see that the NFS server really is seeing the callback channel drop, and we see -ERESTARTSYS from nfsd_cb_done and -EINVAL from nfs_cb_setup_err. I think the server may be spuriously shutting down the callback rpc_client, which does rpc_killall_tasks for any pending callbacks. I started playing with the upstream client, and noticed that if the client is idle with nconnect > 1, the XS_IDLE_DISC_TO (5 minutes) can take down the connection with the callback channel for v4.1. We recently prioritized this first connection, perhaps we can disable the idle timeout for it. There's some weird behavior for nconnect=16, we only get 12 connections at first, then my client usually only primes 5 of them with a SEQUENCE within the next 5 minutes, and tears down the callback connection, then re-connects all 16 again. This whole situation makes delegations a huge net loss in this setup. Can anyone remember why we wanted XS_IDLE_DISC_TO back in the single-connection TCP days? Ben
