On Thu, Nov 17, 2016 at 02:58:12PM -0500, Olga Kornievskaia wrote: > On Thu, Nov 17, 2016 at 2:32 PM, bfields@xxxxxxxxxxxx > <bfields@xxxxxxxxxxxx> wrote: > > On Thu, Nov 17, 2016 at 05:45:52PM +0000, Trond Myklebust wrote: > >> On Thu, 2016-11-17 at 11:31 -0500, J. Bruce Fields wrote: > >> > On Wed, Nov 16, 2016 at 02:55:05PM -0600, Jason L Tibbitts III wrote: > >> > > > >> > > I'm replying to a rather old message, but the issue has just now > >> > > popped > >> > > back up again. > >> > > > >> > > To recap, a client stops being able to access _any_ mount on a > >> > > particular server, and "NFS: nfs4_reclaim_open_state: Lock reclaim > >> > > failed!" appears several hundred times per second in the kernel > >> > > log. > >> > > The load goes up by one for ever process attempting to access any > >> > > mount > >> > > from that particular server. Mounts to other servers are fine, and > >> > > other clients can mount things from that one server without > >> > > problems. > >> > > > >> > > When I kill every process keeping that particular mount active and > >> > > then > >> > > umount it, I see: > >> > > > >> > > NFS: nfs4_reclaim_open_state: unhandled error -10068 > >> > > >> > NFS4ERR_RETRY_UNCACHED_REP. > >> > > >> > So, you're using NFSv4.1 or 4.2, and the server thinks that the > >> > client > >> > has reused a (slot, sequence number) pair, but the server doesn't > >> > have a > >> > cached response to return. > >> > > >> > Hard to know how that happened, and it's not shown in the below. > >> > Sounds like a bug, though. > >> > >> ...or a Ctrl-C.... > > > > How does that happen? > > > > If I may chime in... > > Bruce, when an application sends a Ctrl-C and clients's session slot > has sent out an RPC but didn't process the reply, the client doesn't > know if the server processed that sequence id or not. In that case, > the client doesn't increment the sequence number. Normally the client > would handle getting such an error by retrying again (and resetting > the slots) but I think during recovery operation the client handles > errors differently (by just erroring). I believe the reasoning that we > don't want to be stuck trying to recover from the recovery from the > recovery etc... So in that case the client can end up sending a different rpc reusing the old slot and sequence number? --b. > > Jason, > > The UNCACHED_REP error is really not interesting as it's a consequence > of you having a client that already failed with an error of "unable to > reclaim the locks". I'm surprised that the application doesn't error > at this point with EIO. But that aside, I think I've seen this kind of > behavior due to client't callback channel going down (and not replying > to the CB_RECALLs and then server revoking state). > > > > > --b. > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > > the body of a message to majordomo@xxxxxxxxxxxxxxx > > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
