On Wed, 24 Jun 2020 at 09:54, James Pearson <jcpearson@xxxxxxxxx> wrote: > > On Wed, 10 Jun 2020 at 17:24, Robert Milkowski <rmilkowski@xxxxxxxxx> wrote: > > > > I was hoping someone here might get back with "hey, this has been fixed by > > commit...". > > We also did see it on centos 7.6 > > > > I will try to get it re-produce it and once I can then I'll try to reproduce > > it against upstream. > > I haven't been monitoring this list recently - but we are still having > the same problem with CentOS 7.7 clients and Isilon filers > > I've just 'fixed' one client with the issue in the way you described > earlier in this thread by looking for a process in nfs4_state_manager > via a stack trace - killing that pid with -9 and all the stuck mount > pops back to life I have tcpdumps of when this RENEW/NFSERR_STALE_CLIENTID loop starts taken on different clients - and all show a similar behaviour - from one of these tcpdumps: 27803 2020-06-23 13:40:36 10.78.202.217 -> 10.78.196.220 NFS 194 V4 Call RENEW CID: 0x3091 27804 2020-06-23 13:40:36 10.78.196.220 -> 10.78.202.217 NFS 114 V4 Reply (Call In 27803) RENEW 27815 2020-06-23 13:41:06 10.78.202.217 -> 10.78.196.220 NFS 194 V4 Call RENEW CID: 0x3091 27816 2020-06-23 13:41:06 10.78.196.220 -> 10.78.202.217 NFS 114 V4 Reply (Call In 27815) RENEW 27827 2020-06-23 13:41:36 10.78.202.217 -> 10.78.196.220 NFS 194 V4 Call RENEW CID: 0x3091 27828 2020-06-23 13:41:36 10.78.196.220 -> 10.78.202.217 NFS 114 V4 Reply (Call In 27827) RENEW 27856 2020-06-23 13:42:06 10.78.202.217 -> 10.78.196.220 NFS 194 V4 Call RENEW CID: 0x3091 27857 2020-06-23 13:42:06 10.78.196.220 -> 10.78.202.217 NFS 114 V4 Reply (Call In 27856) RENEW 28071 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 110 V4 NULL Call[Malformed Packet] 28073 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 94 V4 NULL Reply (Call In 28071)[Malformed Packet] 28082 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 110 V4 NULL Call[Malformed Packet] 28084 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 94 V4 NULL Reply (Call In 28082)[Malformed Packet] 28086 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 370 V4 Call EXCHANGE_ID 28087 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 106 V4 Reply (Call In 28086) Status: NFS4ERR_MINOR_VERS_MISMATCH 28092 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 206 V4 Call PUTROOTFH | GETATTR 28093 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 342 V4 Reply (Call In 28092) PUTROOTFH | GETATTR 28095 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 258 V4 Call GETATTR FH: 0x2e47f02d 28096 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 162 V4 Reply (Call In 28095) GETATTR 28097 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 262 V4 Call GETATTR FH: 0x2e47f02d 28098 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 178 V4 Reply (Call In 28097) GETATTR 28099 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 258 V4 Call GETATTR FH: 0x2e47f02d 28100 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 162 V4 Reply (Call In 28099) GETATTR ... 39660 2020-06-23 13:42:53 10.78.202.217 -> 10.78.196.220 NFS 346 V4 Call LOOKUP DH: 0x19cf3646/shared 39661 2020-06-23 13:42:53 10.78.196.220 -> 10.78.202.217 NFS 342 V4 Reply (Call In 39660) LOOKUP 39663 2020-06-23 13:42:53 10.78.202.217 -> 10.78.196.220 NFS 314 V4 Call READLINK 39665 2020-06-23 13:42:53 10.78.196.220 -> 10.78.202.217 NFS 162 V4 Reply (Call In 39663) READLINK 44182 2020-06-23 13:42:56 10.78.202.217 -> 10.78.196.220 NFS 326 V4 Call GETATTR FH: 0xff288ce9 44183 2020-06-23 13:42:56 10.78.196.220 -> 10.78.202.217 NFS 266 V4 Reply (Call In 44182) GETATTR 68152 2020-06-23 13:42:59 10.78.202.217 -> 10.78.196.220 NFS 194 V4 Call RENEW CID: 0x3091 68153 2020-06-23 13:42:59 10.78.196.220 -> 10.78.202.217 NFS 114 V4 Reply (Call In 68152) RENEW Status: NFS4ERR_STALE_CLIENTID 68155 2020-06-23 13:42:59 10.78.202.217 -> 10.78.196.220 NFS 194 V4 Call RENEW CID: 0x3091 68156 2020-06-23 13:42:59 10.78.196.220 -> 10.78.202.217 NFS 114 V4 Reply (Call In 68155) RENEW Status: NFS4ERR_STALE_CLIENTID 73012 2020-06-23 13:43:04 10.78.202.217 -> 10.78.196.220 NFS 194 V4 Call RENEW CID: 0x3091 73013 2020-06-23 13:43:04 10.78.196.220 -> 10.78.202.217 NFS 114 V4 Reply (Call In 73012) RENEW Status: NFS4ERR_STALE_CLIENTID 73270 2020-06-23 13:43:09 10.78.202.217 -> 10.78.196.220 NFS 194 V4 Call RENEW CID: 0x3091 73271 2020-06-23 13:43:09 10.78.196.220 -> 10.78.202.217 NFS 114 V4 Reply (Call In 73270) RENEW Status: NFS4ERR_STALE_CLIENTID 73273 2020-06-23 13:43:14 10.78.202.217 -> 10.78.196.220 NFS 194 V4 Call RENEW CID: 0x3091 73274 2020-06-23 13:43:14 10.78.196.220 -> 10.78.202.217 NFS 114 V4 Reply (Call In 73273) RENEW Status: NFS4ERR_STALE_CLIENTID 73276 2020-06-23 13:43:19 10.78.202.217 -> 10.78.196.220 NFS 194 V4 Call RENEW CID: 0x3091 73277 2020-06-23 13:43:19 10.78.196.220 -> 10.78.202.217 NFS 114 V4 Reply (Call In 73276) RENEW Status: NFS4ERR_STALE_CLIENTID 73279 2020-06-23 13:43:24 10.78.202.217 -> 10.78.196.220 NFS 194 V4 Call RENEW CID: 0x3091 73280 2020-06-23 13:43:24 10.78.196.220 -> 10.78.202.217 NFS 114 V4 Reply (Call In 73279) RENEW Status: NFS4ERR_STALE_CLIENTID i.e. all clients have: 28071 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 110 V4 NULL Call[Malformed Packet] 28073 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 94 V4 NULL Reply (Call In 28071)[Malformed Packet] 28082 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 110 V4 NULL Call[Malformed Packet] 28084 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 94 V4 NULL Reply (Call In 28082)[Malformed Packet] 28086 2020-06-23 13:42:29 10.78.202.217 -> 10.78.196.220 NFS 370 V4 Call EXCHANGE_ID 28087 2020-06-23 13:42:29 10.78.196.220 -> 10.78.202.217 NFS 106 V4 Reply (Call In 28086) Status: NFS4ERR_MINOR_VERS_MISMATCH then 30 seconds later, the RENEW/NFSERR_STALE_CLIENTID loop starts: 68152 2020-06-23 13:42:59 10.78.202.217 -> 10.78.196.220 NFS 194 V4 Call RENEW CID: 0x3091 68153 2020-06-23 13:42:59 10.78.196.220 -> 10.78.202.217 NFS 114 V4 Reply (Call In 68152) RENEW Status: NFS4ERR_STALE_CLIENTID 68155 2020-06-23 13:42:59 10.78.202.217 -> 10.78.196.220 NFS 194 V4 Call RENEW CID: 0x3091 68156 2020-06-23 13:42:59 10.78.196.220 -> 10.78.202.217 NFS 114 V4 Reply (Call In 68155) RENEW Status: NFS4ERR_STALE_CLIENTID Does anyone have any idea if the above might point to something that has already been fixed in the mainline kernel? Thanks James Pearson
