$ wc -l scipy39-intel-* 454 scipy39-intel-qcomp1.1 wc: scipy39-intel-qcomp2: Operation not permitted 363 scipy39-intel-smmic1 365 scipy39-intel-smmic1.1 1182 total $ wc -l scipy39-intel-* 454 scipy39-intel-qcomp1.1 435 scipy39-intel-qcomp2 363 scipy39-intel-smmic1 365 scipy39-intel-smmic1.1 Client: nfs-utils-2.5.4-18.el9.x86_64 kernel-5.14.0-284.30.1.el9_2.x86_64earthg.cora.nwra.com:/export/home/orion /home/orion nfs4 rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acregmin=1,acregmax=1,acdirmin=1,acdirmax=1,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.133,lookupcache=pos,local_lock=none,addr=192.168.1.5 0 0
Server: nfs-utils-1.3.0-0.68.el7.2.x86_64 kernel-3.10.0-1160.99.1.el7.x86_64/ -ro,async,fsid=0 192.168.1.0/24(sec=sys) 192.168.2.0/24(sec=sys) 10.0.0.0/8(sec=krb5) *.nwra.com(sec=krb5) /export/home -rw,async,nohide 192.168.1.0/24(sec=sys) 192.168.2.0/24(sec=sys) 10.0.0.0/8(sec=krb5) *.nwra.com(sec=krb5)
tshark reports the server returning NFS4ERR_NOENT:Frame 41: 318 bytes on wire (2544 bits), 318 bytes captured (2544 bits) on interface 0 Ethernet II, Src: RealtekU_a9:d8:34 (52:54:00:a9:d8:34), Dst: Dell_4d:82:59 (00:24:e8:4d:82:59) Internet Protocol Version 4, Src: 192.168.1.133 (192.168.1.133), Dst: 192.168.1.5 (192.168.1.5) Transmission Control Protocol, Src Port: gdoi (848), Dst Port: nfs (2049), Seq: 5329, Ack: 4625, Len: 264
Remote Procedure Call, Type:Call XID:0xef4cfdaf
Network File System, Ops(3): SEQUENCE, PUTFH, CLOSE
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Tag: <EMPTY>
length: 0
contents: <EMPTY>
minorversion: 2
Operations (count: 3): SEQUENCE, PUTFH, CLOSE
Opcode: SEQUENCE (53)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002f7
slot id: 0
high slot id: 0
cache this?: Yes
Opcode: PUTFH (22)
filehandle
length: 32
[hash (CRC-32): 0xefafdd27]
decode type as: unknown
filehandle:
01000681c370d9f027664f269d1025f5132996603f233624...
Opcode: CLOSE (4)
seqid: 0x00000000
stateid
[StateID Hash: 0x6ae3]
seqid: 0x00000001
Data: 1f100365e983952638000000
[Main Opcode: CLOSE (4)]
Frame 42: 170 bytes on wire (1360 bits), 170 bytes captured (1360 bits)
on interface 0
Ethernet II, Src: Dell_4d:82:59 (00:24:e8:4d:82:59), Dst:
RealtekU_a9:d8:34 (52:54:00:a9:d8:34)
Internet Protocol Version 4, Src: 192.168.1.5 (192.168.1.5), Dst:
192.168.1.133 (192.168.1.133)
Transmission Control Protocol, Src Port: nfs (2049), Dst Port: gdoi
(848), Seq: 4625, Ack: 5593, Len: 116
Remote Procedure Call, Type:Reply XID:0xef4cfdaf
Network File System, Ops(3): SEQUENCE PUTFH CLOSE
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Status: NFS4_OK (0)
Tag: <EMPTY>
length: 0
contents: <EMPTY>
Operations (count: 3)
Opcode: SEQUENCE (53)
Status: NFS4_OK (0)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002f7
slot id: 0
high slot id: 9
target high slot id: 9
status flags: 0x00000000
.... .... .... .... .... .... .... ...0 =
SEQ4_STATUS_CB_PATH_DOWN: Not set
.... .... .... .... .... .... .... ..0. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING: Not set
.... .... .... .... .... .... .... .0.. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED: Not set
.... .... .... .... .... .... .... 0... =
SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED: Not set
.... .... .... .... .... .... ...0 .... =
SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED: Not set
.... .... .... .... .... .... ..0. .... =
SEQ4_STATUS_ADMIN_STATE_REVOKED: Not set
.... .... .... .... .... .... .0.. .... =
SEQ4_STATUS_RECALLABLE_STATE_REVOKED: Not set
.... .... .... .... .... .... 0... .... =
SEQ4_STATUS_LEASE_MOVED: Not set
.... .... .... .... .... ...0 .... .... =
SEQ4_STATUS_RESTART_RECLAIM_NEEDED: Not set
.... .... .... .... .... ..0. .... .... =
SEQ4_STATUS_CB_PATH_DOWN_SESSION: Not set
.... .... .... .... .... .0.. .... .... =
SEQ4_STATUS_BACKCHANNEL_FAULT: Not set
.... .... .... .... .... 0... .... .... =
SEQ4_STATUS_DEVID_CHANGED: Not set
.... .... .... .... ...0 .... .... .... =
SEQ4_STATUS_DEVID_DELETED: Not set
Opcode: PUTFH (22)
Status: NFS4_OK (0)
Opcode: CLOSE (4)
Status: NFS4_OK (0)
stateid
[StateID Hash: 0x6414]
seqid: 0x00000002
Data: 1f100365e983952638000000
[Main Opcode: CLOSE (4)]
Frame 43: 326 bytes on wire (2608 bits), 326 bytes captured (2608 bits)
on interface 0
Ethernet II, Src: RealtekU_a9:d8:34 (52:54:00:a9:d8:34), Dst:
Dell_4d:82:59 (00:24:e8:4d:82:59)
Internet Protocol Version 4, Src: 192.168.1.133 (192.168.1.133), Dst:
192.168.1.5 (192.168.1.5)
Transmission Control Protocol, Src Port: gdoi (848), Dst Port: nfs
(2049), Seq: 5593, Ack: 4741, Len: 272
Remote Procedure Call, Type:Call XID:0xf04cfdaf
Network File System, Ops(5): SEQUENCE, PUTFH, LOOKUP, GETFH, GETATTR
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Tag: <EMPTY>
length: 0
contents: <EMPTY>
minorversion: 2
Operations (count: 5): SEQUENCE, PUTFH, LOOKUP, GETFH, GETATTR
Opcode: SEQUENCE (53)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002f8
slot id: 0
high slot id: 0
cache this?: No
Opcode: PUTFH (22)
filehandle
length: 32
[hash (CRC-32): 0x6fd1a280]
decode type as: unknown
filehandle:
01000681c370d9f027664f269d1025f51329966086b50420...
Opcode: LOOKUP (15)
Name: .git
length: 4
contents: .git
Opcode: GETFH (10)
Opcode: GETATTR (9)
Attr mask[0]: 0x0010011a (TYPE, CHANGE, SIZE, FSID, FILEID)
reqd_attr: TYPE (1)
reqd_attr: CHANGE (3)
reqd_attr: SIZE (4)
reqd_attr: FSID (8)
reco_attr: FILEID (20)
Attr mask[1]: 0x00b0a23a (MODE, NUMLINKS, OWNER,
OWNER_GROUP, RAWDEV, SPACE_USED, TIME_ACCESS, TIME_METADATA,
TIME_MODIFY, MOUNTED_ON_FILEID)
reco_attr: MODE (33)
reco_attr: NUMLINKS (35)
reco_attr: OWNER (36)
reco_attr: OWNER_GROUP (37)
reco_attr: RAWDEV (41)
reco_attr: SPACE_USED (45)
reco_attr: TIME_ACCESS (47)
reco_attr: TIME_METADATA (52)
reco_attr: TIME_MODIFY (53)
reco_attr: MOUNTED_ON_FILEID (55)
[Main Opcode: LOOKUP (15)]
Frame 44: 154 bytes on wire (1232 bits), 154 bytes captured (1232 bits)
on interface 0
Ethernet II, Src: Dell_4d:82:59 (00:24:e8:4d:82:59), Dst:
RealtekU_a9:d8:34 (52:54:00:a9:d8:34)
Internet Protocol Version 4, Src: 192.168.1.5 (192.168.1.5), Dst:
192.168.1.133 (192.168.1.133)
Transmission Control Protocol, Src Port: nfs (2049), Dst Port: gdoi
(848), Seq: 4741, Ack: 5865, Len: 100
Remote Procedure Call, Type:Reply XID:0xf04cfdaf
Network File System, Ops(3): SEQUENCE PUTFH LOOKUP(NFS4ERR_NOENT)
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Status: NFS4ERR_NOENT (2)
Tag: <EMPTY>
length: 0
contents: <EMPTY>
Operations (count: 3)
Opcode: SEQUENCE (53)
Status: NFS4_OK (0)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002f8
slot id: 0
high slot id: 9
target high slot id: 9
status flags: 0x00000000
.... .... .... .... .... .... .... ...0 =
SEQ4_STATUS_CB_PATH_DOWN: Not set
.... .... .... .... .... .... .... ..0. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING: Not set
.... .... .... .... .... .... .... .0.. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED: Not set
.... .... .... .... .... .... .... 0... =
SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED: Not set
.... .... .... .... .... .... ...0 .... =
SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED: Not set
.... .... .... .... .... .... ..0. .... =
SEQ4_STATUS_ADMIN_STATE_REVOKED: Not set
.... .... .... .... .... .... .0.. .... =
SEQ4_STATUS_RECALLABLE_STATE_REVOKED: Not set
.... .... .... .... .... .... 0... .... =
SEQ4_STATUS_LEASE_MOVED: Not set
.... .... .... .... .... ...0 .... .... =
SEQ4_STATUS_RESTART_RECLAIM_NEEDED: Not set
.... .... .... .... .... ..0. .... .... =
SEQ4_STATUS_CB_PATH_DOWN_SESSION: Not set
.... .... .... .... .... .0.. .... .... =
SEQ4_STATUS_BACKCHANNEL_FAULT: Not set
.... .... .... .... .... 0... .... .... =
SEQ4_STATUS_DEVID_CHANGED: Not set
.... .... .... .... ...0 .... .... .... =
SEQ4_STATUS_DEVID_DELETED: Not set
Opcode: PUTFH (22)
Status: NFS4_OK (0)
Opcode: LOOKUP (15)
Status: NFS4ERR_NOENT (2)
[Main Opcode: LOOKUP (15)]
Frame 45: 326 bytes on wire (2608 bits), 326 bytes captured (2608 bits)
on interface 0
Ethernet II, Src: RealtekU_a9:d8:34 (52:54:00:a9:d8:34), Dst:
Dell_4d:82:59 (00:24:e8:4d:82:59)
Internet Protocol Version 4, Src: 192.168.1.133 (192.168.1.133), Dst:
192.168.1.5 (192.168.1.5)
Transmission Control Protocol, Src Port: gdoi (848), Dst Port: nfs
(2049), Seq: 5865, Ack: 4841, Len: 272
Remote Procedure Call, Type:Call XID:0xf14cfdaf
Network File System, Ops(5): SEQUENCE, PUTFH, LOOKUP, GETFH, GETATTR
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Tag: <EMPTY>
length: 0
contents: <EMPTY>
minorversion: 2
Operations (count: 5): SEQUENCE, PUTFH, LOOKUP, GETFH, GETATTR
Opcode: SEQUENCE (53)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002f9
slot id: 0
high slot id: 0
cache this?: No
Opcode: PUTFH (22)
filehandle
length: 32
[hash (CRC-32): 0x6fd1a280]
decode type as: unknown
filehandle:
01000681c370d9f027664f269d1025f51329966086b50420...
Opcode: LOOKUP (15)
Name: .git
length: 4
contents: .git
Opcode: GETFH (10)
Opcode: GETATTR (9)
Attr mask[0]: 0x0010011a (TYPE, CHANGE, SIZE, FSID, FILEID)
reqd_attr: TYPE (1)
reqd_attr: CHANGE (3)
reqd_attr: SIZE (4)
reqd_attr: FSID (8)
reco_attr: FILEID (20)
Attr mask[1]: 0x00b0a23a (MODE, NUMLINKS, OWNER,
OWNER_GROUP, RAWDEV, SPACE_USED, TIME_ACCESS, TIME_METADATA,
TIME_MODIFY, MOUNTED_ON_FILEID)
reco_attr: MODE (33)
reco_attr: NUMLINKS (35)
reco_attr: OWNER (36)
reco_attr: OWNER_GROUP (37)
reco_attr: RAWDEV (41)
reco_attr: SPACE_USED (45)
reco_attr: TIME_ACCESS (47)
reco_attr: TIME_METADATA (52)
reco_attr: TIME_MODIFY (53)
reco_attr: MOUNTED_ON_FILEID (55)
[Main Opcode: LOOKUP (15)]
Frame 46: 154 bytes on wire (1232 bits), 154 bytes captured (1232 bits)
on interface 0
Ethernet II, Src: Dell_4d:82:59 (00:24:e8:4d:82:59), Dst:
RealtekU_a9:d8:34 (52:54:00:a9:d8:34)
Internet Protocol Version 4, Src: 192.168.1.5 (192.168.1.5), Dst:
192.168.1.133 (192.168.1.133)
Transmission Control Protocol, Src Port: nfs (2049), Dst Port: gdoi
(848), Seq: 4841, Ack: 6137, Len: 100
Remote Procedure Call, Type:Reply XID:0xf14cfdaf
Network File System, Ops(3): SEQUENCE PUTFH LOOKUP(NFS4ERR_NOENT)
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Status: NFS4ERR_NOENT (2)
Tag: <EMPTY>
length: 0
contents: <EMPTY>
Operations (count: 3)
Opcode: SEQUENCE (53)
Status: NFS4_OK (0)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002f9
slot id: 0
high slot id: 9
target high slot id: 9
status flags: 0x00000000
.... .... .... .... .... .... .... ...0 =
SEQ4_STATUS_CB_PATH_DOWN: Not set
.... .... .... .... .... .... .... ..0. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING: Not set
.... .... .... .... .... .... .... .0.. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED: Not set
.... .... .... .... .... .... .... 0... =
SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED: Not set
.... .... .... .... .... .... ...0 .... =
SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED: Not set
.... .... .... .... .... .... ..0. .... =
SEQ4_STATUS_ADMIN_STATE_REVOKED: Not set
.... .... .... .... .... .... .0.. .... =
SEQ4_STATUS_RECALLABLE_STATE_REVOKED: Not set
.... .... .... .... .... .... 0... .... =
SEQ4_STATUS_LEASE_MOVED: Not set
.... .... .... .... .... ...0 .... .... =
SEQ4_STATUS_RESTART_RECLAIM_NEEDED: Not set
.... .... .... .... .... ..0. .... .... =
SEQ4_STATUS_CB_PATH_DOWN_SESSION: Not set
.... .... .... .... .... .0.. .... .... =
SEQ4_STATUS_BACKCHANNEL_FAULT: Not set
.... .... .... .... .... 0... .... .... =
SEQ4_STATUS_DEVID_CHANGED: Not set
.... .... .... .... ...0 .... .... .... =
SEQ4_STATUS_DEVID_DELETED: Not set
Opcode: PUTFH (22)
Status: NFS4_OK (0)
Opcode: LOOKUP (15)
Status: NFS4ERR_NOENT (2)
[Main Opcode: LOOKUP (15)]
Frame 47: 326 bytes on wire (2608 bits), 326 bytes captured (2608 bits)
on interface 0
Ethernet II, Src: RealtekU_a9:d8:34 (52:54:00:a9:d8:34), Dst:
Dell_4d:82:59 (00:24:e8:4d:82:59)
Internet Protocol Version 4, Src: 192.168.1.133 (192.168.1.133), Dst:
192.168.1.5 (192.168.1.5)
Transmission Control Protocol, Src Port: gdoi (848), Dst Port: nfs
(2049), Seq: 6137, Ack: 4941, Len: 272
Remote Procedure Call, Type:Call XID:0xf24cfdaf
Network File System, Ops(5): SEQUENCE, PUTFH, LOOKUP, GETFH, GETATTR
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Tag: <EMPTY>
length: 0
contents: <EMPTY>
minorversion: 2
Operations (count: 5): SEQUENCE, PUTFH, LOOKUP, GETFH, GETATTR
Opcode: SEQUENCE (53)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002fa
slot id: 0
high slot id: 0
cache this?: No
Opcode: PUTFH (22)
filehandle
length: 32
[hash (CRC-32): 0x6fd1a280]
decode type as: unknown
filehandle:
01000681c370d9f027664f269d1025f51329966086b50420...
Opcode: LOOKUP (15)
Name: HEAD
length: 4
contents: HEAD
Opcode: GETFH (10)
Opcode: GETATTR (9)
Attr mask[0]: 0x0010011a (TYPE, CHANGE, SIZE, FSID, FILEID)
reqd_attr: TYPE (1)
reqd_attr: CHANGE (3)
reqd_attr: SIZE (4)
reqd_attr: FSID (8)
reco_attr: FILEID (20)
Attr mask[1]: 0x00b0a23a (MODE, NUMLINKS, OWNER,
OWNER_GROUP, RAWDEV, SPACE_USED, TIME_ACCESS, TIME_METADATA,
TIME_MODIFY, MOUNTED_ON_FILEID)
reco_attr: MODE (33)
reco_attr: NUMLINKS (35)
reco_attr: OWNER (36)
reco_attr: OWNER_GROUP (37)
reco_attr: RAWDEV (41)
reco_attr: SPACE_USED (45)
reco_attr: TIME_ACCESS (47)
reco_attr: TIME_METADATA (52)
reco_attr: TIME_MODIFY (53)
reco_attr: MOUNTED_ON_FILEID (55)
[Main Opcode: LOOKUP (15)]
Frame 48: 154 bytes on wire (1232 bits), 154 bytes captured (1232 bits)
on interface 0
Ethernet II, Src: Dell_4d:82:59 (00:24:e8:4d:82:59), Dst:
RealtekU_a9:d8:34 (52:54:00:a9:d8:34)
Internet Protocol Version 4, Src: 192.168.1.5 (192.168.1.5), Dst:
192.168.1.133 (192.168.1.133)
Transmission Control Protocol, Src Port: nfs (2049), Dst Port: gdoi
(848), Seq: 4941, Ack: 6409, Len: 100
Remote Procedure Call, Type:Reply XID:0xf24cfdaf
Network File System, Ops(3): SEQUENCE PUTFH LOOKUP(NFS4ERR_NOENT)
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
Status: NFS4ERR_NOENT (2)
Tag: <EMPTY>
length: 0
contents: <EMPTY>
Operations (count: 3)
Opcode: SEQUENCE (53)
Status: NFS4_OK (0)
sessionid: 1f100365e98395269619000000000000
seqid: 0x000002fa
slot id: 0
high slot id: 9
target high slot id: 9
status flags: 0x00000000
.... .... .... .... .... .... .... ...0 =
SEQ4_STATUS_CB_PATH_DOWN: Not set
.... .... .... .... .... .... .... ..0. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING: Not set
.... .... .... .... .... .... .... .0.. =
SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED: Not set
.... .... .... .... .... .... .... 0... =
SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED: Not set
.... .... .... .... .... .... ...0 .... =
SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED: Not set
.... .... .... .... .... .... ..0. .... =
SEQ4_STATUS_ADMIN_STATE_REVOKED: Not set
.... .... .... .... .... .... .0.. .... =
SEQ4_STATUS_RECALLABLE_STATE_REVOKED: Not set
.... .... .... .... .... .... 0... .... =
SEQ4_STATUS_LEASE_MOVED: Not set
.... .... .... .... .... ...0 .... .... =
SEQ4_STATUS_RESTART_RECLAIM_NEEDED: Not set
.... .... .... .... .... ..0. .... .... =
SEQ4_STATUS_CB_PATH_DOWN_SESSION: Not set
.... .... .... .... .... .0.. .... .... =
SEQ4_STATUS_BACKCHANNEL_FAULT: Not set
.... .... .... .... .... 0... .... .... =
SEQ4_STATUS_DEVID_CHANGED: Not set
.... .... .... .... ...0 .... .... .... =
SEQ4_STATUS_DEVID_DELETED: Not set
Opcode: PUTFH (22)
Status: NFS4_OK (0)
Opcode: LOOKUP (15)
Status: NFS4ERR_NOENT (2)
[Main Opcode: LOOKUP (15)]
I did switch the client from mounting with sec=krb5 to sec=sys. Perhaps
this has left the server in a bad state?
Any ideas what could be going wrong? I've rebooted the client many times, but have not yet had the opportunity to reboot the server.
I've not yet been able to reproduce the issue on any other EL9 (or other) client.
-- Orion Poplawski he/him/his - surely the least important thing about me IT Systems Manager 720-772-5637 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion@xxxxxxxx Boulder, CO 80301 https://www.nwra.com/
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
