----- Original Message ----- > From: "trondmy" <trondmy@xxxxxxxxxxxxxxx> > To: "Tigran Mkrtchyan" <tigran.mkrtchyan@xxxxxxx> > Cc: "linux-nfs" <linux-nfs@xxxxxxxxxxxxxxx> > Sent: Friday, 13 November, 2020 23:45:00 > Subject: Re: [PATCH v3 00/11] Add RDMA support to the pNFS file+flexfiles data channels > On Fri, 2020-11-13 at 22:30 +0100, Mkrtchyan, Tigran wrote: >> >> After more testing, it looks like that client doesn't like >> notification bitmap: >> >> >> [31576.789492] --> _nfs4_proc_getdeviceinfo >> [31576.789503] --> nfs41_call_sync_prepare data->seq_server >> 000000001d17c43e >> [31576.789507] --> nfs4_alloc_slot used_slots=0000 >> highest_used=4294967295 max_slots=16 >> [31576.789510] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 >> slotid=0 >> [31576.789527] encode_sequence: >> sessionid=2910695007:150995712:0:16777216 seqid=92462 slotid=0 >> max_slotid=0 cache_this=0 >> [31576.789991] decode_getdeviceinfo: unsupported notification > > According to this, you appear to be returning a deviceinfo bitmap with > at least one non-zero entry that is not in the first 32-bit word. We > only ask for notifications for NOTIFY_DEVICEID4_CHANGE and > NOTIFY_DEVICEID4_DELETE, so we only expect bitmap[0] to have non-zero > entries. according to packet capture only bitmap[0] has non zero bits set. This is the reply of compound starting from nfs staus code, tag length and so on. 0000 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 35 0010 00 00 00 00 5f ae 7d ad 00 03 00 09 00 00 00 00 0020 00 00 00 01 00 00 00 4c 00 00 00 00 00 00 00 0f 0030 00 00 00 0f 00 00 00 00 00 00 00 2f 00 00 00 00 0040 00 00 00 04 00 00 00 40 00 00 00 01 00 00 00 03 0050 74 63 70 00 00 00 00 16 31 33 31 2e 31 36 39 2e 0060 31 39 31 2e 31 34 33 2e 31 32 35 2e 34 39 00 00 0070 00 00 00 01 00 00 00 04 00 00 00 01 00 10 00 00 0080 00 10 00 00 00 00 00 01 00 00 00 02 00 00 00 06 0090 00 00 00 00 the last 12 bytes : bitmap size, bitmap[0], bitmap[1] This part of code in the didn't change since 2010, and I have no issues to use 5.8 kernel. I am pretty sure, that tests with 5.9 did pass as expected. I will try to bisec it. Tigran. > >> [31576.790003] --> nfs4_alloc_slot used_slots=0001 highest_used=0 >> max_slots=16 >> [31576.790007] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 >> slotid=1 >> [31576.790010] nfs4_free_slot: slotid 1 highest_used_slotid 0 >> [31576.790013] nfs41_sequence_process: Error 0 free the slot >> [31576.790017] nfs4_free_slot: slotid 0 highest_used_slotid >> 4294967295 >> [31576.790030] <-- _nfs4_proc_getdeviceinfo status=-5 >> [31576.790034] nfs4_get_device_info getdevice info returns -5 >> [31576.790084] <-- nfs4_get_device_info d 0000000000000000 >> >> >> Tigran. >> >> >> ----- Original Message ----- >> > From: "Tigran Mkrtchyan" <tigran.mkrtchyan@xxxxxxx> >> > To: "trondmy" <trondmy@xxxxxxxxxxxxxxx> >> > Cc: "linux-nfs" <linux-nfs@xxxxxxxxxxxxxxx> >> > Sent: Friday, 13 November, 2020 13:48:07 >> > Subject: Re: [PATCH v3 00/11] Add RDMA support to the pNFS >> > file+flexfiles data channels >> >> > Hi Trond, >> > >> > whit this changes (3ee69a14f92d74ced2647140b3799511ba4f3fa5) I see >> > an infinite >> > loop >> > of LAYOUTGET->GETDEVICEINFO->LAYOUTRETURN without any attempt to >> > connect to a >> > DS. >> > >> > This is how the response to LAYOUTGET looks like. >> > >> > Network File System, Ops(2): SEQUENCE GETDEVINFO >> > [Program Version: 4] >> > [V4 Procedure: COMPOUND (1)] >> > Status: NFS4_OK (0) >> > Tag: <EMPTY> >> > length: 0 >> > contents: <EMPTY> >> > Operations (count: 2) >> > Opcode: SEQUENCE (53) >> > Opcode: GETDEVINFO (47) >> > Status: NFS4_OK (0) >> > layout type: LAYOUT4_FLEX_FILES (4) >> > r_netid: tcp >> > length: 3 >> > contents: tcp >> > fill bytes: opaque data >> > r_addr: 131.169.191.143.125.49 >> > length: 22 >> > contents: 131.169.191.143.125.49 >> > fill bytes: opaque data >> > version: 4 >> > minorversion: 1 >> > max_rsize: 1048576 >> > max_wsize: 1048576 >> > tightly coupled: Yes >> > notify_mask: 0x00000006 (Change, Delete) >> > notify_type: Change (1) >> > notify_type: Delete (2) >> > [Main Opcode: GETDEVINFO (47)] >> > >> > >> > >> > The MDS is mounted with IPv4. I can provide the full packet trace, >> > if needed. >> > >> > >> > Regards, >> > Tigran. >> > >> > >> > ----- Original Message ----- >> > > From: "trondmy" <trondmy@xxxxxxxxxxxxxxx> >> > > To: "linux-nfs" <linux-nfs@xxxxxxxxxxxxxxx> >> > > Sent: Wednesday, 11 November, 2020 00:42:31 >> > > Subject: Re: [PATCH v3 00/11] Add RDMA support to the pNFS >> > > file+flexfiles data >> > > channels >> > >> > > On Tue, 2020-11-10 at 18:18 -0500, trondmy@xxxxxxxxxx wrote: >> > > > From: Trond Myklebust <trond.myklebust@xxxxxxxxxxxxxxx> >> > > > >> > > > Add support for connecting to the pNFS files/flexfiles data >> > > > servers >> > > > through RDMA, assuming that the GETDEVICEINFO call advertises >> > > > that >> > > > support. >> > > > >> > > > v2: Fix layoutstats encoding for pNFS/flexfiles. >> > > > v3: Move most of the netid handling into the SUNRPC and RDMA >> > > > modules. >> > > > Fix up the mount code to benefit more from automated >> > > > loading of >> > > > SUNRPC transport modules. >> > > > >> > > >> > > Note that one cleanup that I did not perform, but which really >> > > could be >> > > useful should we want to add more transport mechanisms, is to >> > > move the >> > > code to parse stringified addresses (in particular IETF style >> > > "universal addresses") into the transport modules so that the >> > > actual >> > > parsing of mount and pNFS transport info can be automatically >> > > extended >> > > when new transport modules are added. >> > > >> > > -- >> > > Trond Myklebust >> > > Linux NFS client maintainer, Hammerspace >> > > trond.myklebust@xxxxxxxxxxxxxxx > > -- > Trond Myklebust > CTO, Hammerspace Inc > 4984 El Camino Real, Suite 208 > Los Altos, CA 94022 > > www.hammer.space
