On Thu, 7 Mar 2019 at 16:42, Benjamin Coddington <bcodding@xxxxxxxxxx> wrote: > > On 7 Mar 2019, at 2:51, Kanika wrote: > > > Hi Benjamin, > > Thanks for your reply. I suspected so but wanted to confirm. I tried > > block/scsi layout as well but that isn't working either. > > > > Based on an old thread on this list on "how to setup pnfs for block > > layout", I set up an iscsi target on the DS and the initiators on the MDS > > and client. All machines are running 4.15 kernel. Client as an iscsi > > initiator but does not mount the device. MDS also an initiator with an xfs > > filesystem mounted on the device. This mount point is exported as an nfs > > share from the MDS. The client mounts the exported MDS share. > > By default LAYOUT_SCSI is used but the GETDEVICEINFO call keeps failing > > with NFS4ERR_INVAL. As a result all reads/writes from the client are routed > > to the MDS instead of the DS. Anything wrong with my setup? > > The SCSI layout needs to work with whole SCSI devices, not partitions. > > Try putting your XFS on /dev/sda instead of /dev/sda1. > > Ben Hi Ben, Thanks for the pointer. I tried this but still no luck, see the same error for GETDEVICEINFO. Anything else I am missing? On MDS: root@ubuntu1604:~# lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL NAME FSTYPE SIZE MOUNTPOINT LABEL sda xfs 1G /sudosrv <------ iscsi block device sr0 1024M vda 64G ├─vda1 ext4 63G / ├─vda2 1K └─vda5 swap 975M [SWAP] #cat /etc/exports /sudosrv *(rw,sync,fsid=0,no_subtree_check,no_root_squash,pnfs) On client: root@ubuntu18_04_2:~# cat /proc/partitions major minor #blocks name 11 0 1048575 sr0 252 0 67108864 vda 252 1 66107392 vda1 252 2 1 vda2 252 5 998400 vda5 8 0 1048576 sda root@ubuntu18_04_2:~# cat /proc/self/mountstats device 192.168.122.92:/ mounted on /mnt with fstype nfs4 statvers=1.1 ... nfsv4: bm0=0xfdffbfff,bm1=0x40f9be3e,bm2=0x803,acl=0x3,sessions,pnfs=LAYOUT_SCSI blkmapd and blocklayoutdriver loaded on the client root@ubuntu18_04_2:~# ps aux | grep blk root 2485 0.0 0.0 23920 176 ? Ss 21:25 0:00 /usr/sbin/blkmapd -- packet exchange between MDS and client. 94 2019-03-07 21:31:18.480733 192.168.122.83 → 192.168.122.92 NFS 234 V4 Call GETDEVINFO 95 2019-03-07 21:31:18.483495 192.168.122.92 → 192.168.122.83 NFS 158 V4 Reply (Call In 94) GETDEVINFO Status: NFS4ERR_INVAL 96 2019-03-07 21:31:18.483878 192.168.122.83 → 192.168.122.92 NFS 282 V4 Call LAYOUTRETURN 98 2019-03-07 21:31:18.483935 192.168.122.83 → 192.168.122.92 NFS 1478 V4 Call WRITE StateID: 0xcaae Offset: 0 Len: 4096 100 2019-03-07 21:31:18.484204 192.168.122.92 → 192.168.122.83 NFS 170 V4 Reply (Call In 96) LAYOUTRETURN 102 2019-03-07 21:31:18.528822 192.168.122.92 → 192.168.122.83 NFS 246 V4 Reply (Call In 98) WRITE
