Hi Ben, I was just about to reply saying I found the problem. It is exactly what you point out - I was using IET iscsi targets that does not support PRs. The kernel log showed PR command failing with "illegal request". Switching to LIO target fixed the problem. Appreciate your help on the problem and thanks for answering my questions. Thanks, jrk On Mon, 11 Mar 2019 at 16:21, Benjamin Coddington <bcodding@xxxxxxxxxx> wrote: > > On 8 Mar 2019, at 2:05, Kanika wrote: > > > On Thu, 7 Mar 2019 at 16:42, Benjamin Coddington <bcodding@xxxxxxxxxx> > > wrote: > >> > >> On 7 Mar 2019, at 2:51, Kanika wrote: > >> > >>> Hi Benjamin, > >>> Thanks for your reply. I suspected so but wanted to confirm. I tried > >>> block/scsi layout as well but that isn't working either. > >>> > >>> Based on an old thread on this list on "how to setup pnfs for block > >>> layout", I set up an iscsi target on the DS and the initiators on > >>> the MDS > >>> and client. All machines are running 4.15 kernel. Client as an iscsi > >>> initiator but does not mount the device. MDS also an initiator with > >>> an xfs > >>> filesystem mounted on the device. This mount point is exported as an > >>> nfs > >>> share from the MDS. The client mounts the exported MDS share. > >>> By default LAYOUT_SCSI is used but the GETDEVICEINFO call keeps > >>> failing > >>> with NFS4ERR_INVAL. As a result all reads/writes from the client are > >>> routed > >>> to the MDS instead of the DS. Anything wrong with my setup? > >> > >> The SCSI layout needs to work with whole SCSI devices, not > >> partitions. > >> > >> Try putting your XFS on /dev/sda instead of /dev/sda1. > >> > >> Ben > > Hi Ben, > > Thanks for the pointer. I tried this but still no luck, see the same > > error for GETDEVICEINFO. Anything else I am missing? > > It seems like you're hitting: > > nfsd4_scsi_proc_getdeviceinfo(..) > ... > if (sb->s_bdev != sb->s_bdev->bd_contains) > return nfserr_inval; > > But if you're getting past that, then the MDS should be using pr_err to > show > something informative in the kernel log. It's possible your SCSI > devices > doesn't support persistent reservations. > > If there's nothing informative in the log, can you show the output of > > sg_inq --vpd -p 0x83 /dev/sda > > On both the client and the MDS? > > Ben
