I was able to get the pNFS file-layout code functioning with 1 DS, and a MDS and
1 Client. I need to get block layout working for a research project I am
working on, however have so far been unable to get this working.
I would very much appreciate help or suggestions (or a kick in the head if I'm
doing something stupid).
I referenced the wiki pages on client and DS setup and referenced the two
threads I saw on this topic, however cannot get block-layout working.
Below I have shown the steps I have taken to try and get this working (sorry for
the long list).
I have also shown the testing and troubleshooting I have undertaken, and the
results there-of.
Data Server: 192.168.1.2 MDS: 192.168.1.1 Client: 192.168.1.3
On Data-server(s) create folder with mkdir /pnfs
On Data-server(s) Edit /etc/exports. Add the string:
/pnfs *(rw, sync,fsid=0,insecure,no_subtree_check,pnfs,no_root_squash)
On Data-server(s) type parted /dev/sdb
(parted) mklabel gpt
(parted) mklabel 1 <Provide ext4 file-system, and partition start and end>
(parted> print
Model: ATA SSDSA2SH06â (scsi)
Disk /dev/sdb: 64.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End size File system Name Flags
1 1049kB 64.0GB 64.0GB ext4 1
On Data-server(s) type mkfs.ext4 âb 4096 /dev/sdb1
On Data-server(s) type /etc/init.d/iptables stop
On Data-server(s) type /etc/init.d/tgtd start
On Data-server(s) type:
tgtadm âlld iscsi âop new âmode target âtid 1 âT
iqn.2011-03.com.intel:openblock
On Data-server(s) type
tgtadm âlld iscsi âmode logicalunit âop new âtid 1 âlun 1 âbacking-store /dev/sdb1
On Data-server(s) type
tgtadm âlld iscsi âmode target âop bind âtid 1 âinitiator-address ALL
On Data-server(s) type tgtadm âlld iscsi âop show âmode target
On Data-server(s) type mount /dev/sdb1 /pnfs
On Data-server(s) type service nfs restart
On Data-server(s) cd to the ctl directory and type ./ctl âu &
See: âProcessing: MSG_VERS : Version: 8
On MDS copy the spnfsd.conf file from pNFS_Packages/pnfs-nfs-utils/utils to
/etc/spnfsd.conf and edit it for 1 DS and the proper DS-Mount-Directory, DS1_IP,
and DS1_Root.
[General]
Verbosity = 1
Stripe-size = 8192
Dense-striping = 0
Pipefs-Directory = /var/lib/nfs/rpc_pipefs
DS-Mount-Directory = /spnfs
[DataServers]
NumDS = 1
DS1_IP= 192.168.1.2
DS1_PORT= 2049
DS1_ROOT = /pnfs
DS1_ID = 1
On MDS create the /spnfs/192.168.1.2 folder
On MDS edit /etc/exports and add below string:
/ *(rw, sync,fsid=0,insecure,no_subtree_check,pnfs,no_root_squash)
On MDS type mount 192.168.1.2:/pnfs /spnfs/192.168.1.2
See on the DS â the ctl application prints out:
Processing: MSG_VERS : Version: 8
On MDS disable iptables with /etc/init.d/iptables stop
On MDS type service nfs restart â See:
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]
On MDS type modprobe nfsd to start the NFS daemon
On MDS type spnfsd to start service on MDS server.
See recapitulation of /etc/spnfsd.confâ;
On the Client disable iptables with /etc/init.d/iptables stop
On the Client log onto the Data-Server iSCSI target by typing:
iscsi âm discovery ât st âp 192.168.1.2 âl
On the Client start the blkmapd service by typing service blkmapd start
On the Client run /usr/sbin/blkmapd âf
On the Client mount MDS server by typing:
mount ât nfs4 âo minorversion =1 192.168.1.1:/spnfs /mnt
TESTING:
On the Client generate some activity in /mnt (copy file from local client disk
to /mnt)
On the Client cat /proc/self/mountstats and look for LAYOUT and see:
nfsv4: bm0=0xfdffbfff,bm1=0x40f9be3e,acl=0x3,sessions,pnfs=not configured
LAYOUTGET: 0 0 0 0 0 0 0 0
LAYTOUTCOMMIT: 0 0 0 0 0 0 0 0
LAYOUTRETURN: 0 0 0 0 0 0 0 0
On the Client I see in /mnt the file I copied
On the MDS I see in /spnfs (note: not /spnfs/192.168.1.2) the file I copied
from the Client
On the MDS I see in /spnfs/192.168.1.2 a file called 2359299.1868952557
On the DS I see in /pnfs a file called 2359299.1868952557
TROUBLESHOOTING:
Read the new-ish trouble shooting information on the wiki-page.
1 â Ran more traffic and still see the same information from the clientâs
/proc/self/mountstats
2 â Ran /usr/sbin/blkmapd âf on the Client and saw no output
3 â Looked at /var/log/messages on all 3 machines.
Saw nothing on either the MDS or the DS.
On the Client I see the innocuous:
âFS-Cache: Loadedâ
Registering the id_resolver key type
FS-Cache: Netfs ânfsâ registered for caching
4 â On the Client I ran âmountâ and see I have:
192.168.1.1:/spnf on /mnt type nfs4
(rw,minorversion=1,addr192.168.1.1,clientaddr=192.168.1.3)
<Note: no output from the running â/usr/sbin/blkmapd âfâ when I ran the mount
command>
5 â On the Client I verified I have an iSCSI target attached.
6 â On the Client I did an lsmod|grep bllock to see I had blocklayoutdriver
loaded and used by nfs and sunrpc
QUESTIONS:
1 â do you see anything missing or wrong about my procedure?
2 â The MDS mounts the DS using âmount 192.168.1.2:/pnfs /spnfs/192.168.1.2â.
Does the MDS also/instead need to act as an iSCSI initiator and discover/logon
to the DS iSCSI target?
3 â from /proc/self/mountstats on the client I see I do NOT in fact have
pnfs=LAYOUT_BLOCK_VOLUEM so I assume we can suspect we do not have a pNFS mount.
What does this mean and how can I address the issue?
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
