Re: pNFS file layout performance

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I believe Benny is right!  I might have forgotten to load the  nfs file layout driver.  I will check on that as soon as I have a chance.   Hopefully I will have good news to report later.

Thanks,
Helen

----- Original Message -----
From: Daniel.Muntz@xxxxxxx [mailto:Daniel.Muntz@xxxxxxx]
Sent: Thursday, August 05, 2010 04:01 PM
To: Chen, Helen Y; bhalevy@xxxxxxxxxxx <bhalevy@xxxxxxxxxxx>; steved@xxxxxxxxxx <steved@xxxxxxxxxx>; linux-nfs@xxxxxxxxxxxxxxx <linux-nfs@xxxxxxxxxxxxxxx>
Subject: RE: pNFS file layout performance

Anything in the logs indicating the MDS couldn't communicate with the DSs?  Beyond that, if the setup is correct, would it be possible to get network traces?  Nothing immediately comes to mind as to why performance would be poor if the data is landing on the MDS.  Otoh, if the data were landing on the DSs and the client was having to resort to proxy I/O, then spNFS performance would probably be poor (proxy I/O is _not_ spNFS's strong suit :-)

  -Dan

-----Original Message-----
From: linux-nfs-owner@xxxxxxxxxxxxxxx [mailto:linux-nfs-owner@xxxxxxxxxxxxxxx] On Behalf Of Chen, Helen Y
Sent: Wednesday, August 04, 2010 5:21 PM
To: Chen, Helen Y; 'Benny Halevy'; 'steved@xxxxxxxxxx'; 'NFS list'
Subject: RE: pNFS file layout performance

I add a 2nd DS and repeated the benchmark.   It had seemed the test data had been placed on the MDS and not the DS'es?  Any idea why?

Thanks,
Helen

The following are my notes on kernel, configuration set up, and results:

I rebuilt Steve's f13 pnfs kernel source with:
# CONFIG_PNFSD_LOCAL_EXPORT is not set


Configuration set up:
Data Server:
/etc/fstab
tmpfs		/export/spnfs		tmpfs	size=85%	0 0

/etc/exports
/export/spnfs	*(rw,sync,fsid=0,insecure,no_subtree_check,pnfs,no_root_squash)


Meta Data Server:
/etc/fstab
wtb9-10g:/	/spnfs/192.168.96.109	nfs4	minorversion=1,intr,soft,rsize=32768,wsiz
e=32768	0 0
wtb10-10g:/	/spnfs/192.168.96.110	nfs4	minorversion=1,intr,soft,rsize=32768,wsiz
e=32768	0 0
wtb11-10g:/	/spnfs/192.168.96.11	nfs4	minorversion=1,intr,soft,rsize=32768,wsiz
e=32768	0 0

/etc/exports
/export	*(rw,sync,pnfs,fsid=0,insecure,no_subtree_check,no_root_squash)

/etc/spnfsd.conf
[General]
Verbosity = 1
Stripe-size = 8192
Dense-striping = 0
Pipefs-Directory = /var/lib/nfs/rpc_pipefs
DS-Mount-Directory = /spnfs

[DataServers]
NumDS = 2
DS1_IP = 192.168.96.109
DS1_PORT = 2049
DS1_ROOT = /
DS1_ID = 1
DS2_IP = 192.168.96.110
DS2_PORT = 2049
DS2_ROOT = /
DS2_ID = 2


pNFS Client:
/etc/fstab
wtb8-10g:/  /mnt nfs4	minorversion=1,tcp,intr,soft,rsize=32768,wsize=32768,timeo=600 0 


The following is one of my benchmark result:
iozone -i 0 -s 1g -r 32k -f /mnt/1g -c -w


1. on Client:
[root@wtb7 pnfs-tests]# ls -l /mnt
total 132100
-rw-r----- 1 root root 1073741824 Aug  4 16:25 1g

2. on MDS
[root@wtb8 ~]# stat /export/1g
  File: `/export/1g'
  Size: 1073741824	Blocks: 264200     IO Block: 4096   regular file
Device: 805h/2053d	Inode: 491522      Links: 1
Access: (0640/-rw-r-----)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2010-08-04 16:24:48.000000000 -0700
Modify: 2010-08-04 16:25:28.000000000 -0700
Change: 2010-08-04 16:25:28.000000000 -0700

2. on MDS
[root@wtb8 ~]# stat /export/1g
  File: `/export/1g'
  Size: 1073741824	Blocks: 264200     IO Block: 4096   regular file
Device: 805h/2053d	Inode: 491522      Links: 1
Access: (0640/-rw-r-----)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2010-08-04 16:24:48.000000000 -0700
Modify: 2010-08-04 16:25:28.000000000 -0700
Change: 2010-08-04 16:25:28.000000000 -0700

3. op DS1:
[root@wtb9 ~]# stat /export/spnfs
  File: `/export/spnfs'
  Size: 60        	Blocks: 0          IO Block: 4096   directory
Device: 14h/20d	Inode: 8981        Links: 2
Access: (1777/drwxrwxrwt)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2010-08-04 16:26:49.938720191 -0700
Modify: 2010-08-04 16:58:31.959755618 -0700
Change: 2010-08-04 16:58:31.959755618 -0700

4. on DS2
[root@wtb10 ~]# stat /export/spnfs/
  File: `/export/spnfs/'
  Size: 60        	Blocks: 0          IO Block: 4096   directory
Device: 13h/19d	Inode: 8694        Links: 2
Access: (1777/drwxrwxrwt)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2010-08-04 16:26:54.573669020 -0700
Modify: 2010-08-04 16:58:18.525689302 -0700
Change: 2010-08-04 16:58:18.525689302 -0700



________________________________________
From: Chen, Helen Y
Sent: Wednesday, August 04, 2010 3:59 PM
To: 'Benny Halevy'; 'steved@xxxxxxxxxx'; 'NFS list'
Subject: RE: pNFS file layout performance

Steve and Benny,

Thank you very much for your help! I have successfully set up a 3-node testbed and ran some benchmark since.  Unfortunately, throughput results are very poor.

I am running Steve's 2.6.33.5-112.2.2.pnfs.fc13.x86_64 kernel, and exported a 28GB ramfs from my DS to the MDS. I was able to achieve ~400 MB/s over NFS using iozone. I then proceed to run the same test from the pNFS client and got only 50 MB/s.  After detecting that my test data had landed on both the MDS and DS, I assumed disk I/O on the MDS was the bottleneck. So I proceeded to rebuild the kernel with CONFIG_PNFSD_LOCAL_EXPORT disabled, but achieved only 6 MB/s afterward. Is this expected, or am I doing something wrong?  Please let me know if I need to provide further information.

Thanks,
Helen

-----Original Message-----
From: Benny Halevy [mailto:bhalevy.lists@xxxxxxxxx] On Behalf Of Benny Halevy
Sent: Wednesday, May 26, 2010 5:36 AM
To: steved@xxxxxxxxxx; NFS list
Cc: Chen, Helen Y
Subject: Fwd: Re: [pnfs] problem building pnfs-nfs-utils under Fedora 13

Helen, please note that the pnfs@xxxxxxxxxxxxx mailing list was deprecated.
Forwarding to linux-nfs@xxxxxxxxxxxxxxxx

>From a quick glance I'm not sure what went wrong with your build, Steve should know better :-)

Benny

On May. 19, 2010, 20:32 +0300, "Chen, Helen Y" <hycsw@xxxxxxxxxx> wrote:




Has anyone successfully build pnfs enabled nfs utils under Fedora 13?
I am running the kernel from:
http://steved.fedorapeople.org/repos/pnfs/13/x86_64/

I installed libtirpc{,-devel}, tcp_wrappers{,-devel}, libevent{,-devel}, nfs-utils-lib{,-devel}, libgssglue{,-devel}, libblkid{,-devel}, and libcap{,-devel}

per instructions from:
_http://wiki.linux-nfs.org/wiki/index.php/Configuring_pNFS/spnfsd#Kernel_and_nfs-utils_compilati__on_
<http://wiki.linux-nfs.org/wiki/index.php/Configuring_pNFS/spnfsd#Kernel_and_nfs-utils_compilation>

I used libnfsidmap{,-devel} bundled in:
Nfs-utils-lib-devel-1.1.5-1.fc13.X86_64.rpm

Finally, I downloaded _nfs-utils-1.2.2-4.1.pnfs.src.rpm_
<http://steved.fedorapeople.org/repos/pnfs/13/source/nfs-utils-1.2.2-4.1.pnfs.src.rpm>
from _http://steved.fedorapeople.org/repos/pnfs/13/source/_

I am having trouble building these utils. I failed to generate 'configure' when I ran autogen.sh:

/c//leaning up ............. done
//lobotomize//: putting auxiliary files in `.'.
libtoolize: copying file `./ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `aclocal'.
libtoolize: copying file `aclocal/libtool.m4'
libtoolize: copying file `aclocal/ltoptions.m4'
libtoolize: copying file `aclocal/ltsugar.m4'
libtoolize: copying file `aclocal/ltversion.m4'
libtoolize: copying file `aclocal/lt~obsolete.m4'
configure.ac:5: installing `./config.guess'
configure.ac:5: installing `./config.sub'
configure.ac:421: required file `tools/mountstats/Makefile.in' not found
configure.ac:421: required file `tools/nfs-iostat/Makefile.in' not found/

I deleted the two Makefile.in requirements from line 421 in configure.ac because there were only python scripts inside those directories.

When I ran the 'configure' generated after the modification, it failed with the following output:

/checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether to enable maintainer-specific portions of Makefiles... no checking for style of include used by make... GNU checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables...
checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking dependency style of gcc... gcc3 checking how to run the C preprocessor... gcc -E checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for ANSI C header files... no checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... no checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for clnt_tli_create in -ltirpc... yes checking /usr/include/tirpc/netconfig.h usability... yes checking /usr/include/tirpc/netconfig.h presence... yes checking for /usr/include/tirpc/netconfig.h... yes checking for prctl... yes checking for cap_get_proc in -lcap... yes checking sys/capability.h usability... yes checking sys/capability.h presence... yes checking for sys/capability.h... yes checking for libwrap... /

But libwrap is obviously installed based on the locate command:
#locate libwrap
/usr/lib/libwrap.so
/usr/lib/libwrap.so.0
/usr/lib/libwrap.so.0.7.6
/usr/lib64/libwrap.so
/usr/lib64/libwrap.so.0
/usr/lib64/libwrap.so.0.7.6

I am new at this and would appreciate any help you can provide.

Thanks,
Helen









--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [Linux Filesystem Development]     [Linux USB Development]     [Linux Media Development]     [Video for Linux]     [Linux NILFS]     [Linux Audio Users]     [Yosemite Info]     [Linux SCSI]

  Powered by Linux