Hi Thank you for your information. I don't use Infiniband Switch. My environment is the following. --- NFS Server --- InfiniHost III Ex (SDR ) ---- NFS Client. --- Regards. -- Hiroyuki Sato. 2011/4/15 sfaibish <sfaibish@xxxxxxx>: > On Fri, 15 Apr 2011 05:54:45 -0400, Hiroyuki Sato <hiroysato@xxxxxxxxx> > wrote: > >> Dear members. >> >> I'm measuring the performance NFS on IPoIB and NFS/RDMA. >> I would like to ask the following questions. >> >> 1) Benchmark result. >> >> ÂDoes anyone tryied this benchmark?? >> ÂIf so , could you please tell me that result?? >> >> â2) My test result and What is problem?? >> >> ÂI compared that. Please see my test result at below. >> >> ÂI extpected NFS/RDMA faster than NFS IPoIB. >> ÂHowever, NFS/RDMA is slower than NFS IPoIB. >> ÂExpecially rsize,wsize larger equal 32768 >> >> ÂCould you please what is problelm ?? >> >> Thank you for your information. >> >> >> 1, Environment > > First impression is that your IB switch is not configured to > support NFS/RDMA. In which case of course IPoIB is faster as > it can get all the BW from the switch. > >> >> Â(1) Server >> Â ÂCentOS 5.5 x86_64 >> Â ÂKernel: 2.36.8.2 (self build) >> Â ÂOFED: any kernel modules does not use. >> Â Â ÂOnly startup script and nfs-utils used >> Â Â Âfrom OFED-1.5.3 >> Â ÂMemory: 8GB >> Â ÂHCA: Mellanox InfiniHost III Ex. >> >> >> Â(2) Client >> Â ÂCentOS 5.5 x86_64 >> Â ÂKernel: 2.36.8.2 (self build) >> Â ÂOFED: any kernel modules does not use. >> Â Â ÂOnly startup script and nfs-utils used. >> Â Â Âfrom OFED-1.5.3 >> Â ÂMemory: 8GB >> Â ÂHCA: Mellanox InfiniHost III Ex. >> >> >> ÂClient and Server connected directly with Infiniband CX4 cable. >> >> >> 2, Configurations >> >> Â(1) NFS/RDMA >> >> Â Â1-1) Server exports (/etc/exports) >> Â Â Â/dev/shm >> Â192.168.100.0/255.255.255.0(rw,no_root_squash,insecure,fsid=0,sync) >> >> Â Â1-2) Client mount >> Â Â Âmount.rnfs 192.168.100.231:/dev/shm /mnt -i -o >> rdma,port=20049,rsize=32768,wsize=32768,sync,rw >> >> Â(2) IPoIB >> >> Â Â2-1) Server exports(/etc/exports) >> Â Â Â/dev/shm >> Â192.168.100.0/255.255.255.0(rw,no_root_squash,insecure,fsid=0,sync) >> >> Â Â2-2) Client mount >> Â Â Âmount -t nfs Â-o nfsvers=3,rsize=32768,wsize=32768,tcp,sync,rw >> 192.168.100.231:/dev/shm /mnt >> >> >> 3, test result >> >> ÂPlease see attachement file. for more information. >> >> ÂSummary: >> Â 1GB write bs=(64MB) : >> Â Â ÂNFS IPoIB : 661.499MB/sec >> Â Â ÂNFS/RDMA Â: 512.513MB/sec >> >> Â 1GB read bs=(64MB) : >> Â Â ÂNFS IPoIB : 592.250MB/sec >> Â Â ÂNFS/RDMA Â: 1.353MB/sec ( very slow ) >> >> 4, about test tool >> >> ÂThis tool based on Âblockdev-perftest which is included in SCST. >> >> Âthis tool is using fio benchmark tool internally. >> >> ÂI modified this script for file system, >> >> >> ÂThis command execute fio like the follwoing >> >> Â Â fio --rw=write # (and read)\ >> Â Â Â--directory=/mnt/fio-testing \ >> Â Â Â--bs=XXX \ >> Â Â Â--size=1073741824 \ >> Â Â Â--ioengine=psync \ >> Â Â Â--end_fsync=1 \ >> Â Â Â--invalidate=1 \ >> Â Â Â--direct=1 \ >> Â Â Â--name=writeperftest >> >> Â ÂXXX is >> >> Â Â Â67108864 >> Â Â Â33554432 >> Â Â Â16777216 >> Â Â Â8388608 >> Â Â Â... >> Â Â Â... >> >> Sincerely >> >> -- >> Hiroyuki Sato. > > > > -- > Best Regards > > Sorin Faibish > Corporate Distinguished Engineer > Unified Storage Division > Â Â Â ÂEMCÂ > where information lives > > Phone: 508-249-5745 > Cellphone: 617-510-0422 > Email : sfaibish@xxxxxxx > -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
