Dear members.
I'm measuring the performance NFS on IPoIB and NFS/RDMA.
I would like to ask the following questions.
1) Benchmark result.
Does anyone tryied this benchmark??
If so , could you please tell me that result??
â2) My test result and What is problem??
I compared that. Please see my test result at below.
I extpected NFS/RDMA faster than NFS IPoIB.
However, NFS/RDMA is slower than NFS IPoIB.
Expecially rsize,wsize larger equal 32768
Could you please what is problelm ??
Thank you for your information.
1, Environment
(1) Server
CentOS 5.5 x86_64
Kernel: 2.36.8.2 (self build)
OFED: any kernel modules does not use.
Only startup script and nfs-utils used
from OFED-1.5.3
Memory: 8GB
HCA: Mellanox InfiniHost III Ex.
(2) Client
CentOS 5.5 x86_64
Kernel: 2.36.8.2 (self build)
OFED: any kernel modules does not use.
Only startup script and nfs-utils used.
from OFED-1.5.3
Memory: 8GB
HCA: Mellanox InfiniHost III Ex.
Client and Server connected directly with Infiniband CX4 cable.
2, Configurations
(1) NFS/RDMA
1-1) Server exports (/etc/exports)
/dev/shm 192.168.100.0/255.255.255.0(rw,no_root_squash,insecure,fsid=0,sync)
1-2) Client mount
mount.rnfs 192.168.100.231:/dev/shm /mnt -i -o
rdma,port=20049,rsize=32768,wsize=32768,sync,rw
(2) IPoIB
2-1) Server exports(/etc/exports)
/dev/shm 192.168.100.0/255.255.255.0(rw,no_root_squash,insecure,fsid=0,sync)
2-2) Client mount
mount -t nfs -o nfsvers=3,rsize=32768,wsize=32768,tcp,sync,rw
192.168.100.231:/dev/shm /mnt
3, test result
Please see attachement file. for more information.
Summary:
1GB write bs=(64MB) :
NFS IPoIB : 661.499MB/sec
NFS/RDMA : 512.513MB/sec
1GB read bs=(64MB) :
NFS IPoIB : 592.250MB/sec
NFS/RDMA : 1.353MB/sec ( very slow )
4, about test tool
This tool based on blockdev-perftest which is included in SCST.
this tool is using fio benchmark tool internally.
I modified this script for file system,
This command execute fio like the follwoing
fio --rw=write # (and read)\
--directory=/mnt/fio-testing \
--bs=XXX \
--size=1073741824 \
--ioengine=psync \
--end_fsync=1 \
--invalidate=1 \
--direct=1 \
--name=writeperftest
XXX is
67108864
33554432
16777216
8388608
...
...
Sincerely
--
Hiroyuki Sato.
##################################################################################
#
# NFS IPoIB
#
##################################################################################
blocksize W W(avg, W(std, W R R(avg, R(std, R
(bytes) (s) MB/s) MB/s) (IOPS) (s) MB/s) MB/s) (IOPS)
67108864 1.548 661.499 0.000 10.336 1.729 592.250 0.000 9.254
33554432 1.563 655.150 0.000 20.473 1.738 589.183 0.000 18.412
16777216 1.567 653.478 0.000 40.842 1.731 591.566 0.000 36.973
8388608 1.593 642.812 0.000 80.352 1.765 580.170 0.000 72.521
4194304 1.566 653.895 0.000 163.474 1.793 571.110 0.000 142.777
2097152 1.642 623.630 0.000 311.815 1.727 592.936 0.000 296.468
1048576 1.752 584.475 0.000 584.475 1.725 593.623 0.000 593.623
524288 2.009 509.706 0.000 1019.413 1.839 556.824 0.000 1113.649
262144 2.367 432.615 0.000 1730.460 2.128 481.203 0.000 1924.812
131072 2.934 349.012 0.000 2792.093 2.597 394.301 0.000 3154.409
65536 4.323 236.873 0.000 3789.961 3.342 306.403 0.000 4902.454
32768 4.824 212.272 0.000 6792.703 4.526 226.248 0.000 7239.947
16384 6.811 150.345 0.000 9622.082 6.372 160.703 0.000 10284.997
8192 10.747 95.282 0.000 12196.148 10.499 97.533 0.000 12484.237
4096 18.409 55.625 0.000 14239.991 17.611 58.145 0.000 14885.242
2048 34.319 29.838 0.000 15276.902 32.964 31.064 0.000 15904.866
1024 65.221 15.700 0.000 16077.276 62.440 16.400 0.000 16793.338
512 126.783 8.077 0.000 16541.271 121.787 8.408 0.000 17219.835
##################################################################################
#
# NFS RDMA
#
##################################################################################
blocksize W W(avg, W(std, W R R(avg, R(std, R
(bytes) (s) MB/s) MB/s) (IOPS) (s) MB/s) MB/s) (IOPS)
67108864 1.998 512.513 0.000 8.008 756.578 1.353 0.000 0.021
33554432 2.001 511.744 0.000 15.992
** I stoppted this test.
##################################################################################
#
# ib_read_bw
#
##################################################################################
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
2 1000 0.33 0.33
4 1000 0.66 0.66
8 1000 1.32 1.32
16 1000 2.62 2.61
32 1000 5.52 5.22
64 1000 10.90 10.90
128 1000 20.76 20.73
256 1000 41.56 41.38
512 1000 83.03 82.23
1024 1000 168.08 162.31
2048 1000 320.71 318.58
4096 1000 537.50 536.40
8192 1000 758.71 757.27
16384 1000 845.21 844.48
32768 1000 890.46 890.38
65536 1000 912.59 912.51
131072 1000 923.86 923.85
262144 1000 929.80 929.79
524288 1000 931.76 931.75
1048576 1000 933.19 933.19
2097152 1000 933.96 933.95
4194304 1000 934.32 934.32
8388608 1000 934.53 934.53
------------------------------------------------------------------
########################################################################################################################
#
# another NFS/RDMA test rsize,rsize=8192, 16384, 32768
#
########################################################################################################################
**** 2011-04-13 22:18:22 ************
**** size=8192 ************
blocksize W W W W(avg, W(std, W R R R R(avg, R(std, R
(bytes) (s) (s) (s) MB/s) MB/s) (IOPS) (s) (s) (s) MB/s) MB/s) (IOPS)
67108864 2.444 2.439 2.449 418.986 0.700 6.547 1.674 1.683 1.680 609.890 1.360 9.530
33554432 2.454 2.431 2.464 418.029 2.364 13.063 1.688 1.679 1.681 608.561 1.394 19.018
16777216 2.444 2.461 2.457 417.282 1.236 26.080 1.686 1.681 1.683 608.318 0.742 38.020
8388608 2.461 2.482 2.465 414.692 1.526 51.837 1.685 1.688 1.695 606.160 1.502 75.770
4194304 2.475 2.492 2.495 411.691 1.461 102.923 1.696 1.699 1.697 603.300 0.443 150.825
2097152 2.529 2.523 2.536 404.852 0.850 202.426 1.720 1.723 1.724 594.543 0.587 297.271
1048576 2.616 2.609 2.620 391.588 0.681 391.588 1.780 1.775 1.772 576.687 1.071 576.687
524288 2.756 2.787 2.779 369.150 1.753 738.301 1.929 1.924 1.917 532.413 1.363 1064.825
262144 3.204 3.140 3.187 322.340 2.758 1289.361 2.097 2.092 2.113 487.473 2.074 1949.893
131072 3.688 3.676 3.701 277.634 0.768 2221.074 2.473 2.490 2.474 413.074 1.295 3304.591
65536 4.573 4.562 4.576 224.054 0.295 3584.865 3.368 3.354 3.382 304.042 1.032 4864.664
32768 6.854 6.891 6.896 148.831 0.406 4762.595 4.592 4.570 4.638 222.617 1.368 7123.748
16384 11.250 11.188 11.222 91.266 0.206 5841.028 7.045 7.037 7.019 145.586 0.225 9317.495
8192 11.823 11.765 11.805 86.797 0.178 11110.040 9.523 9.520 9.515 107.571 0.037 13769.033
4096 20.751 20.771 20.635 49.424 0.143 12652.454 17.926 17.932 17.917 57.127 0.020 14624.493
2048 39.540 39.485 39.401 25.940 0.038 13281.436 35.677 35.654 35.653 28.715 0.009 14701.863
1024 76.246 76.427 76.217 13.421 0.016 13743.425 71.061 71.065 71.055 14.410 0.001 14756.137
512 142.273 142.352 142.209 7.197 0.003 14739.822 132.722 132.492 132.666 7.721 0.006 15812.455
**** 2011-04-13 23:58:34 ************
**** size=16384 ************
blocksize W W W W(avg, W(std, W R R R R(avg, R(std, R
(bytes) (s) (s) (s) MB/s) MB/s) (IOPS) (s) (s) (s) MB/s) MB/s) (IOPS)
67108864 2.066 2.064 2.053 496.850 1.380 7.763 1.188 1.181 1.187 863.898 2.257 13.498
33554432 2.055 2.057 2.063 497.491 0.821 15.547 1.200 1.184 1.187 860.292 5.001 26.884
16777216 2.062 2.070 2.072 495.167 1.036 30.948 1.194 1.188 1.188 860.509 2.042 53.782
8388608 2.070 2.083 2.086 492.392 1.648 61.549 1.200 1.194 1.200 854.763 2.021 106.845
4194304 2.095 2.111 2.106 486.697 1.548 121.674 1.212 1.211 1.212 845.117 0.329 211.279
2097152 2.138 2.159 2.136 477.549 2.309 238.774 1.244 1.243 1.243 823.593 0.312 411.796
1048576 2.239 2.247 2.230 457.420 1.419 457.420 1.306 1.302 1.303 785.478 1.023 785.478
524288 2.418 2.426 2.402 423.965 1.754 847.931 1.479 1.472 1.482 692.990 1.967 1385.980
262144 2.772 2.753 2.751 371.198 1.270 1484.793 1.630 1.631 1.630 628.092 0.182 2512.370
131072 3.237 3.249 3.235 316.018 0.602 2528.144 1.958 1.950 1.939 525.406 2.101 4203.248
65536 4.426 4.435 4.440 230.960 0.302 3695.368 2.542 2.536 2.514 404.646 1.930 6474.330
32768 6.826 6.824 6.823 150.051 0.027 4801.641 3.718 3.719 3.709 275.615 0.334 8819.679
16384 7.281 7.286 7.284 140.589 0.040 8997.667 5.331 5.339 5.328 192.024 0.167 12289.546
8192 11.798 11.910 11.925 86.214 0.413 11035.416 9.527 9.527 9.537 107.446 0.053 13753.142
4096 20.925 20.802 20.846 49.095 0.120 12568.307 17.955 17.950 17.937 57.056 0.024 14606.295
2048 39.976 39.984 39.841 25.643 0.042 13129.008 35.746 35.731 35.678 28.669 0.023 14678.410
1024 76.671 76.793 76.936 13.333 0.019 13653.360 71.187 71.197 71.138 14.387 0.005 14732.573
512 142.352 142.240 142.347 7.195 0.003 14736.196 132.752 132.504 132.624 7.721 0.006 15812.455
**** 2011-04-14 01:19:53 ************
**** size=32768 ************
blocksize W W W W(avg, W(std, W R R R R(avg, R(std, R
(bytes) (s) (s) (s) MB/s) MB/s) (IOPS) (s) (s) (s) MB/s) MB/s) (IOPS)
67108864 2.007 2.000 2.007 510.810 0.842 7.981 396.626 541.706 812.201 1.911 0.539 0.030
33554432 2.009 2.028 2.016 507.525 1.971 15.860 722.354 902.984 632.485 1.390 0.199 0.043
16777216 2.017 2.030 2.025 505.932 1.339 31.621 1714.260 1263.500 2164.890 0.627 0.139 0.039
8388608 2.032 2.048 2.047 501.394 1.801 62.674 3387.870 4057.120 3246.640 0.290 0.027 0.036
4194304 2.049 2.060 2.049 498.866 1.258
-- stoppted here --
