This is probably less than worthless, but I figured some of you might get a kick out of it. It's not pretty, but it's data. Only includes values for file creation and deletion, no copying or moving. I'll probably post this in a while with all the configs and the source of makefiles.pl (not that it's anything spectacular). If you want to see them before then, just ask. At a later date I'll probably either thread the file creation script or use a more standard benchmark, and take the average of multiple instead of single run values. Test Notes: Extremely raw, tests ran only once per defined type (no averaging). All GLFS tests were fully redundant across two servers. Same hardware in every test Bare disk utilized disk on client, not server No io-threads, read-ahead or write-behind utilized Testing file creation is not concurrent, so io-threads doesn't help. Quick summary of GLFS results in TIME(% over best case for test) format: Type 4000 1k files 400 10k files 40 100k files 100 1MB files Avg time Client 51.779(+53%) 7.247(+31%) 2.695(+42%) 55.966(+43%) = +42.25% Clnt/Srvr 47.045(+25%) 5.802(+5%) 1.897(+0%) 39.177(+0%) = +7.5% Server 37.850(+0%) 5.530(+0%) 2.239(+18%) 45.250(+16%) = +8.5% Summary Notes: Providing redundancy by implementing AFR and unify on the client seems to provide the worst performance (but it might provide better fault tolerance and recovery, as some of my tests show) AFR on servers and Unify on the client provides good performance, but I had some HA recovery problems with it I need to investigate. AFR/Unify on the server as outlined in http://www.gluster.org/docs/index.php/GlusterFS_High_Availability_Storage_with_GlusterFS seems to provide almost best case performance, and I have tested HA failing with it quite a bit. It works, but it's slow. NFS gets stomped in every case, but it's not tuned at all (besides any default tuning RHEL/CentOS may include at the kernel level). Particularly of note is the small file creation times of NFS compared to GLFS. #################################################### 10 root directories, 10 sub directories in each root, 40 files of 1k in each sub directory (4000 1k files). # Base disk # time /root/makefiles.pl /tmp/test/ real 0m0.789s user 0m0.308s sys 0m0.466s # time rm -rf /tmp/test/* real 0m0.249s user 0m0.010s sys 0m0.239s # Untuned NFS # time /root/makefiles.pl /mnt/nfs/test/ real 3m44.871s user 0m0.491s sys 0m1.334s # time rm -rf /mnt/nfs/test/* real 1m19.988s user 0m0.017s sys 0m0.606s # Untuned GLFS (client side AFR and unify) # time /root/makefiles.pl /mnt/glusterfs/test/ real 0m51.779s user 0m0.324s sys 0m0.478s # time rm -rf /mnt/glusterfs/test/* real 0m20.523s user 0m0.029s sys 0m0.274s # Untuned GLFS (server side AFR, client side unify) # time /root/makefiles.pl /mnt/glusterfs/test/ real 0m47.045s user 0m0.250s sys 0m0.543s # time rm -rf /mnt/glusterfs/test/* real 0m21.177s user 0m0.009s sys 0m0.146s # Untuned GLFS (server side AFR and unify) # time /root/makefiles.pl /mnt/glusterfs/test/ real 0m37.850s user 0m0.644s sys 0m0.630s # time rm -rf /mnt/glusterfs/test/* real 0m18.106s user 0m0.011s sys 0m0.167s #################################################### 10 root directories, 1 sub directories in each root, 40 files of 10k in each sub directory. (400 10k files) # Base disk # time /root/makefiles.pl /tmp/test/ 10 10 1 40 real 0m0.156s user 0m0.072s sys 0m0.084s # time rm -rf /tmp/test/* real 0m0.032s user 0m0.001s sys 0m0.031s # Untuned NFS # time /root/makefiles.pl /mnt/nfs/test/ 10 10 1 40 real 0m23.532s user 0m0.085s sys 0m0.222s # time rm -rf /mnt/nfs/test/* real 0m7.685s user 0m0.002s sys 0m0.068s # Untuned GLFS (client side AFR and unify) # time /root/makefiles.pl /mnt/glusterfs/test/ 10 10 1 40 real 0m7.247s user 0m0.075s sys 0m0.085s # time rm -rf /mnt/glusterfs/test/* real 0m1.787s user 0m0.001s sys 0m0.021s # Untuned GLFS (server side AFR, client side unify) # time /root/makefiles.pl /mnt/glusterfs/test/ 10 10 1 40 real 0m5.802s user 0m0.080s sys 0m0.083s # time rm -rf /mnt/glusterfs/test/* real 0m2.142s user 0m0.003s sys 0m0.021s # Untuned GLFS (server side AFR and unify) # time /root/makefiles.pl /mnt/glusterfs/test/ 10 10 1 40 real 0m5.530s user 0m0.065s sys 0m0.068s # time rm -rf /mnt/glusterfs/test/* real 0m1.760s user 0m0.000s sys 0m0.026s #################################################### 1 root directories, 1 sub directories in each root, 40 files of 100k in each sub directory. (40 100k files) # Base disk # time /root/makefiles.pl /tmp/test/ 100 1 1 40 real 0m0.098s user 0m0.052s sys 0m0.046s # time rm -rf /tmp/test/* real 0m0.011s user 0m0.001s sys 0m0.010s # Untuned NFS # time /root/makefiles.pl /mnt/nfs/test/ 100 1 1 40 real 0m3.101s user 0m0.051s sys 0m0.083s # time rm -rf /mnt/nfs/test/* real 0m0.736s user 0m0.001s sys 0m0.014s # Untuned GLFS (client side AFR and unify) # time /root/makefiles.pl /mnt/glusterfs/test/ 100 1 1 40 real 0m2.695s user 0m0.043s sys 0m0.025s # time rm -rf /mnt/glusterfs/test/* real 0m0.190s user 0m0.000s sys 0m0.006s # Untuned GLFS (server side AFR, client side unify) # time /root/makefiles.pl /mnt/glusterfs/test/ 100 1 1 40 real 0m1.897s user 0m0.051s sys 0m0.036s # time rm -rf /mnt/glusterfs/test/* real 0m0.229s user 0m0.001s sys 0m0.005s # Untuned GLFS (server side AFR and unify) # time /root/makefiles.pl /mnt/glusterfs/test/ 100 1 1 40 real 0m2.239s user 0m0.051s sys 0m0.030s # time rm -rf /mnt/glusterfs/test/* real 0m0.185s user 0m0.002s sys 0m0.003s #################################################### 10 root directories, 1 sub directories in each root, 10 files of 1000k in each sub directory. (100 1 MB files) # Base disk # time /root/makefiles.pl /tmp/test/ 1000 10 1 10 real 0m1.653s user 0m0.749s sys 0m0.890s # time rm -rf /tmp/test/* real 0m15.333s user 0m0.001s sys 0m0.133s # Untuned NFS # time /root/makefiles.pl /mnt/nfs/test/ 1000 10 1 10 real 0m19.327s user 0m0.672s sys 0m0.678s # time rm -rf /mnt/nfs/test/* real 0m2.438s user 0m0.002s sys 0m0.058s # Untuned GLFS (client side AFR and unify) # time /root/makefiles.pl /mnt/glusterfs/test/ 1000 10 1 10 real 0m55.966s user 0m0.638s sys 0m0.459s # time rm -rf /mnt/glusterfs/test/* real 0m0.671s user 0m0.002s sys 0m0.006s # Untuned GLFS (server side AFR, client side unify) # time /root/makefiles.pl /mnt/glusterfs/test/ 1000 10 1 10 real 0m39.177s user 0m0.732s sys 0m0.435s # time rm -rf /mnt/glusterfs/test/* real 0m1.088s user 0m0.002s sys 0m0.010s # Untuned GLFS (server side AFR and unify) # time /root/makefiles.pl /mnt/glusterfs/test/ 1000 10 1 10 real 0m45.250s user 0m0.831s sys 0m0.397s # time rm -rf /mnt/glusterfs/test/* real 0m0.872s user 0m0.005s sys 0m0.010s -- - Kevan Benson - A-1 Networks
