| Hello Mathieu, Thanks for replying. Previously, i’ve never notice such throughput performances (around 1GBs for 1 big file) but.... The situation with a « big » set of small files wasn’t amazing but not such bad than today. The problem seems to concern exclusively the size of each file. "proof": [root@node056 tmp]# dd if=/dev/zero of=masterfile bs=1M count=1000 1000+0 enregistrements lus 1000+0 enregistrements écrits 1048576000 octets (1,0 GB) copiés, 2,09139 s, 501 MB/s [root@node056 tmp]# time split -b 1000000 -a 12 masterfile # 1MB per file real 0m42.841s user 0m0.004s sys 0m1.416s [root@node056 tmp]# rm -f xaaaaaaaaa* && sync [root@node056 tmp]# time split -b 5000000 -a 12 masterfile # 5 MB per file real 0m17.801s user 0m0.008s sys 0m1.396s [root@node056 tmp]# rm -f xaaaaaaaaa* && sync [root@node056 tmp]# time split -b 10000000 -a 12 masterfile # 10MB per file real 0m9.686s user 0m0.008s sys 0m1.451s [root@node056 tmp]# rm -f xaaaaaaaaa* && sync [root@node056 tmp]# time split -b 20000000 -a 12 masterfile # 20MB per file real 0m9.717s user 0m0.003s sys 0m1.399s [root@node056 tmp]# rm -f xaaaaaaaaa* && sync [root@node056 tmp]# time split -b 1000000 -a 12 masterfile # 10MB per file real 0m40.283s user 0m0.007s sys 0m1.390s [root@node056 tmp]# rm -f xaaaaaaaaa* && sync Higher is the generated file size, best is the performance (IO throughput and running time)… ifstat output is coherent from both client/node and server side.. a new test: [root@node056 tmp]# dd if=/dev/zero of=masterfile bs=1M count=10000 10000+0 enregistrements lus 10000+0 enregistrements écrits 10485760000 octets (10 GB) copiés, 23,0044 s, 456 MB/s [root@node056 tmp]# rm -f xaaaaaaaaa* && sync [root@node056 tmp]# time split -b 10000000 -a 12 masterfile # 10MB per file real 1m43.216s user 0m0.038s sys 0m13.407s So the performance per file is the same (despite of 10x more files) So, i dont understand why, to get the best performance, i need to create file with a size of 10MB or more. Here are my volume reconfigured options: performance.cache-max-file-size: 64MB performance.read-ahead: on performance.write-behind: on features.quota-deem-statfs: on performance.stat-prefetch: on performance.flush-behind: on features.default-soft-limit: 90% features.quota: on diagnostics.brick-log-level: CRITICAL auth.allow: localhost,127.0.0.1,10.* nfs.disable: on performance.cache-size: 1GB performance.write-behind-window-size: 4MB performance.quick-read: on performance.io-cache: on performance.io-thread-count: 64 nfs.enable-ino32: off It’s not a local cache trouble because: 1- it’s disabled in my mount command mount -t glusterfs -o transport=rdma,direct-io-mode=disable,enable-ino32 ib-storage1:vol_home /home 2- i made my test also playing with /proc/sys/vm/drop_caches 3- I note the same ifstat output from both client and server side which is coherent with the computing of bandwidth (file sizes / time (considering the replication). I think it’s not an infiniband network trouble but here are my [not default] settings: connected mode with MTU set to 65520 Do you confirm my feelings? If yes, do you have any other idea? Thanks again and thanks by advance, Geoffrey -----------------------------------------------
Geoffrey Letessier Responsable informatique & ingénieur système CNRS - UPR 9080 - Laboratoire de Biochimie Théorique Institut de Biologie Physico-Chimique 13, rue Pierre et Marie Curie - 75005 Paris Tel: 01 58 41 50 93 - eMail: geoffrey.letessier@xxxxxxx Le 20 juin 2015 à 09:12, Mathieu Chateau <mathieu.chateau@xxxxxxx> a écrit :
|
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
