Re: small files performance

Alastair Neil <ajneil.tech@xxxxxxxxx> · Tue, 10 Oct 2017 17:59:10 -0400

I just tried setting:

performance.parallel-readdir on
features.cache-invalidation on
features.cache-invalidation-timeout 600
performance.stat-prefetch
performance.cache-invalidation
performance.md-cache-timeout 600
network.inode-lru-limit 50000
performance.cache-invalidation on

and clients could not see their files with ls when accessing via a fuse mount.  The files and directories were there, however, if you accessed them directly. Server are 3.10.5 and the clients are 3.10 and 3.12.

Any ideas?

On 10 October 2017 at 10:53, Gandalf Corvotempesta <gandalf.corvotempesta@xxxxxxxxx> wrote:
2017-10-10 8:25 GMT+02:00 Karan Sandha <ksandha@xxxxxxxxxx>:
Hi Gandalf,
We have multiple tuning to do for small-files which decrease the time for negative lookups , meta-data caching, parallel readdir. Bumping the server and client event threads will help you out in increasing the small file performance. 

gluster v set <vol-name>  group metadata-cache
gluster v set <vol-name> group nl-cache
gluster v set <vol-name> performance.parallel-readdir on (Note : readdir should be on)

This is what i'm getting with suggested parameters.
I'm running "fio" from a mounted gluster client:
172.16.0.12:/gv0 on /mnt2 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

# fio --ioengine=libaio     --filename=fio.test     --size=256M     --direct=1     --rw=randrw     --refill_buffers     --norandommap     --bs=8k     --rwmixread=70     --iodepth=16     --numjobs=16     --runtime=60     --group_reporting     --name=fio-test
fio-test: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=16
...
fio-2.16
Starting 16 processes
fio-test: Laying out IO file(s) (1 file(s) / 256MB)
Jobs: 14 (f=13): [m(5),_(1),m(8),f(1),_(1)] [33.9% done] [1000KB/440KB/0KB /s] [125/55/0 iops] [eta 01m:59s]
fio-test: (groupid=0, jobs=16): err= 0: pid=2051: Tue Oct 10 16:51:46 2017
  read : io=43392KB, bw=733103B/s, iops=89, runt= 60610msec
    slat (usec): min=14, max=1992.5K, avg=177873.67, stdev=382294.06
    clat (usec): min=768, max=6016.8K, avg=1871390.57, stdev=1082220.06
     lat (usec): min=872, max=6630.6K, avg=2049264.23, stdev=1158405.41
    clat percentiles (msec):
     |  1.00th=[   20],  5.00th=[  208], 10.00th=[  457], 20.00th=[  873],
     | 30.00th=[ 1237], 40.00th=[ 1516], 50.00th=[ 1795], 60.00th=[ 2073],
     | 70.00th=[ 2442], 80.00th=[ 2835], 90.00th=[ 3326], 95.00th=[ 3785],
     | 99.00th=[ 4555], 99.50th=[ 4948], 99.90th=[ 5211], 99.95th=[ 5800],
     | 99.99th=[ 5997]
  write: io=18856KB, bw=318570B/s, iops=38, runt= 60610msec
    slat (usec): min=17, max=3428, avg=212.62, stdev=287.88
    clat (usec): min=59, max=6015.6K, avg=1693729.12, stdev=1003122.83
     lat (usec): min=79, max=6015.9K, avg=1693941.74, stdev=1003126.51
    clat percentiles (usec):
     |  1.00th=[  724],  5.00th=[144384], 10.00th=[403456], 20.00th=[765952],
     | 30.00th=[1105920], 40.00th=[1368064], 50.00th=[1630208], 60.00th=[1875968],
     | 70.00th=[2179072], 80.00th=[2572288], 90.00th=[3031040], 95.00th=[3489792],
     | 99.00th=[4227072], 99.50th=[4423680], 99.90th=[4751360], 99.95th=[5210112],
     | 99.99th=[5996544]
    lat (usec) : 100=0.15%, 250=0.05%, 500=0.06%, 750=0.09%, 1000=0.05%
    lat (msec) : 2=0.28%, 4=0.09%, 10=0.15%, 20=0.39%, 50=1.81%
    lat (msec) : 100=1.02%, 250=1.63%, 500=5.59%, 750=6.03%, 1000=7.31%
    lat (msec) : 2000=35.61%, >=2000=39.67%
  cpu          : usr=0.01%, sys=0.01%, ctx=8218, majf=11, minf=295
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=96.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=5424/w=2357/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: io=43392KB, aggrb=715KB/s, minb=715KB/s, maxb=715KB/s, mint=60610msec, maxt=60610msec
  WRITE: io=18856KB, aggrb=311KB/s, minb=311KB/s, maxb=311KB/s, mint=60610msec, maxt=60610msec

_______________________________________________

Gluster-users mailing list

Gluster-users@xxxxxxxxxxx

http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users