On January 16, 2020 11:10:14 AM GMT+02:00, venky evr <venky.evr@xxxxxxxxx> wrote: >Description of problem: > >I am getting bad performance with glusterfs 7 with fio. >Any help is greatly appreicated. > >With glusterfs : > >1 node >1 brick >1 volume (ssd volume) formated with ext4 > >volume created and mounted with default values. > > > >fio --rw=write --ioengine=sync --fdatasync=1 --directory=/mnt/1 >--size=64m --bs=1024 --name=mytest > > > >fio-3.16 >Starting 1 process >mytest2: Laying out IO file (1 file / 64MiB) >Jobs: 1 (f=1): [W(1)][100.0%][w=2759KiB/s][w=2759 IOPS][eta 00m:00s] >mytest2: (groupid=0, jobs=1): err= 0: pid=11970: Thu Jan 16 13:06:50 >2020 > write: IOPS=2632, BW=2633KiB/s (2696kB/s)(64.0MiB/24892msec) > clat (usec): min=50, max=1433, avg=61.43, stdev=11.78 > lat (usec): min=50, max=1435, avg=61.68, stdev=11.82 > clat percentiles (usec): > | 1.00th=[ 54], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 56], > | 30.00th=[ 56], 40.00th=[ 57], 50.00th=[ 58], 60.00th=[ 59], > | 70.00th=[ 64], 80.00th=[ 69], 90.00th=[ 72], 95.00th=[ 79], > | 99.00th=[ 103], 99.50th=[ 121], 99.90th=[ 133], 99.95th=[ 137], > | 99.99th=[ 155] > bw ( KiB/s): min= 1834, max= 2752, per=99.94%, avg=2630.37, >stdev=146.81, samples=49 >iops : min= 1834, max= 2752, avg=2630.37, stdev=146.81, >samples=49 > lat (usec) : 100=98.72%, 250=1.28%, 750=0.01% > lat (msec) : 2=0.01% > fsync/fdatasync/sync_file_range: > sync (usec): min=245, max=13622, avg=315.54, stdev=84.93 > sync percentiles (usec): > | 1.00th=[ 255], 5.00th=[ 262], 10.00th=[ 265], 20.00th=[ 269], > | 30.00th=[ 273], 40.00th=[ 277], 50.00th=[ 281], 60.00th=[ 289], > | 70.00th=[ 310], 80.00th=[ 400], 90.00th=[ 416], 95.00th=[ 433], > | 99.00th=[ 486], 99.50th=[ 519], 99.90th=[ 676], 99.95th=[ 848], > | 99.99th=[ 914] > cpu : usr=1.41%, sys=3.50%, ctx=196761, majf=0, minf=13 >IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >>=64=0.0% >submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >>=64=0.0% >complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >>=64=0.0% > issued rwts: total=0,65536,0,0 short=65535,0,0,0 dropped=0,0,0,0 > latency : target=0, window=0, percentile=100.00%, depth=1 > >Run status group 0 (all jobs): > WRITE: bw=2633KiB/s (2696kB/s), 2633KiB/s-2633KiB/s >(2696kB/s-2696kB/s), io=64.0MiB (67.1MB), run=24892-24892msec > > >Without Glusterfs: > > >Ext4 filesystem created and mounted locally, there is no gluster >involved during this test > > mount /dev/sdb1 /mnt/1 > > >Starting 1 process >mytest: Laying out IO file (1 file / 64MiB) >Jobs: 1 (f=1): [W(1)][100.0%][w=10.2MiB/s][w=10.5k IOPS][eta 00m:00s] >mytest: (groupid=0, jobs=1): err= 0: pid=9308: Thu Jan 16 13:00:20 2020 > write: IOPS=10.2k, BW=9.99MiB/s (10.5MB/s)(64.0MiB/6407msec) > clat (nsec): min=1257, max=831943, avg=2674.65, stdev=10327.82 > lat (nsec): min=1323, max=832038, avg=2783.73, stdev=10330.58 > clat percentiles (nsec): >| 1.00th=[ 1352], 5.00th=[ 1400], 10.00th=[ 1448], 20.00th=[ >1496], >| 30.00th=[ 1560], 40.00th=[ 1656], 50.00th=[ 2320], 60.00th=[ >2448], >| 70.00th=[ 2864], 80.00th=[ 2992], 90.00th=[ 3280], 95.00th=[ >4384], >| 99.00th=[ 8896], 99.50th=[ 11328], 99.90th=[ 20608], >99.95th=[222208], > | 99.99th=[464896] > bw ( KiB/s): min= 8946, max=10536, per=99.92%, avg=10219.75, >stdev=441.53, samples=12 >iops : min= 8946, max=10536, avg=10219.75, stdev=441.53, >samples=12 > lat (usec) : 2=42.86%, 4=50.26%, 10=6.19%, 20=0.57%, 50=0.06% > lat (usec) : 100=0.01%, 250=0.01%, 500=0.04%, 750=0.01%, 1000=0.01% > fsync/fdatasync/sync_file_range: > sync (usec): min=50, max=1174, avg=93.61, stdev=62.99 > sync percentiles (usec): > | 1.00th=[ 52], 5.00th=[ 53], 10.00th=[ 54], 20.00th=[ 55], > | 30.00th=[ 55], 40.00th=[ 56], 50.00th=[ 57], 60.00th=[ 61], > | 70.00th=[ 82], 80.00th=[ 178], 90.00th=[ 194], 95.00th=[ 215], > | 99.00th=[ 253], 99.50th=[ 265], 99.90th=[ 334], 99.95th=[ 627], > | 99.99th=[ 758] > cpu : usr=2.68%, sys=16.98%, ctx=81932, majf=0, minf=16 >IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >>=64=0.0% >submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >>=64=0.0% >complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >>=64=0.0% > issued rwts: total=0,65536,0,0 short=65535,0,0,0 dropped=0,0,0,0 > latency : target=0, window=0, percentile=100.00%, depth=1 > >Run status group 0 (all jobs): > WRITE: bw=9.99MiB/s (10.5MB/s), 9.99MiB/s-9.99MiB/s >(10.5MB/s-10.5MB/s), io=64.0MiB (67.1MB), run=6407-6407msec > >Disk stats (read/write): >sdb: ios=0/98184, merge=0/16369, ticks=0/4804, in_queue=4788, >util=73.71% > > > > > >we tried following tuning parameters, but performance didnt increase. > >performance. Client-io-threads on >performance.io-thread-count 64 >server. Event-threads 4 or 8 >client. Event-threads 4 or 8 Something that come to my mind: 1. Switch to XFS 2. Set I/O scheduler of the brick to none/noop 3. Mount options of the brick : 'noatime' and if there is battery-backed cache 'nobarrier' 4. Set reasonable value for 'vm.dirty_background_bytes' & 'vm.dirty_bytes' to a reasonable values 5. Disable gluster logs or raise them at ERR or higher. Check if any of the described 'profiles' in /var/lib/gluster/groups' match your workload. Best Regards, Strahil Nikolov ________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
