Description of problem:
I am getting bad performance with glusterfs 7 with fio.
Any help is greatly appreicated.
With glusterfs :
1 node
1 brick
1 volume (ssd volume) formated with ext4
volume created and mounted with default values.
fio --rw=write --ioengine=sync --fdatasync=1 --directory=/mnt/1 --size=64m --bs=1024 --name=mytest
fio-3.16
Starting 1 process
mytest2: Laying out IO file (1 file / 64MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=2759KiB/s][w=2759 IOPS][eta 00m:00s]
mytest2: (groupid=0, jobs=1): err= 0: pid=11970: Thu Jan 16 13:06:50 2020
write: IOPS=2632, BW=2633KiB/s (2696kB/s)(64.0MiB/24892msec)
clat (usec): min=50, max=1433, avg=61.43, stdev=11.78
lat (usec): min=50, max=1435, avg=61.68, stdev=11.82
clat percentiles (usec):
| 1.00th=[ 54], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 56],
| 30.00th=[ 56], 40.00th=[ 57], 50.00th=[ 58], 60.00th=[ 59],
| 70.00th=[ 64], 80.00th=[ 69], 90.00th=[ 72], 95.00th=[ 79],
| 99.00th=[ 103], 99.50th=[ 121], 99.90th=[ 133], 99.95th=[ 137],
| 99.99th=[ 155]
bw ( KiB/s): min= 1834, max= 2752, per=99.94%, avg=2630.37, stdev=146.81, samples=49
iops : min= 1834, max= 2752, avg=2630.37, stdev=146.81, samples=49
lat (usec) : 100=98.72%, 250=1.28%, 750=0.01%
lat (msec) : 2=0.01%
fsync/fdatasync/sync_file_range:
sync (usec): min=245, max=13622, avg=315.54, stdev=84.93
sync percentiles (usec):
| 1.00th=[ 255], 5.00th=[ 262], 10.00th=[ 265], 20.00th=[ 269],
| 30.00th=[ 273], 40.00th=[ 277], 50.00th=[ 281], 60.00th=[ 289],
| 70.00th=[ 310], 80.00th=[ 400], 90.00th=[ 416], 95.00th=[ 433],
| 99.00th=[ 486], 99.50th=[ 519], 99.90th=[ 676], 99.95th=[ 848],
| 99.99th=[ 914]
cpu : usr=1.41%, sys=3.50%, ctx=196761, majf=0, minf=13
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,65536,0,0 short=65535,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=2633KiB/s (2696kB/s), 2633KiB/s-2633KiB/s (2696kB/s-2696kB/s), io=64.0MiB (67.1MB), run=24892-24892msec
Without Glusterfs:
Ext4 filesystem created and mounted locally, there is no gluster involved during this test
mount /dev/sdb1 /mnt/1
Starting 1 process
mytest: Laying out IO file (1 file / 64MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=10.2MiB/s][w=10.5k IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=1): err= 0: pid=9308: Thu Jan 16 13:00:20 2020
write: IOPS=10.2k, BW=9.99MiB/s (10.5MB/s)(64.0MiB/6407msec)
clat (nsec): min=1257, max=831943, avg=2674.65, stdev=10327.82
lat (nsec): min=1323, max=832038, avg=2783.73, stdev=10330.58
clat percentiles (nsec):
| 1.00th=[ 1352], 5.00th=[ 1400], 10.00th=[ 1448], 20.00th=[ 1496],
| 30.00th=[ 1560], 40.00th=[ 1656], 50.00th=[ 2320], 60.00th=[ 2448],
| 70.00th=[ 2864], 80.00th=[ 2992], 90.00th=[ 3280], 95.00th=[ 4384],
| 99.00th=[ 8896], 99.50th=[ 11328], 99.90th=[ 20608], 99.95th=[222208],
| 99.99th=[464896]
bw ( KiB/s): min= 8946, max=10536, per=99.92%, avg=10219.75, stdev=441.53, samples=12
iops : min= 8946, max=10536, avg=10219.75, stdev=441.53, samples=12
lat (usec) : 2=42.86%, 4=50.26%, 10=6.19%, 20=0.57%, 50=0.06%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.04%, 750=0.01%, 1000=0.01%
fsync/fdatasync/sync_file_range:
sync (usec): min=50, max=1174, avg=93.61, stdev=62.99
sync percentiles (usec):
| 1.00th=[ 52], 5.00th=[ 53], 10.00th=[ 54], 20.00th=[ 55],
| 30.00th=[ 55], 40.00th=[ 56], 50.00th=[ 57], 60.00th=[ 61],
| 70.00th=[ 82], 80.00th=[ 178], 90.00th=[ 194], 95.00th=[ 215],
| 99.00th=[ 253], 99.50th=[ 265], 99.90th=[ 334], 99.95th=[ 627],
| 99.99th=[ 758]
cpu : usr=2.68%, sys=16.98%, ctx=81932, majf=0, minf=16
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,65536,0,0 short=65535,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=9.99MiB/s (10.5MB/s), 9.99MiB/s-9.99MiB/s (10.5MB/s-10.5MB/s), io=64.0MiB (67.1MB), run=6407-6407msec
Disk stats (read/write):
sdb: ios=0/98184, merge=0/16369, ticks=0/4804, in_queue=4788, util=73.71%
we tried following tuning parameters, but performance didnt increase.
performance. Client-io-threads on
performance.io-thread-count 64
server. Event-threads 4 or 8
client. Event-threads 4 or 8________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
