Gluster might not be doing O_DIRECT from glusterfsd process to brick even when application opens file with O_DIRECT, but you can enable this: http://stackoverflow.com/questions/37892090/enable-direct-i-o-mode-in-glusterfs HTH ----- Original Message ----- > From: "qingwei wei" <tchengwee@xxxxxxxxx> > To: fio@xxxxxxxxxxxxxxx > Sent: Tuesday, January 3, 2017 3:34:14 AM > Subject: Fio engine gfapi question > > Hi, > > I am using fio to test gluster performance using gfapi engine. My > gluster server configuration is as follow: > > Volume Name: test12bgf1 > Type: Distribute > Volume ID: 9cf4ed03-2759-4370-bcee-43881a96a597 > Status: Started > Number of Bricks: 12 > Transport-type: tcp > Bricks: > Brick1: 192.168.99.11:/data/ssd/test12bgf11 > Brick2: 192.168.99.11:/data/ssd/test12bgf12 > Brick3: 192.168.99.11:/data/ssd/test12bgf13 > Brick4: 192.168.99.11:/data/ssd/test12bgf14 > Brick5: 192.168.99.11:/data/ssd/test12bgf15 > Brick6: 192.168.99.11:/data/ssd/test12bgf16 > Brick7: 192.168.99.11:/data/ssd/test12bgf17 > Brick8: 192.168.99.11:/data/ssd/test12bgf18 > Brick9: 192.168.99.11:/data/ssd/test12bgf19 > Brick10: 192.168.99.11:/data/ssd/test12bgf110 > Brick11: 192.168.99.11:/data/ssd/test12bgf111 > Brick12: 192.168.99.11:/data/ssd/test12bgf112 > Options Reconfigured: > performance.stat-prefetch: off > features.shard-block-size: 16MB > features.shard: on > nfs.disable: true > performance.quick-read: off > performance.io-cache: off > performance.read-ahead: off > performance.readdir-ahead: on > > This is s setup without any replication. I did some fio test and found > that the write latency is much better compare to read latency. > > write latency for the test below is 64.32 us > > [root@supermicro fioResults]# fio -group_reporting -ioengine gfapi > -fallocate none -direct 1 -size 1g -filesize 1g -nrfiles 1 -openfiles > 1 -bs 4k -numjobs 1 -iodepth 1 -name test -rw randwrite -volume > test12bgf1 -brick 192.168.99.11 -ramp_time 5 -runtime 600 > test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=gfapi, iodepth=1 > fio-2.15-69-g104e > Starting 1 process > Jobs: 1 (f=0): [f(1)] [100.0% done] [0KB/13057KB/0KB /s] [0/3264/0 > iops] [eta 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=3445: Tue Jan 3 16:30:35 2017 > write: io=748976KB, bw=60183KB/s, iops=15045, runt= 12445msec > clat (usec): min=14, max=3279, avg=63.95, stdev=13.13 > lat (usec): min=14, max=3279, avg=64.32, stdev=13.13 > clat percentiles (usec): > | 1.00th=[ 50], 5.00th=[ 54], 10.00th=[ 56], 20.00th=[ 58], > | 30.00th=[ 60], 40.00th=[ 62], 50.00th=[ 63], 60.00th=[ 65], > | 70.00th=[ 67], 80.00th=[ 69], 90.00th=[ 73], 95.00th=[ 76], > | 99.00th=[ 84], 99.50th=[ 87], 99.90th=[ 94], 99.95th=[ 98], > | 99.99th=[ 106] > lat (usec) : 20=0.09%, 50=0.74%, 100=99.13%, 250=0.03%, 500=0.01% > lat (msec) : 2=0.01%, 4=0.01% > cpu : usr=84.35%, sys=15.36%, ctx=8421, majf=0, minf=11 > IO depths : 1=140.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, > >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > issued : total=r=0/w=187244/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=1 > > Run status group 0 (all jobs): > WRITE: io=748976KB, aggrb=60182KB/s, minb=60182KB/s, maxb=60182KB/s, > mint=12445msec, maxt=12445msec > > write latency for the test below is 334.82 us > > [root@supermicro fioResults]# fio -group_reporting -ioengine gfapi > -fallocate none -direct 1 -size 1g -filesize 1g -nrfiles 1 -openfiles > 1 -bs 4k -numjobs 1 -iodepth 1 -name test -rw randread -volume > test12bgf1 -brick 192.168.99.11 -ramp_time 5 -runtime 600 > test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=gfapi, iodepth=1 > fio-2.15-69-g104e > Starting 1 process > Jobs: 1 (f=0): [f(1)] [100.0% done] [2626KB/0KB/0KB /s] [656/0/0 iops] > [eta 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=3506: Tue Jan 3 16:33:17 2017 > read : io=990236KB, bw=11869KB/s, iops=2967, runt= 83429msec > clat (usec): min=225, max=2153, avg=334.48, stdev=34.85 > lat (usec): min=226, max=2153, avg=334.82, stdev=34.85 > clat percentiles (usec): > | 1.00th=[ 262], 5.00th=[ 282], 10.00th=[ 294], 20.00th=[ 310], > | 30.00th=[ 318], 40.00th=[ 322], 50.00th=[ 330], 60.00th=[ 338], > | 70.00th=[ 354], 80.00th=[ 366], 90.00th=[ 378], 95.00th=[ 394], > | 99.00th=[ 422], 99.50th=[ 430], 99.90th=[ 470], 99.95th=[ 502], > | 99.99th=[ 556] > lat (usec) : 250=0.18%, 500=99.77%, 750=0.05%, 1000=0.01% > lat (msec) : 2=0.01%, 4=0.01% > cpu : usr=14.25%, sys=3.59%, ctx=252187, majf=0, minf=8 > IO depths : 1=105.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, > >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > issued : total=r=247559/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=1 > > Run status group 0 (all jobs): > READ: io=990236KB, aggrb=11869KB/s, minb=11869KB/s, maxb=11869KB/s, > mint=83429msec, maxt=83429msec > > So is this because FIO gfapi write immediately return after data is > written to local socket buffer? > > Regards, > > Cw > -- > To unsubscribe from this list: send the line "unsubscribe fio" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
