I cannot seem to get fio to use cached I/O. I have tried using libaio, posixaio and sync I/O but when I switch from direct=1 to direct=0 I get either the same or worse numbers. When I use iozone in a similar setup my numbers are, as expected, much higher when specifying cached I/O.
I have used both version 1.41 and 1.55 with the same results.
Uname âa
Linux morrison 2.6.32-71.el6.x86_64 #1 SMP Wed Sep 1 01:33:01 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
It does look like the page cache is being used...
[root@morrison disk]# head /proc/meminfo
MemTotal: 24545172 kB
MemFree: 19167944 kB
Buffers: 28040 kB
Cached: 4 406440 kB
SwapCached: 0 kB
fio test using direct=1
[root@morrison disk]# fio --name=1file --ioengine=libaio --direct=0 --size=4g --numjobs=1 --directory=/disk --iodepth=8 --rw=randrw --bs=4k
1file: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=8
Starting 1 process
1file: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m] [100.0% done] [45M/46M /s] [11K/11K iops] [eta 00m:00s]
1file: (groupid=0, jobs=1): err= 0: pid=8084
read : io=2,049MB, bw=45,305KB/s, iops=11,326, runt= 46311msec
slat (usec): min=67, max=1,335, avg=76.71, stdev=20.35
clat (usec): min=38, max=2,625, avg=310.30, stdev=114.25
bw (KB/s) : min=44720, max=46992, per=100.13%, avg=45364.84, stdev=515.30
write: io=2,047MB, bw=45,263KB/s, iops=11,315, runt= 46311msec
slat (usec): min=3, max=591, avg= 6.54, stdev= 2.07
clat (usec): min=36, max=2,675, avg=309.65, stdev=119.88
bw (KB/s) : min=42816, max=48024, per=100.12%, avg=45315.30, stdev=886.71
cpu : usr=5.34%, sys=19.78%, ctx=525848, majf=0, minf=88
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w: total=524527/524049, short=0/0
lat (usec): 50=0.13%, 100=0.66%, 250=21.96%, 500=75.73%, 750=1.03%
lat (usec): 1000=0.17%
lat (msec): 2=0.30%, 4=0.02%
Run status group 0 (all jobs):
READ: io=2,049MB, aggrb=45,304KB/s, minb=46,392KB/s, maxb=46,392KB/s, mint=46311msec, maxt=46311msec
WRITE: io=2,047MB, aggrb=45,263KB/s, minb=46,349KB/s, maxb=46,349KB/s, mint=46311msec, maxt=46311msec
Disk stats (read/write):
rssda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
fio test using direct=0
[root@morrison disk]# fio --name=1file --ioengine=libaio --direct=0 --size=4g --numjobs=1 --directory=/disk --iodepth=8 --rw=randrw --bs=4k
1file: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=8
Starting 1 process
Jobs: 1 (f=1): [m] [100.0% done] [45M/45M /s] [11K/11K iops] [eta 00m:00s]
1file: (groupid=0, jobs=1): err= 0: pid=8089
read : io=2,050MB, bw=43,952KB/s, iops=10,988, runt= 47751msec
slat (usec): min=67, max=1,078, avg=76.88, stdev=21.11
clat (usec): min=39, max=2,989, avg=311.50, stdev=116.29
bw (KB/s) : min= 3, max=46152, per=101.75%, avg=44721.67, stdev=4710.82
write: io=2,046MB, bw=43,885KB/s, iops=10,971, runt= 47751msec
slat (usec): min=3, max=648, avg= 6.66, stdev= 2.04
clat (usec): min=36, max=2,786, avg=310.84, stdev=121.66
bw (KB/s) : min= 3, max=47088, per=101.75%, avg=44653.02, stdev=4759.19
cpu : usr=5.31%, sys=21.87%, ctx=526068, majf=0, minf=89
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w: total=524691/523885, short=0/0
lat (usec): 50=0.12%, 100=0.65%, 250=21.87%, 500=75.84%, 750=1.03%
lat (usec): 1000=0.15%
lat (msec): 2=0.32%, 4=0.02%
Run status group 0 (all jobs):
READ: io=2,050MB, aggrb=43,952KB/s, minb=45,007KB/s, maxb=45,007KB/s, mint=47751msec, maxt=47751msec
WRITE: io=2,046MB, aggrb=43,884KB/s, minb=44,937KB/s, maxb=44,937KB/s, mint=47751msec, maxt=47751msec
Disk stats (read/write):
rssda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
iozone test using direct I/O
[root@morrison disk]# iozone -i8 -r4k -s4g -k8 -w -t1 -O -I
Iozone: Performance Test of File I/O
Version $Revision: 3.321 $
Compiled for 64 bit mode.
Build: linux-AMD64
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.
Run began: Tue May 31 15:34:52 2011
Record Size 4 KB
File size set to 4194304 KB
POSIX Async I/O (no bcopy). Depth 8
Setting no_unlink
OPS Mode. Output is in operations per second.
O_DIRECT feature enabled
Command line used: iozone -i8 -r4k -s4g -k8 -w -t1 -O -I
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 1 process
Each process writes a 4194304 Kbyte file in 4 Kbyte records
Children see throughput for 1 mixed workload = 11788.53 ops/sec
Parent sees throughput for 1 mixed workload = 11788.47 ops/sec
Min throughput per process = 11788.53 ops/sec
Max throughput per process = 11788.53 ops/sec
Avg throughput per process = 11788.53 ops/sec
Min xfer = 1048576.00 ops
iozone test complete.
iozone test using cache I/O
[root@morrison disk]# iozone -i8 -r4k -s4g -k8 -w -t1 -O
Iozone: Performance Test of File I/O
Version $Revision: 3.321 $
Compiled for 64 bit mode.
Build: linux-AMD64
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.
Run began: Tue May 31 15:37:35 2011
Record Size 4 KB
File size set to 4194304 KB
POSIX Async I/O (no bcopy). Depth 8
Setting no_unlink
OPS Mode. Output is in operations per second.
Command line used: iozone -i8 -r4k -s4g -k8 -w -t1 -O
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 1 process
Each process writes a 4194304 Kbyte file in 4 Kbyte records
Children see throughput for 1 mixed workload = 90633.90 ops/sec
Parent sees throughput for 1 mixed workload = 90629.44 ops/sec
Min throughput per process = 90633.90 ops/sec
Max throughput per process = 90633.90 ops/sec
Avg throughput per process = 90633.90 ops/sec
Min xfer = 1048576.00 ops
iozone test complete.
Any help would be appreciated since I much prefer fio to iozone.
Gordon Bookless
--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
