Ok, never mind, it looks like I need to set invalidate=0 for this to work as expected. --- On Tue, 5/31/11, Gordon Bookless <ghhb@xxxxxxxxx> wrote: > From: Gordon Bookless <ghhb@xxxxxxxxx> > Subject: fio does not seem to work correctly when specifying buffered I/O > To: fio@xxxxxxxxxxxxxxx > Date: Tuesday, May 31, 2011, 3:53 PM > I cannot seem to get fio to use > cached I/O. I have tried using libaio, posixaio and sync I/O > but when I switch from direct=1 to direct=0 I get either the > same or worse numbers. When I use iozone in a similar setup > my numbers are, as expected, much higher when specifying > cached I/O. > I have used both version 1.41 and 1.55 with the same > results. > > Uname âa > Linux morrison 2.6.32-71.el6.x86_64 #1 SMP Wed Sep 1 > 01:33:01 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux > > It does look like the page cache is being used... > [root@morrison disk]# head /proc/meminfo > MemTotal:    24545172 kB > MemFree:    19167944 kB > Buffers:       >   28040 kB > Cached:     > 4ÂÂÂ406440 kB > SwapCached:       >  0 kB > > fio test using direct=1 > > [root@morrison disk]# fio --name=1file --ioengine=libaio > --direct=0 --size=4g --numjobs=1 --directory=/disk > --iodepth=8 --rw=randrw --bs=4k > 1file: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, > iodepth=8 > Starting 1 process > 1file: Laying out IO file(s) (1 file(s) / 4096MB) > Jobs: 1 (f=1): [m] [100.0% done] [45M/46M /s] [11K/11K > iops] [eta 00m:00s] > 1file: (groupid=0, jobs=1): err= 0: pid=8084 >  read : io=2,049MB, bw=45,305KB/s, iops=11,326, runt= > 46311msec >   slat (usec): min=67, max=1,335, avg=76.71, > stdev=20.35 >   clat (usec): min=38, max=2,625, avg=310.30, > stdev=114.25 >   bw (KB/s) : min=44720, max=46992, > per=100.13%, avg=45364.84, stdev=515.30 >  write: io=2,047MB, bw=45,263KB/s, iops=11,315, runt= > 46311msec >   slat (usec): min=3, max=591, avg= 6.54, > stdev= 2.07 >   clat (usec): min=36, max=2,675, avg=309.65, > stdev=119.88 >   bw (KB/s) : min=42816, max=48024, > per=100.12%, avg=45315.30, stdev=886.71 >  cpu     : usr=5.34%, > sys=19.78%, ctx=525848, majf=0, minf=88 >  IO depths  : 1=0.1%, 2=0.1%, 4=0.1%, > 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% >  ÂÂÂsubmit  : 0=0.0%, > 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% >  ÂÂÂcomplete : 0=0.0%, 4=100.0%, > 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% >  ÂÂÂissued r/w: total=524527/524049, > short=0/0 >  ÂÂÂlat (usec): 50=0.13%, 100=0.66%, > 250=21.96%, 500=75.73%, 750=1.03% >  ÂÂÂlat (usec): 1000=0.17% >  ÂÂÂlat (msec): 2=0.30%, 4=0.02% > > Run status group 0 (all jobs): > ÂÂÂREAD: io=2,049MB, aggrb=45,304KB/s, > minb=46,392KB/s, maxb=46,392KB/s, mint=46311msec, > maxt=46311msec >  WRITE: io=2,047MB, aggrb=45,263KB/s, > minb=46,349KB/s, maxb=46,349KB/s, mint=46311msec, > maxt=46311msec > > Disk stats (read/write): >  rssda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, > util=0.00% > > fio test using direct=0 > [root@morrison disk]# fio --name=1file --ioengine=libaio > --direct=0 --size=4g --numjobs=1 --directory=/disk > --iodepth=8 --rw=randrw --bs=4k > 1file: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, > iodepth=8 > Starting 1 process > Jobs: 1 (f=1): [m] [100.0% done] [45M/45M /s] [11K/11K > iops] [eta 00m:00s] > 1file: (groupid=0, jobs=1): err= 0: pid=8089 >  read : io=2,050MB, bw=43,952KB/s, iops=10,988, runt= > 47751msec >   slat (usec): min=67, max=1,078, avg=76.88, > stdev=21.11 >   clat (usec): min=39, max=2,989, avg=311.50, > stdev=116.29 >   bw (KB/s) : min=  3, max=46152, > per=101.75%, avg=44721.67, stdev=4710.82 >  write: io=2,046MB, bw=43,885KB/s, iops=10,971, runt= > 47751msec >   slat (usec): min=3, max=648, avg= 6.66, > stdev= 2.04 >   clat (usec): min=36, max=2,786, avg=310.84, > stdev=121.66 >   bw (KB/s) : min=  3, max=47088, > per=101.75%, avg=44653.02, stdev=4759.19 >  cpu     : usr=5.31%, > sys=21.87%, ctx=526068, majf=0, minf=89 >  IO depths  : 1=0.1%, 2=0.1%, 4=0.1%, > 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% >  ÂÂÂsubmit  : 0=0.0%, > 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% >  ÂÂÂcomplete : 0=0.0%, 4=100.0%, > 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% >  ÂÂÂissued r/w: total=524691/523885, > short=0/0 >  ÂÂÂlat (usec): 50=0.12%, 100=0.65%, > 250=21.87%, 500=75.84%, 750=1.03% >  ÂÂÂlat (usec): 1000=0.15% >  ÂÂÂlat (msec): 2=0.32%, 4=0.02% > > Run status group 0 (all jobs): > ÂÂÂREAD: io=2,050MB, aggrb=43,952KB/s, > minb=45,007KB/s, maxb=45,007KB/s, mint=47751msec, > maxt=47751msec >  WRITE: io=2,046MB, aggrb=43,884KB/s, > minb=44,937KB/s, maxb=44,937KB/s, mint=47751msec, > maxt=47751msec > > Disk stats (read/write): >  rssda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, > util=0.00% > > iozone test using direct I/O > [root@morrison disk]# iozone -i8 -r4k -s4g -k8 -w -t1 -O > -I >     Iozone: Performance Test of > File I/O >         > Version $Revision: 3.321 $ >         > Compiled for 64 bit mode. >         > Build: linux-AMD64 > >     Contributors:William Norcott, > Don Capps, Isom Crawford, Kirby Collins >         >  ÂÂÂAl Slater, Scott Rhine, Mike > Wisner, Ken Goss >         >  ÂÂÂSteve Landherr, Brad Smith, Mark > Kelly, Dr. Alain CYR, >         >  ÂÂÂRandy Dunlap, Mark Montague, Dan > Million, Gavin Brebner, >         >  ÂÂÂJean-Marc Zucconi, Jeff Blomberg, > Benny Halevy, >         >  ÂÂÂErik Habbinga, Kris Strecker, > Walter Wong, Joshua Root. > >     Run began: Tue May 31 15:34:52 > 2011 > >     Record Size 4 KB >     File size set to 4194304 KB >     POSIX Async I/O (no bcopy). > Depth 8 >     Setting no_unlink >     OPS Mode. Output is in > operations per second. >     O_DIRECT feature enabled >     Command line used: iozone -i8 > -r4k -s4g -k8 -w -t1 -O -I >     Time Resolution = 0.000001 > seconds. >     Processor cache size set to > 1024 Kbytes. >     Processor cache line size set > to 32 bytes. >     File stride size set to 17 * > record size. >     Throughput test with 1 process >     Each process writes a 4194304 > Kbyte file in 4 Kbyte records > >     Children see throughput for 1 > mixed workload  =ÂÂÂ11788.53 > ops/sec >     Parent sees throughput for 1 > mixed workload > ÂÂÂ=ÂÂÂ11788.47 ops/sec >     Min throughput per > process       >     =ÂÂÂ11788.53 > ops/sec >     Max throughput per > process       >     =ÂÂÂ11788.53 > ops/sec >     Avg throughput per > process       >     =ÂÂÂ11788.53 > ops/sec >     Min xfer   >         >         >  = 1048576.00 ops > > iozone test complete. > > iozone test using cache I/O > [root@morrison disk]# iozone -i8 -r4k -s4g -k8 -w -t1 -O >     Iozone: Performance Test of > File I/O >         > Version $Revision: 3.321 $ >         > Compiled for 64 bit mode. >         > Build: linux-AMD64 > >     Contributors:William Norcott, > Don Capps, Isom Crawford, Kirby Collins >         >  ÂÂÂAl Slater, Scott Rhine, Mike > Wisner, Ken Goss >         >  ÂÂÂSteve Landherr, Brad Smith, Mark > Kelly, Dr. Alain CYR, >         >  ÂÂÂRandy Dunlap, Mark Montague, Dan > Million, Gavin Brebner, >         >  ÂÂÂJean-Marc Zucconi, Jeff Blomberg, > Benny Halevy, >         >  ÂÂÂErik Habbinga, Kris Strecker, > Walter Wong, Joshua Root. > >     Run began: Tue May 31 15:37:35 > 2011 > >     Record Size 4 KB >     File size set to 4194304 KB >     POSIX Async I/O (no bcopy). > Depth 8 >     Setting no_unlink >     OPS Mode. Output is in > operations per second. >     Command line used: iozone -i8 > -r4k -s4g -k8 -w -t1 -O >     Time Resolution = 0.000001 > seconds. >     Processor cache size set to > 1024 Kbytes. >     Processor cache line size set > to 32 bytes. >     File stride size set to 17 * > record size. >     Throughput test with 1 process >     Each process writes a 4194304 > Kbyte file in 4 Kbyte records > >     Children see throughput for 1 > mixed workload  =ÂÂÂ90633.90 > ops/sec >     Parent sees throughput for 1 > mixed workload > ÂÂÂ=ÂÂÂ90629.44 ops/sec >     Min throughput per > process       >     =ÂÂÂ90633.90 > ops/sec >     Max throughput per > process       >     =ÂÂÂ90633.90 > ops/sec >     Avg throughput per > process       >     =ÂÂÂ90633.90 > ops/sec >     Min xfer   >         >         >  = 1048576.00 ops > > iozone test complete. > > > Any help would be appreciated since I much prefer fio to > iozone. > > > Gordon Bookless > -- To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
