On 27 August 2012 03:59, David Rhodes Clymer <david@xxxxxxxxxxxxxx> wrote:
> I've been running bcache on my devel machine, and inspired by the new
> write-up on the bcache wiki, decided to do some simple benchmarking to
> see if I could put some numbers to the performance benefit I see on my
> PC.
>
> I decided to test the various cache modes using fio, and since I use
> postgresql a bit, pgbench (using both postgresql 9.1 & 9.2). I have no
> particular skill with regard to benchmarking, so perhaps I've screwed
> something up, but I was a bit surprised by the numbers.
>
> With pgbench (which is primarily writes) writeback mode was the
> decisive winner, as expected. I saw little difference between the
> other modes, including cache mode "none".
>
> With fio (using the parameters specified on the wiki), cache mode None
> produced the highest IOps & throughput.
>
> Does that make sense? Why would cache mode none be comparable to any
> of the other modes?
>
> ## Environment ##
>
> Linux zapazoid 3.5.0+ #1 SMP Sat Aug 25 12:42:26 EDT 2012 x86_64 GNU/Linux
>
> /dev/mapper/vg0-pg_base on /var/lib/postgresql/9.1/main/base type ext4
> (rw,relatime)
> /dev/mapper/vg0-pg_xlog on /var/lib/postgresql/9.1/main/pg_xlog type
> ext4 (rw,relatime)
> /dev/mapper/vg0-home on /home type ext4 (rw,relatime)
>
> Hardware:
>
> Phenom II X6
> 8GB RAM
> 7200 RPM 320GB HDD
> OCZ Vertex 3 64GB SSD
>
>
> Benchmark Script:
>
> #!/bin/sh
>
> outfile=bcache_bench-results.txt
> pg_bin=/usr/lib/postgresql/9.2/bin/
>
> for cm in writethrough writeback writearound none; do
> echo $cm > /sys/block/sda/sda2/bcache/cache_mode
> cat /sys/block/sda/sda2/bcache/cache_mode
> echo
> # fio benchmark
> echo "Running fio with bcache in mode: $cm"
> cat ./bcache.fio
> fio ./bcache.fio
> echo
>
> # pgbench benchmark
> # First pg 9.1, then 9.2
> for p in 5432 5433; do
> if [ "$p" = 5432 ]; then
> version=9.1
> else
> version=9.2
> fi
> echo "Running pgbench against postgresql $version, with bcache in mode: $cm"
> su - postgres <<EOF
> "${pg_bin}psql" -p $p -c 'create database pg_bench_db;'
> "${pg_bin}pgbench" -p $p -i -s 10 -c 10 -T 300 pg_bench_db
> "${pg_bin}pgbench" -p $p -s 10 -c 10 -T 300 pg_bench_db
> "${pg_bin}psql" -p $p -c 'drop database pg_bench_db;'
> EOF
> done
> done| tee "$outfile"
>
> Fio version: 2.0.8
>
> Fio config:
>
> [test]
> randrepeat=1
> ioengine=libaio
> bs=4k
> ba=4k
> size=8G
> direct=1
> gtod_reduce=1
> norandommap
> iodepth=32
>
>
>
> ## Results ##
>
>
> [writethrough] writeback writearound none
>
> Running fio with bcache in mode: writethrough
> [test]
> randrepeat=1
> ioengine=libaio
> bs=4k
> ba=4k
> size=8G
> direct=1
> gtod_reduce=1
> norandommap
> iodepth=32
> test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
> 2.0.8
> Starting 1 process
>
> test: (groupid=0, jobs=1): err= 0: pid=12901
> read : io=8192.0MB, bw=31633KB/s, iops=7908 , runt=265183msec
> cpu : usr=5.27%, sys=29.96%, ctx=1932374, majf=0, minf=52
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
> issued : total=r=2097152/w=0/d=0, short=r=0/w=0/d=0
>
> Run status group 0 (all jobs):
> READ: io=8192.0MB, aggrb=31633KB/s, minb=31633KB/s, maxb=31633KB/s,
> mint=265183msec, maxt=265183msec
>
> Disk stats (read/write):
> dm-3: ios=2095437/238, merge=0/0, ticks=8387259/33094,
> in_queue=8420690, util=100.00%, aggrios=0/0, aggrmerge=0/0,
> aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
> bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
>
> Running pgbench against postgresql 9.1, with bcache in mode: writethrough
> CREATE DATABASE
> transaction type: TPC-B (sort of)
> scaling factor: 10
> query mode: simple
> number of clients: 10
> number of threads: 1
> duration: 300 s
> number of transactions actually processed: 266049
> tps = 884.810200 (including connections establishing)
> tps = 884.964389 (excluding connections establishing)
> DROP DATABASE
> Running pgbench against postgresql 9.2, with bcache in mode: writethrough
> CREATE DATABASE
> transaction type: TPC-B (sort of)
> scaling factor: 10
> query mode: simple
> number of clients: 10
> number of threads: 1
> duration: 300 s
> number of transactions actually processed: 275057
> tps = 916.809784 (including connections establishing)
> tps = 916.991406 (excluding connections establishing)
> DROP DATABASE
> writethrough [writeback] writearound none
>
> Running fio with bcache in mode: writeback
> [test]
> randrepeat=1
> ioengine=libaio
> bs=4k
> ba=4k
> size=8G
> direct=1
> gtod_reduce=1
> norandommap
> iodepth=32
> test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
> 2.0.8
> Starting 1 process
>
> test: (groupid=0, jobs=1): err= 0: pid=13069
> read : io=8192.0MB, bw=35757KB/s, iops=8939 , runt=234599msec
> cpu : usr=5.34%, sys=29.18%, ctx=2039126, majf=0, minf=51
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
> issued : total=r=2097152/w=0/d=0, short=r=0/w=0/d=0
>
> Run status group 0 (all jobs):
> READ: io=8192.0MB, aggrb=35757KB/s, minb=35757KB/s, maxb=35757KB/s,
> mint=234599msec, maxt=234599msec
>
> Disk stats (read/write):
> dm-3: ios=2095690/75, merge=0/0, ticks=7433820/106,
> in_queue=7434173, util=100.00%, aggrios=0/0, aggrmerge=0/0,
> aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
> bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
>
> Running pgbench against postgresql 9.1, with bcache in mode: writeback
> CREATE DATABASE
> transaction type: TPC-B (sort of)
> scaling factor: 10
> query mode: simple
> number of clients: 10
> number of threads: 1
> duration: 300 s
> number of transactions actually processed: 675318
> tps = 2250.938899 (including connections establishing)
> tps = 2251.295253 (excluding connections establishing)
> DROP DATABASE
> Running pgbench against postgresql 9.2, with bcache in mode: writeback
> CREATE DATABASE
> transaction type: TPC-B (sort of)
> scaling factor: 10
> query mode: simple
> number of clients: 10
> number of threads: 1
> duration: 300 s
> number of transactions actually processed: 823609
> tps = 2741.663194 (including connections establishing)
> tps = 2742.135127 (excluding connections establishing)
> DROP DATABASE
> writethrough writeback [writearound] none
>
> Running fio with bcache in mode: writearound
> [test]
> randrepeat=1
> ioengine=libaio
> bs=4k
> ba=4k
> size=8G
> direct=1
> gtod_reduce=1
> norandommap
> iodepth=32
> test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
> 2.0.8
> Starting 1 process
>
> test: (groupid=0, jobs=1): err= 0: pid=13226
> read : io=8192.0MB, bw=33949KB/s, iops=8487 , runt=247091msec
> cpu : usr=5.47%, sys=32.60%, ctx=1961342, majf=0, minf=51
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
> issued : total=r=2097152/w=0/d=0, short=r=0/w=0/d=0
>
> Run status group 0 (all jobs):
> READ: io=8192.0MB, aggrb=33949KB/s, minb=33949KB/s, maxb=33949KB/s,
> mint=247091msec, maxt=247091msec
>
> Disk stats (read/write):
> dm-3: ios=2094874/58, merge=0/0, ticks=7816729/4730,
> in_queue=7821618, util=100.00%, aggrios=0/0, aggrmerge=0/0,
> aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
> bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
>
> Running pgbench against postgresql 9.1, with bcache in mode: writearound
> CREATE DATABASE
> transaction type: TPC-B (sort of)
> scaling factor: 10
> query mode: simple
> number of clients: 10
> number of threads: 1
> duration: 300 s
> number of transactions actually processed: 264464
> tps = 881.501731 (including connections establishing)
> tps = 881.650022 (excluding connections establishing)
> DROP DATABASE
> Running pgbench against postgresql 9.2, with bcache in mode: writearound
> CREATE DATABASE
> transaction type: TPC-B (sort of)
> scaling factor: 10
> query mode: simple
> number of clients: 10
> number of threads: 1
> duration: 300 s
> number of transactions actually processed: 283804
> tps = 945.962582 (including connections establishing)
> tps = 946.128733 (excluding connections establishing)
> DROP DATABASE
> writethrough writeback writearound [none]
>
> Running fio with bcache in mode: none
> [test]
> randrepeat=1
> ioengine=libaio
> bs=4k
> ba=4k
> size=8G
> direct=1
> gtod_reduce=1
> norandommap
> iodepth=32
> test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
> 2.0.8
> Starting 1 process
>
> test: (groupid=0, jobs=1): err= 0: pid=13367
> read : io=8192.0MB, bw=37969KB/s, iops=9492 , runt=220931msec
> cpu : usr=5.99%, sys=29.32%, ctx=1992824, majf=0, minf=51
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
> issued : total=r=2097152/w=0/d=0, short=r=0/w=0/d=0
>
> Run status group 0 (all jobs):
> READ: io=8192.0MB, aggrb=37969KB/s, minb=37969KB/s, maxb=37969KB/s,
> mint=220931msec, maxt=220931msec
>
> Disk stats (read/write):
> dm-3: ios=2096590/41, merge=0/0, ticks=6995926/4349,
> in_queue=7000464, util=100.00%, aggrios=0/0, aggrmerge=0/0,
> aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
> bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
>
> Running pgbench against postgresql 9.1, with bcache in mode: none
> CREATE DATABASE
> transaction type: TPC-B (sort of)
> scaling factor: 10
> query mode: simple
> number of clients: 10
> number of threads: 1
> duration: 300 s
> number of transactions actually processed: 265415
> tps = 884.688970 (including connections establishing)
> tps = 884.861368 (excluding connections establishing)
> DROP DATABASE
> Running pgbench against postgresql 9.2, with bcache in mode: none
> CREATE DATABASE
> transaction type: TPC-B (sort of)
> scaling factor: 10
> query mode: simple
> number of clients: 10
> number of threads: 1
> duration: 300 s
> number of transactions actually processed: 282727
> tps = 942.373702 (including connections establishing)
> tps = 942.546713 (excluding connections establishing)
> DROP DATABASE
>
>
>
> -davidc
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Hmm cache mode none shouldn't do that from my experiences...
You could you try turning up the number of jobs?
ie: numjobs=16 or something
Also I find bcache prefers higher queue depths (on the order of 64 or 128)
Joseph.
--
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846
--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
