Hi Mike/Warren,
Thanks for helping out here. I am running the below fio command to test this with 4 jobs and a iodepth of 128
fio --time_based --name=benchmark --size=4G --filename=/mnt/test.bin --ioengine=libaio --randrepeat=0 --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=randwrite --blocksize=4k --group_reportin
The QEMU instance is created using nova, the settings I can see in the config are below:
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writeback'/>
<auth username='$$'>
<secret type='ceph' uuid='$$'/>
</auth>
<source protocol='rbd' name='ssd_volume/volume-$$'>
<host name='$$' port='6789'/>
<host name='$$' port='6789'/>
<host name='$$' port='6789'/>
</source>
<target dev='vde' bus='virtio'/>
<serial>$$</serial>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</disk>
The below shows the output from running Fio:
# fio --time_based --name=benchmark --size=4G --filename=/mnt/test.bin --ioengine=libaio --randrepeat=0 --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=randwrite --blocksize=4k --group_reporting
fio: time_based requires a runtime/timeout setting
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
...
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.0.13
Starting 4 processes
Jobs: 3 (f=3): [_www] [99.7% done] [0K/36351K/0K /s] [0 /9087 /0 iops] [eta 00m:03s]
benchmark: (groupid=0, jobs=4): err= 0: pid=8547: Thu Nov 19 05:16:31 2015
write: io=16384MB, bw=19103KB/s, iops=4775 , runt=878269msec
slat (usec): min=4 , max=2339.4K, avg=807.17, stdev=12460.02
clat (usec): min=1 , max=2469.6K, avg=106265.05, stdev=138893.39
lat (usec): min=67 , max=2469.8K, avg=107073.04, stdev=139377.68
clat percentiles (usec):
| 1.00th=[ 1928], 5.00th=[ 9408], 10.00th=[12352], 20.00th=[18816],
| 30.00th=[43776], 40.00th=[64768], 50.00th=[78336], 60.00th=[89600],
| 70.00th=[102912], 80.00th=[123392], 90.00th=[216064], 95.00th=[370688],
| 99.00th=[733184], 99.50th=[782336], 99.90th=[1044480], 99.95th=[2088960],
| 99.99th=[2342912]
bw (KB/s) : min= 4, max=14968, per=26.11%, avg=4987.39, stdev=1947.67
lat (usec) : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.05%, 250=0.30%
lat (usec) : 500=0.24%, 750=0.11%, 1000=0.08%
lat (msec) : 2=0.23%, 4=0.46%, 10=4.47%, 20=15.08%, 50=11.28%
lat (msec) : 100=35.47%, 250=23.52%, 500=5.92%, 750=1.96%, 1000=0.70%
lat (msec) : 2000=0.06%, >=2000=0.06%
cpu : usr=0.62%, sys=2.42%, ctx=1602209, majf=1, minf=101
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=0/w=4194304/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=16384MB, aggrb=19102KB/s, minb=19102KB/s, maxb=19102KB/s, mint=878269msec, maxt=878269msec
Disk stats (read/write):
vde: ios=1119/4330437, merge=0/105599, ticks=556/121755054, in_queue=121749666, util=99.86
The below shows lspci from within the guest:
# lspci | grep -i scsi
00:04.0 SCSI storage controller: Red Hat, Inc Virtio block devic
Thanks
On Wed, Nov 18, 2015 at 7:05 PM, Warren Wang - ISD <Warren.Wang@xxxxxxxxxxx> wrote:
What were you using for iodepth and numjobs? If you’re getting an average of 2ms per operation, and you’re single threaded, I’d expect about 500 IOPS / thread, until you hit the limit of your QEMU setup, which may be a single IO thread. That’s also what I think Mike is alluding to.
Warren
From: Sean Redmond <sean.redmond1@xxxxxxxxx<mailto:sean.redmond1@xxxxxxxxx>>
Date: Wednesday, November 18, 2015 at 6:39 AM
To: "ceph-users@xxxxxxxx<mailto:ceph-users@xxxxxxxx>" <ceph-users@xxxxxxxx<mailto:ceph-users@xxxxxxxx>>
Subject: All SSD Pool - Odd Performance
This email and any files transmitted with it are confidential and intended solely for the individual or entity to whom they are addressed. If you have received this email in error destroy it immediately. *** Walmart Confidential ***
Hi,
I have a performance question for anyone running an SSD only pool. Let me detail the setup first.
12 X Dell PowerEdge R630 ( 2 X 2620v3 64Gb RAM)
8 X intel DC 3710 800GB
Dual port Solarflare 10GB/s NIC (one front and one back)
Ceph 0.94.5
Ubuntu 14.04 (3.13.0-68-generic)
The above is in one pool that is used for QEMU guests, A 4k FIO test on the SSD directly yields around 55k Iops, the same test inside a QEMU guest seems to hit a limit around 4k Iops. If I deploy multiple guests they can all reach 4K Iops simultaneously.
I don't see any evidence of a bottle neck on the OSD hosts,Is this limit inside the guest expected or I am just not looking deep enough yet?
Thanks
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
