Yes, reads will be affected a lot for mix read/write scenarios as Ceph is serializing ops on a PG. Write path is inefficient and that is affecting reads in
turn.
Hope you are following all the config settings (shards/threads, pg numbers etc. etc.) already discussed in the community.
You may want to try with bigger QD and see if it is improving or not.
BTW, try with jewel or latest master (if not already) and you should see mix read/write performance improvement as write performance has improved in Jewel..
Thanks & Regards
Somnath
From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx]
On Behalf Of min fang
Sent: Monday, May 02, 2016 8:18 PM
To: ceph-users
Subject: [ceph-users] performance drop a lot when running fio mix read/write
Hi, I run randow fio with rwmixread=70, and found read iops is 707, write is 303. (reference the following). This value is less than random write and read value. The 4K random write IOPs is 529 and 4k randread
IOPs is 11343. Apart from rw type is different, other parameters are all same.
I do not understand why mix write and read will have so huge impact on performance. All random IOs. thanks.
fio -filename=/dev/rbd2 -direct=1 -iodepth 64 -thread -rw=randrw -rwmixread=70 -ioengine=libaio -bs=4k -size=100G -numjobs=1 -runtime=1000 -group_reporting -name=mytest1
mytest1: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.8
time 7423 cycles_start=1062103697308843
Starting 1 thread
Jobs: 1 (f=1): [m(1)] [100.0% done] [2144KB/760KB/0KB /s] [536/190/0 iops] [eta 00m:00s]
mytest1: (groupid=0, jobs=1): err= 0: pid=7425: Sat Apr 30 08:55:14 2016
read : io=2765.2MB, bw=2830.5KB/s, iops=707, runt=1000393msec
slat (usec): min=2, max=268, avg= 8.93, stdev= 4.17
clat (usec): min=203, max=1939.9K, avg=34039.43, stdev=93674.48
lat (usec): min=207, max=1939.9K, avg=34048.93, stdev=93674.50
clat percentiles (usec):
| 1.00th=[ 516], 5.00th=[ 836], 10.00th=[ 1112], 20.00th=[ 1448],
| 30.00th=[ 1736], 40.00th=[ 6944], 50.00th=[13376], 60.00th=[17280],
| 70.00th=[21888], 80.00th=[30848], 90.00th=[49920], 95.00th=[103936],
| 99.00th=[552960], 99.50th=[675840], 99.90th=[880640], 99.95th=[954368],
| 99.99th=[1105920]
bw (KB /s): min= 350, max= 5944, per=100.00%, avg=2837.77, stdev=1272.84
write: io=1184.8MB, bw=1212.8KB/s, iops=303, runt=1000393msec
slat (usec): min=2, max=310, avg= 9.35, stdev= 4.50
clat (msec): min=5, max=2210, avg=131.60, stdev=226.47
lat (msec): min=5, max=2210, avg=131.61, stdev=226.47
clat percentiles (msec):
| 1.00th=[ 9], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 20],
| 30.00th=[ 25], 40.00th=[ 34], 50.00th=[ 44], 60.00th=[ 61],
| 70.00th=[ 84], 80.00th=[ 125], 90.00th=[ 449], 95.00th=[ 709],
| 99.00th=[ 1037], 99.50th=[ 1139], 99.90th=[ 1369], 99.95th=[ 1450],
| 99.99th=[ 1663]
bw (KB /s): min= 40, max= 2562, per=100.00%, avg=1215.62, stdev=564.19
lat (usec) : 250=0.01%, 500=0.60%, 750=1.94%, 1000=2.95%
lat (msec) : 2=18.69%, 4=2.46%, 10=4.21%, 20=22.05%, 50=26.40%
lat (msec) : 100=9.65%, 250=4.64%, 500=2.76%, 750=2.13%, 1000=1.11%
lat (msec) : 2000=0.39%, >=2000=0.01%
cpu : usr=0.83%, sys=1.47%, ctx=971080, majf=0, minf=1
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=707885/w=303294/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: io=2765.2MB, aggrb=2830KB/s, minb=2830KB/s, maxb=2830KB/s, mint=1000393msec, maxt=1000393msec
WRITE: io=1184.8MB, aggrb=1212KB/s, minb=1212KB/s, maxb=1212KB/s, mint=1000393msec, maxt=1000393msec
Disk stats (read/write):
rbd2: ios=707885/303293, merge=0/0, ticks=24085792/39904840, in_queue=64045864, util=100.00%
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this
message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy
any and all copies of this message in your possession (whether hard copies or electronically stored copies).
|
