Hi Mourik Jan,
> So, ran the fio commands, and pasted output (as it's quite a lot)
here:
> I hope someone here can draw some conclusions from this output...
Now you know, it sort of performs similar to other enterprise drives.
And you know your ceph solution will never perform beyond this ;) And if
your ceph overhead is more than standard, you maybe able be to identify
at a later time where the problem is.
With the 4k, 128k, 1024k and 4096k test results I am trying to have
data, that will explain ceph performance under different use cases.
This is a result from a samsung sm863 1.92TB
write-4k-seq: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
randwrite-4k-seq: (g=1): rw=randwrite, bs=(R) 4096B-4096B, (W)
4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
read-4k-seq: (g=2): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
randread-4k-seq: (g=3): rw=randread, bs=(R) 4096B-4096B, (W)
4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
rw-4k-seq: (g=4): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
randrw-4k-seq: (g=5): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
(T) 4096B-4096B, ioengine=libaio, iodepth=1
write-128k-seq: (g=6): rw=write, bs=(R) 128KiB-128KiB, (W)
128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1
randwrite-128k-seq: (g=7): rw=randwrite, bs=(R) 128KiB-128KiB, (W)
128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1
read-128k-seq: (g=8): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB,
(T) 128KiB-128KiB, ioengine=libaio, iodepth=1
randread-128k-seq: (g=9): rw=randread, bs=(R) 128KiB-128KiB, (W)
128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1
rw-128k-seq: (g=10): rw=rw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T)
128KiB-128KiB, ioengine=libaio, iodepth=1
randrw-128k-seq: (g=11): rw=randrw, bs=(R) 128KiB-128KiB, (W)
128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1
write-1024k-seq: (g=12): rw=write, bs=(R) 1024KiB-1024KiB, (W)
1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
randwrite-1024k-seq: (g=13): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W)
1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
read-1024k-seq: (g=14): rw=read, bs=(R) 1024KiB-1024KiB, (W)
1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
randread-1024k-seq: (g=15): rw=randread, bs=(R) 1024KiB-1024KiB, (W)
1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
rw-1024k-seq: (g=16): rw=rw, bs=(R) 1024KiB-1024KiB, (W)
1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
randrw-1024k-seq: (g=17): rw=randrw, bs=(R) 1024KiB-1024KiB, (W)
1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
write-4096k-seq: (g=18): rw=write, bs=(R) 4096KiB-4096KiB, (W)
4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1
randwrite-4096k-seq: (g=19): rw=randwrite, bs=(R) 4096KiB-4096KiB, (W)
4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1
read-4096k-seq: (g=20): rw=read, bs=(R) 4096KiB-4096KiB, (W)
4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1
randread-4096k-seq: (g=21): rw=randread, bs=(R) 4096KiB-4096KiB, (W)
4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1
rw-4096k-seq: (g=22): rw=rw, bs=(R) 4096KiB-4096KiB, (W)
4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1
randrw-4096k-seq: (g=23): rw=randrw, bs=(R) 4096KiB-4096KiB, (W)
4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1
fio-3.1
Starting 24 processes
write-4k-seq: (groupid=0, jobs=1): err= 0: pid=11515: Mon Aug 26
15:45:05 2019
write: IOPS=30.8k, BW=120MiB/s (126MB/s)(21.1GiB/180001msec)
slat (nsec): min=3762, max=75591, avg=7173.31, stdev=3526.20
clat (nsec): min=440, max=1057.9k, avg=23654.15, stdev=4570.40
lat (usec): min=23, max=1062, avg=31.02, stdev= 4.94
clat percentiles (nsec):
| 1.00th=[15680], 5.00th=[19840], 10.00th=[21120],
20.00th=[21888],
| 30.00th=[22144], 40.00th=[22400], 50.00th=[22656],
60.00th=[22912],
| 70.00th=[23168], 80.00th=[24448], 90.00th=[28544],
95.00th=[29568],
| 99.00th=[47360], 99.50th=[49920], 99.90th=[51968],
99.95th=[52480],
| 99.99th=[84480]
bw ( KiB/s): min=117983, max=146636, per=100.00%, avg=123958.14,
stdev=7366.26, samples=360
iops : min=29495, max=36659, avg=30989.17, stdev=1841.60,
samples=360
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 10=0.01%, 20=5.10%, 50=94.45%, 100=0.45%, 250=0.01%
lat (usec) : 500=0.01%, 750=0.01%
lat (msec) : 2=0.01%
cpu : usr=5.57%, sys=25.51%, ctx=8452083, majf=0, minf=7
IO depths : 1=117.5%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=0,5538482,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randwrite-4k-seq: (groupid=1, jobs=1): err= 0: pid=11560: Mon Aug 26
15:45:05 2019
write: IOPS=29.5k, BW=115MiB/s (121MB/s)(20.2GiB/180001msec)
slat (usec): min=3, max=126, avg= 8.06, stdev= 3.59
clat (nsec): min=457, max=419756, avg=23739.16, stdev=4377.60
lat (usec): min=23, max=428, avg=31.99, stdev= 4.78
clat percentiles (usec):
| 1.00th=[ 16], 5.00th=[ 21], 10.00th=[ 22], 20.00th=[
23],
| 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 23], 60.00th=[
23],
| 70.00th=[ 24], 80.00th=[ 24], 90.00th=[ 29], 95.00th=[
30],
| 99.00th=[ 48], 99.50th=[ 50], 99.90th=[ 52], 99.95th=[
52],
| 99.99th=[ 101]
bw ( KiB/s): min=112000, max=144697, per=100.00%, avg=118764.70,
stdev=6220.53, samples=360
iops : min=28000, max=36174, avg=29690.82, stdev=1555.19,
samples=360
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 10=0.01%, 20=3.33%, 50=96.16%, 100=0.50%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=5.60%, sys=27.07%, ctx=8880261, majf=0, minf=5
IO depths : 1=119.3%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=0,5306473,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-4k-seq: (groupid=2, jobs=1): err= 0: pid=11589: Mon Aug 26 15:45:05
2019
read: IOPS=29.4k, BW=115MiB/s (121MB/s)(20.2GiB/180001msec)
slat (usec): min=3, max=256, avg= 7.64, stdev= 3.53
clat (nsec): min=449, max=1125.7k, avg=24003.96, stdev=4493.67
lat (usec): min=23, max=1130, avg=31.81, stdev= 4.75
clat percentiles (nsec):
| 1.00th=[15424], 5.00th=[20864], 10.00th=[21376],
20.00th=[21888],
| 30.00th=[22144], 40.00th=[22144], 50.00th=[22400],
60.00th=[22656],
| 70.00th=[22912], 80.00th=[28288], 90.00th=[29312],
95.00th=[30336],
| 99.00th=[45312], 99.50th=[48896], 99.90th=[50944],
99.95th=[51456],
| 99.99th=[55040]
bw ( KiB/s): min=113771, max=149349, per=100.00%, avg=118606.15,
stdev=5880.22, samples=360
iops : min=28442, max=37337, avg=29651.20, stdev=1470.06,
samples=360
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=2.26%, 50=97.44%, 100=0.30%
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%
cpu : usr=4.94%, sys=27.39%, ctx=9280107, majf=0, minf=6
IO depths : 1=117.2%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=5299598,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randread-4k-seq: (groupid=3, jobs=1): err= 0: pid=12050: Mon Aug 26
15:45:05 2019
read: IOPS=24.5k, BW=95.9MiB/s (101MB/s)(16.9GiB/180001msec)
slat (nsec): min=3693, max=76505, avg=5960.68, stdev=1820.53
clat (nsec): min=675, max=1052.6k, avg=33198.13, stdev=12212.51
lat (usec): min=27, max=1057, avg=39.34, stdev=12.55
clat percentiles (usec):
| 1.00th=[ 27], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[
29],
| 30.00th=[ 29], 40.00th=[ 30], 50.00th=[ 30], 60.00th=[
31],
| 70.00th=[ 33], 80.00th=[ 35], 90.00th=[ 40], 95.00th=[
45],
| 99.00th=[ 106], 99.50th=[ 110], 99.90th=[ 116], 99.95th=[
118],
| 99.99th=[ 123]
bw ( KiB/s): min=83500, max=110084, per=100.00%, avg=98894.61,
stdev=3387.20, samples=360
iops : min=20875, max=27521, avg=24723.30, stdev=846.78,
samples=360
lat (nsec) : 750=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=96.78%, 100=1.84%
lat (usec) : 250=1.37%, 750=0.01%
lat (msec) : 2=0.01%
cpu : usr=7.31%, sys=19.03%, ctx=4440118, majf=0, minf=6
IO depths : 1=116.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=4418876,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
rw-4k-seq: (groupid=4, jobs=1): err= 0: pid=12073: Mon Aug 26 15:45:05
2019
read: IOPS=15.0k, BW=62.4MiB/s (65.4MB/s)(10.0GiB/180001msec)
slat (nsec): min=3661, max=75395, avg=5554.24, stdev=2712.89
clat (nsec): min=487, max=143971, avg=23601.85, stdev=4187.61
lat (usec): min=22, max=148, avg=29.32, stdev= 4.52
clat percentiles (nsec):
| 1.00th=[15296], 5.00th=[20352], 10.00th=[20608],
20.00th=[21120],
| 30.00th=[21632], 40.00th=[21888], 50.00th=[22144],
60.00th=[22656],
| 70.00th=[23680], 80.00th=[25728], 90.00th=[28800],
95.00th=[30592],
| 99.00th=[40704], 99.50th=[41728], 99.90th=[45824],
99.95th=[48896],
| 99.99th=[52992]
bw ( KiB/s): min=57859, max=73862, per=100.00%, avg=64353.67,
stdev=5203.87, samples=360
iops : min=14464, max=18465, avg=16088.08, stdev=1300.94,
samples=360
write: IOPS=15.0k, BW=62.5MiB/s (65.5MB/s)(10.0GiB/180001msec)
slat (nsec): min=3761, max=77015, avg=5723.77, stdev=2746.22
clat (nsec): min=459, max=579225, avg=24813.39, stdev=4312.80
lat (usec): min=23, max=584, avg=30.70, stdev= 4.74
clat percentiles (nsec):
| 1.00th=[16512], 5.00th=[21120], 10.00th=[21632],
20.00th=[22144],
| 30.00th=[22656], 40.00th=[22912], 50.00th=[23168],
60.00th=[23936],
| 70.00th=[25472], 80.00th=[28032], 90.00th=[29824],
95.00th=[31360],
| 99.00th=[41728], 99.50th=[42752], 99.90th=[47360],
99.95th=[50432],
| 99.99th=[96768]
bw ( KiB/s): min=57775, max=74101, per=100.00%, avg=64475.76,
stdev=5298.07, samples=360
iops : min=14443, max=18525, avg=16118.61, stdev=1324.51,
samples=360
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 10=0.01%, 20=2.56%, 50=97.39%, 100=0.04%, 250=0.01%
lat (usec) : 750=0.01%
cpu : usr=7.05%, sys=21.91%, ctx=7267832, majf=0, minf=6
IO depths : 1=116.3%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=2874379,2879839,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randrw-4k-seq: (groupid=5, jobs=1): err= 0: pid=12211: Mon Aug 26
15:45:05 2019
read: IOPS=13.8k, BW=53.0MiB/s (56.6MB/s)(9719MiB/180001msec)
slat (usec): min=3, max=442, avg= 5.29, stdev= 1.73
clat (nsec): min=695, max=1834.6k, avg=33487.10, stdev=27390.90
lat (usec): min=27, max=1838, avg=38.94, stdev=27.53
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[
28],
| 30.00th=[ 28], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[
30],
| 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 38], 95.00th=[
44],
| 99.00th=[ 109], 99.50th=[ 116], 99.90th=[ 396], 99.95th=[
457],
| 99.99th=[ 881]
bw ( KiB/s): min=49604, max=62173, per=100.00%, avg=55691.34,
stdev=2474.35, samples=360
iops : min=12401, max=15543, avg=13922.50, stdev=618.60,
samples=360
write: IOPS=13.8k, BW=54.0MiB/s (56.6MB/s)(9720MiB/180001msec)
slat (nsec): min=3751, max=58621, avg=5456.94, stdev=1739.89
clat (usec): min=5, max=128, avg=25.19, stdev= 4.45
lat (usec): min=23, max=136, avg=30.81, stdev= 5.27
clat percentiles (nsec):
| 1.00th=[20864], 5.00th=[21632], 10.00th=[21888],
20.00th=[22400],
| 30.00th=[22912], 40.00th=[23168], 50.00th=[23680],
60.00th=[24192],
| 70.00th=[25472], 80.00th=[26496], 90.00th=[30336],
95.00th=[36608],
| 99.00th=[41728], 99.50th=[43776], 99.90th=[49408],
99.95th=[51968],
| 99.99th=[61184]
bw ( KiB/s): min=49244, max=61663, per=100.00%, avg=55697.41,
stdev=2464.11, samples=360
iops : min=12311, max=15415, avg=13924.00, stdev=616.03,
samples=360
lat (nsec) : 750=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.03%, 50=98.22%, 100=0.86%
lat (usec) : 250=0.70%, 500=0.15%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%
cpu : usr=8.46%, sys=18.38%, ctx=4985736, majf=0, minf=6
IO depths : 1=117.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=2488163,2488417,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
write-128k-seq: (groupid=6, jobs=1): err= 0: pid=12242: Mon Aug 26
15:45:05 2019
write: IOPS=3160, BW=395MiB/s (414MB/s)(69.5GiB/180001msec)
slat (nsec): min=16332, max=72183, avg=31691.27, stdev=3994.93
clat (usec): min=241, max=2658, avg=281.36, stdev=55.40
lat (usec): min=276, max=2692, avg=313.44, stdev=55.62
clat percentiles (usec):
| 1.00th=[ 265], 5.00th=[ 265], 10.00th=[ 269], 20.00th=[
277],
| 30.00th=[ 277], 40.00th=[ 277], 50.00th=[ 277], 60.00th=[
281],
| 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 289], 95.00th=[
293],
| 99.00th=[ 318], 99.50th=[ 326], 99.90th=[ 343], 99.95th=[
2212],
| 99.99th=[ 2606]
bw ( KiB/s): min=398848, max=416129, per=100.00%, avg=407445.96,
stdev=3324.34, samples=360
iops : min= 3116, max= 3251, avg=3182.76, stdev=25.94,
samples=360
lat (usec) : 250=0.01%, 500=99.91%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.06%
cpu : usr=4.72%, sys=11.74%, ctx=570098, majf=0, minf=7
IO depths : 1=116.4%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=0,568940,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randwrite-128k-seq: (groupid=7, jobs=1): err= 0: pid=12269: Mon Aug 26
15:45:05 2019
write: IOPS=3149, BW=394MiB/s (413MB/s)(69.2GiB/180001msec)
slat (nsec): min=16078, max=73647, avg=31696.03, stdev=3965.26
clat (usec): min=236, max=2643, avg=281.93, stdev=50.94
lat (usec): min=275, max=2675, avg=314.03, stdev=51.17
clat percentiles (usec):
| 1.00th=[ 265], 5.00th=[ 265], 10.00th=[ 269], 20.00th=[
277],
| 30.00th=[ 277], 40.00th=[ 277], 50.00th=[ 277], 60.00th=[
281],
| 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 289], 95.00th=[
297],
| 99.00th=[ 326], 99.50th=[ 347], 99.90th=[ 379], 99.95th=[
1729],
| 99.99th=[ 2606]
bw ( KiB/s): min=398848, max=415547, per=100.00%, avg=406058.61,
stdev=3217.07, samples=360
iops : min= 3116, max= 3246, avg=3171.89, stdev=25.13,
samples=360
lat (usec) : 250=0.01%, 500=99.92%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.03%, 4=0.04%
cpu : usr=4.84%, sys=11.77%, ctx=568040, majf=0, minf=6
IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=0,566991,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-128k-seq: (groupid=8, jobs=1): err= 0: pid=12297: Mon Aug 26
15:45:05 2019
read: IOPS=3258, BW=407MiB/s (427MB/s)(71.6GiB/180001msec)
slat (usec): min=14, max=201, avg=29.15, stdev= 4.09
clat (usec): min=98, max=662, avg=274.56, stdev= 8.35
lat (usec): min=264, max=686, avg=304.11, stdev= 9.97
clat percentiles (usec):
| 1.00th=[ 258], 5.00th=[ 262], 10.00th=[ 265], 20.00th=[
269],
| 30.00th=[ 273], 40.00th=[ 273], 50.00th=[ 277], 60.00th=[
277],
| 70.00th=[ 277], 80.00th=[ 281], 90.00th=[ 281], 95.00th=[
285],
| 99.00th=[ 302], 99.50th=[ 314], 99.90th=[ 322], 99.95th=[
326],
| 99.99th=[ 334]
bw ( KiB/s): min=415225, max=440860, per=100.00%, avg=420009.72,
stdev=3918.49, samples=360
iops : min= 3243, max= 3444, avg=3280.92, stdev=30.53,
samples=360
lat (usec) : 100=0.01%, 250=0.05%, 500=99.95%, 750=0.01%
cpu : usr=2.40%, sys=13.84%, ctx=587372, majf=0, minf=6
IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=586463,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randread-128k-seq: (groupid=9, jobs=1): err= 0: pid=12316: Mon Aug 26
15:45:05 2019
read: IOPS=3029, BW=379MiB/s (397MB/s)(66.6GiB/180001msec)
slat (usec): min=10, max=103, avg=28.55, stdev= 4.15
clat (usec): min=237, max=978, avg=297.79, stdev=30.25
lat (usec): min=272, max=1005, avg=326.75, stdev=30.75
clat percentiles (usec):
| 1.00th=[ 269], 5.00th=[ 269], 10.00th=[ 277], 20.00th=[
281],
| 30.00th=[ 281], 40.00th=[ 285], 50.00th=[ 285], 60.00th=[
289],
| 70.00th=[ 293], 80.00th=[ 310], 90.00th=[ 359], 95.00th=[
371],
| 99.00th=[ 379], 99.50th=[ 383], 99.90th=[ 400], 99.95th=[
416],
| 99.99th=[ 490]
bw ( KiB/s): min=382976, max=396202, per=100.00%, avg=390503.28,
stdev=2581.76, samples=360
iops : min= 2992, max= 3095, avg=3050.45, stdev=19.98,
samples=360
lat (usec) : 250=0.01%, 500=99.99%, 750=0.01%, 1000=0.01%
cpu : usr=2.36%, sys=12.93%, ctx=546079, majf=0, minf=5
IO depths : 1=116.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=545272,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
rw-128k-seq: (groupid=10, jobs=1): err= 0: pid=12422: Mon Aug 26
15:45:05 2019
read: IOPS=1406, BW=176MiB/s (184MB/s)(30.9GiB/180001msec)
slat (nsec): min=13910, max=82839, avg=29020.55, stdev=4086.77
clat (usec): min=231, max=3707, avg=361.02, stdev=246.76
lat (usec): min=268, max=3739, avg=390.44, stdev=246.77
clat percentiles (usec):
| 1.00th=[ 258], 5.00th=[ 262], 10.00th=[ 269], 20.00th=[
273],
| 30.00th=[ 273], 40.00th=[ 277], 50.00th=[ 277], 60.00th=[
281],
| 70.00th=[ 289], 80.00th=[ 302], 90.00th=[ 537], 95.00th=[
930],
| 99.00th=[ 1483], 99.50th=[ 1516], 99.90th=[ 1713], 99.95th=[
1926],
| 99.99th=[ 2343]
bw ( KiB/s): min=131878, max=199741, per=100.00%, avg=181277.06,
stdev=7492.15, samples=360
iops : min= 1030, max= 1560, avg=1415.71, stdev=58.46,
samples=360
write: IOPS=1407, BW=176MiB/s (185MB/s)(30.9GiB/180001msec)
slat (usec): min=15, max=462, avg=31.27, stdev= 4.11
clat (usec): min=243, max=2656, avg=282.69, stdev=55.31
lat (usec): min=276, max=2684, avg=314.36, stdev=55.53
clat percentiles (usec):
| 1.00th=[ 265], 5.00th=[ 265], 10.00th=[ 269], 20.00th=[
277],
| 30.00th=[ 277], 40.00th=[ 277], 50.00th=[ 281], 60.00th=[
281],
| 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 289], 95.00th=[
297],
| 99.00th=[ 326], 99.50th=[ 330], 99.90th=[ 400], 99.95th=[
2147],
| 99.99th=[ 2606]
bw ( KiB/s): min=133688, max=197841, per=100.00%, avg=181452.40,
stdev=7546.36, samples=360
iops : min= 1044, max= 1545, avg=1417.08, stdev=58.93,
samples=360
lat (usec) : 250=0.01%, 500=94.34%, 750=2.38%, 1000=0.97%
lat (msec) : 2=2.26%, 4=0.05%
cpu : usr=3.00%, sys=11.40%, ctx=507392, majf=0, minf=8
IO depths : 1=117.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=253133,253388,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randrw-128k-seq: (groupid=11, jobs=1): err= 0: pid=12481: Mon Aug 26
15:45:05 2019
read: IOPS=1420, BW=178MiB/s (186MB/s)(31.2GiB/180001msec)
slat (nsec): min=15881, max=72688, avg=29081.99, stdev=4063.08
clat (usec): min=251, max=2443, avg=353.64, stdev=194.49
lat (usec): min=281, max=2466, avg=383.13, stdev=194.56
clat percentiles (usec):
| 1.00th=[ 269], 5.00th=[ 273], 10.00th=[ 277], 20.00th=[
281],
| 30.00th=[ 285], 40.00th=[ 285], 50.00th=[ 289], 60.00th=[
293],
| 70.00th=[ 306], 80.00th=[ 355], 90.00th=[ 502], 95.00th=[
668],
| 99.00th=[ 1369], 99.50th=[ 1467], 99.90th=[ 1844], 99.95th=[
1876],
| 99.99th=[ 1942]
bw ( KiB/s): min=167936, max=199967, per=100.00%, avg=183163.65,
stdev=5287.72, samples=360
iops : min= 1312, max= 1562, avg=1430.45, stdev=41.25,
samples=360
write: IOPS=1425, BW=178MiB/s (187MB/s)(31.3GiB/180001msec)
slat (nsec): min=16678, max=69100, avg=31409.40, stdev=4041.71
clat (usec): min=240, max=2605, avg=280.68, stdev=15.29
lat (usec): min=277, max=2636, avg=312.50, stdev=16.10
clat percentiles (usec):
| 1.00th=[ 265], 5.00th=[ 265], 10.00th=[ 269], 20.00th=[
277],
| 30.00th=[ 277], 40.00th=[ 277], 50.00th=[ 277], 60.00th=[
281],
| 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 289], 95.00th=[
297],
| 99.00th=[ 322], 99.50th=[ 326], 99.90th=[ 347], 99.95th=[
359],
| 99.99th=[ 412]
bw ( KiB/s): min=166040, max=202752, per=100.00%, avg=183802.21,
stdev=5258.34, samples=360
iops : min= 1297, max= 1584, avg=1435.43, stdev=41.04,
samples=360
lat (usec) : 250=0.01%, 500=95.00%, 750=2.82%, 1000=0.87%
lat (msec) : 2=1.30%, 4=0.01%
cpu : usr=3.32%, sys=11.54%, ctx=513256, majf=0, minf=6
IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=255775,256663,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
write-1024k-seq: (groupid=12, jobs=1): err= 0: pid=12511: Mon Aug 26
15:45:05 2019
write: IOPS=463, BW=463MiB/s (486MB/s)(81.4GiB/180002msec)
slat (usec): min=72, max=182, avg=117.73, stdev=13.96
clat (usec): min=1924, max=13454, avg=2034.72, stdev=361.66
lat (usec): min=2031, max=13537, avg=2153.08, stdev=361.97
clat percentiles (usec):
| 1.00th=[ 1958], 5.00th=[ 1975], 10.00th=[ 1991], 20.00th=[
1991],
| 30.00th=[ 2008], 40.00th=[ 2008], 50.00th=[ 2024], 60.00th=[
2024],
| 70.00th=[ 2040], 80.00th=[ 2040], 90.00th=[ 2057], 95.00th=[
2057],
| 99.00th=[ 2089], 99.50th=[ 2114], 99.90th=[ 9503], 99.95th=[
9634],
| 99.99th=[ 9896]
bw ( KiB/s): min=458752, max=496484, per=100.00%, avg=477576.15,
stdev=7844.90, samples=360
iops : min= 448, max= 484, avg=465.70, stdev= 7.62,
samples=360
lat (msec) : 2=30.24%, 4=69.50%, 10=0.25%, 20=0.01%
cpu : usr=3.01%, sys=3.96%, ctx=84629, majf=0, minf=6
IO depths : 1=116.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=0,83363,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randwrite-1024k-seq: (groupid=13, jobs=1): err= 0: pid=12533: Mon Aug 26
15:45:05 2019
write: IOPS=464, BW=464MiB/s (487MB/s)(81.6GiB/180001msec)
slat (usec): min=69, max=191, avg=116.30, stdev=13.70
clat (usec): min=1925, max=7096, avg=2029.60, stdev=243.97
lat (usec): min=2031, max=7231, avg=2146.51, stdev=244.60
clat percentiles (usec):
| 1.00th=[ 1958], 5.00th=[ 1975], 10.00th=[ 1991], 20.00th=[
1991],
| 30.00th=[ 2008], 40.00th=[ 2008], 50.00th=[ 2024], 60.00th=[
2024],
| 70.00th=[ 2040], 80.00th=[ 2040], 90.00th=[ 2057], 95.00th=[
2057],
| 99.00th=[ 2073], 99.50th=[ 2114], 99.90th=[ 6783], 99.95th=[
6849],
| 99.99th=[ 6915]
bw ( KiB/s): min=464896, max=493344, per=100.00%, avg=478765.46,
stdev=6158.78, samples=360
iops : min= 454, max= 481, avg=466.85, stdev= 6.00,
samples=360
lat (msec) : 2=28.52%, 4=71.22%, 10=0.26%
cpu : usr=3.06%, sys=3.92%, ctx=84773, majf=0, minf=6
IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=0,83570,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-1024k-seq: (groupid=14, jobs=1): err= 0: pid=12560: Mon Aug 26
15:45:05 2019
read: IOPS=467, BW=468MiB/s (491MB/s)(82.2GiB/180002msec)
slat (usec): min=74, max=173, avg=100.38, stdev= 8.93
clat (usec): min=1852, max=2618, avg=2030.43, stdev=30.12
lat (usec): min=1944, max=2720, avg=2131.49, stdev=32.83
clat percentiles (usec):
| 1.00th=[ 1926], 5.00th=[ 1975], 10.00th=[ 1991], 20.00th=[
2008],
| 30.00th=[ 2024], 40.00th=[ 2024], 50.00th=[ 2040], 60.00th=[
2040],
| 70.00th=[ 2040], 80.00th=[ 2057], 90.00th=[ 2057], 95.00th=[
2073],
| 99.00th=[ 2073], 99.50th=[ 2089], 99.90th=[ 2147], 99.95th=[
2180],
| 99.99th=[ 2311]
bw ( KiB/s): min=474838, max=517192, per=100.00%, avg=482440.02,
stdev=6096.02, samples=360
iops : min= 463, max= 505, avg=470.42, stdev= 5.95,
samples=360
lat (msec) : 2=14.64%, 4=85.36%
cpu : usr=0.57%, sys=5.91%, ctx=84982, majf=0, minf=6
IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=84223,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randread-1024k-seq: (groupid=15, jobs=1): err= 0: pid=12579: Mon Aug 26
15:45:05 2019
read: IOPS=481, BW=481MiB/s (505MB/s)(84.6GiB/180002msec)
slat (usec): min=72, max=156, avg=99.50, stdev= 9.67
clat (usec): min=1852, max=2373, avg=1971.06, stdev=65.28
lat (usec): min=1951, max=2478, avg=2071.25, stdev=66.85
clat percentiles (usec):
| 1.00th=[ 1876], 5.00th=[ 1893], 10.00th=[ 1909], 20.00th=[
1926],
| 30.00th=[ 1942], 40.00th=[ 1942], 50.00th=[ 1958], 60.00th=[
1958],
| 70.00th=[ 1975], 80.00th=[ 2008], 90.00th=[ 2073], 95.00th=[
2147],
| 99.00th=[ 2147], 99.50th=[ 2147], 99.90th=[ 2212], 99.95th=[
2245],
| 99.99th=[ 2311]
bw ( KiB/s): min=485376, max=518218, per=100.00%, avg=496212.19,
stdev=5915.15, samples=360
iops : min= 474, max= 506, avg=483.80, stdev= 5.80,
samples=360
lat (msec) : 2=79.15%, 4=20.85%
cpu : usr=0.55%, sys=5.94%, ctx=87565, majf=0, minf=5
IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=86622,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
rw-1024k-seq: (groupid=16, jobs=1): err= 0: pid=12602: Mon Aug 26
15:45:05 2019
read: IOPS=211, BW=212MiB/s (222MB/s)(37.2GiB/180001msec)
slat (usec): min=73, max=176, avg=101.99, stdev= 9.64
clat (usec): min=1931, max=6860, avg=2501.48, stdev=587.24
lat (usec): min=2026, max=6961, avg=2604.16, stdev=587.28
clat percentiles (usec):
| 1.00th=[ 1958], 5.00th=[ 1991], 10.00th=[ 2008], 20.00th=[
2024],
| 30.00th=[ 2040], 40.00th=[ 2040], 50.00th=[ 2073], 60.00th=[
2507],
| 70.00th=[ 2802], 80.00th=[ 3228], 90.00th=[ 3490], 95.00th=[
3556],
| 99.00th=[ 3654], 99.50th=[ 3851], 99.90th=[ 5211], 99.95th=[
5735],
| 99.99th=[ 6456]
bw ( KiB/s): min=168272, max=271546, per=100.00%, avg=218455.45,
stdev=17227.11, samples=360
iops : min= 164, max= 265, avg=212.88, stdev=16.77,
samples=360
write: IOPS=206, BW=207MiB/s (217MB/s)(36.4GiB/180001msec)
slat (usec): min=75, max=178, avg=118.75, stdev=12.18
clat (usec): min=1935, max=10175, avg=2034.53, stdev=278.16
lat (usec): min=2036, max=10254, avg=2153.95, stdev=278.51
clat percentiles (usec):
| 1.00th=[ 1958], 5.00th=[ 1975], 10.00th=[ 1991], 20.00th=[
1991],
| 30.00th=[ 2008], 40.00th=[ 2024], 50.00th=[ 2024], 60.00th=[
2024],
| 70.00th=[ 2040], 80.00th=[ 2040], 90.00th=[ 2057], 95.00th=[
2057],
| 99.00th=[ 2073], 99.50th=[ 2114], 99.90th=[ 7111], 99.95th=[
9634],
| 99.99th=[10028]
bw ( KiB/s): min=163096, max=270877, per=100.00%, avg=213343.87,
stdev=16525.87, samples=360
iops : min= 159, max= 264, avg=207.88, stdev=16.13,
samples=360
lat (msec) : 2=15.78%, 4=83.91%, 10=0.30%, 20=0.01%
cpu : usr=1.57%, sys=4.59%, ctx=76136, majf=0, minf=7
IO depths : 1=117.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=38137,37246,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randrw-1024k-seq: (groupid=17, jobs=1): err= 0: pid=12621: Mon Aug 26
15:45:05 2019
read: IOPS=222, BW=222MiB/s (233MB/s)(39.1GiB/180002msec)
slat (usec): min=73, max=152, avg=101.01, stdev= 9.85
clat (usec): min=1858, max=5794, avg=2242.66, stdev=409.82
lat (usec): min=1951, max=5894, avg=2344.36, stdev=409.83
clat percentiles (usec):
| 1.00th=[ 1876], 5.00th=[ 1909], 10.00th=[ 1926], 20.00th=[
1942],
| 30.00th=[ 1958], 40.00th=[ 1975], 50.00th=[ 2040], 60.00th=[
2147],
| 70.00th=[ 2343], 80.00th=[ 2573], 90.00th=[ 2868], 95.00th=[
3097],
| 99.00th=[ 3621], 99.50th=[ 3720], 99.90th=[ 3982], 99.95th=[
4178],
| 99.99th=[ 4817]
bw ( KiB/s): min=189935, max=274432, per=100.00%, avg=229282.93,
stdev=14988.39, samples=360
iops : min= 185, max= 268, avg=223.41, stdev=14.61,
samples=360
write: IOPS=221, BW=222MiB/s (232MB/s)(38.0GiB/180002msec)
slat (usec): min=75, max=176, avg=118.54, stdev=12.46
clat (usec): min=1926, max=7003, avg=2025.44, stdev=144.51
lat (usec): min=2032, max=7088, avg=2144.64, stdev=145.19
clat percentiles (usec):
| 1.00th=[ 1958], 5.00th=[ 1958], 10.00th=[ 1991], 20.00th=[
1991],
| 30.00th=[ 2008], 40.00th=[ 2024], 50.00th=[ 2024], 60.00th=[
2024],
| 70.00th=[ 2040], 80.00th=[ 2040], 90.00th=[ 2057], 95.00th=[
2057],
| 99.00th=[ 2073], 99.50th=[ 2089], 99.90th=[ 3916], 99.95th=[
6652],
| 99.99th=[ 6915]
bw ( KiB/s): min=186368, max=260654, per=100.00%, avg=228583.23,
stdev=14200.29, samples=360
iops : min= 182, max= 254, avg=222.73, stdev=13.83,
samples=360
lat (msec) : 2=34.68%, 4=65.22%, 10=0.10%
cpu : usr=1.70%, sys=4.81%, ctx=80799, majf=0, minf=6
IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=40027,39906,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
write-4096k-seq: (groupid=18, jobs=1): err= 0: pid=12649: Mon Aug 26
15:45:05 2019
write: IOPS=117, BW=470MiB/s (493MB/s)(82.6GiB/180008msec)
slat (usec): min=359, max=868, avg=560.54, stdev=47.08
clat (usec): min=7707, max=19363, avg=7941.94, stdev=743.06
lat (usec): min=8158, max=19836, avg=8503.08, stdev=744.61
clat percentiles (usec):
| 1.00th=[ 7767], 5.00th=[ 7767], 10.00th=[ 7767], 20.00th=[
7832],
| 30.00th=[ 7832], 40.00th=[ 7898], 50.00th=[ 7898], 60.00th=[
7898],
| 70.00th=[ 7898], 80.00th=[ 7898], 90.00th=[ 7898], 95.00th=[
7898],
| 99.00th=[14615], 99.50th=[15401], 99.90th=[15533],
99.95th=[15664],
| 99.99th=[16057]
bw ( KiB/s): min=466944, max=498498, per=100.00%, avg=484736.70,
stdev=7540.84, samples=360
iops : min= 114, max= 121, avg=117.88, stdev= 1.82,
samples=360
lat (msec) : 10=98.96%, 20=1.04%
cpu : usr=1.99%, sys=4.84%, ctx=21926, majf=0, minf=6
IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=0,21158,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randwrite-4096k-seq: (groupid=19, jobs=1): err= 0: pid=12751: Mon Aug 26
15:45:05 2019
write: IOPS=117, BW=471MiB/s (494MB/s)(82.9GiB/180001msec)
slat (usec): min=391, max=859, avg=561.31, stdev=46.23
clat (usec): min=7709, max=16870, avg=7919.13, stdev=536.92
lat (usec): min=8137, max=17416, avg=8481.01, stdev=538.50
clat percentiles (usec):
| 1.00th=[ 7767], 5.00th=[ 7767], 10.00th=[ 7767], 20.00th=[
7832],
| 30.00th=[ 7832], 40.00th=[ 7898], 50.00th=[ 7898], 60.00th=[
7898],
| 70.00th=[ 7898], 80.00th=[ 7898], 90.00th=[ 7898], 95.00th=[
7898],
| 99.00th=[12256], 99.50th=[12911], 99.90th=[14877],
99.95th=[15533],
| 99.99th=[16450]
bw ( KiB/s): min=467879, max=498498, per=100.00%, avg=485950.74,
stdev=6245.26, samples=360
iops : min= 114, max= 121, avg=118.17, stdev= 1.49,
samples=360
lat (msec) : 10=98.96%, 20=1.04%
cpu : usr=2.05%, sys=4.80%, ctx=21985, majf=0, minf=5
IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=0,21210,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-4096k-seq: (groupid=20, jobs=1): err= 0: pid=12865: Mon Aug 26
15:45:05 2019
read: IOPS=120, BW=482MiB/s (505MB/s)(84.7GiB/180005msec)
slat (usec): min=375, max=585, avg=462.83, stdev=22.62
clat (usec): min=7372, max=8219, avg=7837.59, stdev=72.82
lat (usec): min=7833, max=8683, avg=8301.03, stdev=74.07
clat percentiles (usec):
| 1.00th=[ 7504], 5.00th=[ 7767], 10.00th=[ 7767], 20.00th=[
7767],
| 30.00th=[ 7832], 40.00th=[ 7832], 50.00th=[ 7832], 60.00th=[
7832],
| 70.00th=[ 7898], 80.00th=[ 7898], 90.00th=[ 7898], 95.00th=[
7898],
| 99.00th=[ 7963], 99.50th=[ 7963], 99.90th=[ 8094], 99.95th=[
8160],
| 99.99th=[ 8160]
bw ( KiB/s): min=484296, max=523423, per=100.00%, avg=496606.33,
stdev=5140.76, samples=360
iops : min= 118, max= 127, avg=120.75, stdev= 1.21,
samples=360
lat (msec) : 10=100.00%
cpu : usr=0.10%, sys=5.70%, ctx=22379, majf=0, minf=6
IO depths : 1=116.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=21673,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randread-4096k-seq: (groupid=21, jobs=1): err= 0: pid=12886: Mon Aug 26
15:45:05 2019
read: IOPS=124, BW=497MiB/s (521MB/s)(87.3GiB/180003msec)
slat (usec): min=358, max=616, avg=467.77, stdev=24.22
clat (usec): min=7345, max=8130, avg=7580.08, stdev=143.91
lat (usec): min=7782, max=8578, avg=8048.48, stdev=144.73
clat percentiles (usec):
| 1.00th=[ 7373], 5.00th=[ 7439], 10.00th=[ 7439], 20.00th=[
7504],
| 30.00th=[ 7504], 40.00th=[ 7504], 50.00th=[ 7570], 60.00th=[
7570],
| 70.00th=[ 7570], 80.00th=[ 7635], 90.00th=[ 7832], 95.00th=[
7963],
| 99.00th=[ 7963], 99.50th=[ 8029], 99.90th=[ 8029], 99.95th=[
8029],
| 99.99th=[ 8094]
bw ( KiB/s): min=499712, max=523423, per=100.00%, avg=512080.41,
stdev=4112.59, samples=360
iops : min= 122, max= 127, avg=124.64, stdev= 0.98,
samples=360
lat (msec) : 10=100.00%
cpu : usr=0.11%, sys=5.97%, ctx=22989, majf=0, minf=5
IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=22349,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
rw-4096k-seq: (groupid=22, jobs=1): err= 0: pid=12909: Mon Aug 26
15:45:05 2019
read: IOPS=58, BW=234MiB/s (245MB/s)(41.1GiB/180006msec)
slat (usec): min=346, max=562, avg=452.82, stdev=26.35
clat (usec): min=7728, max=16713, avg=8327.62, stdev=806.21
lat (usec): min=8137, max=17156, avg=8781.03, stdev=807.21
clat percentiles (usec):
| 1.00th=[ 7767], 5.00th=[ 7767], 10.00th=[ 7832], 20.00th=[
7832],
| 30.00th=[ 7832], 40.00th=[ 7898], 50.00th=[ 7898], 60.00th=[
8291],
| 70.00th=[ 8356], 80.00th=[ 8979], 90.00th=[ 9241], 95.00th=[
9372],
| 99.00th=[10159], 99.50th=[14877], 99.90th=[16450],
99.95th=[16581],
| 99.99th=[16712]
bw ( KiB/s): min=164168, max=339264, per=100.00%, avg=241040.85,
stdev=33169.14, samples=360
iops : min= 40, max= 82, avg=58.44, stdev= 8.03,
samples=360
write: IOPS=57, BW=229MiB/s (240MB/s)(40.2GiB/180006msec)
slat (usec): min=389, max=875, avg=574.61, stdev=47.69
clat (usec): min=7718, max=18868, avg=7927.09, stdev=664.09
lat (usec): min=8142, max=19410, avg=8502.30, stdev=666.26
clat percentiles (usec):
| 1.00th=[ 7767], 5.00th=[ 7767], 10.00th=[ 7767], 20.00th=[
7832],
| 30.00th=[ 7832], 40.00th=[ 7898], 50.00th=[ 7898], 60.00th=[
7898],
| 70.00th=[ 7898], 80.00th=[ 7898], 90.00th=[ 7898], 95.00th=[
7898],
| 99.00th=[10814], 99.50th=[14746], 99.90th=[15008],
99.95th=[15270],
| 99.99th=[15533]
bw ( KiB/s): min=148945, max=320128, per=100.00%, avg=235860.97,
stdev=31961.69, samples=360
iops : min= 36, max= 78, avg=57.19, stdev= 7.77,
samples=360
lat (msec) : 10=98.93%, 20=1.07%
cpu : usr=1.06%, sys=5.11%, ctx=21483, majf=0, minf=7
IO depths : 1=116.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=10520,10295,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randrw-4096k-seq: (groupid=23, jobs=1): err= 0: pid=12929: Mon Aug 26
15:45:05 2019
read: IOPS=59, BW=236MiB/s (248MB/s)(41.5GiB/180008msec)
slat (usec): min=346, max=567, avg=463.00, stdev=26.69
clat (usec): min=7373, max=10851, avg=7817.33, stdev=441.45
lat (usec): min=7760, max=11352, avg=8280.93, stdev=442.88
clat percentiles (usec):
| 1.00th=[ 7373], 5.00th=[ 7439], 10.00th=[ 7439], 20.00th=[
7504],
| 30.00th=[ 7504], 40.00th=[ 7570], 50.00th=[ 7635], 60.00th=[
7701],
| 70.00th=[ 7898], 80.00th=[ 8094], 90.00th=[ 8455], 95.00th=[
8717],
| 99.00th=[ 9372], 99.50th=[ 9503], 99.90th=[ 9765],
99.95th=[10290],
| 99.99th=[10814]
bw ( KiB/s): min=132129, max=332332, per=100.00%, avg=243420.59,
stdev=32701.57, samples=360
iops : min= 32, max= 81, avg=59.02, stdev= 7.95,
samples=360
write: IOPS=60, BW=241MiB/s (252MB/s)(42.3GiB/180008msec)
slat (usec): min=402, max=885, avg=588.78, stdev=48.47
clat (usec): min=7696, max=16431, avg=7891.27, stdev=398.70
lat (usec): min=8171, max=16917, avg=8480.66, stdev=400.14
clat percentiles (usec):
| 1.00th=[ 7767], 5.00th=[ 7767], 10.00th=[ 7767], 20.00th=[
7832],
| 30.00th=[ 7832], 40.00th=[ 7832], 50.00th=[ 7898], 60.00th=[
7898],
| 70.00th=[ 7898], 80.00th=[ 7898], 90.00th=[ 7898], 95.00th=[
7898],
| 99.00th=[ 8029], 99.50th=[12256], 99.90th=[12387],
99.95th=[12387],
| 99.99th=[14877]
bw ( KiB/s): min=165829, max=355096, per=100.00%, avg=248270.02,
stdev=31688.91, samples=360
iops : min= 40, max= 86, avg=60.21, stdev= 7.69,
samples=360
lat (msec) : 10=99.59%, 20=0.41%
cpu : usr=1.15%, sys=5.40%, ctx=22140, majf=0, minf=6
IO depths : 1=116.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=10625,10836,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s),
io=21.1GiB (22.7GB), run=180001-180001msec
Run status group 1 (all jobs):
WRITE: bw=115MiB/s (121MB/s), 115MiB/s-115MiB/s (121MB/s-121MB/s),
io=20.2GiB (21.7GB), run=180001-180001msec
Run status group 2 (all jobs):
READ: bw=115MiB/s (121MB/s), 115MiB/s-115MiB/s (121MB/s-121MB/s),
io=20.2GiB (21.7GB), run=180001-180001msec
Run status group 3 (all jobs):
READ: bw=95.9MiB/s (101MB/s), 95.9MiB/s-95.9MiB/s (101MB/s-101MB/s),
io=16.9GiB (18.1GB), run=180001-180001msec
Run status group 4 (all jobs):
READ: bw=62.4MiB/s (65.4MB/s), 62.4MiB/s-62.4MiB/s
(65.4MB/s-65.4MB/s), io=10.0GiB (11.8GB), run=180001-180001msec
WRITE: bw=62.5MiB/s (65.5MB/s), 62.5MiB/s-62.5MiB/s
(65.5MB/s-65.5MB/s), io=10.0GiB (11.8GB), run=180001-180001msec
Run status group 5 (all jobs):
READ: bw=53.0MiB/s (56.6MB/s), 53.0MiB/s-53.0MiB/s
(56.6MB/s-56.6MB/s), io=9719MiB (10.2GB), run=180001-180001msec
WRITE: bw=54.0MiB/s (56.6MB/s), 54.0MiB/s-54.0MiB/s
(56.6MB/s-56.6MB/s), io=9720MiB (10.2GB), run=180001-180001msec
Run status group 6 (all jobs):
WRITE: bw=395MiB/s (414MB/s), 395MiB/s-395MiB/s (414MB/s-414MB/s),
io=69.5GiB (74.6GB), run=180001-180001msec
Run status group 7 (all jobs):
WRITE: bw=394MiB/s (413MB/s), 394MiB/s-394MiB/s (413MB/s-413MB/s),
io=69.2GiB (74.3GB), run=180001-180001msec
Run status group 8 (all jobs):
READ: bw=407MiB/s (427MB/s), 407MiB/s-407MiB/s (427MB/s-427MB/s),
io=71.6GiB (76.9GB), run=180001-180001msec
Run status group 9 (all jobs):
READ: bw=379MiB/s (397MB/s), 379MiB/s-379MiB/s (397MB/s-397MB/s),
io=66.6GiB (71.5GB), run=180001-180001msec
Run status group 10 (all jobs):
READ: bw=176MiB/s (184MB/s), 176MiB/s-176MiB/s (184MB/s-184MB/s),
io=30.9GiB (33.2GB), run=180001-180001msec
WRITE: bw=176MiB/s (185MB/s), 176MiB/s-176MiB/s (185MB/s-185MB/s),
io=30.9GiB (33.2GB), run=180001-180001msec
Run status group 11 (all jobs):
READ: bw=178MiB/s (186MB/s), 178MiB/s-178MiB/s (186MB/s-186MB/s),
io=31.2GiB (33.5GB), run=180001-180001msec
WRITE: bw=178MiB/s (187MB/s), 178MiB/s-178MiB/s (187MB/s-187MB/s),
io=31.3GiB (33.6GB), run=180001-180001msec
Run status group 12 (all jobs):
WRITE: bw=463MiB/s (486MB/s), 463MiB/s-463MiB/s (486MB/s-486MB/s),
io=81.4GiB (87.4GB), run=180002-180002msec
Run status group 13 (all jobs):
WRITE: bw=464MiB/s (487MB/s), 464MiB/s-464MiB/s (487MB/s-487MB/s),
io=81.6GiB (87.6GB), run=180001-180001msec
Run status group 14 (all jobs):
READ: bw=468MiB/s (491MB/s), 468MiB/s-468MiB/s (491MB/s-491MB/s),
io=82.2GiB (88.3GB), run=180002-180002msec
Run status group 15 (all jobs):
READ: bw=481MiB/s (505MB/s), 481MiB/s-481MiB/s (505MB/s-505MB/s),
io=84.6GiB (90.8GB), run=180002-180002msec
Run status group 16 (all jobs):
READ: bw=212MiB/s (222MB/s), 212MiB/s-212MiB/s (222MB/s-222MB/s),
io=37.2GiB (39.0GB), run=180001-180001msec
WRITE: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s),
io=36.4GiB (39.1GB), run=180001-180001msec
Run status group 17 (all jobs):
READ: bw=222MiB/s (233MB/s), 222MiB/s-222MiB/s (233MB/s-233MB/s),
io=39.1GiB (41.0GB), run=180002-180002msec
WRITE: bw=222MiB/s (232MB/s), 222MiB/s-222MiB/s (232MB/s-232MB/s),
io=38.0GiB (41.8GB), run=180002-180002msec
Run status group 18 (all jobs):
WRITE: bw=470MiB/s (493MB/s), 470MiB/s-470MiB/s (493MB/s-493MB/s),
io=82.6GiB (88.7GB), run=180008-180008msec
Run status group 19 (all jobs):
WRITE: bw=471MiB/s (494MB/s), 471MiB/s-471MiB/s (494MB/s-494MB/s),
io=82.9GiB (88.0GB), run=180001-180001msec
Run status group 20 (all jobs):
READ: bw=482MiB/s (505MB/s), 482MiB/s-482MiB/s (505MB/s-505MB/s),
io=84.7GiB (90.9GB), run=180005-180005msec
Run status group 21 (all jobs):
READ: bw=497MiB/s (521MB/s), 497MiB/s-497MiB/s (521MB/s-521MB/s),
io=87.3GiB (93.7GB), run=180003-180003msec
Run status group 22 (all jobs):
READ: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s),
io=41.1GiB (44.1GB), run=180006-180006msec
WRITE: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s),
io=40.2GiB (43.2GB), run=180006-180006msec
Run status group 23 (all jobs):
READ: bw=236MiB/s (248MB/s), 236MiB/s-236MiB/s (248MB/s-248MB/s),
io=41.5GiB (44.6GB), run=180008-180008msec
WRITE: bw=241MiB/s (252MB/s), 241MiB/s-241MiB/s (252MB/s-252MB/s),
io=42.3GiB (45.4GB), run=180008-180008msec
Disk stats (read/write):
sdb: ios=20762156/22203113, merge=0/0, ticks=4810773/4551307,
in_queue=9339055, util=85.66%
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
