We were able to isolate an individual Micron 5200 and perform Vitaliy's
tests in his spreadsheet.
An interesting item - write cache changes do NOT require a power cycle
to take effect, at least on a Micron 5200.
The complete results from fio are included at the end of this message
for the individual tests, for both write enabled and disabled.
The shortened version of the results:
Journal IOPS (sync=1)
------------------------------------------------------------------------
-------------
Write cache ON:
write: IOPS=19.7k, BW=76.0MiB/s (80.7MB/s)(4618MiB/60001msec)
lat (usec): min=42, max=1273, avg=50.18, stdev= 6.40
Write cache OFF:
write: IOPS=32.3k, BW=126MiB/s (132MB/s)(7560MiB/60001msec)
lat (usec): min=25, max=7079, avg=30.55, stdev= 7.94
Journal IOPS (fsync=1)
------------------------------------------------------------------------
-------------
Write cache ON:
write: IOPS=16.9k, BW=66.2MiB/s (69.4MB/s)(3971MiB/60001msec)
lat (usec): min=24, max=5068, avg=31.77, stdev= 7.82
Write cache OFF:
write: IOPS=32.1k, BW=126MiB/s (132MB/s)(7533MiB/60001msec)
lat (usec): min=24, max=7076, avg=29.41, stdev= 7.52
Parallel random (sync)
------------------------------------------------------------------------
-------------
Write cache ON:
write: IOPS=43.9k, BW=172MiB/s (180MB/s)(10.1GiB/60001msec)
lat (usec): min=220, max=14767, avg=727.61, stdev=313.36
Write cache OFF:
write: IOPS=44.3k, BW=173MiB/s (181MB/s)(10.1GiB/60001msec)
lat (usec): min=134, max=4941, avg=721.96, stdev=311.46
Parallel random (fsync)
------------------------------------------------------------------------
-------------
Write cache ON:
write: IOPS=44.4k, BW=173MiB/s (182MB/s)(10.2GiB/60001msec)
lat (usec): min=109, max=4349, avg=703.01, stdev=303.69
Write cache OFF:
write: IOPS=44.6k, BW=174MiB/s (183MB/s)(10.2GiB/60001msec)
lat (usec): min=26, max=7288, avg=716.32, stdev=300.48
Non-txn random
------------------------------------------------------------------------
-------------
Write cache ON:
write: IOPS=43.1k, BW=168MiB/s (177MB/s)(9.87GiB/60004msec)
lat (usec): min=350, max=41703, avg=2967.89, stdev=1682.28
Write cache OFF:
write: IOPS=43.4k, BW=170MiB/s (178MB/s)(9.93GiB/60004msec)
lat (usec): min=177, max=42795, avg=2947.52, stdev=1666.24
Linear write
------------------------------------------------------------------------
-------------
Write cache ON:
write: IOPS=126, BW=505MiB/s (530MB/s)(29.6GiB/60027msec)
lat (msec): min=226, max=281, avg=253.26, stdev= 3.51
Write cache OFF:
write: IOPS=126, BW=507MiB/s (531MB/s)(29.8GiB/60254msec)
lat (msec): min=7, max=492, avg=252.52, stdev=13.16
So, we can determine that some improvement can be seen with the write
cache disabled (specifically on a Micron 5200), it is not enough that
will likely change much in terms of Ceph's performance unless journal
latency, IOPS, and bandwidth are a bottleneck.
The "Journal IOPS (sync=1)" test shows the most dramatic difference,
where disabling the write cache reduces the I/O latency by 39% (a
reduction from 50.18us to 30.55us with a difference of 0.02ms) which
respectively raises the IOPS and throughput of synchronous I/O.
The "Journal IOPS (fsync=1)" test also shows a dramatic difference, but
in terms of IOPS and throughput (approximately +90%), not latency.
Hope this helps! I would love to hear feedback.
Eric
###################################
# Journal IOPS (sync=1)
###################################
# Write cache ENABLED
hdparm -W 1 /dev/sde
fio --ioengine=libaio -sync=1 --direct=1 --name=test --bs=4k --iodepth=1
--readwrite=write --runtime 60 --filename=/dev/sde
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=75.6MiB/s][r=0,w=19.3k
IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=38269: Tue Jan 21 14:20:01 2020
write: IOPS=19.7k, BW=76.0MiB/s (80.7MB/s)(4618MiB/60001msec)
slat (usec): min=2, max=180, avg= 4.43, stdev= 1.86
clat (nsec): min=1950, max=1262.3k, avg=45662.55, stdev=5778.88
lat (usec): min=42, max=1273, avg=50.18, stdev= 6.40
clat percentiles (usec):
| 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[
43],
| 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 44], 60.00th=[
45],
| 70.00th=[ 47], 80.00th=[ 48], 90.00th=[ 51], 95.00th=[
55],
| 99.00th=[ 66], 99.50th=[ 74], 99.90th=[ 91], 99.95th=[
104],
| 99.99th=[ 167]
bw ( KiB/s): min=70152, max=81704, per=100.00%, avg=78835.97,
stdev=2929.71, samples=119
iops : min=17538, max=20426, avg=19708.98, stdev=732.40,
samples=119
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=88.53%
lat (usec) : 100=11.41%, 250=0.06%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%
cpu : usr=3.07%, sys=13.62%, ctx=1182324, majf=0, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwts: total=0,1182284,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=76.0MiB/s (80.7MB/s), 76.0MiB/s-76.0MiB/s
(80.7MB/s-80.7MB/s), io=4618MiB (4843MB), run=60001-60001msec
Disk stats (read/write):
sde: ios=0/1179332, merge=0/0, ticks=0/0, in_queue=53791, util=89.80%
# Write cache DISABLED
hdparm -W 0 /dev/sde
fio --ioengine=libaio -sync=1 --direct=1 --name=test --bs=4k --iodepth=1
--readwrite=write --runtime 60 --filename=/dev/sde
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=127MiB/s][r=0,w=32.5k IOPS][eta
00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=37681: Tue Jan 21 14:18:31 2020
write: IOPS=32.3k, BW=126MiB/s (132MB/s)(7560MiB/60001msec)
slat (usec): min=2, max=119, avg= 4.02, stdev= 2.35
clat (usec): min=3, max=7075, avg=26.45, stdev= 7.46
lat (usec): min=25, max=7079, avg=30.55, stdev= 7.94
clat percentiles (usec):
| 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[
24],
| 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 25], 60.00th=[
25],
| 70.00th=[ 26], 80.00th=[ 27], 90.00th=[ 34], 95.00th=[
39],
| 99.00th=[ 49], 99.50th=[ 54], 99.90th=[ 67], 99.95th=[
80],
| 99.99th=[ 135]
bw ( KiB/s): min=125264, max=133205, per=99.98%, avg=129000.82,
stdev=1305.91, samples=119
iops : min=31316, max=33301, avg=32250.18, stdev=326.48,
samples=119
lat (usec) : 4=0.01%, 10=0.01%, 20=0.17%, 50=98.96%, 100=0.85%
lat (usec) : 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 10=0.01%
cpu : usr=5.02%, sys=19.89%, ctx=1935347, majf=0, minf=19
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwts: total=0,1935348,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s),
io=7560MiB (7927MB), run=60001-60001msec
Disk stats (read/write):
sde: ios=0/1929843, merge=0/0, ticks=0/7, in_queue=51540, util=86.03%
###################################
# Journal IOPS (fsync=1)
###################################
# Write cache ENABLED
hdparm -W 1 /dev/sde
fio --ioengine=libaio -fsync=1 --direct=1 --name=test --bs=4k
--iodepth=1 --readwrite=write --runtime 60 --filename=/dev/sde
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=67.6MiB/s][r=0,w=17.3k
IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=53286: Tue Jan 21 14:26:11 2020
write: IOPS=16.9k, BW=66.2MiB/s (69.4MB/s)(3971MiB/60001msec)
slat (usec): min=2, max=138, avg= 5.30, stdev= 2.66
clat (nsec): min=1175, max=5063.8k, avg=26367.70, stdev=7131.05
lat (usec): min=24, max=5068, avg=31.77, stdev= 7.82
clat percentiles (usec):
| 1.00th=[ 23], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[
25],
| 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 25], 60.00th=[
26],
| 70.00th=[ 26], 80.00th=[ 26], 90.00th=[ 31], 95.00th=[
38],
| 99.00th=[ 47], 99.50th=[ 53], 99.90th=[ 66], 99.95th=[
80],
| 99.99th=[ 139]
bw ( KiB/s): min=63728, max=74376, per=100.00%, avg=67777.82,
stdev=2506.07, samples=119
iops : min=15932, max=18594, avg=16944.44, stdev=626.53,
samples=119
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.15%, 50=99.09%
lat (usec) : 100=0.73%, 250=0.03%, 500=0.01%, 750=0.01%
lat (msec) : 10=0.01%
fsync/fdatasync/sync_file_range:
sync (nsec): min=45, max=21704, avg=90.66, stdev=242.08
sync percentiles (nsec):
| 1.00th=[ 59], 5.00th=[ 70], 10.00th=[ 72], 20.00th=[
75],
| 30.00th=[ 76], 40.00th=[ 77], 50.00th=[ 78], 60.00th=[
80],
| 70.00th=[ 81], 80.00th=[ 86], 90.00th=[ 113], 95.00th=[
131],
| 99.00th=[ 229], 99.50th=[ 286], 99.90th=[ 398], 99.95th=[
540],
| 99.99th=[16768]
cpu : usr=4.13%, sys=21.90%, ctx=2033277, majf=0, minf=43
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwts: total=0,1016651,0,1016651 short=0,0,0,0
dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=66.2MiB/s (69.4MB/s), 66.2MiB/s-66.2MiB/s
(69.4MB/s-69.4MB/s), io=3971MiB (4164MB), run=60001-60001msec
Disk stats (read/write):
sde: ios=0/2027538, merge=0/0, ticks=0/5, in_queue=49335, util=82.37%
# Write cache DISABLED
hdparm -W 0 /dev/sde
fio --ioengine=libaio -fsync=1 --direct=1 --name=test --bs=4k
--iodepth=1 --readwrite=write --runtime 60 --filename=/dev/sde
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=127MiB/s][r=0,w=32.5k IOPS][eta
00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=53949: Tue Jan 21 14:28:08 2020
write: IOPS=32.1k, BW=126MiB/s (132MB/s)(7533MiB/60001msec)
slat (nsec): min=2811, max=82814, avg=4027.35, stdev=1621.66
clat (nsec): min=921, max=7072.8k, avg=25289.52, stdev=7275.27
lat (usec): min=24, max=7076, avg=29.41, stdev= 7.52
clat percentiles (usec):
| 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 24], 20.00th=[
24],
| 30.00th=[ 24], 40.00th=[ 25], 50.00th=[ 25], 60.00th=[
25],
| 70.00th=[ 25], 80.00th=[ 26], 90.00th=[ 29], 95.00th=[
33],
| 99.00th=[ 42], 99.50th=[ 49], 99.90th=[ 64], 99.95th=[
74],
| 99.99th=[ 139]
bw ( KiB/s): min=122896, max=133088, per=100.00%, avg=128554.66,
stdev=2600.79, samples=119
iops : min=30724, max=33272, avg=32138.66, stdev=650.19,
samples=119
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.11%, 50=99.43%
lat (usec) : 100=0.43%, 250=0.02%, 500=0.01%, 750=0.01%
lat (msec) : 4=0.01%, 10=0.01%
fsync/fdatasync/sync_file_range:
sync (nsec): min=25, max=23077, avg=53.70, stdev=113.94
sync percentiles (nsec):
| 1.00th=[ 41], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[
46],
| 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 48], 60.00th=[
48],
| 70.00th=[ 51], 80.00th=[ 53], 90.00th=[ 68], 95.00th=[
84],
| 99.00th=[ 127], 99.50th=[ 137], 99.90th=[ 167], 99.95th=[
187],
| 99.99th=[ 446]
cpu : usr=6.25%, sys=23.03%, ctx=1928286, majf=0, minf=19
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwts: total=0,1928343,0,1928342 short=0,0,0,0
dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s),
io=7533MiB (7898MB), run=60001-60001msec
Disk stats (read/write):
sde: ios=0/1923034, merge=0/0, ticks=0/7, in_queue=50356, util=84.05%
###################################
# Parallel random (sync)
###################################
# Write cache ENABLED
hdparm -W 1 /dev/sde
fio --ioengine=libaio --sync 1 --direct=1 --name=test --bs=4k
--iodepth=32 --readwrite=randwrite --runtime 60 --filename=/dev/sde
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=32
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=185MiB/s][r=0,w=47.5k IOPS][eta
00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2179: Tue Jan 21 14:31:13 2020
write: IOPS=43.9k, BW=172MiB/s (180MB/s)(10.1GiB/60001msec)
slat (nsec): min=1908, max=263064, avg=5245.93, stdev=6753.97
clat (usec): min=190, max=14511, avg=722.29, stdev=313.68
lat (usec): min=220, max=14767, avg=727.61, stdev=313.36
clat percentiles (usec):
| 1.00th=[ 437], 5.00th=[ 461], 10.00th=[ 482], 20.00th=[
506],
| 30.00th=[ 537], 40.00th=[ 586], 50.00th=[ 644], 60.00th=[
701],
| 70.00th=[ 766], 80.00th=[ 865], 90.00th=[ 1037], 95.00th=[
1237],
| 99.00th=[ 2089], 99.50th=[ 2507], 99.90th=[ 3032], 99.95th=[
3228],
| 99.99th=[ 3818]
bw ( KiB/s): min=127488, max=191488, per=99.93%, avg=175628.84,
stdev=15728.90, samples=119
iops : min=31872, max=47872, avg=43907.21, stdev=3932.22,
samples=119
lat (usec) : 250=0.01%, 500=17.19%, 750=50.25%, 1000=21.08%
lat (msec) : 2=10.32%, 4=1.14%, 10=0.01%, 20=0.01%
cpu : usr=4.47%, sys=24.21%, ctx=82461, majf=0, minf=1189
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>=64=0.0%
issued rwts: total=0,2636233,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=172MiB/s (180MB/s), 172MiB/s-172MiB/s (180MB/s-180MB/s),
io=10.1GiB (10.8GB), run=60001-60001msec
Disk stats (read/write):
sde: ios=0/2626505, merge=0/0, ticks=0/243501, in_queue=1638201,
util=98.59%
# Write cache DISABLED
hdparm -W 0 /dev/sde
fio --ioengine=libaio --sync 1 --direct=1 --name=test --bs=4k
--iodepth=32 --readwrite=randwrite --runtime 60 --filename=/dev/sde
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=32
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=174MiB/s][r=0,w=44.6k IOPS][eta
00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2679: Tue Jan 21 14:32:43 2020
write: IOPS=44.3k, BW=173MiB/s (181MB/s)(10.1GiB/60001msec)
slat (usec): min=2, max=277, avg= 4.56, stdev= 1.69
clat (usec): min=131, max=4937, avg=717.32, stdev=311.45
lat (usec): min=134, max=4941, avg=721.96, stdev=311.46
clat percentiles (usec):
| 1.00th=[ 375], 5.00th=[ 383], 10.00th=[ 404], 20.00th=[
465],
| 30.00th=[ 529], 40.00th=[ 594], 50.00th=[ 660], 60.00th=[
725],
| 70.00th=[ 807], 80.00th=[ 914], 90.00th=[ 1074], 95.00th=[
1221],
| 99.00th=[ 1860], 99.50th=[ 2376], 99.90th=[ 2933], 99.95th=[
3097],
| 99.99th=[ 3425]
bw ( KiB/s): min=131393, max=188032, per=99.99%, avg=177060.72,
stdev=12247.06, samples=120
iops : min=32848, max=47008, avg=44265.15, stdev=3061.80,
samples=120
lat (usec) : 250=0.01%, 500=25.29%, 750=37.97%, 1000=23.10%
lat (msec) : 2=12.77%, 4=0.84%, 10=0.01%
cpu : usr=7.21%, sys=30.12%, ctx=2457978, majf=0, minf=2723
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>=64=0.0%
issued rwts: total=0,2656202,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=173MiB/s (181MB/s), 173MiB/s-173MiB/s (181MB/s-181MB/s),
io=10.1GiB (10.9GB), run=60001-60001msec
Disk stats (read/write):
sde: ios=0/2648092, merge=0/0, ticks=0/385081, in_queue=1901845,
util=99.82%
###################################
# Parallel random (fsync)
###################################
# Write cache ENABLED
hdparm -W 1 /dev/sde
fio -ioengine=libaio -fsync=32 -direct=1 -name=test -bs=4k -iodepth=32
-readwrite=randwrite -runtime 60 -filename=/dev/sde
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=32
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=180MiB/s][r=0,w=46.2k IOPS][eta
00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=3284: Tue Jan 21 14:35:18 2020
write: IOPS=44.4k, BW=173MiB/s (182MB/s)(10.2GiB/60001msec)
slat (usec): min=2, max=310, avg= 3.66, stdev= 1.85
clat (usec): min=103, max=4346, avg=699.27, stdev=304.08
lat (usec): min=109, max=4349, avg=703.01, stdev=303.69
clat percentiles (usec):
| 1.00th=[ 137], 5.00th=[ 424], 10.00th=[ 445], 20.00th=[
482],
| 30.00th=[ 529], 40.00th=[ 586], 50.00th=[ 644], 60.00th=[
701],
| 70.00th=[ 775], 80.00th=[ 865], 90.00th=[ 1029], 95.00th=[
1205],
| 99.00th=[ 1860], 99.50th=[ 2343], 99.90th=[ 2900], 99.95th=[
3064],
| 99.99th=[ 3425]
bw ( KiB/s): min=135680, max=189440, per=99.95%, avg=177495.28,
stdev=12530.84, samples=119
iops : min=33920, max=47360, avg=44373.77, stdev=3132.68,
samples=119
lat (usec) : 250=2.95%, 500=21.37%, 750=42.66%, 1000=21.96%
lat (msec) : 2=10.24%, 4=0.81%, 10=0.01%
fsync/fdatasync/sync_file_range:
sync (nsec): min=70, max=18732, avg=285.51, stdev=311.45
sync percentiles (nsec):
| 1.00th=[ 92], 5.00th=[ 113], 10.00th=[ 137], 20.00th=[
179],
| 30.00th=[ 219], 40.00th=[ 262], 50.00th=[ 290], 60.00th=[
314],
| 70.00th=[ 330], 80.00th=[ 350], 90.00th=[ 418], 95.00th=[
466],
| 99.00th=[ 532], 99.50th=[ 548], 99.90th=[ 724], 99.95th=[
4192],
| 99.99th=[17280]
cpu : usr=5.24%, sys=18.64%, ctx=175057, majf=0, minf=2785
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=103.1%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>=64=0.0%
issued rwts: total=0,2663737,0,83241 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=173MiB/s (182MB/s), 173MiB/s-173MiB/s (182MB/s-182MB/s),
io=10.2GiB (10.9GB), run=60001-60001msec
Disk stats (read/write):
sde: ios=0/2737160, merge=0/0, ticks=0/64338, in_queue=854478,
util=94.59%
# Write cache DISABLED
hdparm -W 0 /dev/sde
fio -ioengine=libaio -fsync=32 -direct=1 -name=test -bs=4k -iodepth=32
-readwrite=randwrite -runtime 60 -filename=/dev/sde
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=32
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=180MiB/s][r=0,w=46.2k IOPS][eta
00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=5331: Tue Jan 21 14:37:00 2020
write: IOPS=44.6k, BW=174MiB/s (183MB/s)(10.2GiB/60001msec)
slat (usec): min=2, max=332, avg= 4.34, stdev= 2.95
clat (usec): min=22, max=7285, avg=711.90, stdev=300.51
lat (usec): min=26, max=7288, avg=716.32, stdev=300.48
clat percentiles (usec):
| 1.00th=[ 371], 5.00th=[ 392], 10.00th=[ 412], 20.00th=[
469],
| 30.00th=[ 529], 40.00th=[ 594], 50.00th=[ 660], 60.00th=[
717],
| 70.00th=[ 799], 80.00th=[ 898], 90.00th=[ 1057], 95.00th=[
1205],
| 99.00th=[ 1762], 99.50th=[ 2212], 99.90th=[ 2900], 99.95th=[
3130],
| 99.99th=[ 4113]
bw ( KiB/s): min=142432, max=187944, per=99.99%, avg=178456.13,
stdev=10264.83, samples=120
iops : min=35608, max=46986, avg=44614.05, stdev=2566.17,
samples=120
lat (usec) : 50=0.02%, 100=0.01%, 250=0.01%, 500=24.81%, 750=39.11%
lat (usec) : 1000=23.30%
lat (msec) : 2=12.06%, 4=0.67%, 10=0.01%
fsync/fdatasync/sync_file_range:
sync (nsec): min=20, max=24154, avg=72.68, stdev=314.56
sync percentiles (nsec):
| 1.00th=[ 44], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[
56],
| 30.00th=[ 60], 40.00th=[ 62], 50.00th=[ 63], 60.00th=[
64],
| 70.00th=[ 66], 80.00th=[ 68], 90.00th=[ 84], 95.00th=[
102],
| 99.00th=[ 133], 99.50th=[ 153], 99.90th=[ 247], 99.95th=[
3280],
| 99.99th=[18304]
cpu : usr=7.32%, sys=28.57%, ctx=2392052, majf=0, minf=351
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=103.1%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>=64=0.0%
issued rwts: total=0,2677096,0,83659 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=174MiB/s (183MB/s), 174MiB/s-174MiB/s (183MB/s-183MB/s),
io=10.2GiB (10.0GB), run=60001-60001msec
Disk stats (read/write):
sde: ios=0/2667686, merge=0/0, ticks=0/361398, in_queue=1899749,
util=99.81%
###################################
# Non-txn random
###################################
# Write cache ENABLED
hdparm -W 1 /dev/sde
fio --ioengine=libaio --direct=1 --name=test --bs=4k --iodepth=128
--readwrite=randwrite --runtime 60 --filename=/dev/sde
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=128
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=173MiB/s][r=0,w=44.2k IOPS][eta
00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=5863: Tue Jan 21 14:39:42 2020
write: IOPS=43.1k, BW=168MiB/s (177MB/s)(9.87GiB/60004msec)
slat (nsec): min=1486, max=268843, avg=2980.57, stdev=2666.13
clat (usec): min=348, max=41700, avg=2964.83, stdev=1682.29
lat (usec): min=350, max=41703, avg=2967.89, stdev=1682.28
clat percentiles (usec):
| 1.00th=[ 570], 5.00th=[ 865], 10.00th=[ 1123], 20.00th=[
1549],
| 30.00th=[ 1991], 40.00th=[ 2409], 50.00th=[ 2835], 60.00th=[
3261],
| 70.00th=[ 3654], 80.00th=[ 4080], 90.00th=[ 4686], 95.00th=[
5342],
| 99.00th=[ 8029], 99.50th=[10814], 99.90th=[17433],
99.95th=[19792],
| 99.99th=[24773]
bw ( KiB/s): min=145144, max=187920, per=99.99%, avg=172411.80,
stdev=9582.90, samples=120
iops : min=36286, max=46980, avg=43102.94, stdev=2395.76,
samples=120
lat (usec) : 500=0.41%, 750=2.70%, 1000=4.43%
lat (msec) : 2=22.74%, 4=47.73%, 10=21.38%, 20=0.56%, 50=0.05%
cpu : usr=9.25%, sys=26.84%, ctx=2308725, majf=0, minf=349
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%,
>=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.1%
issued rwts: total=0,2586513,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
WRITE: bw=168MiB/s (177MB/s), 168MiB/s-168MiB/s (177MB/s-177MB/s),
io=9.87GiB (10.6GB), run=60004-60004msec
Disk stats (read/write):
sde: ios=0/2577017, merge=0/0, ticks=0/6339012, in_queue=7635424,
util=99.82%
# Write cache DISABLED
hdparm -W 0 /dev/sde
fio --ioengine=libaio --direct=1 --name=test --bs=4k --iodepth=128
--readwrite=randwrite --runtime 60 --filename=/dev/sde
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=128
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=173MiB/s][r=0,w=44.2k IOPS][eta
00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=6444: Tue Jan 21 14:41:18 2020
write: IOPS=43.4k, BW=170MiB/s (178MB/s)(9.93GiB/60004msec)
slat (nsec): min=1525, max=370532, avg=3344.44, stdev=2412.72
clat (usec): min=172, max=42791, avg=2944.07, stdev=1666.24
lat (usec): min=177, max=42795, avg=2947.52, stdev=1666.24
clat percentiles (usec):
| 1.00th=[ 553], 5.00th=[ 832], 10.00th=[ 1090], 20.00th=[
1549],
| 30.00th=[ 1991], 40.00th=[ 2409], 50.00th=[ 2835], 60.00th=[
3228],
| 70.00th=[ 3621], 80.00th=[ 4047], 90.00th=[ 4621], 95.00th=[
5276],
| 99.00th=[ 8094], 99.50th=[10552], 99.90th=[16909],
99.95th=[19268],
| 99.99th=[24511]
bw ( KiB/s): min=134704, max=187328, per=99.99%, avg=173594.69,
stdev=11418.81, samples=120
iops : min=33676, max=46832, avg=43398.65, stdev=2854.76,
samples=120
lat (usec) : 250=0.01%, 500=0.54%, 750=3.06%, 1000=4.49%
lat (msec) : 2=22.04%, 4=48.82%, 10=20.45%, 20=0.55%, 50=0.04%
cpu : usr=8.06%, sys=25.75%, ctx=2265281, majf=0, minf=8051
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%,
>=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.1%
issued rwts: total=0,2604374,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
WRITE: bw=170MiB/s (178MB/s), 170MiB/s-170MiB/s (178MB/s-178MB/s),
io=9.93GiB (10.7GB), run=60004-60004msec
Disk stats (read/write):
sde: ios=0/2594742, merge=0/0, ticks=0/6323399, in_queue=7637489,
util=99.82%
###################################
# Linear Write
###################################
# Write cache ENABLED
hdparm -W 1 /dev/sde
fio --ioengine=libaio --sync 1 --direct=1 --name=test --bs=4M
--iodepth=32 --readwrite=write --runtime 60 --filename=/dev/sde
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=444MiB/s][r=0,w=111 IOPS][eta
00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=8159: Tue Jan 21 14:43:17 2020
write: IOPS=126, BW=505MiB/s (530MB/s)(29.6GiB/60027msec)
slat (usec): min=98, max=2656, avg=248.53, stdev=99.88
clat (msec): min=226, max=281, avg=253.01, stdev= 3.52
lat (msec): min=226, max=281, avg=253.26, stdev= 3.51
clat percentiles (msec):
| 1.00th=[ 247], 5.00th=[ 249], 10.00th=[ 249], 20.00th=[
251],
| 30.00th=[ 251], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[
253],
| 70.00th=[ 255], 80.00th=[ 255], 90.00th=[ 257], 95.00th=[
259],
| 99.00th=[ 264], 99.50th=[ 266], 99.90th=[ 275], 99.95th=[
279],
| 99.99th=[ 284]
bw ( KiB/s): min=262144, max=524288, per=99.61%, avg=515497.52,
stdev=40828.47, samples=120
iops : min= 64, max= 128, avg=125.82, stdev= 9.96,
samples=120
lat (msec) : 250=18.18%, 500=81.82%
cpu : usr=2.54%, sys=0.63%, ctx=291, majf=0, minf=600
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>=64=0.0%
issued rwts: total=0,7584,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=505MiB/s (530MB/s), 505MiB/s-505MiB/s (530MB/s-530MB/s),
io=29.6GiB (31.8GB), run=60027-60027msec
Disk stats (read/write):
sde: ios=0/30810, merge=0/0, ticks=0/7663827, in_queue=7680115,
util=99.81%
# Write cache DISABLED
hdparm -W 0 /dev/sde
fio --ioengine=libaio --sync 1 --direct=1 --name=test --bs=4M
--iodepth=32 --readwrite=write --runtime 60 --filename=/dev/sde
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=505MiB/s][r=0,w=126 IOPS][eta
00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=8486: Tue Jan 21 14:44:43 2020
write: IOPS=126, BW=507MiB/s (531MB/s)(29.8GiB/60254msec)
slat (usec): min=100, max=533, avg=237.75, stdev=45.78
clat (msec): min=7, max=492, avg=252.28, stdev=13.16
lat (msec): min=7, max=492, avg=252.52, stdev=13.16
clat percentiles (msec):
| 1.00th=[ 251], 5.00th=[ 251], 10.00th=[ 251], 20.00th=[
251],
| 30.00th=[ 251], 40.00th=[ 251], 50.00th=[ 253], 60.00th=[
253],
| 70.00th=[ 253], 80.00th=[ 253], 90.00th=[ 255], 95.00th=[
257],
| 99.00th=[ 264], 99.50th=[ 268], 99.90th=[ 439], 99.95th=[
468],
| 99.99th=[ 493]
bw ( KiB/s): min=499712, max=524288, per=99.99%, avg=518885.49,
stdev=5490.08, samples=120
iops : min= 122, max= 128, avg=126.63, stdev= 1.37,
samples=120
lat (msec) : 10=0.01%, 20=0.01%, 50=0.05%, 100=0.09%, 250=6.22%
lat (msec) : 500=93.61%
cpu : usr=2.70%, sys=0.50%, ctx=7641, majf=0, minf=74
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
>=64=0.0%
issued rwts: total=0,7634,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=507MiB/s (531MB/s), 507MiB/s-507MiB/s (531MB/s-531MB/s),
io=29.8GiB (32.0GB), run=60254-60254msec
Disk stats (read/write):
sde: ios=0/30659, merge=0/0, ticks=0/7621094, in_queue=7653144,
util=99.82%
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
