Hi Andrey,
Thanks again for your tips .. the psync thingy in particular. I need to
verify if that applies to PostgreSQL, because it brings huge gains
compared to sync!
Here is the summary of my latest numbers:
1) engine=libaio
Individual NVMes:
iops=7350.4K
usr=8.25%, sys=64.76%, ctx=57533651
MD (RAID-0) over NVMes:
iops=4112.8K
usr=4.98%, sys=94.81%, ctx=12736
=> MD reaches 55% of perf compared to non-MD.
2) engine=sync
Individual NVMes:
IOPS=6657k
usr=0.56%, sys=4.43%, ctx=200588483
MD (RAID-0) over NVMes:
IOPS=1467k
usr=0.07%, sys=4.13%, ctx=46545978
=> MD reaches 22% of perf compared to non-MD.
3) engine=psync
Individual NVMes:
IOPS=7086k
usr=0.60%, sys=4.43%, ctx=214720330
MD (RAID-0) over NVMes:
IOPS=4154k
usr=0.46%, sys=5.81%, ctx=124737165
=> MD reaches 58% of perf compared to non-MD.
==================
Are the CPU load numbers reported by FIO reliable?
I mean, compare the load between libaio and sync/psync!
Cheers,
/Tobias
oberstet@svr-psql19:~/scm/parcit/RA/adr/system/docs$ cat
best_sync_individual_nvmes.fio
[global]
group_reporting
size=30G
ioengine=sync
iodepth=1
thread=1
direct=1
time_based=1
randrepeat=0
norandommap=1
disable_lat=1
bs=4k
runtime=30
[randread-individual-nvmes]
stonewall
filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1:/dev/nvme10n1:/dev/nvme11n1:/dev/nvme12n1:/dev/nvme13n1:/dev/nvme14n1:/dev/nvme15n1
rw=randread
numjobs=2800
oberstet@svr-psql19:~/scm/parcit/RA/adr/system/docs$ cat
best_sync_md_over_nvmes.fio
[global]
group_reporting
size=30G
ioengine=sync
iodepth=1
thread=1
direct=1
time_based=1
randrepeat=0
norandommap=1
disable_lat=1
bs=4k
runtime=30
[randread-md-over-nvmes]
stonewall
filename=/dev/md1
rw=randread
numjobs=2800
oberstet@svr-psql19:~/scm/parcit/RA/adr/system/docs$ sudo
/opt/fio/bin/fio best_sync_individual_nvmes.fio
randread-individual-nvmes: (g=0): rw=randread,
bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=sync, iodepth=1
...
fio-2.17-17-g9cf1
Starting 2800 threads
Jobs: 2747 (f=28032):
[f(9),_(1),f(27),_(3),f(20),_(1),f(2),_(1),f(57),_(1),f(250),_(1),f(108),_(1),f(48),_(1),f(26),_(1),f(14),_(2),f(444),_(1),f(36),_(1),f(193),_(1),f(100),_(1),f(26),_(1),f(40),_(1),f(1),_(1),f(19),_(2),f(36),_(1),f(77),_(1),f(20),_(1),f(37),_(1),f(6),_(1),f(8),_(1),f(45),_(1),f(3),_(1),f(10),_(1),f(38),_(1),f(7),_(1),f(16),_(1),f(10),_(1),f(3),_(1),f(3),_(2),f(11),_(1),f(26),_(1),f(39),_(1),f(5),_(1),f(15),_(1),f(90),_(1),f(80),_(1),f(87),_(1),f(67),_(1),f(91),_(1),f(9),_(1),f(35),E(1),f(166),_(1),f(78),_(1),f(152),_(1),f(57)][100.0%][r=18.7GiB/s,w=0KiB/s][r=4885k,w=0
IOPS][eta 00m:00s]
randread-individual-nvmes: (groupid=0, jobs=2800): err= 0: pid=8021: Mon
Jan 23 20:51:43 2017
read: IOPS=6657k, BW=25.5GiB/s (27.3GB/s)(762GiB/30012msec)
clat (usec): min=31, max=35890, avg=403.07, stdev=587.78
clat percentiles (usec):
| 1.00th=[ 112], 5.00th=[ 131], 10.00th=[ 145], 20.00th=[ 167],
| 30.00th=[ 187], 40.00th=[ 211], 50.00th=[ 237], 60.00th=[ 270],
| 70.00th=[ 318], 80.00th=[ 406], 90.00th=[ 676], 95.00th=[ 1336],
| 99.00th=[ 3280], 99.50th=[ 4016], 99.90th=[ 5536], 99.95th=[ 6304],
| 99.99th=[ 9536]
lat (usec) : 50=0.01%, 100=0.18%, 250=54.00%, 500=31.18%, 750=5.73%
lat (usec) : 1000=2.24%
lat (msec) : 2=3.63%, 4=2.52%, 10=0.50%, 20=0.01%, 50=0.01%
cpu : usr=0.56%, sys=4.43%, ctx=200588483, majf=0, minf=2797
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=199803621,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=25.5GiB/s (27.3GB/s), 25.5GiB/s-25.5GiB/s
(27.3GB/s-27.3GB/s), io=762GiB (818GB), run=30012-30012msec
Disk stats (read/write):
nvme0n1: ios=12474932/0, merge=0/0, ticks=3440096/0,
in_queue=3545768, util=97.54%
nvme1n1: ios=12488816/0, merge=0/0, ticks=6811092/0,
in_queue=7420304, util=97.96%
nvme2n1: ios=12488737/0, merge=0/0, ticks=4947416/0,
in_queue=5379024, util=97.12%
nvme3n1: ios=12488626/0, merge=0/0, ticks=4578888/0,
in_queue=4696164, util=96.85%
nvme4n1: ios=12488514/0, merge=0/0, ticks=3848360/0,
in_queue=4189952, util=97.85%
nvme5n1: ios=12488384/0, merge=0/0, ticks=2872728/0,
in_queue=2946696, util=96.89%
nvme6n1: ios=12488271/0, merge=0/0, ticks=2480536/0,
in_queue=2544704, util=96.92%
nvme7n1: ios=12488165/0, merge=0/0, ticks=4038500/0,
in_queue=4154768, util=96.91%
nvme8n1: ios=12488052/0, merge=0/0, ticks=4553428/0,
in_queue=4675568, util=97.22%
nvme9n1: ios=12487937/0, merge=0/0, ticks=5487888/0,
in_queue=5956252, util=97.72%
nvme10n1: ios=12486833/0, merge=0/0, ticks=6234216/0,
in_queue=6402356, util=97.54%
nvme11n1: ios=12486699/0, merge=0/0, ticks=4646856/0,
in_queue=5042628, util=97.76%
nvme12n1: ios=12486586/0, merge=0/0, ticks=5331000/0,
in_queue=5478728, util=97.59%
nvme13n1: ios=12486467/0, merge=0/0, ticks=3464404/0,
in_queue=3715416, util=98.27%
nvme14n1: ios=12486358/0, merge=0/0, ticks=2576312/0,
in_queue=2641952, util=97.49%
nvme15n1: ios=12486251/0, merge=0/0, ticks=4135908/0,
in_queue=4270008, util=97.69%
oberstet@svr-psql19:~/scm/parcit/RA/adr/system/docs$ sudo
/opt/fio/bin/fio best_sync_md_over_nvmes.fio
randread-md-over-nvmes: (g=0): rw=randread,
bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=sync, iodepth=1
...
fio-2.17-17-g9cf1
Starting 2800 threads
Jobs: 2800 (f=2800): [r(2800)][100.0%][r=5764MiB/s,w=0KiB/s][r=1476k,w=0
IOPS][eta 00m:00s]
randread-md-over-nvmes: (groupid=0, jobs=2800): err= 0: pid=11137: Mon
Jan 23 20:52:30 2017
read: IOPS=1467k, BW=5732MiB/s (6011MB/s)(169GiB/30116msec)
clat (usec): min=27, max=33113, avg=124.27, stdev=112.85
clat percentiles (usec):
| 1.00th=[ 77], 5.00th=[ 84], 10.00th=[ 86], 20.00th=[ 88],
| 30.00th=[ 93], 40.00th=[ 101], 50.00th=[ 104], 60.00th=[ 107],
| 70.00th=[ 115], 80.00th=[ 133], 90.00th=[ 177], 95.00th=[ 227],
| 99.00th=[ 370], 99.50th=[ 506], 99.90th=[ 2096], 99.95th=[ 2544],
| 99.99th=[ 2960]
lat (usec) : 50=0.04%, 100=36.72%, 250=60.00%, 500=2.73%, 750=0.22%
lat (usec) : 1000=0.07%
lat (msec) : 2=0.12%, 4=0.11%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=0.07%, sys=4.13%, ctx=46545978, majf=0, minf=2797
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=44193488,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=5732MiB/s (6011MB/s), 5732MiB/s-5732MiB/s
(6011MB/s-6011MB/s), io=169GiB (181GB), run=30116-30116msec
Disk stats (read/write):
md1: ios=44010950/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=2762093/0, aggrmerge=0/0, aggrticks=280663/0,
aggrin_queue=284837, aggrutil=99.12%
nvme15n1: ios=2766734/0, merge=0/0, ticks=264808/0, in_queue=267732,
util=98.68%
nvme6n1: ios=2761142/0, merge=0/0, ticks=288704/0, in_queue=291288,
util=98.76%
nvme9n1: ios=2759118/0, merge=0/0, ticks=275752/0, in_queue=282288,
util=98.95%
nvme11n1: ios=2762423/0, merge=0/0, ticks=264996/0, in_queue=271464,
util=98.91%
nvme2n1: ios=2764361/0, merge=0/0, ticks=281520/0, in_queue=288924,
util=99.12%
nvme14n1: ios=2760515/0, merge=0/0, ticks=264796/0, in_queue=266752,
util=98.61%
nvme5n1: ios=2761756/0, merge=0/0, ticks=280020/0, in_queue=282840,
util=98.92%
nvme8n1: ios=2763138/0, merge=0/0, ticks=279332/0, in_queue=280624,
util=98.53%
nvme10n1: ios=2764117/0, merge=0/0, ticks=291264/0, in_queue=293444,
util=98.67%
nvme1n1: ios=2761579/0, merge=0/0, ticks=275872/0, in_queue=282080,
util=98.90%
nvme13n1: ios=2759948/0, merge=0/0, ticks=280080/0, in_queue=286324,
util=99.05%
nvme4n1: ios=2763271/0, merge=0/0, ticks=279592/0, in_queue=287944,
util=98.96%
nvme7n1: ios=2759669/0, merge=0/0, ticks=280708/0, in_queue=284056,
util=98.88%
nvme0n1: ios=2761263/0, merge=0/0, ticks=296868/0, in_queue=300408,
util=98.78%
nvme12n1: ios=2763077/0, merge=0/0, ticks=288264/0, in_queue=290264,
util=98.71%
nvme3n1: ios=2761377/0, merge=0/0, ticks=298040/0, in_queue=300960,
util=98.74%
oberstet@svr-psql19:~/scm/parcit/RA/adr/system/docs$
=================
Changing engine to psync, leaving everything else unchanged:
oberstet@svr-psql19:~/scm/parcit/RA/adr/system/docs$ sudo
/opt/fio/bin/fio best_sync_individual_nvmes.fio
randread-individual-nvmes: (g=0): rw=randread,
bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=psync, iodepth=1
...
fio-2.17-17-g9cf1
Starting 2800 threads
Jobs: 2771 (f=40464):
[f(8),_(1),f(14),_(1),f(30),_(1),f(6),_(1),f(4),_(1),f(7),_(1),f(14),_(1),f(6),_(1),f(62),_(1),f(3),_(1),f(167),_(1),f(309),_(1),f(269),_(1),f(47),_(1),f(206),_(1),f(26),_(1),f(56),_(2),f(4),_(1),f(39),_(1),f(148),_(1),f(148),_(1),f(4),_(1),f(63),_(1),f(27),_(1),f(19),_(1),f(314),_(1),f(189),_(1),f(205),_(1),f(377)][100.0%][r=25.7GiB/s,w=0KiB/s][r=6726k,w=0
IOPS][eta 00m:00s]
randread-individual-nvmes: (groupid=0, jobs=2800): err= 0: pid=14753:
Mon Jan 23 20:58:45 2017
read: IOPS=7086k, BW=27.4GiB/s (29.3GB/s)(811GiB/30010msec)
clat (usec): min=34, max=57916, avg=381.14, stdev=524.36
clat percentiles (usec):
| 1.00th=[ 121], 5.00th=[ 145], 10.00th=[ 159], 20.00th=[ 185],
| 30.00th=[ 207], 40.00th=[ 229], 50.00th=[ 255], 60.00th=[ 286],
| 70.00th=[ 326], 80.00th=[ 394], 90.00th=[ 564], 95.00th=[ 988],
| 99.00th=[ 2928], 99.50th=[ 3632], 99.90th=[ 5344], 99.95th=[ 6688],
| 99.99th=[11200]
lat (usec) : 50=0.01%, 100=0.08%, 250=48.03%, 500=39.59%, 750=5.69%
lat (usec) : 1000=1.66%
lat (msec) : 2=2.69%, 4=1.91%, 10=0.32%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=0.60%, sys=4.43%, ctx=214720330, majf=0, minf=2797
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=212658246,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=27.4GiB/s (29.3GB/s), 27.4GiB/s-27.4GiB/s
(29.3GB/s-29.3GB/s), io=811GiB (871GB), run=30010-30010msec
Disk stats (read/write):
nvme0n1: ios=13204662/0, merge=0/0, ticks=5579056/0,
in_queue=5713604, util=97.16%
nvme1n1: ios=13292212/0, merge=0/0, ticks=3336164/0,
in_queue=3661216, util=97.52%
nvme2n1: ios=13292063/0, merge=0/0, ticks=3097888/0,
in_queue=3359552, util=97.09%
nvme3n1: ios=13291900/0, merge=0/0, ticks=2973176/0,
in_queue=3072764, util=96.31%
nvme4n1: ios=13291734/0, merge=0/0, ticks=4962684/0,
in_queue=5434620, util=97.02%
nvme5n1: ios=13291540/0, merge=0/0, ticks=7857284/0,
in_queue=8108332, util=96.75%
nvme6n1: ios=13291403/0, merge=0/0, ticks=3160292/0,
in_queue=3249508, util=96.46%
nvme7n1: ios=13291270/0, merge=0/0, ticks=5593256/0,
in_queue=5748080, util=96.42%
nvme8n1: ios=13291057/0, merge=0/0, ticks=3345216/0,
in_queue=3450892, util=96.81%
nvme9n1: ios=13290897/0, merge=0/0, ticks=3102344/0,
in_queue=3394168, util=97.38%
nvme10n1: ios=13290753/0, merge=0/0, ticks=3050116/0,
in_queue=3129208, util=96.74%
nvme11n1: ios=13290570/0, merge=0/0, ticks=6353996/0,
in_queue=6956272, util=97.59%
nvme12n1: ios=13290405/0, merge=0/0, ticks=3268144/0,
in_queue=3372100, util=97.04%
nvme13n1: ios=13290255/0, merge=0/0, ticks=3037220/0,
in_queue=3297944, util=97.78%
nvme14n1: ios=13290110/0, merge=0/0, ticks=8279264/0,
in_queue=8503324, util=97.47%
nvme15n1: ios=13289722/0, merge=0/0, ticks=3361284/0,
in_queue=3467660, util=97.22%
oberstet@svr-psql19:~/scm/parcit/RA/adr/system/docs$ sudo
/opt/fio/bin/fio best_sync_md_over_nvmes.fio
randread-md-over-nvmes: (g=0): rw=randread,
bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=psync, iodepth=1
...
fio-2.17-17-g9cf1
Starting 2800 threads
Jobs: 2367 (f=2342):
[_(1),r(2),_(1),r(38),_(10),r(1),_(1),r(2),_(2),r(2),_(11),r(1),_(1),r(5),_(1),E(1),r(2),f(2),E(1),r(1),f(3),r(19),f(1),r(87),_(1),r(234),_(1),r(13),_(1),r(29),f(1),_(1),r(17),E(1),r(9),E(1),r(9),E(1),r(3),E(1),r(6),_(1),r(16),E(1),r(2),_(1),r(8),E(1),r(30),_(1),r(15),E(1),r(11),f(1),r(27),f(1),r(11),E(1),r(13),_(1),r(27),E(1),r(31),E(1),r(32),E(1),r(6),_(1),r(26),E(1),r(18),E(1),r(5),_(1),E(1),r(16),f(1),r(1),f(1),r(3),f(3),r(3),f(2),r(1),f(3),r(1),f(1),r(1),f(1),r(1),f(4),r(3),f(5),r(1),f(12),E(1),r(2),f(3),r(2),f(1),_(1),f(8),r(1),f(9),r(1),f(1),r(1),f(2),r(1),f(4),r(1),f(7),r(2),f(5),r(1),f(2),r(1),f(2),r(1),f(2),_(1),f(1),r(1),f(2),r(1),f(2),r(1),f(5),r(1),f(1),r(2),f(1),r(4),f(1),r(1),f(5),r(1),f(1),r(2),f(1),r(1),E(1),r(1),f(3),r(2),f(5),r(1),f(1),r(2),f(1),r(1),f(1),r(2),_(1),f(9),E(1),f(3),_(2),f(11),_(1),f(3),_(1),f(4),_(2),f(1),_(1),f(7),_(1),f(3),_(2),f(7),_(1),f(4),_(1),f(4),_(1),f(5),_(1),f(3),_(1),f(12),_(1),f(12),_(1),f(4),_(1),f(2),_(1),f(7),_(1),f(1),_(1),f(15),_(2),f(1),_(1),f(2),_(1),f(10),_(1),f(2),_(1),f(12),_(1),f(10),_(1),f(5),_(1),f(6),_(2),f(6),_(1),f(2),_(1),f(13),_(1),f(6),_(1),f(21),_(1),f(2),_(1),f(1),_(2),f(1),_(1),f(26),_(1),f(1),_(1),f(1),E(1),f(6),_(1),f(3),_(1),f(2),_(1),f(2),_(1),f(3),_(1),f(10),_(1),f(8),_(1),f(11),_(1),f(7),_(1),f(2),_(1),f(4),_(1),f(5),_(1),f(4),_(1),f(8),_(1),f(6),_(1),f(5),_(1),f(9),_(2),f(3),_(1),f(1),_(1),f(13),_(1),f(3),_(1),f(2),_(1),f(1),_(1),f(5),_(1),f(14),_(1),f(4),_(1),f(5),_(1),f(12),_(1),f(1),_(2),f(1),_(1),f(3),_(1),f(2),_(3),f(2),_(1),f(3),_(1),f(5),_(1),f(7),_(3),f(19),_(1),f(4),_(1),f(6),_(1),f(9),_(1),f(9),_(2),f(2),_(2),f(22),_(1),f(69),_(1),f(17),_(1),f(26),_(1),f(1),_(1),f(5),_(1),f(3),_(1),f(9),_(1),f(19),_(1),f(11),_(2),f(7),_(1),f(21),_(1),f(3),_(1),f(6),_(1),f(10),_(1),f(2),_(1),f(26),_(1),f(7),_(1),f(1),_(2),f(2),_(1),f(8),_(1),f(20),_(1),f(15),_(2),f(2),_(1),f(11),_(1),f(8),_(1),f(14),_(1),f(10),_(1),f(6),_(1),f(2),_(1),f(25),_(1),f(2),_(1),f(1),_(1),f(4),_(1),f(42),_(1),f(5),_(2),f(14),_(2),f(2),_(2),f(7),_(1),f(2),_(1),f(2),_(2),
f(12),_(1),f(15),_(1),f(2),_(1),f(1),_(1),f(2),_(1),f(4),_(1),f(6),_(1),f(8),_(4),f(2),_(3),f(4),_(1),f(1),_(1),f(1),_(1),f(4),_(1),f(18),_(2),f(1),_(1),f(1),_(2),f(11),_(1),f(20),_(1),f(7),_(1),f(4),_(1),f(6),_(1),f(4),_(1),f(11),_(2),f(3),_(1),f(1),_(1),f(1),_(1),f(8),_(1),f(2),_(1),f(2),_(1),f(4),_(2),f(3),_(1),f(4),_(1),E(1),_(1),f(1),_(1),f(1),_(1),E(1),_(3),f(2),_(5),f(1),_(1),E(1),f(1),_(1),f(2),_(1),f(5),_(2),f(2),_(1),E(1),f(2),_(1),f(3),E(1),f(1),_(2),f(10),_(1),f(1),_(4),f(1),_(1),f(2),_(2),f(3),_(1),f(2),_(3),f(1),_(3),f(1),_(2),f(2),E(1),f(2),_(1),f(1),_(3),f(1),_(1),f(2),E(1),f(9),_(1),f(1),E(1),f(1),_(1),f(1),_(1),f(1),E(1),f(1),E(1),_(1),f(3),E(1),f(1),_(2),f(1),_(1),E(1),f(1),_(2),f(3),_(1),f(1),_(1),f(3),_(1),f(2),_(2),f(2),_(1),f(2),_(3),f(2),_(2),f(8),_(1),f(1),_(2),f(1),_(1),f(3),_(2),f(1),_(1),f(1),_(1),f(1),_(1),f(1),_(1),f(1),_(1),f(3),_(1),f(5),_(2),f(6),_(2),f(1),_(1),f(9),_(1),f(3),_(1),f(7),_(1),f(1),_(2),f(1),_(1),f(2),_(1),f(5),_(2),f(4),_(1),f(1),_(2),f(3),_(3),f(12),_(1),f(2),_(3),f(3),_(1),f(3),_(1),f(1),_(2),f(3),_(1),f(2),_(1),f(3),_(1),f(3),_(2),f(1),_(1),f(2),_(2),f(9),E(1),f(1),E(1),f(5),_(1),E(1),f(7),_(1),f(1),_(1),f(4),_(2),f(2),_(1),f(3),_(3),f(14),_(1),f(10),_(1),f(1),_(1),f(1),_(1),E(1),f(2),E(1),f(1),_(1),f(1),_(3),f(6),_(1),f(4),E(1),f(4),_(4),f(3),_(1),f(1),_(3),f(1),_(1),f(1),E(1),f(2),_(1),f(2),_(1),f(2),_(1),f(1),E(1),_(1),E(1),f(1),_(2),f(1),_(1),f(2),_(1),f(2),_(9),f(1),_(3),f(3),_(1),f(1),_(1),f(3),_(2),f(3),_(2),f(2),_(1),f(2),_(1),f(1),_(2),f(1),_(2),f(2)][0.5%][r=15.2GiB/s,w=0KiB/s][r=3960k,w=0
IOPS][eta 01h:38m:47s]
randread-md-over-nvmes: (groupid=0, jobs=2800): err= 0: pid=17756: Mon
Jan 23 20:59:22 2017
read: IOPS=4154k, BW=15.9GiB/s (17.2GB/s)(476GiB/30015msec)
clat (usec): min=38, max=264790, avg=669.08, stdev=954.35
clat percentiles (usec):
| 1.00th=[ 149], 5.00th=[ 207], 10.00th=[ 262], 20.00th=[ 342],
| 30.00th=[ 410], 40.00th=[ 470], 50.00th=[ 532], 60.00th=[ 604],
| 70.00th=[ 684], 80.00th=[ 788], 90.00th=[ 956], 95.00th=[ 1160],
| 99.00th=[ 4512], 99.50th=[ 7392], 99.90th=[12480], 99.95th=[14400],
| 99.99th=[19072]
lat (usec) : 50=0.01%, 100=0.04%, 250=8.86%, 500=35.57%, 750=32.34%
lat (usec) : 1000=14.64%
lat (msec) : 2=6.53%, 4=0.91%, 10=0.89%, 20=0.22%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=0.46%, sys=5.81%, ctx=124737165, majf=0, minf=2797
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=124675330,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=15.9GiB/s (17.2GB/s), 15.9GiB/s-15.9GiB/s
(17.2GB/s-17.2GB/s), io=476GiB (511GB), run=30015-30015msec
Disk stats (read/write):
md1: ios=124675330/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=7792208/0, aggrmerge=0/0, aggrticks=1051705/0,
aggrin_queue=1120720, aggrutil=100.00%
nvme15n1: ios=7790429/0, merge=0/0, ticks=1048276/0,
in_queue=1090348, util=100.00%
nvme6n1: ios=7792474/0, merge=0/0, ticks=999284/0, in_queue=1035092,
util=100.00%
nvme9n1: ios=7792704/0, merge=0/0, ticks=1033208/0, in_queue=1151824,
util=100.00%
nvme11n1: ios=7792344/0, merge=0/0, ticks=1103896/0,
in_queue=1231748, util=100.00%
nvme2n1: ios=7791972/0, merge=0/0, ticks=1001928/0, in_queue=1121472,
util=100.00%
nvme14n1: ios=7795323/0, merge=0/0, ticks=1154676/0,
in_queue=1190940, util=100.00%
nvme5n1: ios=7784969/0, merge=0/0, ticks=1048052/0, in_queue=1081964,
util=100.00%
nvme8n1: ios=7792042/0, merge=0/0, ticks=1080976/0, in_queue=1112776,
util=100.00%
nvme10n1: ios=7786642/0, merge=0/0, ticks=1018484/0,
in_queue=1054712, util=100.00%
nvme1n1: ios=7793892/0, merge=0/0, ticks=1072588/0, in_queue=1194612,
util=100.00%
nvme13n1: ios=7792651/0, merge=0/0, ticks=1040368/0,
in_queue=1157356, util=100.00%
nvme4n1: ios=7794567/0, merge=0/0, ticks=1065096/0, in_queue=1198308,
util=100.00%
nvme7n1: ios=7794169/0, merge=0/0, ticks=1061900/0, in_queue=1104168,
util=100.00%
nvme0n1: ios=7794534/0, merge=0/0, ticks=1039064/0, in_queue=1071864,
util=100.00%
nvme12n1: ios=7796809/0, merge=0/0, ticks=1044664/0,
in_queue=1081852, util=100.00%
nvme3n1: ios=7789809/0, merge=0/0, ticks=1014828/0, in_queue=1052484,
util=100.00%
oberstet@svr-psql19:~/scm/parcit/RA/adr/system/docs$
��.n��������+%������w��{.n�������^n�r������&��z�ޗ�zf���h���~����������_��+v���)ߣ�
