On 3/18/2019 11:32 PM, Sagi Grimberg wrote:
>
>> I agree that blk is not the most successful name, we were trying to find
>> something that would work for general storage applications. I think
>> rdma_dim would work as it is completion based but then when we want to
>> use it for nvme it will probably require code duplication.
>
> Lets worry about reuse NVMe when it is actually applicable.
>
>>> Also in the internal review Yamin added a table that summarize all the
>>> testing that were done using NVMeoF (I guess it somehow didn't get to
>>> this RFC).
>>>
>>> I guess we can do the same for iSER to get more confidence and then
>>> set both to create modifiable cq (if HCA supports, of course).
>>>
>>> Agreed ?
>>>
>> I think that adding a flag in create_cq will be less clean as it will
>> require more work for anyone writing applications that should not have
>> to consider this feature.
>>
>> Based on the results I saw during testing I would set it to work by
>> default as I could not find a use case where it significantly reduces
>> performance and in many cases it is a large improvement. It should be
>> more of an opt out situation.
>
> By detailed performance results I meant:
> 1. Full latency histogram for QD=1 both for single queue and multi-queue
> (including max, 99.99% and 99.999% percentiles)
> 2. latency vs. IOPs graph/table for both single queue and multi-queue
> 3. At least some measurement/analysis of how well the algorithm is
> adapting to workload change dynamically and how quickly.
> 4. Test also with real NVMe devices.
>
> Also, we need to separate the host side moderation and the target
> side moderation to understand if/how they effect each other.
>
> Its very easy to show that for high stress workloads you can get an
> improvement as obviously there is a clear win for interrupt moderation,
> however, if this was the only metric that is interesting, we wouldn't
> need it to be adaptive.
>
> As I said before, this adds entropy to the equation which in certain use
> cases can make more harm than good, and we need to quantify where is the
> impact and understand how important they are compared to the extremely
> niche use-case of a single host pushing 2M-8M IOPs.
I ran extensive tests between two hosts using FIO and NVMeoF with and
without DIM on each side separately and together. I am attaching the
results of the tests. The main point I would like to make is that when
used on both sides especially we see improvements across the board in
both IOPS and latency.
>
>> Performance improvement (ConnectX-5 100GbE, x86) running FIO
>> benchmark over
>> NVMf between two equal end-hosts with 56 cores across a
>> Mellanox switch
>> using null_blk device:
>>
>> IO READS before:
>> blk size | BW | IOPS | 99th percentile latency
>> 512B | 3.2GiB | 6.6M | 1549 usec
>> 4k | 7.2GiB | 1.8M | 7177 usec
>> 64k | 10.7GiB | 176k | 82314 usec
>
> I've seen this before, why are we not getting 100Gb/s for 4k with CX5?
> I recall we used to get it with CX4.
It depends on the host server and whether or not it causes backpressure
on the PCI because of the amount of interrupts.
>
>> IO READS after:
>> blk size | BW | IOPS | 99th percentile latency
>> 512B | 4.2GiB | 8.6M | 1729 usec
>> 4k | 10.5GiB | 2.7M | 5669 usec
>> 64k | 10.7GiB | 176k | 102000 usec
>>
>> IO WRITES before:
>> blk size | BW | IOPS | 99th percentile latency
>> 512B | 3GiB | 6.2M | 2573 usec
>> 4k | 7.2GiB | 1.8M | 5342 usec
>> 64k | 10.7GiB | 176k | 62129 usec
>>
>> IO WRITES after:
>> blk size | BW | IOPS | 99th percentile latency
>> 512B | 4.2GiB | 8.6M | 938 usec
>> 4k | 10.2GiB | 2.68M | 2769 usec
>> 64k | 10.6GiB | 173k | 87557 usec
>
> The fact that the 64k 99% latency is substantially higher (20+
> milliseconds) without any BW benefit, while its not a very interesting
> measurement, gives me an indication that a more detailed analysis needs
> to be made here to understand where are the trade-offs.
>
>> It doesn't really make a difference to me how the option is implemented
>> but I think it makes more sense to have it dealt with by us such as in a
>> module parameter and not something like a flag that has a larger radius
>> of effect.
>
> I was suggesting a sysctl global parameter for global behavior and of
> someone wants to override it it can add a CQ flag (which follows the
> common net params exactly).
With DIM Without DIM
Type=read,Size=512B,NumJobs=1,depth=1 Type=read,Size=512B,NumJobs=1,depth=1
read: IOPS=56.7k, BW=27.7MiB/s (29.0MB/s)(277MiB/10001msec | read: IOPS=47.0k, BW=22.0MiB/s (24.1MB/s)(230MiB/10001msec
slat (nsec): min=2087, max=43251, avg=2462.19, stdev=1301 | slat (nsec): min=1981, max=53196, avg=3048.29, stdev=2391
clat (nsec): min=623, max=108655, avg=14747.79, stdev=468 | clat (nsec): min=442, max=107156, avg=17658.37, stdev=647
lat (usec): min=12, max=110, avg=17.25, stdev= 5.05 | lat (usec): min=13, max=109, avg=20.78, stdev= 7.18
| 1.00th=[12992], 5.00th=[12992], 10.00th=[13120], 20. | | 1.00th=[13632], 5.00th=[13760], 10.00th=[13888], 20.
| 30.00th=[13120], 40.00th=[13248], 50.00th=[13248], 60. | | 30.00th=[14144], 40.00th=[14144], 50.00th=[14400], 60.
| 70.00th=[13504], 80.00th=[13888], 90.00th=[16768], 95. | | 70.00th=[17024], 80.00th=[19072], 90.00th=[27776], 95.
| 99.00th=[35584], 99.50th=[36608], 99.90th=[42240], 99. | | 99.00th=[40704], 99.50th=[45824], 99.90th=[51456], 99.
| 99.99th=[53504] | | 99.99th=[58624]
Type=read,Size=512B,NumJobs=1,depth=16 Type=read,Size=512B,NumJobs=1,depth=16
read: IOPS=299k, BW=146MiB/s (153MB/s)(1459MiB/10001msec) | read: IOPS=334k, BW=163MiB/s (171MB/s)(1629MiB/10001msec)
slat (nsec): min=1634, max=45257, avg=1958.38, stdev=657. | slat (nsec): min=1564, max=28780, avg=2162.48, stdev=588.
clat (usec): min=38, max=171, avg=51.28, stdev=13.39 | clat (usec): min=17, max=118, avg=45.50, stdev= 6.30
lat (usec): min=40, max=172, avg=53.27, stdev=13.41 | lat (usec): min=20, max=120, avg=47.70, stdev= 6.64
| 1.00th=[ 40], 5.00th=[ 41], 10.00th=[ 41], 20. | | 1.00th=[41728], 5.00th=[42752], 10.00th=[42752], 20.
| 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 46], 60. | | 30.00th=[43264], 40.00th=[43776], 50.00th=[43776], 60.
| 70.00th=[ 57], 80.00th=[ 62], 90.00th=[ 70], 95. | | 70.00th=[44288], 80.00th=[44288], 90.00th=[46336], 95.
| 99.00th=[ 97], 99.50th=[ 109], 99.90th=[ 127], 99. | | 99.00th=[72192], 99.50th=[73216], 99.90th=[77312], 99.
| 99.99th=[ 149] | | 99.99th=[86528]
Type=read,Size=512B,NumJobs=1,depth=64 Type=read,Size=512B,NumJobs=1,depth=64
read: IOPS=370k, BW=181MiB/s (189MB/s)(1806MiB/10001msec) | read: IOPS=344k, BW=168MiB/s (176MB/s)(1682MiB/10001msec)
slat (nsec): min=1601, max=43074, avg=1927.33, stdev=232. | slat (nsec): min=1607, max=30650, avg=2093.56, stdev=233.
clat (usec): min=56, max=378, avg=170.80, stdev= 4.52 | clat (usec): min=19, max=238, avg=183.42, stdev= 2.22
lat (usec): min=57, max=380, avg=172.77, stdev= 4.55 | lat (usec): min=21, max=241, avg=185.55, stdev= 2.23
| 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20. | | 1.00th=[ 180], 5.00th=[ 182], 10.00th=[ 182], 20.
| 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60. | | 30.00th=[ 182], 40.00th=[ 184], 50.00th=[ 184], 60.
| 70.00th=[ 172], 80.00th=[ 172], 90.00th=[ 176], 95. | | 70.00th=[ 184], 80.00th=[ 186], 90.00th=[ 186], 95.
| 99.00th=[ 184], 99.50th=[ 194], 99.90th=[ 221], 99. | | 99.00th=[ 190], 99.50th=[ 192], 99.90th=[ 196], 99.
| 99.99th=[ 245] | | 99.99th=[ 210]
Type=read,Size=512B,NumJobs=16,depth=1 Type=read,Size=512B,NumJobs=16,depth=1
read: IOPS=917k, BW=448MiB/s (469MB/s)(4478MiB/10002msec) | read: IOPS=907k, BW=443MiB/s (465MB/s)(4431MiB/10001msec)
slat (nsec): min=2003, max=73317, avg=2440.64, stdev=588. | slat (nsec): min=1925, max=69082, avg=2505.79, stdev=869.
clat (nsec): min=597, max=172124, avg=14561.65, stdev=210 | clat (nsec): min=476, max=112521, avg=14672.56, stdev=182
lat (usec): min=11, max=174, avg=17.05, stdev= 2.20 | lat (usec): min=12, max=131, avg=17.23, stdev= 2.08
| 1.00th=[12096], 5.00th=[13120], 10.00th=[13376], 20. | | 1.00th=[11584], 5.00th=[13248], 10.00th=[13376], 20.
| 30.00th=[13888], 40.00th=[14016], 50.00th=[14144], 60. | | 30.00th=[13888], 40.00th=[14016], 50.00th=[14272], 60.
| 70.00th=[14528], 80.00th=[14912], 90.00th=[15680], 95. | | 70.00th=[14784], 80.00th=[15296], 90.00th=[16512], 95.
| 99.00th=[27008], 99.50th=[28288], 99.90th=[33536], 99. | | 99.00th=[22400], 99.50th=[24448], 99.90th=[32384], 99.
| 99.99th=[35584] | 99.99th=[35584]
Type=read,Size=512B,NumJobs=16,depth=16 Type=read,Size=512B,NumJobs=16,depth=16
read: IOPS=4627k, BW=2259MiB/s (2369MB/s)(22.1GiB/10002mse | read: IOPS=4055k, BW=1980MiB/s (2076MB/s)(19.3GiB/10003mse
slat (nsec): min=1620, max=95729, avg=2408.03, stdev=483. | slat (nsec): min=1572, max=239521, avg=2836.25, stdev=282
clat (usec): min=22, max=466, avg=52.52, stdev= 8.65 | clat (usec): min=9, max=616, avg=59.85, stdev=55.40
lat (usec): min=24, max=474, avg=54.98, stdev= 8.95 | lat (usec): min=15, max=630, avg=62.74, stdev=57.98
| 1.00th=[ 42], 5.00th=[ 43], 10.00th=[ 44], 20. | | 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 44], 20.
| 30.00th=[ 48], 40.00th=[ 52], 50.00th=[ 53], 60. | | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 46], 60.
| 70.00th=[ 57], 80.00th=[ 59], 90.00th=[ 61], 95. | | 70.00th=[ 48], 80.00th=[ 52], 90.00th=[ 60], 95.
| 99.00th=[ 73], 99.50th=[ 79], 99.90th=[ 139], 99. | | 99.00th=[ 326], 99.50th=[ 343], 99.90th=[ 367], 99.
| 99.99th=[ 281] | | 99.99th=[ 519]
Type=read,Size=512B,NumJobs=16,depth=64 Type=read,Size=512B,NumJobs=16,depth=64
read: IOPS=4878k, BW=2382MiB/s (2497MB/s)(23.3GiB/10001mse | read: IOPS=3451k, BW=1685MiB/s (1767MB/s)(16.5GiB/10001mse
slat (nsec): min=1611, max=181900, avg=2357.92, stdev=159 | slat (nsec): min=1569, max=204726, avg=3322.74, stdev=274
clat (usec): min=28, max=934, avg=207.19, stdev=38.57 | clat (usec): min=5, max=1543, avg=292.82, stdev=220.59
lat (usec): min=29, max=944, avg=209.60, stdev=39.02 | lat (usec): min=14, max=1561, avg=296.20, stdev=223.03
| 1.00th=[ 172], 5.00th=[ 174], 10.00th=[ 174], 20. | | 1.00th=[ 176], 5.00th=[ 178], 10.00th=[ 180], 20.
| 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 202], 60. | | 30.00th=[ 188], 40.00th=[ 227], 50.00th=[ 243], 60.
| 70.00th=[ 215], 80.00th=[ 233], 90.00th=[ 251], 95. | | 70.00th=[ 249], 80.00th=[ 285], 90.00th=[ 685], 95.
| 99.00th=[ 314], 99.50th=[ 322], 99.90th=[ 338], 99. | | 99.00th=[ 1270], 99.50th=[ 1319], 99.90th=[ 1385], 99.
| 99.99th=[ 783] | | 99.99th=[ 1450]
Type=read,Size=512B,NumJobs=56,depth=1 Type=read,Size=512B,NumJobs=56,depth=1
read: IOPS=2311k, BW=1128MiB/s (1183MB/s)(11.0GiB/10002mse | read: IOPS=2351k, BW=1148MiB/s (1204MB/s)(11.2GiB/10002mse
slat (usec): min=2, max=860, avg= 3.68, stdev= 2.24 | slat (nsec): min=1744, max=502891, avg=3525.21, stdev=199
clat (nsec): min=604, max=4342.7k, avg=19742.21, stdev=70 | clat (nsec): min=456, max=781278, avg=19563.02, stdev=418
lat (usec): min=11, max=4366, avg=23.53, stdev= 7.52 | lat (usec): min=11, max=785, avg=23.18, stdev= 4.79
| 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20. | | 1.00th=[13120], 5.00th=[14912], 10.00th=[15936], 20.
| 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 20], 60. | | 30.00th=[17792], 40.00th=[18304], 50.00th=[19072], 60.
| 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 24], 95. | | 70.00th=[20608], 80.00th=[21632], 90.00th=[23680], 95.
| 99.00th=[ 35], 99.50th=[ 39], 99.90th=[ 49], 99. | | 99.00th=[31872], 99.50th=[35072], 99.90th=[41728], 99.
| 99.99th=[ 330] | | 99.99th=[56064]
Type=read,Size=512B,NumJobs=56,depth=16 Type=read,Size=512B,NumJobs=56,depth=16
read: IOPS=8152k, BW=3981MiB/s (4174MB/s)(38.9GiB/10002mse | read: IOPS=7234k, BW=3532MiB/s (3704MB/s)(34.5GiB/10002mse
slat (nsec): min=1658, max=2406.5k, avg=5086.35, stdev=72 | slat (nsec): min=1588, max=13073k, avg=5296.03, stdev=602
clat (usec): min=10, max=3565, avg=104.02, stdev=51.20 | clat (usec): min=9, max=13311, avg=117.66, stdev=77.29
lat (usec): min=23, max=3568, avg=109.21, stdev=53.58 | lat (usec): min=18, max=13322, avg=123.07, stdev=80.61
| 1.00th=[ 64], 5.00th=[ 68], 10.00th=[ 69], 20. | | 1.00th=[ 63], 5.00th=[ 67], 10.00th=[ 69], 20.
| 30.00th=[ 73], 40.00th=[ 74], 50.00th=[ 76], 60. | | 30.00th=[ 72], 40.00th=[ 76], 50.00th=[ 81], 60.
| 70.00th=[ 91], 80.00th=[ 163], 90.00th=[ 188], 95. | | 70.00th=[ 108], 80.00th=[ 184], 90.00th=[ 247], 95.
| 99.00th=[ 221], 99.50th=[ 231], 99.90th=[ 359], 99. | | 99.00th=[ 338], 99.50th=[ 383], 99.90th=[ 494], 99.
| 99.99th=[ 799] | | 99.99th=[ 594]
Type=read,Size=512B,NumJobs=56,depth=64 Type=read,Size=512B,NumJobs=56,depth=64
read: IOPS=8218k, BW=4013MiB/s (4208MB/s)(39.2GiB/10002mse | read: IOPS=7769k, BW=3793MiB/s (3978MB/s)(37.1GiB/10002mse
slat (nsec): min=1673, max=1678.1k, avg=5096.07, stdev=74 | slat (nsec): min=1583, max=3785.4k, avg=5319.20, stdev=81
clat (usec): min=34, max=2534, avg=430.13, stdev=202.15 | clat (usec): min=2, max=5212, avg=455.02, stdev=334.71
lat (usec): min=61, max=2582, avg=435.33, stdev=204.56 | lat (usec): min=17, max=5214, avg=460.45, stdev=338.64
| 1.00th=[ 277], 5.00th=[ 285], 10.00th=[ 289], 20. | | 1.00th=[ 265], 5.00th=[ 273], 10.00th=[ 281], 20.
| 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 310], 60. | | 30.00th=[ 293], 40.00th=[ 297], 50.00th=[ 302], 60.
| 70.00th=[ 322], 80.00th=[ 725], 90.00th=[ 758], 95. | | 70.00th=[ 310], 80.00th=[ 437], 90.00th=[ 1074], 95.
| 99.00th=[ 816], 99.50th=[ 824], 99.90th=[ 848], 99. | | 99.00th=[ 1401], 99.50th=[ 1991], 99.90th=[ 2245], 99.
| 99.99th=[ 889] | | 99.99th=[ 2442]
Type=read,Size=4K,NumJobs=1,depth=1 Type=read,Size=4K,NumJobs=1,depth=1
read: IOPS=52.8k, BW=206MiB/s (216MB/s)(2065MiB/10001msec) | read: IOPS=54.4k, BW=213MiB/s (223MB/s)(2126MiB/10001msec)
slat (nsec): min=2019, max=42477, avg=2671.11, stdev=1827 | slat (nsec): min=2044, max=28781, avg=2556.03, stdev=1441
clat (nsec): min=605, max=96758, avg=15743.80, stdev=5503 | clat (nsec): min=621, max=70566, avg=15359.54, stdev=5250
lat (nsec): min=13458, max=99251, avg=18489.01, stdev=61 | lat (nsec): min=13302, max=73823, avg=17966.25, stdev=56
| 1.00th=[13248], 5.00th=[13376], 10.00th=[13376], 20. | | 1.00th=[13120], 5.00th=[13248], 10.00th=[13248], 20.
| 30.00th=[13504], 40.00th=[13632], 50.00th=[13632], 60. | | 30.00th=[13376], 40.00th=[13376], 50.00th=[13376], 60.
| 70.00th=[13888], 80.00th=[14912], 90.00th=[19584], 95. | | 70.00th=[13632], 80.00th=[14144], 90.00th=[19328], 95.
| 99.00th=[36608], 99.50th=[38656], 99.90th=[48384], 99. | | 99.00th=[35584], 99.50th=[36608], 99.90th=[45312], 99.
| 99.99th=[69120] | | 99.99th=[53504]
Type=read,Size=4K,NumJobs=1,depth=16 Type=read,Size=4K,NumJobs=1,depth=16
read: IOPS=306k, BW=1197MiB/s (1255MB/s)(11.7GiB/10001msec | read: IOPS=334k, BW=1306MiB/s (1370MB/s)(12.8GiB/10001msec
slat (nsec): min=1621, max=44989, avg=2060.53, stdev=583. | slat (nsec): min=1696, max=45538, avg=2168.76, stdev=208.
clat (usec): min=36, max=179, avg=49.85, stdev=11.62 | clat (usec): min=18, max=148, avg=45.39, stdev= 1.47
lat (usec): min=38, max=181, avg=51.95, stdev=11.64 | lat (usec): min=20, max=192, avg=47.59, stdev= 1.48
| 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 43], 20. | | 1.00th=[43776], 5.00th=[44288], 10.00th=[44288], 20.
| 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 44], 60. | | 30.00th=[44800], 40.00th=[45312], 50.00th=[45312], 60.
| 70.00th=[ 53], 80.00th=[ 58], 90.00th=[ 64], 95. | | 70.00th=[45312], 80.00th=[45824], 90.00th=[45824], 95.
| 99.00th=[ 95], 99.50th=[ 103], 99.90th=[ 127], 99. | | 99.00th=[49408], 99.50th=[54528], 99.90th=[64256], 99.
| 99.99th=[ 149] | | 99.99th=[70144]
Type=read,Size=4K,NumJobs=1,depth=64 Type=read,Size=4K,NumJobs=1,depth=64
read: IOPS=353k, BW=1381MiB/s (1448MB/s)(13.5GiB/10001msec | read: IOPS=338k, BW=1319MiB/s (1383MB/s)(12.9GiB/10001msec
slat (nsec): min=1636, max=31675, avg=2040.73, stdev=224. | slat (nsec): min=1621, max=52632, avg=2145.29, stdev=209.
clat (usec): min=44, max=244, avg=178.71, stdev= 2.85 | clat (usec): min=36, max=236, avg=187.02, stdev= 2.05
lat (usec): min=46, max=246, avg=180.79, stdev= 2.87 | lat (usec): min=38, max=238, avg=189.20, stdev= 2.06
| 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 178], 20. | | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 186], 20.
| 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 180], 60. | | 30.00th=[ 186], 40.00th=[ 188], 50.00th=[ 188], 60.
| 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 182], 95. | | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 190], 95.
| 99.00th=[ 190], 99.50th=[ 194], 99.90th=[ 202], 99. | | 99.00th=[ 192], 99.50th=[ 196], 99.90th=[ 200], 99.
| 99.99th=[ 210] | | 99.99th=[ 217]
Type=read,Size=4K,NumJobs=16,depth=1 Type=read,Size=4K,NumJobs=16,depth=1
read: IOPS=878k, BW=3430MiB/s (3597MB/s)(33.5GiB/10001msec | read: IOPS=869k, BW=3396MiB/s (3561MB/s)(33.2GiB/10002msec
slat (nsec): min=1947, max=53618, avg=2438.66, stdev=659. | slat (nsec): min=1890, max=70947, avg=2497.58, stdev=813.
clat (nsec): min=593, max=420818, avg=15328.26, stdev=153 | clat (nsec): min=590, max=79209, avg=15456.14, stdev=1968
lat (usec): min=13, max=423, avg=17.82, stdev= 1.67 | lat (usec): min=12, max=110, avg=18.01, stdev= 2.18
| 1.00th=[12736], 5.00th=[14144], 10.00th=[14272], 20. | | 1.00th=[12352], 5.00th=[13888], 10.00th=[14144], 20.
| 30.00th=[14656], 40.00th=[14912], 50.00th=[15040], 60. | | 30.00th=[14528], 40.00th=[14784], 50.00th=[15040], 60.
| 70.00th=[15424], 80.00th=[15936], 90.00th=[16768], 95. | | 70.00th=[15552], 80.00th=[16192], 90.00th=[17280], 95.
| 99.00th=[20352], 99.50th=[22912], 99.90th=[27264], 99. | | 99.00th=[23936], 99.50th=[25728], 99.90th=[33536], 99.
| 99.99th=[70144] | | 99.99th=[36096]
Type=read,Size=4K,NumJobs=16,depth=16 Type=read,Size=4K,NumJobs=16,depth=16
read: IOPS=2677k, BW=10.2GiB/s (10.0GB/s)(102GiB/10001msec | read: IOPS=1843k, BW=7201MiB/s (7551MB/s)(70.3GiB/10002mse
slat (nsec): min=1634, max=169095, avg=2282.01, stdev=978 | slat (nsec): min=1582, max=104435, avg=3170.92, stdev=170
clat (usec): min=23, max=416, avg=92.99, stdev=14.19 | clat (usec): min=40, max=554, avg=135.18, stdev=32.57
lat (usec): min=29, max=419, avg=95.32, stdev=14.22 | lat (usec): min=42, max=557, avg=138.42, stdev=32.59
| 1.00th=[ 75], 5.00th=[ 78], 10.00th=[ 80], 20. | | 1.00th=[ 68], 5.00th=[ 92], 10.00th=[ 104], 20.
| 30.00th=[ 87], 40.00th=[ 90], 50.00th=[ 94], 60. | | 30.00th=[ 125], 40.00th=[ 130], 50.00th=[ 135], 60.
| 70.00th=[ 98], 80.00th=[ 99], 90.00th=[ 101], 95. | | 70.00th=[ 141], 80.00th=[ 147], 90.00th=[ 161], 95.
| 99.00th=[ 169], 99.50th=[ 174], 99.90th=[ 192], 99. | | 99.00th=[ 262], 99.50th=[ 306], 99.90th=[ 392], 99.
| 99.99th=[ 273] | | 99.99th=[ 461]
Type=read,Size=4K,NumJobs=16,depth=64 Type=read,Size=4K,NumJobs=16,depth=64
read: IOPS=2673k, BW=10.2GiB/s (10.9GB/s)(102GiB/10001msec | read: IOPS=1917k, BW=7487MiB/s (7850MB/s)(73.1GiB/10001mse
slat (nsec): min=1620, max=148710, avg=2532.78, stdev=332 | slat (nsec): min=1566, max=653412, avg=3347.14, stdev=385
clat (usec): min=19, max=1229, avg=380.20, stdev=82.66 | clat (usec): min=43, max=1445, avg=530.39, stdev=127.84
lat (usec): min=23, max=1232, avg=382.77, stdev=83.63 | lat (usec): min=45, max=1619, avg=533.81, stdev=128.49
| 1.00th=[ 318], 5.00th=[ 330], 10.00th=[ 343], 20. | | 1.00th=[ 188], 5.00th=[ 351], 10.00th=[ 420], 20.
| 30.00th=[ 351], 40.00th=[ 355], 50.00th=[ 363], 60. | | 30.00th=[ 486], 40.00th=[ 502], 50.00th=[ 519], 60.
| 70.00th=[ 371], 80.00th=[ 383], 90.00th=[ 388], 95. | | 70.00th=[ 553], 80.00th=[ 578], 90.00th=[ 644], 95.
| 99.00th=[ 725], 99.50th=[ 742], 99.90th=[ 750], 99. | | 99.00th=[ 1012], 99.50th=[ 1037], 99.90th=[ 1090], 99.
| 99.99th=[ 857] | | 99.99th=[ 1205]
Type=read,Size=4K,NumJobs=56,depth=1 Type=read,Size=4K,NumJobs=56,depth=1
read: IOPS=1606k, BW=6273MiB/s (6578MB/s)(61.3GiB/10002mse | read: IOPS=1571k, BW=6138MiB/s (6436MB/s)(59.9GiB/10001mse
slat (nsec): min=1955, max=745694, avg=2980.95, stdev=139 | slat (nsec): min=1693, max=505053, avg=2771.65, stdev=110
clat (nsec): min=633, max=1012.2k, avg=31216.71, stdev=89 | clat (nsec): min=479, max=1348.1k, avg=32311.24, stdev=36
lat (usec): min=13, max=1015, avg=34.28, stdev= 9.05 | lat (usec): min=15, max=1350, avg=35.15, stdev= 3.81
| 1.00th=[ 22], 5.00th=[ 27], 10.00th=[ 29], 20. | | 1.00th=[26752], 5.00th=[28800], 10.00th=[29568], 20.
| 30.00th=[ 30], 40.00th=[ 31], 50.00th=[ 31], 60. | | 30.00th=[30848], 40.00th=[31360], 50.00th=[31872], 60.
| 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 35], 95. | | 70.00th=[33024], 80.00th=[33536], 90.00th=[35584], 95.
| 99.00th=[ 43], 99.50th=[ 46], 99.90th=[ 70], 99. | | 99.00th=[42240], 99.50th=[43776], 99.90th=[49920], 99.
| 99.99th=[ 474] | | 99.99th=[59136]
Type=read,Size=4K,NumJobs=56,depth=16 Type=read,Size=4K,NumJobs=56,depth=16
read: IOPS=2591k, BW=9.88GiB/s (10.6GB/s)(98.9GiB/10002mse | read: IOPS=2009k, BW=7846MiB/s (8227MB/s)(76.6GiB/10001mse
slat (nsec): min=1655, max=626114, avg=3005.44, stdev=179 | slat (nsec): min=1566, max=379585, avg=3204.59, stdev=230
clat (usec): min=25, max=1308, avg=342.16, stdev=55.44 | clat (usec): min=44, max=1761, avg=442.22, stdev=244.27
lat (usec): min=27, max=1311, avg=345.24, stdev=55.45 | lat (usec): min=46, max=1764, avg=445.50, stdev=244.24
| 1.00th=[ 151], 5.00th=[ 192], 10.00th=[ 318], 20. | | 1.00th=[ 161], 5.00th=[ 194], 10.00th=[ 215], 20.
| 30.00th=[ 330], 40.00th=[ 338], 50.00th=[ 343], 60. | | 30.00th=[ 273], 40.00th=[ 306], 50.00th=[ 351], 60.
| 70.00th=[ 359], 80.00th=[ 367], 90.00th=[ 388], 95. | | 70.00th=[ 506], 80.00th=[ 644], 90.00th=[ 832], 95.
| 99.00th=[ 478], 99.50th=[ 506], 99.90th=[ 635], 99. | | 99.00th=[ 1156], 99.50th=[ 1221], 99.90th=[ 1336], 99.
| 99.99th=[ 955] | | 99.99th=[ 1450]
Type=read,Size=4K,NumJobs=56,depth=64 Type=read,Size=4K,NumJobs=56,depth=64
read: IOPS=2652k, BW=10.1GiB/s (10.9GB/s)(101GiB/10003msec | read: IOPS=1801k, BW=7036MiB/s (7378MB/s)(68.7GiB/10002mse
slat (nsec): min=1638, max=1266.7k, avg=14402.06, stdev=5 | slat (nsec): min=1586, max=2410.2k, avg=20136.27, stdev=8
clat (usec): min=23, max=7119, avg=1336.05, stdev=712.32 | clat (usec): min=58, max=8215, avg=1968.75, stdev=947.22
lat (usec): min=30, max=7224, avg=1350.51, stdev=725.25 | lat (usec): min=60, max=8632, avg=1988.96, stdev=962.84
| 1.00th=[ 570], 5.00th=[ 644], 10.00th=[ 652], 20. | | 1.00th=[ 449], 5.00th=[ 734], 10.00th=[ 865], 20.
| 30.00th=[ 685], 40.00th=[ 1221], 50.00th=[ 1303], 60. | | 30.00th=[ 1418], 40.00th=[ 1729], 50.00th=[ 1909], 60.
| 70.00th=[ 1401], 80.00th=[ 1926], 90.00th=[ 2409], 95. | | 70.00th=[ 2212], 80.00th=[ 2507], 90.00th=[ 3326], 95.
| 99.00th=[ 3359], 99.50th=[ 3589], 99.90th=[ 4047], 99. | | 99.00th=[ 4817], 99.50th=[ 5080], 99.90th=[ 5735], 99.
| 99.99th=[ 5080] | | 99.99th=[ 6587]
Type=read,Size=64K,NumJobs=1,depth=1 Type=read,Size=64K,NumJobs=1,depth=1
read: IOPS=29.6k, BW=1849MiB/s (1939MB/s)(18.1GiB/10001mse | read: IOPS=29.8k, BW=1864MiB/s (1955MB/s)(18.2GiB/10001mse
slat (nsec): min=4727, max=45982, avg=6068.77, stdev=2849 | slat (nsec): min=4768, max=95484, avg=6119.74, stdev=2779
clat (usec): min=3, max=110, avg=27.22, stdev= 7.91 | clat (nsec): min=1555, max=75556, avg=26881.73, stdev=781
lat (usec): min=25, max=122, avg=33.34, stdev= 9.17 | lat (usec): min=25, max=102, avg=33.05, stdev= 9.07
| 1.00th=[21888], 5.00th=[21888], 10.00th=[21888], 20. | | 1.00th=[21632], 5.00th=[21632], 10.00th=[21632], 20.
| 30.00th=[22144], 40.00th=[22400], 50.00th=[22400], 60. | | 30.00th=[21888], 40.00th=[22144], 50.00th=[22144], 60.
| 70.00th=[28288], 80.00th=[35072], 90.00th=[41216], 95. | | 70.00th=[28032], 80.00th=[34048], 90.00th=[40704], 95.
| 99.00th=[47872], 99.50th=[50432], 99.90th=[63744], 99. | | 99.00th=[47360], 99.50th=[49920], 99.90th=[59648], 99.
| 99.99th=[78336] | | 99.99th=[70144]
Type=read,Size=64K,NumJobs=1,depth=16 Type=read,Size=64K,NumJobs=1,depth=16
read: IOPS=168k, BW=10.3GiB/s (11.0GB/s)(103GiB/10001msec) | read: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10001msec)
slat (nsec): min=3795, max=48145, avg=4685.87, stdev=473. | slat (nsec): min=3756, max=65673, avg=4930.28, stdev=334.
clat (usec): min=78, max=259, avg=90.18, stdev= 7.63 | clat (usec): min=83, max=243, avg=87.86, stdev= 2.04
lat (usec): min=83, max=300, avg=94.90, stdev= 7.60 | lat (usec): min=87, max=310, avg=92.83, stdev= 2.05
| 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 85], 20. | | 1.00th=[ 86], 5.00th=[ 87], 10.00th=[ 87], 20.
| 30.00th=[ 88], 40.00th=[ 88], 50.00th=[ 89], 60. | | 30.00th=[ 87], 40.00th=[ 88], 50.00th=[ 88], 60.
| 70.00th=[ 92], 80.00th=[ 93], 90.00th=[ 96], 95. | | 70.00th=[ 88], 80.00th=[ 89], 90.00th=[ 91], 95.
| 99.00th=[ 127], 99.50th=[ 137], 99.90th=[ 174], 99. | | 99.00th=[ 96], 99.50th=[ 99], 99.90th=[ 108], 99.
| 99.99th=[ 196] | | 99.99th=[ 114]
Type=read,Size=64K,NumJobs=1,depth=64 Type=read,Size=64K,NumJobs=1,depth=64
read: IOPS=171k, BW=10.4GiB/s (11.2GB/s)(104GiB/10001msec) | read: IOPS=168k, BW=10.2GiB/s (10.0GB/s)(102GiB/10001msec)
slat (usec): min=2, max=384, avg= 3.57, stdev= 1.13 | slat (usec): min=2, max=390, avg= 4.15, stdev= 1.56
clat (usec): min=50, max=983, avg=370.03, stdev=11.50 | clat (usec): min=37, max=947, avg=376.95, stdev=15.70
lat (usec): min=53, max=1059, avg=373.64, stdev=11.40 | lat (usec): min=42, max=988, avg=381.15, stdev=16.25
| 1.00th=[ 351], 5.00th=[ 359], 10.00th=[ 363], 20. | | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 367], 20.
| 30.00th=[ 367], 40.00th=[ 367], 50.00th=[ 371], 60. | 30.00th=[ 367], 40.00th=[ 367], 50.00th=[ 371], 60.
| 70.00th=[ 371], 80.00th=[ 375], 90.00th=[ 379], 95. | | 70.00th=[ 388], 80.00th=[ 392], 90.00th=[ 400], 95.
| 99.00th=[ 400], 99.50th=[ 408], 99.90th=[ 433], 99. | | 99.00th=[ 412], 99.50th=[ 416], 99.90th=[ 429], 99.
| 99.99th=[ 644] | | 99.99th=[ 668]
Type=read,Size=64K,NumJobs=16,depth=1 Type=read,Size=64K,NumJobs=16,depth=1
read: IOPS=171k, BW=10.4GiB/s (11.2GB/s)(104GiB/10001msec) | read: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10002msec)
slat (nsec): min=4077, max=68679, avg=5394.03, stdev=2968 | slat (nsec): min=3480, max=74756, avg=5273.21, stdev=2961
clat (nsec): min=686, max=194443, avg=87548.33, stdev=162 | clat (nsec): min=616, max=453786, avg=87271.90, stdev=161
lat (usec): min=27, max=208, avg=93.00, stdev=16.25 | lat (usec): min=26, max=458, avg=92.60, stdev=16.20
| 1.00th=[ 42], 5.00th=[ 61], 10.00th=[ 71], 20. | | 1.00th=[ 37], 5.00th=[ 57], 10.00th=[ 68], 20.
| 30.00th=[ 81], 40.00th=[ 84], 50.00th=[ 87], 60. | | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 90], 60.
| 70.00th=[ 94], 80.00th=[ 99], 90.00th=[ 111], 95. | | 70.00th=[ 95], 80.00th=[ 98], 90.00th=[ 104], 95.
| 99.00th=[ 127], 99.50th=[ 133], 99.90th=[ 143], 99. | | 99.00th=[ 126], 99.50th=[ 130], 99.90th=[ 143], 99.
| 99.99th=[ 161] | 99.99th=[ 161]
Type=read,Size=64K,NumJobs=16,depth=16 Type=read,Size=64K,NumJobs=16,depth=16
read: IOPS=175k, BW=10.7GiB/s (11.4GB/s)(107GiB/10002msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec)
slat (nsec): min=4066, max=75561, avg=5259.83, stdev=1080 | slat (nsec): min=3866, max=57337, avg=5204.47, stdev=1220
clat (usec): min=118, max=3688, avg=1460.39, stdev=456.30 | clat (usec): min=116, max=3985, avg=1453.77, stdev=349.92
lat (usec): min=132, max=3693, avg=1465.70, stdev=456.22 | lat (usec): min=128, max=3990, avg=1459.03, stdev=349.93
| 1.00th=[ 1172], 5.00th=[ 1172], 10.00th=[ 1172], 20. | | 1.00th=[ 1254], 5.00th=[ 1270], 10.00th=[ 1270], 20.
| 30.00th=[ 1205], 40.00th=[ 1270], 50.00th=[ 1270], 60. | | 30.00th=[ 1352], 40.00th=[ 1352], 50.00th=[ 1352], 60.
| 70.00th=[ 1352], 80.00th=[ 1401], 90.00th=[ 2376], 95. | | 70.00th=[ 1369], 80.00th=[ 1401], 90.00th=[ 1467], 95.
| 99.00th=[ 2606], 99.50th=[ 2737], 99.90th=[ 2769], 99. | | 99.00th=[ 2737], 99.50th=[ 2769], 99.90th=[ 2769], 99.
| 99.99th=[ 2802] | 99.99th=[ 2802]
Type=read,Size=64K,NumJobs=16,depth=64 Type=read,Size=64K,NumJobs=16,depth=64
read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec) read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec)
slat (usec): min=2, max=890, avg= 4.27, stdev= 3.77 | slat (usec): min=2, max=1185, avg= 9.71, stdev=60.22
clat (usec): min=108, max=11645, avg=5850.95, stdev=186.5 | clat (usec): min=116, max=16490, avg=5824.81, stdev=1345.
lat (usec): min=113, max=12034, avg=5855.28, stdev=186.3 | lat (usec): min=133, max=16494, avg=5834.58, stdev=1367.
| 1.00th=[ 5800], 5.00th=[ 5800], 10.00th=[ 5800], 20. | | 1.00th=[ 5407], 5.00th=[ 5407], 10.00th=[ 5473], 20.
| 30.00th=[ 5866], 40.00th=[ 5866], 50.00th=[ 5866], 60. | | 30.00th=[ 5473], 40.00th=[ 5473], 50.00th=[ 5473], 60.
| 70.00th=[ 5866], 80.00th=[ 5866], 90.00th=[ 5866], 95. | | 70.00th=[ 5473], 80.00th=[ 5473], 90.00th=[ 5538], 95.
| 99.00th=[ 5932], 99.50th=[ 6063], 99.90th=[ 6980], 99. | | 99.00th=[10945], 99.50th=[10945], 99.90th=[11076], 99.
| 99.99th=[11207] | | 99.99th=[11338]
Type=read,Size=64K,NumJobs=56,depth=1 Type=read,Size=64K,NumJobs=56,depth=1
read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10001msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec)
slat (usec): min=4, max=111, avg= 4.93, stdev= 1.11 | slat (nsec): min=3801, max=63985, avg=4896.45, stdev=1489
clat (usec): min=41, max=531, avg=314.40, stdev=75.66 | clat (usec): min=29, max=550, avg=314.45, stdev=76.21
lat (usec): min=50, max=535, avg=319.39, stdev=75.67 | lat (usec): min=35, max=555, avg=319.41, stdev=76.15
| 1.00th=[ 174], 5.00th=[ 176], 10.00th=[ 178], 20. | | 1.00th=[ 174], 5.00th=[ 178], 10.00th=[ 194], 20.
| 30.00th=[ 306], 40.00th=[ 306], 50.00th=[ 306], 60. | | 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 306], 60.
| 70.00th=[ 310], 80.00th=[ 318], 90.00th=[ 461], 95. | | 70.00th=[ 310], 80.00th=[ 322], 90.00th=[ 457], 95.
| 99.00th=[ 478], 99.50th=[ 482], 99.90th=[ 490], 99. | | 99.00th=[ 474], 99.50th=[ 482], 99.90th=[ 490], 99.
| 99.99th=[ 502] | | 99.99th=[ 506]
Type=read,Size=64K,NumJobs=56,depth=16 Type=read,Size=64K,NumJobs=56,depth=16
read: IOPS=175k, BW=10.7GiB/s (11.4GB/s)(107GiB/10006msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec)
slat (usec): min=4, max=176, avg= 5.48, stdev= 2.61 | slat (usec): min=3, max=251, avg= 5.40, stdev= 2.34
clat (usec): min=718, max=14322, avg=5122.23, stdev=2028. | clat (usec): min=236, max=14486, avg=5106.35, stdev=1241.
lat (usec): min=744, max=14328, avg=5127.78, stdev=2028. | lat (usec): min=265, max=14491, avg=5111.80, stdev=1241.
| 1.00th=[ 2343], 5.00th=[ 2376], 10.00th=[ 2442], 20. | | 1.00th=[ 2442], 5.00th=[ 2540], 10.00th=[ 4883], 20.
| 30.00th=[ 4752], 40.00th=[ 4752], 50.00th=[ 4948], 60. | | 30.00th=[ 4948], 40.00th=[ 5080], 50.00th=[ 5080], 60.
| 70.00th=[ 5080], 80.00th=[ 5145], 90.00th=[ 9503], 95. | | 70.00th=[ 5080], 80.00th=[ 5145], 90.00th=[ 7373], 95.
| 99.00th=[ 9896], 99.50th=[ 9896], 99.90th=[10028], 99. | | 99.00th=[ 9896], 99.50th=[ 9896], 99.90th=[ 9896], 99.
| 99.99th=[10028] | 99.99th=[10028]
Type=read,Size=64K,NumJobs=56,depth=64 Type=read,Size=64K,NumJobs=56,depth=64
read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10017msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10018msec)
slat (usec): min=2, max=9087, avg=220.21, stdev=768.63 | slat (usec): min=2, max=10184, avg=189.22, stdev=637.78
clat (usec): min=172, max=64296, avg=20233.61, stdev=8148 | clat (usec): min=347, max=61952, avg=20252.15, stdev=7056
lat (usec): min=186, max=64319, avg=20453.88, stdev=8308 | lat (usec): min=351, max=65141, avg=20441.44, stdev=7203
| 1.00th=[ 9110], 5.00th=[ 9110], 10.00th=[ 9110], 20. | | 1.00th=[ 9503], 5.00th=[ 9765], 10.00th=[ 9896], 20.
| 30.00th=[18220], 40.00th=[18744], 50.00th=[19006], 60. | | 30.00th=[19530], 40.00th=[19530], 50.00th=[19792], 60.
| 70.00th=[27395], 80.00th=[27395], 90.00th=[28705], 95. | | 70.00th=[20317], 80.00th=[20579], 90.00th=[29492], 95.
| 99.00th=[38011], 99.50th=[40109], 99.90th=[44303], 99. | | 99.00th=[41157], 99.50th=[58983], 99.90th=[59507], 99.
| 99.99th=[54789] | | 99.99th=[61080]
Type=write,Size=512B,NumJobs=1,depth=1 Type=write,Size=512B,NumJobs=1,depth=1
write: IOPS=57.7k, BW=28.2MiB/s (29.6MB/s)(282MiB/10001msec | write: IOPS=56.0k, BW=27.4MiB/s (28.7MB/s)(274MiB/10001msec
slat (nsec): min=2139, max=32784, avg=2527.92, stdev=1359 | slat (nsec): min=2112, max=43957, avg=2561.65, stdev=1423
clat (nsec): min=624, max=106340, avg=14332.09, stdev=474 | clat (nsec): min=633, max=60031, avg=14812.68, stdev=5638
lat (usec): min=12, max=108, avg=16.90, stdev= 5.16 | lat (nsec): min=12369, max=72124, avg=17416.39, stdev=60
| 1.00th=[12480], 5.00th=[12480], 10.00th=[12608], 20. | | 1.00th=[12352], 5.00th=[12352], 10.00th=[12480], 20.
| 30.00th=[12608], 40.00th=[12736], 50.00th=[12736], 60. | | 30.00th=[12480], 40.00th=[12608], 50.00th=[12608], 60.
| 70.00th=[12864], 80.00th=[13376], 90.00th=[16512], 95. | | 70.00th=[12864], 80.00th=[14656], 90.00th=[20096], 95.
| 99.00th=[34560], 99.50th=[35584], 99.90th=[42240], 99. | | 99.00th=[35072], 99.50th=[36608], 99.90th=[47360], 99.
| 99.99th=[53504] | | 99.99th=[54016]
Type=write,Size=512B,NumJobs=1,depth=16 Type=write,Size=512B,NumJobs=1,depth=16
write: IOPS=306k, BW=149MiB/s (156MB/s)(1492MiB/10001msec) | write: IOPS=328k, BW=160MiB/s (168MB/s)(1601MiB/10001msec)
slat (nsec): min=1650, max=31443, avg=2000.75, stdev=584. | slat (nsec): min=1641, max=32941, avg=2199.97, stdev=236.
clat (usec): min=38, max=193, avg=50.03, stdev=12.10 | clat (usec): min=14, max=113, avg=46.29, stdev= 1.27
lat (usec): min=40, max=195, avg=52.07, stdev=12.12 | lat (usec): min=16, max=115, avg=48.52, stdev= 1.28
| 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20. | | 1.00th=[43776], 5.00th=[44800], 10.00th=[45312], 20.
| 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 45], 60. | | 30.00th=[45824], 40.00th=[45824], 50.00th=[46336], 60.
| 70.00th=[ 52], 80.00th=[ 59], 90.00th=[ 69], 95. | | 70.00th=[46336], 80.00th=[46848], 90.00th=[47360], 95.
| 99.00th=[ 93], 99.50th=[ 100], 99.90th=[ 126], 99. | | 99.00th=[50432], 99.50th=[53504], 99.90th=[57600], 99.
| 99.99th=[ 145] | | 99.99th=[63744]
Type=write,Size=512B,NumJobs=1,depth=64 Type=write,Size=512B,NumJobs=1,depth=64
write: IOPS=363k, BW=177MiB/s (186MB/s)(1771MiB/10001msec) | write: IOPS=337k, BW=164MiB/s (172MB/s)(1644MiB/10001msec)
slat (nsec): min=1644, max=41021, avg=1961.04, stdev=238. | slat (nsec): min=1642, max=43437, avg=2146.30, stdev=234.
clat (usec): min=71, max=241, avg=174.18, stdev= 2.48 | clat (usec): min=16, max=235, avg=187.62, stdev= 2.46
lat (usec): min=73, max=243, avg=176.18, stdev= 2.50 | lat (usec): min=18, max=237, avg=189.80, stdev= 2.48
| 1.00th=[ 172], 5.00th=[ 172], 10.00th=[ 172], 20. | | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 186], 20.
| 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 174], 60. | | 30.00th=[ 186], 40.00th=[ 188], 50.00th=[ 188], 60.
| 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 178], 95. | | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 192], 95.
| 99.00th=[ 186], 99.50th=[ 190], 99.90th=[ 192], 99. | | 99.00th=[ 194], 99.50th=[ 196], 99.90th=[ 200], 99.
| 99.99th=[ 198] | | 99.99th=[ 204]
Type=write,Size=512B,NumJobs=16,depth=1 Type=write,Size=512B,NumJobs=16,depth=1
write: IOPS=938k, BW=458MiB/s (480MB/s)(4582MiB/10002msec) | write: IOPS=914k, BW=446MiB/s (468MB/s)(4462MiB/10001msec)
slat (nsec): min=2061, max=61336, avg=2520.86, stdev=499. | slat (nsec): min=1818, max=83376, avg=2566.34, stdev=824.
clat (nsec): min=595, max=164756, avg=14051.16, stdev=104 | clat (nsec): min=465, max=343253, avg=14459.07, stdev=181
lat (usec): min=11, max=168, avg=16.62, stdev= 1.19 | lat (usec): min=11, max=345, avg=17.08, stdev= 2.09
| 1.00th=[11712], 5.00th=[12864], 10.00th=[13248], 20. | | 1.00th=[12224], 5.00th=[12992], 10.00th=[13248], 20.
| 30.00th=[13632], 40.00th=[13760], 50.00th=[13888], 60. | | 30.00th=[13632], 40.00th=[13760], 50.00th=[14016], 60.
| 70.00th=[14272], 80.00th=[14528], 90.00th=[15040], 95. | | 70.00th=[14528], 80.00th=[15040], 90.00th=[16320], 95.
| 99.00th=[17280], 99.50th=[18048], 99.90th=[23680], 99. | | 99.00th=[21888], 99.50th=[23936], 99.90th=[32384], 99.
| 99.99th=[29568] | | 99.99th=[35584]
Type=write,Size=512B,NumJobs=16,depth=16 Type=write,Size=512B,NumJobs=16,depth=16
write: IOPS=4631k, BW=2261MiB/s (2371MB/s)(22.1GiB/10001mse | write: IOPS=3218k, BW=1571MiB/s (1648MB/s)(15.3GiB/10001mse
slat (nsec): min=1666, max=183449, avg=2373.16, stdev=548 | slat (nsec): min=1621, max=154997, avg=3548.60, stdev=323
clat (usec): min=18, max=440, avg=52.52, stdev= 9.27 | clat (usec): min=14, max=473, avg=75.44, stdev=65.86
lat (usec): min=20, max=444, avg=54.94, stdev= 9.57 | lat (usec): min=17, max=489, avg=79.05, stdev=68.82
| 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 44], 20. | | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 46], 20.
| 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 52], 60. | | 30.00th=[ 47], 40.00th=[ 48], 50.00th=[ 52], 60.
| 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 62], 95. | | 70.00th=[ 61], 80.00th=[ 64], 90.00th=[ 178], 95.
| 99.00th=[ 74], 99.50th=[ 79], 99.90th=[ 89], 99. | | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 351], 99.
| 99.99th=[ 355] | | 99.99th=[ 367]
Type=write,Size=512B,NumJobs=16,depth=64 Type=write,Size=512B,NumJobs=16,depth=64
write: IOPS=5021k, BW=2452MiB/s (2571MB/s)(23.9GiB/10002mse | write: IOPS=3973k, BW=1940MiB/s (2034MB/s)(18.0GiB/10002mse
slat (nsec): min=1670, max=80008, avg=2293.73, stdev=923. | slat (nsec): min=1627, max=171980, avg=2926.70, stdev=209
clat (usec): min=24, max=1715, avg=201.23, stdev=28.44 | clat (usec): min=10, max=1606, avg=254.29, stdev=176.36
lat (usec): min=26, max=1733, avg=203.56, stdev=28.76 | lat (usec): min=16, max=1615, avg=257.27, stdev=178.32
| 1.00th=[ 176], 5.00th=[ 178], 10.00th=[ 180], 20. | | 1.00th=[ 184], 5.00th=[ 186], 10.00th=[ 188], 20.
| 30.00th=[ 182], 40.00th=[ 184], 50.00th=[ 208], 60. | | 30.00th=[ 194], 40.00th=[ 202], 50.00th=[ 237], 60.
| 70.00th=[ 212], 80.00th=[ 215], 90.00th=[ 219], 95. | | 70.00th=[ 255], 80.00th=[ 260], 90.00th=[ 293], 95.
| 99.00th=[ 293], 99.50th=[ 297], 99.90th=[ 314], 99. | | 99.00th=[ 1385], 99.50th=[ 1418], 99.90th=[ 1467], 99.
| 99.99th=[ 898] | | 99.99th=[ 1500]
Type=write,Size=512B,NumJobs=56,depth=1 Type=write,Size=512B,NumJobs=56,depth=1
write: IOPS=2332k, BW=1139MiB/s (1194MB/s)(11.1GiB/10003mse | write: IOPS=2369k, BW=1157MiB/s (1213MB/s)(11.3GiB/10002mse
slat (usec): min=2, max=720, avg= 3.74, stdev= 2.14 | slat (nsec): min=1823, max=501953, avg=3567.43, stdev=201
clat (nsec): min=624, max=5616.3k, avg=19430.25, stdev=63 | clat (nsec): min=449, max=1722.5k, avg=19318.77, stdev=42
lat (usec): min=10, max=5623, avg=23.28, stdev= 6.86 | lat (usec): min=11, max=1724, avg=22.98, stdev= 4.83
| 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 16], 20. | | 1.00th=[12992], 5.00th=[14656], 10.00th=[15680], 20.
| 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 19], 60. | | 30.00th=[17536], 40.00th=[18048], 50.00th=[18816], 60.
| 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 24], 95. | | 70.00th=[20352], 80.00th=[21632], 90.00th=[23424], 95.
| 99.00th=[ 33], 99.50th=[ 37], 99.90th=[ 47], 99. | | 99.00th=[31872], 99.50th=[35584], 99.90th=[41728], 99.
| 99.99th=[ 310] | | 99.99th=[54016]
Type=write,Size=512B,NumJobs=56,depth=16 Type=write,Size=512B,NumJobs=56,depth=16
write: IOPS=8099k, BW=3955MiB/s (4147MB/s)(38.6GiB/10002mse | write: IOPS=6237k, BW=3046MiB/s (3193MB/s)(29.7GiB/10002mse
slat (nsec): min=1742, max=11042k, avg=5139.82, stdev=693 | slat (nsec): min=1640, max=2381.5k, avg=5462.05, stdev=67
clat (nsec): min=1973, max=13062k, avg=104665.53, stdev=5 | clat (usec): min=5, max=10435, avg=137.19, stdev=82.68
lat (usec): min=31, max=13064, avg=109.91, stdev=54.98 | lat (usec): min=15, max=10447, avg=142.77, stdev=85.58
| 1.00th=[ 66], 5.00th=[ 69], 10.00th=[ 71], 20. | | 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 70], 20.
| 30.00th=[ 74], 40.00th=[ 75], 50.00th=[ 76], 60. | | 30.00th=[ 85], 40.00th=[ 96], 50.00th=[ 104], 60.
| 70.00th=[ 88], 80.00th=[ 167], 90.00th=[ 188], 95. | | 70.00th=[ 149], 80.00th=[ 210], 90.00th=[ 265], 95.
| 99.00th=[ 215], 99.50th=[ 223], 99.90th=[ 375], 99. | | 99.00th=[ 363], 99.50th=[ 396], 99.90th=[ 553], 99.
| 99.99th=[ 898] | | 99.99th=[ 873]
Type=write,Size=512B,NumJobs=56,depth=64 Type=write,Size=512B,NumJobs=56,depth=64
write: IOPS=8134k, BW=3972MiB/s (4165MB/s)(38.8GiB/10002mse | write: IOPS=7488k, BW=3656MiB/s (3834MB/s)(35.7GiB/10002mse
slat (nsec): min=1797, max=599123, avg=5146.84, stdev=693 | slat (nsec): min=1666, max=7929.3k, avg=5382.53, stdev=12
clat (usec): min=42, max=1474, avg=434.58, stdev=198.63 | clat (usec): min=14, max=12517, avg=472.19, stdev=331.30
lat (usec): min=49, max=1477, avg=439.83, stdev=201.01 | lat (usec): min=17, max=12520, avg=477.70, stdev=334.84
| 1.00th=[ 281], 5.00th=[ 289], 10.00th=[ 297], 20. | | 1.00th=[ 265], 5.00th=[ 277], 10.00th=[ 289], 20.
| 30.00th=[ 310], 40.00th=[ 310], 50.00th=[ 314], 60. | | 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 314], 60.
| 70.00th=[ 326], 80.00th=[ 725], 90.00th=[ 758], 95. | | 70.00th=[ 347], 80.00th=[ 562], 90.00th=[ 1057], 95.
| 99.00th=[ 799], 99.50th=[ 816], 99.90th=[ 840], 99. | | 99.00th=[ 1434], 99.50th=[ 1696], 99.90th=[ 2040], 99.
| 99.99th=[ 922] | | 99.99th=[ 2474]
Type=write,Size=4K,NumJobs=1,depth=1 Type=write,Size=4K,NumJobs=1,depth=1
write: IOPS=54.1k, BW=211MiB/s (222MB/s)(2115MiB/10001msec) | write: IOPS=55.8k, BW=218MiB/s (229MB/s)(2180MiB/10000msec)
slat (nsec): min=2154, max=32419, avg=2620.30, stdev=1579 | slat (nsec): min=2138, max=44613, avg=2552.61, stdev=1404
clat (nsec): min=639, max=171519, avg=15378.78, stdev=558 | clat (nsec): min=627, max=624863, avg=14910.38, stdev=517
lat (usec): min=13, max=173, avg=18.04, stdev= 6.03 | lat (usec): min=13, max=627, avg=17.50, stdev= 5.55
| 1.00th=[12864], 5.00th=[12992], 10.00th=[12992], 20. | | 1.00th=[12736], 5.00th=[12864], 10.00th=[12864], 20.
| 30.00th=[13120], 40.00th=[13120], 50.00th=[13248], 60. | | 30.00th=[12992], 40.00th=[12992], 50.00th=[12992], 60.
| 70.00th=[13376], 80.00th=[15424], 90.00th=[20864], 95. | | 70.00th=[13248], 80.00th=[13632], 90.00th=[20352], 95.
| 99.00th=[36096], 99.50th=[38144], 99.90th=[48384], 99. | | 99.00th=[35584], 99.50th=[37120], 99.90th=[44800], 99.
| 99.99th=[69120] | | 99.99th=[53504]
Type=write,Size=4K,NumJobs=1,depth=16 Type=write,Size=4K,NumJobs=1,depth=16
write: IOPS=285k, BW=1112MiB/s (1166MB/s)(10.9GiB/10001msec | write: IOPS=325k, BW=1271MiB/s (1332MB/s)(12.4GiB/10001msec
slat (nsec): min=1784, max=29113, avg=2129.85, stdev=618. | slat (nsec): min=1697, max=32986, avg=2232.67, stdev=205.
clat (usec): min=40, max=179, avg=53.75, stdev=12.39 | clat (nsec): min=14620, max=89670, avg=46638.22, stdev=11
lat (usec): min=42, max=182, avg=55.91, stdev=12.40 | lat (nsec): min=16814, max=91833, avg=48907.48, stdev=11
| 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 44], 20. | | 1.00th=[44800], 5.00th=[45312], 10.00th=[45824], 20.
| 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 50], 60. | | 30.00th=[46336], 40.00th=[46336], 50.00th=[46336], 60.
| 70.00th=[ 58], 80.00th=[ 64], 90.00th=[ 72], 95. | | 70.00th=[46848], 80.00th=[46848], 90.00th=[47360], 95.
| 99.00th=[ 96], 99.50th=[ 105], 99.90th=[ 129], 99. | | 99.00th=[50944], 99.50th=[53504], 99.90th=[58112], 99.
| 99.99th=[ 147] | | 99.99th=[63744]
Type=write,Size=4K,NumJobs=1,depth=64 Type=write,Size=4K,NumJobs=1,depth=64
write: IOPS=340k, BW=1327MiB/s (1391MB/s)(12.0GiB/10001msec | write: IOPS=322k, BW=1259MiB/s (1320MB/s)(12.3GiB/10001msec
slat (nsec): min=1701, max=34828, avg=2128.31, stdev=233. | slat (nsec): min=1697, max=33655, avg=2273.54, stdev=208.
clat (usec): min=52, max=233, avg=185.96, stdev= 1.94 | clat (usec): min=18, max=238, avg=195.95, stdev= 2.07
lat (usec): min=54, max=235, avg=188.13, stdev= 1.95 | lat (usec): min=20, max=241, avg=198.26, stdev= 2.08
| 1.00th=[ 182], 5.00th=[ 184], 10.00th=[ 184], 20. | | 1.00th=[ 192], 5.00th=[ 194], 10.00th=[ 194], 20.
| 30.00th=[ 186], 40.00th=[ 186], 50.00th=[ 186], 60. | | 30.00th=[ 196], 40.00th=[ 196], 50.00th=[ 196], 60.
| 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 188], 95. | | 70.00th=[ 198], 80.00th=[ 198], 90.00th=[ 198], 95.
| 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 196], 99. | | 99.00th=[ 202], 99.50th=[ 204], 99.90th=[ 208], 99.
| 99.99th=[ 202] | | 99.99th=[ 212]
Type=write,Size=4K,NumJobs=16,depth=1 Type=write,Size=4K,NumJobs=16,depth=1
write: IOPS=883k, BW=3449MiB/s (3616MB/s)(33.7GiB/10002msec | write: IOPS=855k, BW=3339MiB/s (3502MB/s)(32.6GiB/10001msec
slat (nsec): min=2093, max=58113, avg=2489.62, stdev=384. | slat (nsec): min=1832, max=65363, avg=2579.38, stdev=826.
clat (nsec): min=601, max=198998, avg=15168.57, stdev=132 | clat (nsec): min=472, max=134403, avg=15646.36, stdev=196
lat (usec): min=12, max=201, avg=17.70, stdev= 1.39 | lat (usec): min=12, max=136, avg=18.28, stdev= 2.21
| 1.00th=[12608], 5.00th=[13760], 10.00th=[14144], 20. | | 1.00th=[13120], 5.00th=[13760], 10.00th=[14016], 20.
| 30.00th=[14656], 40.00th=[14784], 50.00th=[15040], 60. | | 30.00th=[14656], 40.00th=[14912], 50.00th=[15168], 60.
| 70.00th=[15424], 80.00th=[15808], 90.00th=[16320], 95. | | 70.00th=[15936], 80.00th=[16512], 90.00th=[17536], 95.
| 99.00th=[18816], 99.50th=[20096], 99.90th=[25984], 99. | | 99.00th=[23680], 99.50th=[25728], 99.90th=[33536], 99.
| 99.99th=[44800] | | 99.99th=[37120]
Type=write,Size=4K,NumJobs=16,depth=16 Type=write,Size=4K,NumJobs=16,depth=16
write: IOPS=2677k, BW=10.2GiB/s (10.0GB/s)(102GiB/10001msec | write: IOPS=1826k, BW=7133MiB/s (7480MB/s)(69.7GiB/10001mse
slat (nsec): min=1663, max=57295, avg=2380.16, stdev=1242 | slat (nsec): min=1639, max=113071, avg=3312.71, stdev=236
clat (usec): min=37, max=345, avg=92.86, stdev=12.80 | clat (usec): min=40, max=454, avg=136.35, stdev=35.22
lat (usec): min=41, max=351, avg=95.28, stdev=12.74 | lat (usec): min=43, max=456, avg=139.72, stdev=35.40
| 1.00th=[ 69], 5.00th=[ 75], 10.00th=[ 78], 20. | | 1.00th=[ 72], 5.00th=[ 91], 10.00th=[ 101], 20.
| 30.00th=[ 86], 40.00th=[ 92], 50.00th=[ 96], 60. | | 30.00th=[ 121], 40.00th=[ 127], 50.00th=[ 133], 60.
| 70.00th=[ 99], 80.00th=[ 101], 90.00th=[ 104], 95. | | 70.00th=[ 143], 80.00th=[ 149], 90.00th=[ 182], 95.
| 99.00th=[ 133], 99.50th=[ 151], 99.90th=[ 178], 99. | | 99.00th=[ 258], 99.50th=[ 273], 99.90th=[ 314], 99.
| 99.99th=[ 212] | | 99.99th=[ 367]
Type=write,Size=4K,NumJobs=16,depth=64 Type=write,Size=4K,NumJobs=16,depth=64
write: IOPS=2691k, BW=10.3GiB/s (11.0GB/s)(103GiB/10001msec | write: IOPS=1875k, BW=7325MiB/s (7680MB/s)(71.5GiB/10001mse
slat (nsec): min=1672, max=299471, avg=2451.55, stdev=255 | slat (nsec): min=1638, max=627057, avg=3232.91, stdev=324
clat (usec): min=70, max=965, avg=377.69, stdev=68.69 | clat (usec): min=49, max=1357, avg=542.32, stdev=162.71
lat (usec): min=77, max=980, avg=380.18, stdev=69.11 | lat (usec): min=52, max=1384, avg=545.61, stdev=162.80
| 1.00th=[ 223], 5.00th=[ 285], 10.00th=[ 314], 20. | | 1.00th=[ 202], 5.00th=[ 277], 10.00th=[ 330], 20.
| 30.00th=[ 351], 40.00th=[ 363], 50.00th=[ 375], 60. | | 30.00th=[ 453], 40.00th=[ 498], 50.00th=[ 537], 60.
| 70.00th=[ 396], 80.00th=[ 412], 90.00th=[ 433], 95. | | 70.00th=[ 635], 80.00th=[ 685], 90.00th=[ 758], 95.
| 99.00th=[ 693], 99.50th=[ 717], 99.90th=[ 758], 99. | | 99.00th=[ 922], 99.50th=[ 955], 99.90th=[ 1029], 99.
| 99.99th=[ 807] | | 99.99th=[ 1123]
Type=write,Size=4K,NumJobs=56,depth=1 Type=write,Size=4K,NumJobs=56,depth=1
write: IOPS=1417k, BW=5534MiB/s (5803MB/s)(54.1GiB/10002mse | write: IOPS=1416k, BW=5533MiB/s (5802MB/s)(54.0GiB/10002mse
slat (usec): min=2, max=722, avg= 3.06, stdev= 1.31 | slat (nsec): min=1846, max=533231, avg=2881.71, stdev=111
clat (nsec): min=647, max=796809, avg=35776.52, stdev=697 | clat (nsec): min=498, max=4453.2k, avg=36057.87, stdev=38
lat (usec): min=14, max=799, avg=38.92, stdev= 7.08 | lat (usec): min=15, max=4456, avg=39.01, stdev= 4.01
| 1.00th=[ 28], 5.00th=[ 32], 10.00th=[ 33], 20. | | 1.00th=[29312], 5.00th=[32128], 10.00th=[33024], 20.
| 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 36], 60. | | 30.00th=[34560], 40.00th=[35072], 50.00th=[35584], 60.
| 70.00th=[ 37], 80.00th=[ 38], 90.00th=[ 40], 95. | | 70.00th=[37120], 80.00th=[37632], 90.00th=[38656], 95.
| 99.00th=[ 50], 99.50th=[ 53], 99.90th=[ 69], 99. | | 99.00th=[49408], 99.50th=[52480], 99.90th=[56576], 99.
| 99.99th=[ 367] | | 99.99th=[64768]
Type=write,Size=4K,NumJobs=56,depth=16 Type=write,Size=4K,NumJobs=56,depth=16
write: IOPS=2470k, BW=9649MiB/s (10.1GB/s)(94.3GiB/10002mse | write: IOPS=1616k, BW=6314MiB/s (6621MB/s)(61.7GiB/10001mse
slat (nsec): min=1714, max=353380, avg=3021.76, stdev=221 | slat (nsec): min=1649, max=738008, avg=3245.81, stdev=163
clat (usec): min=44, max=1490, avg=359.06, stdev=129.14 | clat (usec): min=44, max=5046, avg=550.36, stdev=143.07
lat (usec): min=47, max=1493, avg=362.15, stdev=129.13 | lat (usec): min=46, max=5049, avg=553.68, stdev=143.03
| 1.00th=[ 212], 5.00th=[ 235], 10.00th=[ 247], 20. | | 1.00th=[ 215], 5.00th=[ 343], 10.00th=[ 404], 20.
| 30.00th=[ 277], 40.00th=[ 285], 50.00th=[ 302], 60. | | 30.00th=[ 490], 40.00th=[ 510], 50.00th=[ 529], 60.
| 70.00th=[ 392], 80.00th=[ 478], 90.00th=[ 570], 95. | | 70.00th=[ 586], 80.00th=[ 660], 90.00th=[ 758], 95.
| 99.00th=[ 717], 99.50th=[ 750], 99.90th=[ 840], 99. | | 99.00th=[ 938], 99.50th=[ 996], 99.90th=[ 1106], 99.
| 99.99th=[ 1004] | | 99.99th=[ 1221]
Type=write,Size=4K,NumJobs=56,depth=64 Type=write,Size=4K,NumJobs=56,depth=64
write: IOPS=2093k, BW=8177MiB/s (8574MB/s)(79.9GiB/10003mse | write: IOPS=1625k, BW=6348MiB/s (6656MB/s)(62.0GiB/10003mse
slat (nsec): min=1735, max=4573.4k, avg=18118.68, stdev=6 | slat (nsec): min=1656, max=1933.3k, avg=21620.58, stdev=7
clat (usec): min=40, max=8076, avg=1693.04, stdev=804.74 | clat (usec): min=66, max=8663, avg=2182.95, stdev=879.90
lat (usec): min=44, max=8079, avg=1711.22, stdev=817.33 | lat (usec): min=69, max=8669, avg=2204.64, stdev=895.51
| 1.00th=[ 375], 5.00th=[ 644], 10.00th=[ 766], 20. | | 1.00th=[ 693], 5.00th=[ 971], 10.00th=[ 1074], 20.
| 30.00th=[ 1205], 40.00th=[ 1516], 50.00th=[ 1647], 60. | | 30.00th=[ 1942], 40.00th=[ 2114], 50.00th=[ 2212], 60.
| 70.00th=[ 1909], 80.00th=[ 2278], 90.00th=[ 2835], 95. | | 70.00th=[ 2343], 80.00th=[ 2507], 90.00th=[ 3392], 95.
| 99.00th=[ 4015], 99.50th=[ 4293], 99.90th=[ 4883], 99. | | 99.00th=[ 4883], 99.50th=[ 5211], 99.90th=[ 5800], 99.
| 99.99th=[ 5735] | | 99.99th=[ 6587]
Type=write,Size=64K,NumJobs=1,depth=1 Type=write,Size=64K,NumJobs=1,depth=1
write: IOPS=23.3k, BW=1455MiB/s (1526MB/s)(14.2GiB/10001mse | write: IOPS=23.3k, BW=1454MiB/s (1525MB/s)(14.2GiB/10001mse
slat (usec): min=4, max=109, avg= 6.71, stdev= 3.24 | slat (nsec): min=4849, max=39702, avg=6880.02, stdev=3240
clat (usec): min=14, max=124, avg=35.67, stdev= 9.52 | clat (usec): min=17, max=133, avg=35.53, stdev= 9.39
lat (usec): min=33, max=159, avg=42.42, stdev=10.65 | lat (usec): min=33, max=146, avg=42.46, stdev=10.52
| 1.00th=[29312], 5.00th=[29312], 10.00th=[29312], 20. | | 1.00th=[29312], 5.00th=[29312], 10.00th=[29568], 20.
| 30.00th=[29568], 40.00th=[29824], 50.00th=[30080], 60. | | 30.00th=[29568], 40.00th=[29824], 50.00th=[29824], 60.
| 70.00th=[36608], 80.00th=[44800], 90.00th=[49920], 95. | | 70.00th=[36608], 80.00th=[43776], 90.00th=[49408], 95.
| 99.00th=[68096], 99.50th=[72192], 99.90th=[86528], 99. | | 99.00th=[68096], 99.50th=[72192], 99.90th=[85504], 99.
| 99.99th=[91648] | | 99.99th=[90624]
Type=write,Size=64K,NumJobs=1,depth=16 Type=write,Size=64K,NumJobs=1,depth=16
write: IOPS=141k, BW=8811MiB/s (9239MB/s)(86.1GiB/10001msec | write: IOPS=146k, BW=9109MiB/s (9552MB/s)(88.0GiB/10001msec
slat (nsec): min=3960, max=44206, avg=5332.37, stdev=776. | slat (nsec): min=3787, max=44724, avg=5539.44, stdev=622.
clat (usec): min=83, max=307, avg=107.82, stdev=13.00 | clat (usec): min=76, max=228, avg=103.90, stdev=11.08
lat (usec): min=88, max=312, avg=113.19, stdev=13.00 | lat (usec): min=82, max=233, avg=109.48, stdev=11.10
| 1.00th=[ 92], 5.00th=[ 96], 10.00th=[ 98], 20. | | 1.00th=[ 90], 5.00th=[ 93], 10.00th=[ 95], 20.
| 30.00th=[ 102], 40.00th=[ 103], 50.00th=[ 105], 60. | | 30.00th=[ 98], 40.00th=[ 99], 50.00th=[ 101], 60.
| 70.00th=[ 110], 80.00th=[ 113], 90.00th=[ 121], 95. | | 70.00th=[ 105], 80.00th=[ 109], 90.00th=[ 120], 95.
| 99.00th=[ 163], 99.50th=[ 174], 99.90th=[ 204], 99. | | 99.00th=[ 141], 99.50th=[ 149], 99.90th=[ 182], 99.
| 99.99th=[ 241] | | 99.99th=[ 200]
Type=write,Size=64K,NumJobs=1,depth=64 Type=write,Size=64K,NumJobs=1,depth=64
write: IOPS=144k, BW=8971MiB/s (9407MB/s)(87.6GiB/10001msec | write: IOPS=139k, BW=8708MiB/s (9131MB/s)(85.0GiB/10001msec
slat (nsec): min=2732, max=48101, avg=4533.01, stdev=1509 | slat (nsec): min=2676, max=64177, avg=5388.79, stdev=1847
clat (usec): min=162, max=1015, avg=440.98, stdev=74.73 | clat (usec): min=39, max=894, avg=453.54, stdev=56.76
lat (usec): min=174, max=1022, avg=445.55, stdev=74.68 | lat (usec): min=45, max=898, avg=458.97, stdev=57.24
| 1.00th=[ 322], 5.00th=[ 326], 10.00th=[ 338], 20. | | 1.00th=[ 338], 5.00th=[ 355], 10.00th=[ 367], 20.
| 30.00th=[ 400], 40.00th=[ 412], 50.00th=[ 437], 60. | | 30.00th=[ 408], 40.00th=[ 457], 50.00th=[ 478], 60.
| 70.00th=[ 482], 80.00th=[ 502], 90.00th=[ 537], 95. | | 70.00th=[ 494], 80.00th=[ 498], 90.00th=[ 506], 95.
| 99.00th=[ 635], 99.50th=[ 693], 99.90th=[ 799], 99. | | 99.00th=[ 545], 99.50th=[ 562], 99.90th=[ 734], 99.
| 99.99th=[ 898] | | 99.99th=[ 848]
Type=write,Size=64K,NumJobs=16,depth=1 Type=write,Size=64K,NumJobs=16,depth=1
write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10001msec) write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10001msec)
slat (nsec): min=4432, max=41303, avg=5462.66, stdev=782. | slat (nsec): min=4383, max=73755, avg=5607.60, stdev=1278
clat (usec): min=23, max=214, avg=85.24, stdev= 5.53 | clat (usec): min=22, max=424, avg=85.24, stdev= 8.07
lat (usec): min=37, max=220, avg=90.75, stdev= 5.58 | lat (usec): min=38, max=430, avg=90.90, stdev= 8.07
| 1.00th=[ 68], 5.00th=[ 80], 10.00th=[ 82], 20. | | 1.00th=[ 68], 5.00th=[ 76], 10.00th=[ 79], 20.
| 30.00th=[ 84], 40.00th=[ 85], 50.00th=[ 86], 60. | | 30.00th=[ 82], 40.00th=[ 83], 50.00th=[ 84], 60.
| 70.00th=[ 87], 80.00th=[ 88], 90.00th=[ 90], 95. | | 70.00th=[ 87], 80.00th=[ 91], 90.00th=[ 98], 95.
| 99.00th=[ 103], 99.50th=[ 110], 99.90th=[ 123], 99. | | 99.00th=[ 106], 99.50th=[ 112], 99.90th=[ 124], 99.
| 99.99th=[ 137] | 99.99th=[ 137]
Type=write,Size=64K,NumJobs=16,depth=16 Type=write,Size=64K,NumJobs=16,depth=16
write: IOPS=176k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) write: IOPS=176k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec)
slat (nsec): min=4024, max=70542, avg=6729.45, stdev=1425 | slat (nsec): min=3611, max=70931, avg=6507.12, stdev=1389
clat (usec): min=103, max=4145, avg=1447.24, stdev=446.02 | clat (usec): min=109, max=3915, avg=1451.13, stdev=409.41
lat (usec): min=120, max=4152, avg=1454.01, stdev=446.19 | lat (usec): min=129, max=3920, avg=1457.68, stdev=409.50
| 1.00th=[ 474], 5.00th=[ 1237], 10.00th=[ 1254], 20. | | 1.00th=[ 1254], 5.00th=[ 1270], 10.00th=[ 1270], 20.
| 30.00th=[ 1270], 40.00th=[ 1270], 50.00th=[ 1270], 60. | | 30.00th=[ 1270], 40.00th=[ 1270], 50.00th=[ 1287], 60.
| 70.00th=[ 1369], 80.00th=[ 1450], 90.00th=[ 2540], 95. | | 70.00th=[ 1352], 80.00th=[ 1369], 90.00th=[ 2540], 95.
| 99.00th=[ 2671], 99.50th=[ 2704], 99.90th=[ 2933], 99. | | 99.00th=[ 2737], 99.50th=[ 2737], 99.90th=[ 2769], 99.
| 99.99th=[ 3916] | | 99.99th=[ 2769]
Type=write,Size=64K,NumJobs=16,depth=64 Type=write,Size=64K,NumJobs=16,depth=64
write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10004msec) write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10004msec)
slat (usec): min=2, max=6195, avg=13.26, stdev=111.30 | slat (nsec): min=3042, max=96073, avg=6866.24, stdev=2170
clat (usec): min=112, max=16931, avg=5822.91, stdev=1645. | clat (usec): min=124, max=10103, avg=5850.40, stdev=798.1
lat (usec): min=118, max=17733, avg=5836.21, stdev=1665. | lat (usec): min=131, max=10109, avg=5857.31, stdev=798.1
| 1.00th=[ 4015], 5.00th=[ 4293], 10.00th=[ 4424], 20. | | 1.00th=[ 4752], 5.00th=[ 4883], 10.00th=[ 4948], 20.
| 30.00th=[ 4883], 40.00th=[ 5080], 50.00th=[ 5276], 60. | | 30.00th=[ 5276], 40.00th=[ 5407], 50.00th=[ 5669], 60.
| 70.00th=[ 6063], 80.00th=[ 6587], 90.00th=[ 7570], 95. | | 70.00th=[ 6259], 80.00th=[ 6587], 90.00th=[ 7046], 95.
| 99.00th=[11600], 99.50th=[12387], 99.90th=[14091], 99. | | 99.00th=[ 7767], 99.50th=[ 7898], 99.90th=[ 8455], 99.
| 99.99th=[15795] | | 99.99th=[ 8848]
Type=write,Size=64K,NumJobs=56,depth=1 Type=write,Size=64K,NumJobs=56,depth=1
write: IOPS=176k, BW=10.7GiB/s (11.5GB/s)(107GiB/10001msec) | write: IOPS=175k, BW=10.7GiB/s (11.4GB/s)(107GiB/10002msec)
slat (nsec): min=4406, max=47421, avg=6031.08, stdev=1752 | slat (nsec): min=4029, max=63780, avg=5771.34, stdev=1537
clat (usec): min=49, max=915, avg=312.16, stdev=47.66 | clat (usec): min=32, max=979, avg=314.28, stdev=57.25
lat (usec): min=57, max=924, avg=318.25, stdev=47.66 | lat (usec): min=40, max=985, avg=320.11, stdev=57.29
| 1.00th=[ 194], 5.00th=[ 281], 10.00th=[ 289], 20. | | 1.00th=[ 186], 5.00th=[ 293], 10.00th=[ 297], 20.
| 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 306], 60. | | 30.00th=[ 297], 40.00th=[ 302], 50.00th=[ 302], 60.
| 70.00th=[ 310], 80.00th=[ 310], 90.00th=[ 326], 95. | | 70.00th=[ 310], 80.00th=[ 310], 90.00th=[ 326], 95.
| 99.00th=[ 469], 99.50th=[ 482], 99.90th=[ 611], 99. | | 99.00th=[ 603], 99.50th=[ 611], 99.90th=[ 627], 99.
| 99.99th=[ 644] | | 99.99th=[ 750]
Type=write,Size=64K,NumJobs=56,depth=16 Type=write,Size=64K,NumJobs=56,depth=16
write: IOPS=176k, BW=10.7GiB/s (11.5GB/s)(107GiB/10005msec) | write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10005msec)
slat (usec): min=3, max=102, avg= 7.88, stdev= 2.26 | slat (nsec): min=3542, max=89310, avg=7585.26, stdev=2460
clat (usec): min=157, max=13620, avg=5092.13, stdev=1280. | clat (usec): min=190, max=14367, avg=5116.17, stdev=1396.
lat (usec): min=183, max=13626, avg=5100.08, stdev=1280. | lat (usec): min=198, max=14375, avg=5123.82, stdev=1396.
| 1.00th=[ 2442], 5.00th=[ 2540], 10.00th=[ 2540], 20. | | 1.00th=[ 2540], 5.00th=[ 2540], 10.00th=[ 2573], 20.
| 30.00th=[ 5080], 40.00th=[ 5080], 50.00th=[ 5080], 60. | | 30.00th=[ 5080], 40.00th=[ 5080], 50.00th=[ 5145], 60.
| 70.00th=[ 5145], 80.00th=[ 5538], 90.00th=[ 6652], 95. | | 70.00th=[ 5145], 80.00th=[ 5669], 90.00th=[ 6849], 95.
| 99.00th=[ 8586], 99.50th=[ 9241], 99.90th=[10552], 99. | | 99.00th=[ 8979], 99.50th=[ 9634], 99.90th=[11207], 99.
| 99.99th=[12256] | | 99.99th=[12780]
Type=write,Size=64K,NumJobs=56,depth=64 Type=write,Size=64K,NumJobs=56,depth=64
write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10013msec) | write: IOPS=174k, BW=10.6GiB/s (11.4GB/s)(107GiB/10011msec)
slat (usec): min=2, max=36257, avg=176.13, stdev=892.90 | slat (usec): min=3, max=32839, avg=176.71, stdev=925.53
clat (usec): min=184, max=83223, avg=20322.48, stdev=6000 | clat (usec): min=189, max=132076, avg=20366.61, stdev=728
lat (usec): min=188, max=83231, avg=20498.69, stdev=6090 | lat (usec): min=218, max=132086, avg=20543.39, stdev=740
| 1.00th=[ 7832], 5.00th=[10028], 10.00th=[13173], 20. | | 1.00th=[ 7898], 5.00th=[ 9110], 10.00th=[10945], 20.
| 30.00th=[18220], 40.00th=[19792], 50.00th=[20317], 60. | | 30.00th=[17957], 40.00th=[19530], 50.00th=[20317], 60.
| 70.00th=[21103], 80.00th=[22938], 90.00th=[26870], 95. | | 70.00th=[21103], 80.00th=[23462], 90.00th=[28705], 95.
| 99.00th=[40633], 99.50th=[45351], 99.90th=[54264], 99. | | 99.00th=[45876], 99.50th=[50594], 99.90th=[63177], 99.
| 99.99th=[67634] | | 99.99th=[84411]
With DIM on initiator side Without DIM
Type=read,Size=512B,NumJobs=1,depth=1 Type=read,Size=512B,NumJobs=1,depth=1
read: IOPS=46.7k, BW=22.8MiB/s (23.9MB/s)(228MiB/10001msec | read: IOPS=47.0k, BW=22.0MiB/s (24.1MB/s)(230MiB/10001msec
slat (nsec): min=2008, max=43561, avg=3175.68, stdev=2353 | slat (nsec): min=1981, max=53196, avg=3048.29, stdev=2391
clat (nsec): min=456, max=103914, avg=17680.68, stdev=638 | clat (nsec): min=442, max=107156, avg=17658.37, stdev=647
lat (usec): min=13, max=106, avg=20.93, stdev= 7.10 | lat (usec): min=13, max=109, avg=20.78, stdev= 7.18
| 1.00th=[12864], 5.00th=[12992], 10.00th=[14016], 20. | | 1.00th=[13632], 5.00th=[13760], 10.00th=[13888], 20.
| 30.00th=[14144], 40.00th=[14272], 50.00th=[14400], 60. | | 30.00th=[14144], 40.00th=[14144], 50.00th=[14400], 60.
| 70.00th=[17280], 80.00th=[19072], 90.00th=[28032], 95. | | 70.00th=[17024], 80.00th=[19072], 90.00th=[27776], 95.
| 99.00th=[40192], 99.50th=[44800], 99.90th=[50944], 99. | | 99.00th=[40704], 99.50th=[45824], 99.90th=[51456], 99.
| 99.99th=[60160] | | 99.99th=[58624]
Type=read,Size=512B,NumJobs=1,depth=16 Type=read,Size=512B,NumJobs=1,depth=16
read: IOPS=281k, BW=137MiB/s (144MB/s)(1371MiB/10001msec) | read: IOPS=334k, BW=163MiB/s (171MB/s)(1629MiB/10001msec)
slat (nsec): min=1558, max=56900, avg=2362.92, stdev=2075 | slat (nsec): min=1564, max=28780, avg=2162.48, stdev=588.
clat (usec): min=32, max=158, avg=54.17, stdev= 9.58 | clat (usec): min=17, max=118, avg=45.50, stdev= 6.30
lat (usec): min=34, max=160, avg=56.59, stdev= 9.72 | lat (usec): min=20, max=120, avg=47.70, stdev= 6.64
| 1.00th=[ 40], 5.00th=[ 41], 10.00th=[ 43], 20. | | 1.00th=[41728], 5.00th=[42752], 10.00th=[42752], 20.
| 30.00th=[ 47], 40.00th=[ 53], 50.00th=[ 56], 60. | | 30.00th=[43264], 40.00th=[43776], 50.00th=[43776], 60.
| 70.00th=[ 60], 80.00th=[ 62], 90.00th=[ 65], 95. | | 70.00th=[44288], 80.00th=[44288], 90.00th=[46336], 95.
| 99.00th=[ 80], 99.50th=[ 85], 99.90th=[ 99], 99. | | 99.00th=[72192], 99.50th=[73216], 99.90th=[77312], 99.
| 99.99th=[ 133] | | 99.99th=[86528]
Type=read,Size=512B,NumJobs=1,depth=64 Type=read,Size=512B,NumJobs=1,depth=64
read: IOPS=377k, BW=184MiB/s (193MB/s)(1840MiB/10001msec) | read: IOPS=344k, BW=168MiB/s (176MB/s)(1682MiB/10001msec)
slat (nsec): min=1591, max=41737, avg=1885.11, stdev=245. | slat (nsec): min=1607, max=30650, avg=2093.56, stdev=233.
clat (usec): min=29, max=222, avg=167.69, stdev= 1.31 | clat (usec): min=19, max=238, avg=183.42, stdev= 2.22
lat (usec): min=32, max=224, avg=169.61, stdev= 1.33 | lat (usec): min=21, max=241, avg=185.55, stdev= 2.23
| 1.00th=[ 165], 5.00th=[ 167], 10.00th=[ 167], 20. | | 1.00th=[ 180], 5.00th=[ 182], 10.00th=[ 182], 20.
| 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 167], 60. | | 30.00th=[ 182], 40.00th=[ 184], 50.00th=[ 184], 60.
| 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 169], 95. | | 70.00th=[ 184], 80.00th=[ 186], 90.00th=[ 186], 95.
| 99.00th=[ 172], 99.50th=[ 172], 99.90th=[ 178], 99. | | 99.00th=[ 190], 99.50th=[ 192], 99.90th=[ 196], 99.
| 99.99th=[ 196] | | 99.99th=[ 210]
Type=read,Size=512B,NumJobs=16,depth=1 Type=read,Size=512B,NumJobs=16,depth=1
read: IOPS=906k, BW=442MiB/s (464MB/s)(4423MiB/10001msec) | read: IOPS=907k, BW=443MiB/s (465MB/s)(4431MiB/10001msec)
slat (nsec): min=1919, max=71597, avg=2515.26, stdev=515. | slat (nsec): min=1925, max=69082, avg=2505.79, stdev=869.
clat (nsec): min=678, max=1006.2k, avg=14689.14, stdev=14 | clat (nsec): min=476, max=112521, avg=14672.56, stdev=182
lat (usec): min=12, max=1008, avg=17.26, stdev= 1.66 | lat (usec): min=12, max=131, avg=17.23, stdev= 2.08
| 1.00th=[11712], 5.00th=[12736], 10.00th=[13376], 20. | | 1.00th=[11584], 5.00th=[13248], 10.00th=[13376], 20.
| 30.00th=[14144], 40.00th=[14400], 50.00th=[14656], 60. | | 30.00th=[13888], 40.00th=[14016], 50.00th=[14272], 60.
| 70.00th=[15168], 80.00th=[15424], 90.00th=[15936], 95. | | 70.00th=[14784], 80.00th=[15296], 90.00th=[16512], 95.
| 99.00th=[18048], 99.50th=[18816], 99.90th=[23424], 99. | | 99.00th=[22400], 99.50th=[24448], 99.90th=[32384], 99.
| 99.99th=[62208] | | 99.99th=[35584]
Type=read,Size=512B,NumJobs=16,depth=16 Type=read,Size=512B,NumJobs=16,depth=16
read: IOPS=4317k, BW=2108MiB/s (2211MB/s)(20.6GiB/10001mse | read: IOPS=4055k, BW=1980MiB/s (2076MB/s)(19.3GiB/10003mse
slat (nsec): min=1572, max=75553, avg=2642.25, stdev=1686 | slat (nsec): min=1572, max=239521, avg=2836.25, stdev=282
clat (usec): min=21, max=1043, avg=56.25, stdev=12.46 | clat (usec): min=9, max=616, avg=59.85, stdev=55.40
lat (usec): min=23, max=1046, avg=58.94, stdev=13.08 | lat (usec): min=15, max=630, avg=62.74, stdev=57.98
| 1.00th=[ 42], 5.00th=[ 43], 10.00th=[ 44], 20. | | 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 44], 20.
| 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 53], 60. | | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 46], 60.
| 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 72], 95. | | 70.00th=[ 48], 80.00th=[ 52], 90.00th=[ 60], 95.
| 99.00th=[ 87], 99.50th=[ 112], 99.90th=[ 124], 99. | | 99.00th=[ 326], 99.50th=[ 343], 99.90th=[ 367], 99.
| 99.99th=[ 289] | | 99.99th=[ 519]
Type=read,Size=512B,NumJobs=16,depth=64 Type=read,Size=512B,NumJobs=16,depth=64
read: IOPS=4484k, BW=2189MiB/s (2296MB/s)(21.4GiB/10001mse | read: IOPS=3451k, BW=1685MiB/s (1767MB/s)(16.5GiB/10001mse
slat (nsec): min=1566, max=71087, avg=2603.51, stdev=2276 | slat (nsec): min=1569, max=204726, avg=3322.74, stdev=274
clat (usec): min=19, max=1784, avg=225.36, stdev=53.73 | clat (usec): min=5, max=1543, avg=292.82, stdev=220.59
lat (usec): min=26, max=1807, avg=228.01, stdev=54.38 | lat (usec): min=14, max=1561, avg=296.20, stdev=223.03
| 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 176], 20. | | 1.00th=[ 176], 5.00th=[ 178], 10.00th=[ 180], 20.
| 30.00th=[ 182], 40.00th=[ 184], 50.00th=[ 231], 60. | | 30.00th=[ 188], 40.00th=[ 227], 50.00th=[ 243], 60.
| 70.00th=[ 247], 80.00th=[ 269], 90.00th=[ 302], 95. | | 70.00th=[ 249], 80.00th=[ 285], 90.00th=[ 685], 95.
| 99.00th=[ 375], 99.50th=[ 388], 99.90th=[ 490], 99. | | 99.00th=[ 1270], 99.50th=[ 1319], 99.90th=[ 1385], 99.
| 99.99th=[ 1631] | | 99.99th=[ 1450]
Type=read,Size=512B,NumJobs=56,depth=1 Type=read,Size=512B,NumJobs=56,depth=1
read: IOPS=2301k, BW=1123MiB/s (1178MB/s)(10.0GiB/10002mse | read: IOPS=2351k, BW=1148MiB/s (1204MB/s)(11.2GiB/10002mse
slat (nsec): min=1723, max=958519, avg=3130.18, stdev=300 | slat (nsec): min=1744, max=502891, avg=3525.21, stdev=199
clat (nsec): min=456, max=987494, avg=20564.41, stdev=170 | clat (nsec): min=456, max=781278, avg=19563.02, stdev=418
lat (usec): min=12, max=989, avg=23.77, stdev=17.43 | lat (usec): min=11, max=785, avg=23.18, stdev= 4.79
| 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20. | | 1.00th=[13120], 5.00th=[14912], 10.00th=[15936], 20.
| 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 20], 60. | | 30.00th=[17792], 40.00th=[18304], 50.00th=[19072], 60.
| 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 24], 95. | | 70.00th=[20608], 80.00th=[21632], 90.00th=[23680], 95.
| 99.00th=[ 37], 99.50th=[ 45], 99.90th=[ 367], 99. | | 99.00th=[31872], 99.50th=[35072], 99.90th=[41728], 99.
| 99.99th=[ 545] | | 99.99th=[56064]
Type=read,Size=512B,NumJobs=56,depth=16 Type=read,Size=512B,NumJobs=56,depth=16
read: IOPS=7830k, BW=3823MiB/s (4009MB/s)(37.3GiB/10002mse | read: IOPS=7234k, BW=3532MiB/s (3704MB/s)(34.5GiB/10002mse
slat (nsec): min=1591, max=2609.7M, avg=5126.50, stdev=10 | slat (nsec): min=1588, max=13073k, avg=5296.03, stdev=602
clat (usec): min=16, max=3527.6k, avg=108.61, stdev=6148. | clat (usec): min=9, max=13311, avg=117.66, stdev=77.29
lat (usec): min=25, max=3527.6k, avg=113.83, stdev=6235. | lat (usec): min=18, max=13322, avg=123.07, stdev=80.61
| 1.00th=[ 47], 5.00th=[ 54], 10.00th=[ 63], 20. | | 1.00th=[ 63], 5.00th=[ 67], 10.00th=[ 69], 20.
| 30.00th=[ 74], 40.00th=[ 77], 50.00th=[ 80], 60. | | 30.00th=[ 72], 40.00th=[ 76], 50.00th=[ 81], 60.
| 70.00th=[ 97], 80.00th=[ 100], 90.00th=[ 105], 95. | | 70.00th=[ 108], 80.00th=[ 184], 90.00th=[ 247], 95.
| 99.00th=[ 457], 99.50th=[ 519], 99.90th=[ 611], 99. | | 99.00th=[ 338], 99.50th=[ 383], 99.90th=[ 494], 99.
| 99.99th=[ 889] | | 99.99th=[ 594]
Type=read,Size=512B,NumJobs=56,depth=64 Type=read,Size=512B,NumJobs=56,depth=64
read: IOPS=8456k, BW=4129MiB/s (4330MB/s)(40.3GiB/10002mse | read: IOPS=7769k, BW=3793MiB/s (3978MB/s)(37.1GiB/10002mse
slat (nsec): min=1607, max=2432.8k, avg=4995.25, stdev=87 | slat (nsec): min=1583, max=3785.4k, avg=5319.20, stdev=81
clat (usec): min=28, max=3427, avg=418.02, stdev=272.37 | clat (usec): min=2, max=5212, avg=455.02, stdev=334.71
lat (usec): min=34, max=3431, avg=423.12, stdev=275.63 | lat (usec): min=17, max=5214, avg=460.45, stdev=338.64
| 1.00th=[ 281], 5.00th=[ 293], 10.00th=[ 302], 20. | | 1.00th=[ 265], 5.00th=[ 273], 10.00th=[ 281], 20.
| 30.00th=[ 314], 40.00th=[ 322], 50.00th=[ 383], 60. | | 30.00th=[ 293], 40.00th=[ 297], 50.00th=[ 302], 60.
| 70.00th=[ 408], 80.00th=[ 412], 90.00th=[ 429], 95. | | 70.00th=[ 310], 80.00th=[ 437], 90.00th=[ 1074], 95.
| 99.00th=[ 1909], 99.50th=[ 1991], 99.90th=[ 2180], 99. | | 99.00th=[ 1401], 99.50th=[ 1991], 99.90th=[ 2245], 99.
| 99.99th=[ 2442] | 99.99th=[ 2442]
Type=read,Size=4K,NumJobs=1,depth=1 Type=read,Size=4K,NumJobs=1,depth=1
read: IOPS=43.6k, BW=170MiB/s (178MB/s)(1701MiB/10001msec) | read: IOPS=54.4k, BW=213MiB/s (223MB/s)(2126MiB/10001msec)
slat (nsec): min=2041, max=41259, avg=3359.33, stdev=2676 | slat (nsec): min=2044, max=28781, avg=2556.03, stdev=1441
clat (nsec): min=449, max=95097, avg=18989.95, stdev=6853 | clat (nsec): min=621, max=70566, avg=15359.54, stdev=5250
lat (nsec): min=14883, max=99275, avg=22427.07, stdev=76 | lat (nsec): min=13302, max=73823, avg=17966.25, stdev=56
| 1.00th=[14400], 5.00th=[14656], 10.00th=[14656], 20. | | 1.00th=[13120], 5.00th=[13248], 10.00th=[13248], 20.
| 30.00th=[14912], 40.00th=[15040], 50.00th=[15296], 60. | | 30.00th=[13376], 40.00th=[13376], 50.00th=[13376], 60.
| 70.00th=[19072], 80.00th=[22144], 90.00th=[30592], 95. | | 70.00th=[13632], 80.00th=[14144], 90.00th=[19328], 95.
| 99.00th=[43776], 99.50th=[46848], 99.90th=[52480], 99. | | 99.00th=[35584], 99.50th=[36608], 99.90th=[45312], 99.
| 99.99th=[63744] | | 99.99th=[53504]
Type=read,Size=4K,NumJobs=1,depth=16 Type=read,Size=4K,NumJobs=1,depth=16
read: IOPS=345k, BW=1346MiB/s (1411MB/s)(13.1GiB/10001msec | read: IOPS=334k, BW=1306MiB/s (1370MB/s)(12.8GiB/10001msec
slat (nsec): min=1584, max=67990, avg=2073.10, stdev=588. | slat (nsec): min=1696, max=45538, avg=2168.76, stdev=208.
clat (usec): min=32, max=154, avg=44.06, stdev= 4.57 | clat (usec): min=18, max=148, avg=45.39, stdev= 1.47
lat (usec): min=35, max=156, avg=46.17, stdev= 4.62 | lat (usec): min=20, max=192, avg=47.59, stdev= 1.48
| 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 43], 20. | | 1.00th=[43776], 5.00th=[44288], 10.00th=[44288], 20.
| 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 43], 60. | | 30.00th=[44800], 40.00th=[45312], 50.00th=[45312], 60.
| 70.00th=[ 44], 80.00th=[ 44], 90.00th=[ 45], 95. | | 70.00th=[45312], 80.00th=[45824], 90.00th=[45824], 95.
| 99.00th=[ 67], 99.50th=[ 74], 99.90th=[ 86], 99. | | 99.00th=[49408], 99.50th=[54528], 99.90th=[64256], 99.
| 99.99th=[ 117] | | 99.99th=[70144]
Type=read,Size=4K,NumJobs=1,depth=64 Type=read,Size=4K,NumJobs=1,depth=64
read: IOPS=257k, BW=1004MiB/s (1053MB/s)(9.80GiB/10001msec | read: IOPS=338k, BW=1319MiB/s (1383MB/s)(12.9GiB/10001msec
slat (nsec): min=1562, max=50661, avg=2614.87, stdev=2515 | slat (nsec): min=1621, max=52632, avg=2145.29, stdev=209.
clat (usec): min=49, max=325, avg=245.85, stdev=11.72 | clat (usec): min=36, max=236, avg=187.02, stdev= 2.05
lat (usec): min=50, max=328, avg=248.54, stdev=11.76 | lat (usec): min=38, max=238, avg=189.20, stdev= 2.06
| 1.00th=[ 219], 5.00th=[ 227], 10.00th=[ 231], 20. | | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 186], 20.
| 30.00th=[ 241], 40.00th=[ 243], 50.00th=[ 245], 60. | | 30.00th=[ 186], 40.00th=[ 188], 50.00th=[ 188], 60.
| 70.00th=[ 253], 80.00th=[ 255], 90.00th=[ 262], 95. | | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 190], 95.
| 99.00th=[ 273], 99.50th=[ 273], 99.90th=[ 281], 99. | | 99.00th=[ 192], 99.50th=[ 196], 99.90th=[ 200], 99.
| 99.99th=[ 285] | | 99.99th=[ 217]
Type=read,Size=4K,NumJobs=16,depth=1 Type=read,Size=4K,NumJobs=16,depth=1
read: IOPS=849k, BW=3318MiB/s (3479MB/s)(32.4GiB/10002msec | read: IOPS=869k, BW=3396MiB/s (3561MB/s)(33.2GiB/10002msec
slat (nsec): min=1894, max=65881, avg=2560.26, stdev=657. | slat (nsec): min=1890, max=70947, avg=2497.58, stdev=813.
clat (nsec): min=516, max=708489, avg=15808.69, stdev=196 | clat (nsec): min=590, max=79209, avg=15456.14, stdev=1968
lat (usec): min=13, max=710, avg=18.42, stdev= 2.19 | lat (usec): min=12, max=110, avg=18.01, stdev= 2.18
| 1.00th=[13120], 5.00th=[14016], 10.00th=[14272], 20. | | 1.00th=[12352], 5.00th=[13888], 10.00th=[14144], 20.
| 30.00th=[15040], 40.00th=[15424], 50.00th=[15680], 60. | | 30.00th=[14528], 40.00th=[14784], 50.00th=[15040], 60.
| 70.00th=[16320], 80.00th=[16768], 90.00th=[17280], 95. | | 70.00th=[15552], 80.00th=[16192], 90.00th=[17280], 95.
| 99.00th=[19840], 99.50th=[21120], 99.90th=[34560], 99. | | 99.00th=[23936], 99.50th=[25728], 99.90th=[33536], 99.
| 99.99th=[76288] | | 99.99th=[36096]
Type=read,Size=4K,NumJobs=16,depth=16 Type=read,Size=4K,NumJobs=16,depth=16
read: IOPS=2593k, BW=9.89GiB/s (10.6GB/s)(98.9GiB/10002mse | read: IOPS=1843k, BW=7201MiB/s (7551MB/s)(70.3GiB/10002mse
slat (nsec): min=1568, max=163092, avg=2599.50, stdev=145 | slat (nsec): min=1582, max=104435, avg=3170.92, stdev=170
clat (usec): min=34, max=4287.1k, avg=95.73, stdev=9113.1 | clat (usec): min=40, max=554, avg=135.18, stdev=32.57
lat (usec): min=36, max=4287.1k, avg=98.38, stdev=9113.1 | lat (usec): min=42, max=557, avg=138.42, stdev=32.59
| 1.00th=[ 43], 5.00th=[ 53], 10.00th=[ 57], 20. | | 1.00th=[ 68], 5.00th=[ 92], 10.00th=[ 104], 20.
| 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 73], 60. | | 30.00th=[ 125], 40.00th=[ 130], 50.00th=[ 135], 60.
| 70.00th=[ 80], 80.00th=[ 88], 90.00th=[ 96], 95. | | 70.00th=[ 141], 80.00th=[ 147], 90.00th=[ 161], 95.
| 99.00th=[ 104], 99.50th=[ 106], 99.90th=[ 126], 99. | | 99.00th=[ 262], 99.50th=[ 306], 99.90th=[ 392], 99.
| 99.99th=[ 176] | | 99.99th=[ 461]
Type=read,Size=4K,NumJobs=16,depth=64 Type=read,Size=4K,NumJobs=16,depth=64
read: IOPS=2646k, BW=10.1GiB/s (10.8GB/s)(101GiB/10001msec | read: IOPS=1917k, BW=7487MiB/s (7850MB/s)(73.1GiB/10001mse
slat (nsec): min=1566, max=100447, avg=2310.80, stdev=263 | slat (nsec): min=1566, max=653412, avg=3347.14, stdev=385
clat (usec): min=21, max=1440, avg=384.23, stdev=61.42 | clat (usec): min=43, max=1445, avg=530.39, stdev=127.84
lat (usec): min=24, max=1443, avg=386.58, stdev=62.18 | lat (usec): min=45, max=1619, avg=533.81, stdev=128.49
| 1.00th=[ 330], 5.00th=[ 347], 10.00th=[ 347], 20. | | 1.00th=[ 188], 5.00th=[ 351], 10.00th=[ 420], 20.
| 30.00th=[ 363], 40.00th=[ 371], 50.00th=[ 379], 60. | | 30.00th=[ 486], 40.00th=[ 502], 50.00th=[ 519], 60.
| 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 396], 95. | | 70.00th=[ 553], 80.00th=[ 578], 90.00th=[ 644], 95.
| 99.00th=[ 717], 99.50th=[ 725], 99.90th=[ 750], 99. | | 99.00th=[ 1012], 99.50th=[ 1037], 99.90th=[ 1090], 99.
| 99.99th=[ 766] | | 99.99th=[ 1205]
Type=read,Size=4K,NumJobs=56,depth=1 Type=read,Size=4K,NumJobs=56,depth=1
read: IOPS=1514k, BW=5912MiB/s (6200MB/s)(57.7GiB/10002mse | read: IOPS=1571k, BW=6138MiB/s (6436MB/s)(59.9GiB/10001mse
slat (nsec): min=1722, max=911960, avg=2705.80, stdev=141 | slat (nsec): min=1693, max=505053, avg=2771.65, stdev=110
clat (nsec): min=466, max=4359.5k, avg=33747.42, stdev=23 | clat (nsec): min=479, max=1348.1k, avg=32311.24, stdev=36
lat (usec): min=13, max=4362, avg=36.52, stdev=23.37 | lat (usec): min=15, max=1350, avg=35.15, stdev= 3.81
| 1.00th=[ 20], 5.00th=[ 24], 10.00th=[ 27], 20. | | 1.00th=[26752], 5.00th=[28800], 10.00th=[29568], 20.
| 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 34], 60. | | 30.00th=[30848], 40.00th=[31360], 50.00th=[31872], 60.
| 70.00th=[ 35], 80.00th=[ 36], 90.00th=[ 37], 95. | | 70.00th=[33024], 80.00th=[33536], 90.00th=[35584], 95.
| 99.00th=[ 48], 99.50th=[ 143], 99.90th=[ 433], 99. | | 99.00th=[42240], 99.50th=[43776], 99.90th=[49920], 99.
| 99.99th=[ 594] | | 99.99th=[59136]
Type=read,Size=4K,NumJobs=56,depth=16 Type=read,Size=4K,NumJobs=56,depth=16
read: IOPS=2381k, BW=9299MiB/s (9751MB/s)(97.7GiB/10756mse | read: IOPS=2009k, BW=7846MiB/s (8227MB/s)(76.6GiB/10001mse
slat (nsec): min=1573, max=1580.7M, avg=2676.88, stdev=44 | slat (nsec): min=1566, max=379585, avg=3204.59, stdev=230
clat (usec): min=17, max=4303.4k, avg=355.77, stdev=27892 | clat (usec): min=44, max=1761, avg=442.22, stdev=244.27
lat (usec): min=20, max=4303.4k, avg=358.50, stdev=27901 | lat (usec): min=46, max=1764, avg=445.50, stdev=244.24
| 1.00th=[ 44], 5.00th=[ 55], 10.00th=[ 62], 20. | | 1.00th=[ 161], 5.00th=[ 194], 10.00th=[ 215], 20.
| 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 81], 60. | | 30.00th=[ 273], 40.00th=[ 306], 50.00th=[ 351], 60.
| 70.00th=[ 109], 80.00th=[ 135], 90.00th=[ 167], 95. | | 70.00th=[ 506], 80.00th=[ 644], 90.00th=[ 832], 95.
| 99.00th=[ 285], 99.50th=[ 400], 99.90th=[ 506], 99. | | 99.00th=[ 1156], 99.50th=[ 1221], 99.90th=[ 1336], 99.
| 99.99th=[ 1172] | | 99.99th=[ 1450]
Type=read,Size=4K,NumJobs=56,depth=64 Type=read,Size=4K,NumJobs=56,depth=64
read: IOPS=2496k, BW=9749MiB/s (10.2GB/s)(95.2GiB/10003mse | read: IOPS=1801k, BW=7036MiB/s (7378MB/s)(68.7GiB/10002mse
slat (nsec): min=1568, max=2721.3M, avg=14509.11, stdev=1 | slat (nsec): min=1586, max=2410.2k, avg=20136.27, stdev=8
clat (usec): min=35, max=2725.6k, avg=1420.77, stdev=1481 | clat (usec): min=58, max=8215, avg=1968.75, stdev=947.22
lat (usec): min=42, max=2725.9k, avg=1435.33, stdev=1491 | lat (usec): min=60, max=8632, avg=1988.96, stdev=962.84
| 1.00th=[ 570], 5.00th=[ 611], 10.00th=[ 676], 20. | | 1.00th=[ 449], 5.00th=[ 734], 10.00th=[ 865], 20.
| 30.00th=[ 742], 40.00th=[ 1221], 50.00th=[ 1369], 60. | | 30.00th=[ 1418], 40.00th=[ 1729], 50.00th=[ 1909], 60.
| 70.00th=[ 1434], 80.00th=[ 1500], 90.00th=[ 2147], 95. | | 70.00th=[ 2212], 80.00th=[ 2507], 90.00th=[ 3326], 95.
| 99.00th=[ 3163], 99.50th=[ 3261], 99.90th=[ 3654], 99. | | 99.00th=[ 4817], 99.50th=[ 5080], 99.90th=[ 5735], 99.
| 99.99th=[ 4686] | | 99.99th=[ 6587]
Type=read,Size=64K,NumJobs=1,depth=1 Type=read,Size=64K,NumJobs=1,depth=1
read: IOPS=29.7k, BW=1854MiB/s (1944MB/s)(18.1GiB/10001mse | read: IOPS=29.8k, BW=1864MiB/s (1955MB/s)(18.2GiB/10001mse
slat (nsec): min=4746, max=53468, avg=6046.72, stdev=2806 | slat (nsec): min=4768, max=95484, avg=6119.74, stdev=2779
clat (usec): min=3, max=132, avg=27.14, stdev= 7.87 | clat (nsec): min=1555, max=75556, avg=26881.73, stdev=781
lat (usec): min=25, max=137, avg=33.23, stdev= 9.11 | lat (usec): min=25, max=102, avg=33.05, stdev= 9.07
| 1.00th=[21888], 5.00th=[21888], 10.00th=[21888], 20. | | 1.00th=[21632], 5.00th=[21632], 10.00th=[21632], 20.
| 30.00th=[22144], 40.00th=[22144], 50.00th=[22400], 60. | | 30.00th=[21888], 40.00th=[22144], 50.00th=[22144], 60.
| 70.00th=[28288], 80.00th=[35072], 90.00th=[41216], 95. | | 70.00th=[28032], 80.00th=[34048], 90.00th=[40704], 95.
| 99.00th=[47360], 99.50th=[49920], 99.90th=[61696], 99. | | 99.00th=[47360], 99.50th=[49920], 99.90th=[59648], 99.
| 99.99th=[70144] | 99.99th=[70144]
Type=read,Size=64K,NumJobs=1,depth=16 Type=read,Size=64K,NumJobs=1,depth=16
read: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10001msec) read: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10001msec)
slat (nsec): min=3300, max=48724, avg=4702.06, stdev=397. | slat (nsec): min=3756, max=65673, avg=4930.28, stdev=334.
clat (usec): min=59, max=241, avg=88.09, stdev= 4.24 | clat (usec): min=83, max=243, avg=87.86, stdev= 2.04
lat (usec): min=65, max=277, avg=92.83, stdev= 4.22 | lat (usec): min=87, max=310, avg=92.83, stdev= 2.05
| 1.00th=[ 82], 5.00th=[ 83], 10.00th=[ 83], 20. | | 1.00th=[ 86], 5.00th=[ 87], 10.00th=[ 87], 20.
| 30.00th=[ 87], 40.00th=[ 88], 50.00th=[ 88], 60. | | 30.00th=[ 87], 40.00th=[ 88], 50.00th=[ 88], 60.
| 70.00th=[ 91], 80.00th=[ 92], 90.00th=[ 94], 95. | | 70.00th=[ 88], 80.00th=[ 89], 90.00th=[ 91], 95.
| 99.00th=[ 98], 99.50th=[ 99], 99.90th=[ 111], 99. | | 99.00th=[ 96], 99.50th=[ 99], 99.90th=[ 108], 99.
| 99.99th=[ 145] | | 99.99th=[ 114]
Type=read,Size=64K,NumJobs=1,depth=64 Type=read,Size=64K,NumJobs=1,depth=64
read: IOPS=172k, BW=10.5GiB/s (11.2GB/s)(105GiB/10001msec) | read: IOPS=168k, BW=10.2GiB/s (10.0GB/s)(102GiB/10001msec)
slat (usec): min=2, max=420, avg= 4.14, stdev= 3.46 | slat (usec): min=2, max=390, avg= 4.15, stdev= 1.56
clat (usec): min=299, max=1058, avg=368.25, stdev=18.60 | clat (usec): min=37, max=947, avg=376.95, stdev=15.70
lat (usec): min=306, max=1099, avg=372.44, stdev=18.47 | lat (usec): min=42, max=988, avg=381.15, stdev=16.25
| 1.00th=[ 326], 5.00th=[ 343], 10.00th=[ 351], 20. | | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 367], 20.
| 30.00th=[ 359], 40.00th=[ 367], 50.00th=[ 367], 60. | | 30.00th=[ 367], 40.00th=[ 367], 50.00th=[ 371], 60.
| 70.00th=[ 375], 80.00th=[ 379], 90.00th=[ 388], 95. | | 70.00th=[ 388], 80.00th=[ 392], 90.00th=[ 400], 95.
| 99.00th=[ 433], 99.50th=[ 445], 99.90th=[ 465], 99. | | 99.00th=[ 412], 99.50th=[ 416], 99.90th=[ 429], 99.
| 99.99th=[ 758] | | 99.99th=[ 668]
Type=read,Size=64K,NumJobs=16,depth=1 Type=read,Size=64K,NumJobs=16,depth=1
read: IOPS=150k, BW=9390MiB/s (9847MB/s)(105GiB/11428msec) | read: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10002msec)
slat (nsec): min=3750, max=58669, avg=5120.99, stdev=2896 | slat (nsec): min=3480, max=74756, avg=5273.21, stdev=2961
clat (nsec): min=494, max=2276.2M, avg=88365.74, stdev=17 | clat (nsec): min=616, max=453786, avg=87271.90, stdev=161
lat (usec): min=26, max=2276.2k, avg=93.55, stdev=1737.1 | lat (usec): min=26, max=458, avg=92.60, stdev=16.20
| 1.00th=[ 36], 5.00th=[ 55], 10.00th=[ 68], 20. | | 1.00th=[ 37], 5.00th=[ 57], 10.00th=[ 68], 20.
| 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 89], 60. | | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 90], 60.
| 70.00th=[ 95], 80.00th=[ 98], 90.00th=[ 103], 95. | | 70.00th=[ 95], 80.00th=[ 98], 90.00th=[ 104], 95.
| 99.00th=[ 124], 99.50th=[ 129], 99.90th=[ 141], 99. | | 99.00th=[ 126], 99.50th=[ 130], 99.90th=[ 143], 99.
| 99.99th=[ 165] | | 99.99th=[ 161]
Type=read,Size=64K,NumJobs=16,depth=16 Type=read,Size=64K,NumJobs=16,depth=16
read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10003msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec)
slat (nsec): min=3284, max=53892, avg=4935.30, stdev=1716 | slat (nsec): min=3866, max=57337, avg=5204.47, stdev=1220
clat (usec): min=102, max=4744, avg=1455.23, stdev=493.14 | clat (usec): min=116, max=3985, avg=1453.77, stdev=349.92
lat (usec): min=116, max=4749, avg=1460.22, stdev=493.10 | lat (usec): min=128, max=3990, avg=1459.03, stdev=349.93
| 1.00th=[ 1172], 5.00th=[ 1172], 10.00th=[ 1188], 20. | | 1.00th=[ 1254], 5.00th=[ 1270], 10.00th=[ 1270], 20.
| 30.00th=[ 1287], 40.00th=[ 1352], 50.00th=[ 1352], 60. | | 30.00th=[ 1352], 40.00th=[ 1352], 50.00th=[ 1352], 60.
| 70.00th=[ 1369], 80.00th=[ 1385], 90.00th=[ 1401], 95. | | 70.00th=[ 1369], 80.00th=[ 1401], 90.00th=[ 1467], 95.
| 99.00th=[ 3818], 99.50th=[ 3851], 99.90th=[ 3851], 99. | | 99.00th=[ 2737], 99.50th=[ 2769], 99.90th=[ 2769], 99.
| 99.99th=[ 3884] | | 99.99th=[ 2802]
Type=read,Size=64K,NumJobs=16,depth=64 Type=read,Size=64K,NumJobs=16,depth=64
read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec) read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec)
slat (usec): min=2, max=803, avg=16.08, stdev=83.27 | slat (usec): min=2, max=1185, avg= 9.71, stdev=60.22
clat (usec): min=109, max=15039, avg=5820.28, stdev=1740. | clat (usec): min=116, max=16490, avg=5824.81, stdev=1345.
lat (usec): min=116, max=15042, avg=5836.41, stdev=1769. | lat (usec): min=133, max=16494, avg=5834.58, stdev=1367.
| 1.00th=[ 4686], 5.00th=[ 4686], 10.00th=[ 4752], 20. | | 1.00th=[ 5407], 5.00th=[ 5407], 10.00th=[ 5473], 20.
| 30.00th=[ 4752], 40.00th=[ 5080], 50.00th=[ 5080], 60. | | 30.00th=[ 5473], 40.00th=[ 5473], 50.00th=[ 5473], 60.
| 70.00th=[ 5473], 80.00th=[ 5866], 90.00th=[ 9503], 95. | | 70.00th=[ 5473], 80.00th=[ 5473], 90.00th=[ 5538], 95.
| 99.00th=[10552], 99.50th=[10945], 99.90th=[10945], 99. | | 99.00th=[10945], 99.50th=[10945], 99.90th=[11076], 99.
| 99.99th=[11076] | | 99.99th=[11338]
Type=read,Size=64K,NumJobs=56,depth=1 Type=read,Size=64K,NumJobs=56,depth=1
read: IOPS=160k, BW=9979MiB/s (10.5GB/s)(107GiB/10962msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec)
slat (nsec): min=3668, max=66111, avg=4685.24, stdev=1778 | slat (nsec): min=3801, max=63985, avg=4896.45, stdev=1489
clat (usec): min=36, max=3544.7k, avg=317.51, stdev=6531. | clat (usec): min=29, max=550, avg=314.45, stdev=76.21
lat (usec): min=42, max=3544.7k, avg=322.25, stdev=6531. | lat (usec): min=35, max=555, avg=319.41, stdev=76.15
| 1.00th=[ 161], 5.00th=[ 169], 10.00th=[ 174], 20. | | 1.00th=[ 174], 5.00th=[ 178], 10.00th=[ 194], 20.
| 30.00th=[ 285], 40.00th=[ 293], 50.00th=[ 297], 60. | | 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 306], 60.
| 70.00th=[ 306], 80.00th=[ 318], 90.00th=[ 445], 95. | | 70.00th=[ 310], 80.00th=[ 322], 90.00th=[ 457], 95.
| 99.00th=[ 603], 99.50th=[ 611], 99.90th=[ 627], 99. | | 99.00th=[ 474], 99.50th=[ 482], 99.90th=[ 490], 99.
| 99.99th=[ 644] | | 99.99th=[ 506]
Type=read,Size=64K,NumJobs=56,depth=16 Type=read,Size=64K,NumJobs=56,depth=16
read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10007msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec)
slat (usec): min=3, max=171, avg= 4.99, stdev= 2.81 | slat (usec): min=3, max=251, avg= 5.40, stdev= 2.34
clat (usec): min=784, max=14310, avg=5117.02, stdev=1736. | clat (usec): min=236, max=14486, avg=5106.35, stdev=1241.
lat (usec): min=884, max=14315, avg=5122.07, stdev=1736. | lat (usec): min=265, max=14491, avg=5111.80, stdev=1241.
| 1.00th=[ 2409], 5.00th=[ 2442], 10.00th=[ 2474], 20. | | 1.00th=[ 2442], 5.00th=[ 2540], 10.00th=[ 4883], 20.
| 30.00th=[ 4883], 40.00th=[ 4948], 50.00th=[ 4948], 60. | | 30.00th=[ 4948], 40.00th=[ 5080], 50.00th=[ 5080], 60.
| 70.00th=[ 5014], 80.00th=[ 5145], 90.00th=[ 7439], 95. | | 70.00th=[ 5080], 80.00th=[ 5145], 90.00th=[ 7373], 95.
| 99.00th=[ 9896], 99.50th=[ 9896], 99.90th=[10159], 99. | | 99.00th=[ 9896], 99.50th=[ 9896], 99.90th=[ 9896], 99.
| 99.99th=[10290] | | 99.99th=[10028]
Type=read,Size=64K,NumJobs=56,depth=64 Type=read,Size=64K,NumJobs=56,depth=64
read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10016msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10018msec)
slat (usec): min=2, max=3494.5k, avg=194.57, stdev=10422. | slat (usec): min=2, max=10184, avg=189.22, stdev=637.78
clat (usec): min=138, max=3513.0k, avg=20256.38, stdev=11 | clat (usec): min=347, max=61952, avg=20252.15, stdev=7056
lat (usec): min=178, max=3513.0k, avg=20451.00, stdev=11 | lat (usec): min=351, max=65141, avg=20441.44, stdev=7203
| 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20. | | 1.00th=[ 9503], 5.00th=[ 9765], 10.00th=[ 9896], 20.
| 30.00th=[ 10], 40.00th=[ 15], 50.00th=[ 19], 60. | | 30.00th=[19530], 40.00th=[19530], 50.00th=[19792], 60.
| 70.00th=[ 20], 80.00th=[ 20], 90.00th=[ 28], 95. | | 70.00th=[20317], 80.00th=[20579], 90.00th=[29492], 95.
| 99.00th=[ 40], 99.50th=[ 40], 99.90th=[ 3473], 99. | | 99.00th=[41157], 99.50th=[58983], 99.90th=[59507], 99.
| 99.99th=[ 3507] | | 99.99th=[61080]
Type=write,Size=512B,NumJobs=1,depth=1 Type=write,Size=512B,NumJobs=1,depth=1
write: IOPS=46.5k, BW=22.7MiB/s (23.8MB/s)(227MiB/10001msec | write: IOPS=56.0k, BW=27.4MiB/s (28.7MB/s)(274MiB/10001msec
slat (nsec): min=2038, max=46160, avg=3144.53, stdev=2438 | slat (nsec): min=2112, max=43957, avg=2561.65, stdev=1423
clat (nsec): min=449, max=133529, avg=17730.81, stdev=670 | clat (nsec): min=633, max=60031, avg=14812.68, stdev=5638
lat (usec): min=13, max=136, avg=20.95, stdev= 7.40 | lat (nsec): min=12369, max=72124, avg=17416.39, stdev=60
| 1.00th=[13504], 5.00th=[13504], 10.00th=[13632], 20. | | 1.00th=[12352], 5.00th=[12352], 10.00th=[12480], 20.
| 30.00th=[13760], 40.00th=[13888], 50.00th=[14016], 60. | | 30.00th=[12480], 40.00th=[12608], 50.00th=[12608], 60.
| 70.00th=[17792], 80.00th=[20608], 90.00th=[28800], 95. | | 70.00th=[12864], 80.00th=[14656], 90.00th=[20096], 95.
| 99.00th=[41728], 99.50th=[45824], 99.90th=[51456], 99. | | 99.00th=[35072], 99.50th=[36608], 99.90th=[47360], 99.
| 99.99th=[64256] | | 99.99th=[54016]
Type=write,Size=512B,NumJobs=1,depth=16 Type=write,Size=512B,NumJobs=1,depth=16
write: IOPS=255k, BW=125MiB/s (131MB/s)(1248MiB/10001msec) | write: IOPS=328k, BW=160MiB/s (168MB/s)(1601MiB/10001msec)
slat (nsec): min=1628, max=69589, avg=2649.87, stdev=2876 | slat (nsec): min=1641, max=32941, avg=2199.97, stdev=236.
clat (usec): min=23, max=182, avg=59.49, stdev= 8.44 | clat (usec): min=14, max=113, avg=46.29, stdev= 1.27
lat (usec): min=25, max=184, avg=62.20, stdev= 8.26 | lat (usec): min=16, max=115, avg=48.52, stdev= 1.28
| 1.00th=[ 45], 5.00th=[ 46], 10.00th=[ 48], 20. | | 1.00th=[43776], 5.00th=[44800], 10.00th=[45312], 20.
| 30.00th=[ 58], 40.00th=[ 59], 50.00th=[ 60], 60. | | 30.00th=[45824], 40.00th=[45824], 50.00th=[46336], 60.
| 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 70], 95. | | 70.00th=[46336], 80.00th=[46848], 90.00th=[47360], 95.
| 99.00th=[ 85], 99.50th=[ 90], 99.90th=[ 103], 99. | | 99.00th=[50432], 99.50th=[53504], 99.90th=[57600], 99.
| 99.99th=[ 139] | | 99.99th=[63744]
Type=write,Size=512B,NumJobs=1,depth=64 Type=write,Size=512B,NumJobs=1,depth=64
write: IOPS=262k, BW=128MiB/s (134MB/s)(1277MiB/10001msec) | write: IOPS=337k, BW=164MiB/s (172MB/s)(1644MiB/10001msec)
slat (nsec): min=1626, max=45520, avg=2718.39, stdev=3134 | slat (nsec): min=1642, max=43437, avg=2146.30, stdev=234.
clat (usec): min=31, max=304, avg=241.50, stdev=10.69 | clat (usec): min=16, max=235, avg=187.62, stdev= 2.46
lat (usec): min=33, max=307, avg=244.27, stdev=10.71 | lat (usec): min=18, max=237, avg=189.80, stdev= 2.48
| 1.00th=[ 221], 5.00th=[ 225], 10.00th=[ 229], 20. | | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 186], 20.
| 30.00th=[ 235], 40.00th=[ 237], 50.00th=[ 241], 60. | | 30.00th=[ 186], 40.00th=[ 188], 50.00th=[ 188], 60.
| 70.00th=[ 247], 80.00th=[ 251], 90.00th=[ 258], 95. | | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 192], 95.
| 99.00th=[ 265], 99.50th=[ 265], 99.90th=[ 269], 99. | | 99.00th=[ 194], 99.50th=[ 196], 99.90th=[ 200], 99.
| 99.99th=[ 277] | | 99.99th=[ 204]
Type=write,Size=512B,NumJobs=16,depth=1 Type=write,Size=512B,NumJobs=16,depth=1
write: IOPS=886k, BW=433MiB/s (454MB/s)(4327MiB/10002msec) | write: IOPS=914k, BW=446MiB/s (468MB/s)(4462MiB/10001msec)
slat (nsec): min=1827, max=51604, avg=2633.04, stdev=591. | slat (nsec): min=1818, max=83376, avg=2566.34, stdev=824.
clat (nsec): min=515, max=695999, avg=14905.52, stdev=209 | clat (nsec): min=465, max=343253, avg=14459.07, stdev=181
lat (usec): min=9, max=698, avg=17.60, stdev= 2.26 | lat (usec): min=11, max=345, avg=17.08, stdev= 2.09
| 1.00th=[12736], 5.00th=[13376], 10.00th=[13632], 20. | | 1.00th=[12224], 5.00th=[12992], 10.00th=[13248], 20.
| 30.00th=[14400], 40.00th=[14528], 50.00th=[14784], 60. | | 30.00th=[13632], 40.00th=[13760], 50.00th=[14016], 60.
| 70.00th=[15168], 80.00th=[15552], 90.00th=[16064], 95. | | 70.00th=[14528], 80.00th=[15040], 90.00th=[16320], 95.
| 99.00th=[18560], 99.50th=[19840], 99.90th=[28544], 99. | | 99.00th=[21888], 99.50th=[23936], 99.90th=[32384], 99.
| 99.99th=[87552] | | 99.99th=[35584]
Type=write,Size=512B,NumJobs=16,depth=16 Type=write,Size=512B,NumJobs=16,depth=16
write: IOPS=4155k, BW=2029MiB/s (2127MB/s)(19.8GiB/10002mse | write: IOPS=3218k, BW=1571MiB/s (1648MB/s)(15.3GiB/10001mse
slat (nsec): min=1622, max=171986, avg=2797.12, stdev=184 | slat (nsec): min=1621, max=154997, avg=3548.60, stdev=323
clat (usec): min=18, max=1041, avg=58.37, stdev=10.11 | clat (usec): min=14, max=473, avg=75.44, stdev=65.86
lat (usec): min=20, max=1043, avg=61.22, stdev=10.54 | lat (usec): min=17, max=489, avg=79.05, stdev=68.82
| 1.00th=[ 44], 5.00th=[ 45], 10.00th=[ 47], 20. | | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 46], 20.
| 30.00th=[ 50], 40.00th=[ 56], 50.00th=[ 60], 60. | | 30.00th=[ 47], 40.00th=[ 48], 50.00th=[ 52], 60.
| 70.00th=[ 65], 80.00th=[ 67], 90.00th=[ 73], 95. | | 70.00th=[ 61], 80.00th=[ 64], 90.00th=[ 178], 95.
| 99.00th=[ 81], 99.50th=[ 82], 99.90th=[ 86], 99. | | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 351], 99.
| 99.99th=[ 293] | | 99.99th=[ 367]
Type=write,Size=512B,NumJobs=16,depth=64 Type=write,Size=512B,NumJobs=16,depth=64
write: IOPS=4304k, BW=2102MiB/s (2204MB/s)(20.5GiB/10001mse | write: IOPS=3973k, BW=1940MiB/s (2034MB/s)(18.0GiB/10002mse
slat (nsec): min=1621, max=167405, avg=2712.72, stdev=195 | slat (nsec): min=1627, max=171980, avg=2926.70, stdev=209
clat (usec): min=23, max=1767, avg=234.76, stdev=51.39 | clat (usec): min=10, max=1606, avg=254.29, stdev=176.36
lat (usec): min=25, max=1788, avg=237.52, stdev=52.04 | lat (usec): min=16, max=1615, avg=257.27, stdev=178.32
| 1.00th=[ 180], 5.00th=[ 182], 10.00th=[ 184], 20. | | 1.00th=[ 184], 5.00th=[ 186], 10.00th=[ 188], 20.
| 30.00th=[ 188], 40.00th=[ 215], 50.00th=[ 237], 60. | | 30.00th=[ 194], 40.00th=[ 202], 50.00th=[ 237], 60.
| 70.00th=[ 260], 80.00th=[ 265], 90.00th=[ 310], 95. | | 70.00th=[ 255], 80.00th=[ 260], 90.00th=[ 293], 95.
| 99.00th=[ 359], 99.50th=[ 404], 99.90th=[ 412], 99. | | 99.00th=[ 1385], 99.50th=[ 1418], 99.90th=[ 1467], 99.
| 99.99th=[ 1565] | | 99.99th=[ 1500]
Type=write,Size=512B,NumJobs=56,depth=1 Type=write,Size=512B,NumJobs=56,depth=1
write: IOPS=2325k, BW=1135MiB/s (1191MB/s)(11.1GiB/10001mse | write: IOPS=2369k, BW=1157MiB/s (1213MB/s)(11.3GiB/10002mse
slat (nsec): min=1761, max=894949, avg=3151.04, stdev=221 | slat (nsec): min=1823, max=501953, avg=3567.43, stdev=201
clat (nsec): min=443, max=1452.0k, avg=20283.73, stdev=15 | clat (nsec): min=449, max=1722.5k, avg=19318.77, stdev=42
lat (usec): min=12, max=1454, avg=23.51, stdev=16.01 | lat (usec): min=11, max=1724, avg=22.98, stdev= 4.83
| 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20. | | 1.00th=[12992], 5.00th=[14656], 10.00th=[15680], 20.
| 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 20], 60. | | 30.00th=[17536], 40.00th=[18048], 50.00th=[18816], 60.
| 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 24], 95. | | 70.00th=[20352], 80.00th=[21632], 90.00th=[23424], 95.
| 99.00th=[ 34], 99.50th=[ 40], 99.90th=[ 347], 99. | | 99.00th=[31872], 99.50th=[35584], 99.90th=[41728], 99.
| 99.99th=[ 519] | | 99.99th=[54016]
Type=write,Size=512B,NumJobs=56,depth=16 Type=write,Size=512B,NumJobs=56,depth=16
write: IOPS=7222k, BW=3526MiB/s (3698MB/s)(34.4GiB/10001mse | write: IOPS=6237k, BW=3046MiB/s (3193MB/s)(29.7GiB/10002mse
slat (nsec): min=1630, max=325931, avg=4224.26, stdev=656 | slat (nsec): min=1640, max=2381.5k, avg=5462.05, stdev=67
clat (usec): min=15, max=5606, avg=119.11, stdev=48.84 | clat (usec): min=5, max=10435, avg=137.19, stdev=82.68
lat (usec): min=26, max=5610, avg=123.43, stdev=49.91 | lat (usec): min=15, max=10447, avg=142.77, stdev=85.58
| 1.00th=[ 68], 5.00th=[ 74], 10.00th=[ 78], 20. | | 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 70], 20.
| 30.00th=[ 93], 40.00th=[ 99], 50.00th=[ 104], 60. | | 30.00th=[ 85], 40.00th=[ 96], 50.00th=[ 104], 60.
| 70.00th=[ 126], 80.00th=[ 145], 90.00th=[ 176], 95. | | 70.00th=[ 149], 80.00th=[ 210], 90.00th=[ 265], 95.
| 99.00th=[ 318], 99.50th=[ 351], 99.90th=[ 420], 99. | | 99.00th=[ 363], 99.50th=[ 396], 99.90th=[ 553], 99.
| 99.99th=[ 603] | | 99.99th=[ 873]
Type=write,Size=512B,NumJobs=56,depth=64 Type=write,Size=512B,NumJobs=56,depth=64
write: IOPS=8407k, BW=4105MiB/s (4304MB/s)(40.1GiB/10002mse | write: IOPS=7488k, BW=3656MiB/s (3834MB/s)(35.7GiB/10002mse
slat (nsec): min=1642, max=22197k, avg=4988.96, stdev=171 | slat (nsec): min=1666, max=7929.3k, avg=5382.53, stdev=12
clat (usec): min=21, max=22624, avg=420.46, stdev=251.29 | clat (usec): min=14, max=12517, avg=472.19, stdev=331.30
lat (usec): min=25, max=22628, avg=425.55, stdev=253.99 | lat (usec): min=17, max=12520, avg=477.70, stdev=334.84
| 1.00th=[ 277], 5.00th=[ 293], 10.00th=[ 302], 20. | | 1.00th=[ 265], 5.00th=[ 277], 10.00th=[ 289], 20.
| 30.00th=[ 322], 40.00th=[ 338], 50.00th=[ 359], 60. | | 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 314], 60.
| 70.00th=[ 404], 80.00th=[ 441], 90.00th=[ 482], 95. | | 70.00th=[ 347], 80.00th=[ 562], 90.00th=[ 1057], 95.
| 99.00th=[ 1729], 99.50th=[ 1860], 99.90th=[ 2024], 99. | | 99.00th=[ 1434], 99.50th=[ 1696], 99.90th=[ 2040], 99.
| 99.99th=[ 2606] | | 99.99th=[ 2474]
Type=write,Size=4K,NumJobs=1,depth=1 Type=write,Size=4K,NumJobs=1,depth=1
write: IOPS=44.4k, BW=173MiB/s (182MB/s)(1735MiB/10001msec) | write: IOPS=55.8k, BW=218MiB/s (229MB/s)(2180MiB/10000msec)
slat (nsec): min=2009, max=36370, avg=3363.25, stdev=2555 | slat (nsec): min=2138, max=44613, avg=2552.61, stdev=1404
clat (nsec): min=453, max=106416, avg=18501.98, stdev=678 | clat (nsec): min=627, max=624863, avg=14910.38, stdev=517
lat (usec): min=14, max=109, avg=21.94, stdev= 7.54 | lat (usec): min=13, max=627, avg=17.50, stdev= 5.55
| 1.00th=[12864], 5.00th=[14016], 10.00th=[14144], 20. | | 1.00th=[12736], 5.00th=[12864], 10.00th=[12864], 20.
| 30.00th=[14272], 40.00th=[14400], 50.00th=[14656], 60. | | 30.00th=[12992], 40.00th=[12992], 50.00th=[12992], 60.
| 70.00th=[18816], 80.00th=[21632], 90.00th=[30080], 95. | | 70.00th=[13248], 80.00th=[13632], 90.00th=[20352], 95.
| 99.00th=[42240], 99.50th=[46336], 99.90th=[51456], 99. | | 99.00th=[35584], 99.50th=[37120], 99.90th=[44800], 99.
| 99.99th=[64768] | | 99.99th=[53504]
Type=write,Size=4K,NumJobs=1,depth=16 Type=write,Size=4K,NumJobs=1,depth=16
write: IOPS=245k, BW=956MiB/s (1002MB/s)(9556MiB/10001msec) | write: IOPS=325k, BW=1271MiB/s (1332MB/s)(12.4GiB/10001msec
slat (nsec): min=1646, max=69111, avg=2966.78, stdev=2405 | slat (nsec): min=1697, max=32986, avg=2232.67, stdev=205.
clat (usec): min=28, max=146, avg=61.95, stdev= 5.61 | clat (nsec): min=14620, max=89670, avg=46638.22, stdev=11
lat (usec): min=31, max=148, avg=64.97, stdev= 5.64 | lat (nsec): min=16814, max=91833, avg=48907.48, stdev=11
| 1.00th=[ 51], 5.00th=[ 56], 10.00th=[ 57], 20. | | 1.00th=[44800], 5.00th=[45312], 10.00th=[45824], 20.
| 30.00th=[ 59], 40.00th=[ 60], 50.00th=[ 62], 60. | | 30.00th=[46336], 40.00th=[46336], 50.00th=[46336], 60.
| 70.00th=[ 65], 80.00th=[ 66], 90.00th=[ 69], 95. | | 70.00th=[46848], 80.00th=[46848], 90.00th=[47360], 95.
| 99.00th=[ 81], 99.50th=[ 83], 99.90th=[ 90], 99. | | 99.00th=[50944], 99.50th=[53504], 99.90th=[58112], 99.
| 99.99th=[ 108] | | 99.99th=[63744]
Type=write,Size=4K,NumJobs=1,depth=64 Type=write,Size=4K,NumJobs=1,depth=64
write: IOPS=350k, BW=1367MiB/s (1434MB/s)(13.4GiB/10001msec | write: IOPS=322k, BW=1259MiB/s (1320MB/s)(12.3GiB/10001msec
slat (nsec): min=1669, max=39786, avg=2055.37, stdev=197. | slat (nsec): min=1697, max=33655, avg=2273.54, stdev=208.
clat (usec): min=21, max=222, avg=180.47, stdev= 1.67 | clat (usec): min=18, max=238, avg=195.95, stdev= 2.07
lat (usec): min=23, max=224, avg=182.56, stdev= 1.69 | lat (usec): min=20, max=241, avg=198.26, stdev= 2.08
| 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 180], 20. | | 1.00th=[ 192], 5.00th=[ 194], 10.00th=[ 194], 20.
| 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 182], 60. | | 30.00th=[ 196], 40.00th=[ 196], 50.00th=[ 196], 60.
| 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 184], 95. | | 70.00th=[ 198], 80.00th=[ 198], 90.00th=[ 198], 95.
| 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190], 99. | | 99.00th=[ 202], 99.50th=[ 204], 99.90th=[ 208], 99.
| 99.99th=[ 194] | | 99.99th=[ 212]
Type=write,Size=4K,NumJobs=16,depth=1 Type=write,Size=4K,NumJobs=16,depth=1
write: IOPS=851k, BW=3325MiB/s (3486MB/s)(32.5GiB/10001msec | write: IOPS=855k, BW=3339MiB/s (3502MB/s)(32.6GiB/10001msec
slat (nsec): min=1844, max=67596, avg=2580.48, stdev=494. | slat (nsec): min=1832, max=65363, avg=2579.38, stdev=826.
clat (nsec): min=670, max=351867, avg=15732.23, stdev=195 | clat (nsec): min=472, max=134403, avg=15646.36, stdev=196
lat (usec): min=12, max=354, avg=18.37, stdev= 2.08 | lat (usec): min=12, max=136, avg=18.28, stdev= 2.21
| 1.00th=[12608], 5.00th=[14016], 10.00th=[14400], 20. | | 1.00th=[13120], 5.00th=[13760], 10.00th=[14016], 20.
| 30.00th=[15168], 40.00th=[15296], 50.00th=[15552], 60. | | 30.00th=[14656], 40.00th=[14912], 50.00th=[15168], 60.
| 70.00th=[16064], 80.00th=[16512], 90.00th=[17024], 95. | | 70.00th=[15936], 80.00th=[16512], 90.00th=[17536], 95.
| 99.00th=[20096], 99.50th=[22144], 99.90th=[34048], 99. | | 99.00th=[23680], 99.50th=[25728], 99.90th=[33536], 99.
| 99.99th=[89600] | | 99.99th=[37120]
Type=write,Size=4K,NumJobs=16,depth=16 Type=write,Size=4K,NumJobs=16,depth=16
write: IOPS=1885k, BW=7364MiB/s (7722MB/s)(71.9GiB/10001mse | write: IOPS=1826k, BW=7133MiB/s (7480MB/s)(69.7GiB/10001mse
slat (nsec): min=1629, max=135927, avg=2178.04, stdev=783 | slat (nsec): min=1639, max=113071, avg=3312.71, stdev=236
clat (usec): min=40, max=387, avg=133.25, stdev=31.90 | clat (usec): min=40, max=454, avg=136.35, stdev=35.22
lat (usec): min=44, max=391, avg=135.47, stdev=31.86 | lat (usec): min=43, max=456, avg=139.72, stdev=35.40
| 1.00th=[ 89], 5.00th=[ 99], 10.00th=[ 103], 20. | | 1.00th=[ 72], 5.00th=[ 91], 10.00th=[ 101], 20.
| 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 131], 60. | | 30.00th=[ 121], 40.00th=[ 127], 50.00th=[ 133], 60.
| 70.00th=[ 139], 80.00th=[ 143], 90.00th=[ 176], 95. | | 70.00th=[ 143], 80.00th=[ 149], 90.00th=[ 182], 95.
| 99.00th=[ 253], 99.50th=[ 269], 99.90th=[ 289], 99. | | 99.00th=[ 258], 99.50th=[ 273], 99.90th=[ 314], 99.
| 99.99th=[ 310] | | 99.99th=[ 367]
Type=write,Size=4K,NumJobs=16,depth=64 Type=write,Size=4K,NumJobs=16,depth=64
write: IOPS=481k, BW=1880MiB/s (1971MB/s)(23.6GiB/12877msec | write: IOPS=1875k, BW=7325MiB/s (7680MB/s)(71.5GiB/10001mse
slat (nsec): min=1635, max=3356.0M, avg=3206.62, stdev=13 | slat (nsec): min=1638, max=627057, avg=3232.91, stdev=324
clat (usec): min=27, max=7636.9k, avg=1985.08, stdev=8553 | clat (usec): min=49, max=1357, avg=542.32, stdev=162.71
lat (usec): min=29, max=7636.9k, avg=1988.34, stdev=8555 | lat (usec): min=52, max=1384, avg=545.61, stdev=162.80
| 1.00th=[ 178], 5.00th=[ 180], 10.00th=[ 180 | | 1.00th=[ 202], 5.00th=[ 277], 10.00th=[ 330], 20.
| 20.00th=[ 182], 30.00th=[ 192], 40.00th=[ 233 | | 30.00th=[ 453], 40.00th=[ 498], 50.00th=[ 537], 60.
| 50.00th=[ 235], 60.00th=[ 239], 70.00th=[ 247 | | 70.00th=[ 635], 80.00th=[ 685], 90.00th=[ 758], 95.
| 80.00th=[ 253], 90.00th=[ 262], 95.00th=[ 269 | | 99.00th=[ 922], 99.50th=[ 955], 99.90th=[ 1029], 99.
| 99.00th=[ 330], 99.50th=[ 338], 99.90th=[ 506 | | 99.99th=[ 1123]
| 99.95th=[ 832], 99.99th=[4328522] <
Type=write,Size=4K,NumJobs=56,depth=1 Type=write,Size=4K,NumJobs=56,depth=1
write: IOPS=1456k, BW=5686MiB/s (5962MB/s)(55.5GiB/10002mse | write: IOPS=1416k, BW=5533MiB/s (5802MB/s)(54.0GiB/10002mse
slat (nsec): min=1774, max=827924, avg=2804.02, stdev=137 | slat (nsec): min=1846, max=533231, avg=2881.71, stdev=111
clat (nsec): min=462, max=3982.0k, avg=35108.12, stdev=23 | clat (nsec): min=498, max=4453.2k, avg=36057.87, stdev=38
lat (usec): min=13, max=3987, avg=37.98, stdev=23.37 | lat (usec): min=15, max=4456, avg=39.01, stdev= 4.01
| 1.00th=[ 21], 5.00th=[ 27], 10.00th=[ 29], 20. | | 1.00th=[29312], 5.00th=[32128], 10.00th=[33024], 20.
| 30.00th=[ 33], 40.00th=[ 34], 50.00th=[ 35], 60. | | 30.00th=[34560], 40.00th=[35072], 50.00th=[35584], 60.
| 70.00th=[ 36], 80.00th=[ 37], 90.00th=[ 38], 95. | | 70.00th=[37120], 80.00th=[37632], 90.00th=[38656], 95.
| 99.00th=[ 52], 99.50th=[ 141], 99.90th=[ 437], 99. | | 99.00th=[49408], 99.50th=[52480], 99.90th=[56576], 99.
| 99.99th=[ 603] | | 99.99th=[64768]
Type=write,Size=4K,NumJobs=56,depth=16 Type=write,Size=4K,NumJobs=56,depth=16
write: IOPS=1662k, BW=6493MiB/s (6808MB/s)(75.8GiB/11961mse | write: IOPS=1616k, BW=6314MiB/s (6621MB/s)(61.7GiB/10001mse
slat (nsec): min=1629, max=3178.7M, avg=3532.32, stdev=16 | slat (nsec): min=1649, max=738008, avg=3245.81, stdev=163
clat (usec): min=20, max=4303.6k, avg=500.29, stdev=35989 | clat (usec): min=44, max=5046, avg=550.36, stdev=143.07
lat (usec): min=27, max=4303.6k, avg=503.87, stdev=36029 | lat (usec): min=46, max=5049, avg=553.68, stdev=143.03
| 1.00th=[ 60], 5.00th=[ 68], 10.00th=[ 75 | | 1.00th=[ 215], 5.00th=[ 343], 10.00th=[ 404], 20.
| 20.00th=[ 89], 30.00th=[ 97], 40.00th=[ 101 | | 30.00th=[ 490], 40.00th=[ 510], 50.00th=[ 529], 60.
| 50.00th=[ 109], 60.00th=[ 125], 70.00th=[ 139 | | 70.00th=[ 586], 80.00th=[ 660], 90.00th=[ 758], 95.
| 80.00th=[ 153], 90.00th=[ 182], 95.00th=[ 215 | | 99.00th=[ 938], 99.50th=[ 996], 99.90th=[ 1106], 99.
| 99.00th=[ 297], 99.50th=[ 326], 99.90th=[ 392 | | 99.99th=[ 1221]
| 99.95th=[ 429], 99.99th=[2298479] <
Type=write,Size=4K,NumJobs=56,depth=64 Type=write,Size=4K,NumJobs=56,depth=64
write: IOPS=699k, BW=2730MiB/s (2862MB/s)(33.6GiB/12594msec | write: IOPS=1625k, BW=6348MiB/s (6656MB/s)(62.0GiB/10003mse
slat (nsec): min=1628, max=4304.3M, avg=40740.65, stdev=1 | slat (nsec): min=1656, max=1933.3k, avg=21620.58, stdev=7
clat (usec): min=27, max=12592k, avg=4999.61, stdev=14711 | clat (usec): min=66, max=8663, avg=2182.95, stdev=879.90
lat (usec): min=43, max=12592k, avg=5040.40, stdev=14795 | lat (usec): min=69, max=8669, avg=2204.64, stdev=895.51
| 1.00th=[ 178], 5.00th=[ 180], 10.00th=[ 182 | | 1.00th=[ 693], 5.00th=[ 971], 10.00th=[ 1074], 20.
| 20.00th=[ 184], 30.00th=[ 229], 40.00th=[ 245 | | 30.00th=[ 1942], 40.00th=[ 2114], 50.00th=[ 2212], 60.
| 50.00th=[ 247], 60.00th=[ 251], 70.00th=[ 255 | | 70.00th=[ 2343], 80.00th=[ 2507], 90.00th=[ 3392], 95.
| 80.00th=[ 260], 90.00th=[ 273], 95.00th=[ 334 | | 99.00th=[ 4883], 99.50th=[ 5211], 99.90th=[ 5800], 99.
| 99.00th=[ 383], 99.50th=[ 388], 99.90th=[3003122 | | 99.99th=[ 6587]
| 99.95th=[4328522], 99.99th=[5402264] <
Type=write,Size=64K,NumJobs=1,depth=1 Type=write,Size=64K,NumJobs=1,depth=1
write: IOPS=23.3k, BW=1458MiB/s (1529MB/s)(14.2GiB/10001mse | write: IOPS=23.3k, BW=1454MiB/s (1525MB/s)(14.2GiB/10001mse
slat (nsec): min=4814, max=41610, avg=6697.07, stdev=3203 | slat (nsec): min=4849, max=39702, avg=6880.02, stdev=3240
clat (usec): min=3, max=130, avg=35.61, stdev= 9.49 | clat (usec): min=17, max=133, avg=35.53, stdev= 9.39
lat (usec): min=33, max=136, avg=42.35, stdev=10.60 | lat (usec): min=33, max=146, avg=42.46, stdev=10.52
| 1.00th=[29312], 5.00th=[29312], 10.00th=[29312], 20. | | 1.00th=[29312], 5.00th=[29312], 10.00th=[29568], 20.
| 30.00th=[29568], 40.00th=[29824], 50.00th=[30080], 60. | | 30.00th=[29568], 40.00th=[29824], 50.00th=[29824], 60.
| 70.00th=[37120], 80.00th=[43776], 90.00th=[49920], 95. | | 70.00th=[36608], 80.00th=[43776], 90.00th=[49408], 95.
| 99.00th=[69120], 99.50th=[72192], 99.90th=[86528], 99. | | 99.00th=[68096], 99.50th=[72192], 99.90th=[85504], 99.
| 99.99th=[90624] | 99.99th=[90624]
Type=write,Size=64K,NumJobs=1,depth=16 Type=write,Size=64K,NumJobs=1,depth=16
write: IOPS=140k, BW=8738MiB/s (9162MB/s)(85.3GiB/10001msec | write: IOPS=146k, BW=9109MiB/s (9552MB/s)(88.0GiB/10001msec
slat (nsec): min=3279, max=79959, avg=5945.90, stdev=3397 | slat (nsec): min=3787, max=44724, avg=5539.44, stdev=622.
clat (usec): min=63, max=272, avg=108.05, stdev=10.23 | clat (usec): min=76, max=228, avg=103.90, stdev=11.08
lat (usec): min=67, max=276, avg=114.06, stdev=10.46 | lat (usec): min=82, max=233, avg=109.48, stdev=11.10
| 1.00th=[ 91], 5.00th=[ 97], 10.00th=[ 99], 20. | | 1.00th=[ 90], 5.00th=[ 93], 10.00th=[ 95], 20.
| 30.00th=[ 102], 40.00th=[ 103], 50.00th=[ 105], 60. | | 30.00th=[ 98], 40.00th=[ 99], 50.00th=[ 101], 60.
| 70.00th=[ 112], 80.00th=[ 116], 90.00th=[ 123], 95. | | 70.00th=[ 105], 80.00th=[ 109], 90.00th=[ 120], 95.
| 99.00th=[ 139], 99.50th=[ 145], 99.90th=[ 169], 99. | | 99.00th=[ 141], 99.50th=[ 149], 99.90th=[ 182], 99.
| 99.99th=[ 206] | | 99.99th=[ 200]
Type=write,Size=64K,NumJobs=1,depth=64 Type=write,Size=64K,NumJobs=1,depth=64
write: IOPS=145k, BW=9085MiB/s (9526MB/s)(88.7GiB/10001msec | write: IOPS=139k, BW=8708MiB/s (9131MB/s)(85.0GiB/10001msec
slat (usec): min=2, max=164, avg= 5.45, stdev= 3.82 | slat (nsec): min=2676, max=64177, avg=5388.79, stdev=1847
clat (usec): min=235, max=952, avg=434.38, stdev=42.96 | clat (usec): min=39, max=894, avg=453.54, stdev=56.76
lat (usec): min=239, max=957, avg=439.89, stdev=42.76 | lat (usec): min=45, max=898, avg=458.97, stdev=57.24
| 1.00th=[ 371], 5.00th=[ 383], 10.00th=[ 392], 20. | | 1.00th=[ 338], 5.00th=[ 355], 10.00th=[ 367], 20.
| 30.00th=[ 408], 40.00th=[ 416], 50.00th=[ 424], 60. | | 30.00th=[ 408], 40.00th=[ 457], 50.00th=[ 478], 60.
| 70.00th=[ 445], 80.00th=[ 461], 90.00th=[ 494], 95. | | 70.00th=[ 494], 80.00th=[ 498], 90.00th=[ 506], 95.
| 99.00th=[ 570], 99.50th=[ 586], 99.90th=[ 652], 99. | | 99.00th=[ 545], 99.50th=[ 562], 99.90th=[ 734], 99.
| 99.99th=[ 881] | | 99.99th=[ 848]
Type=write,Size=64K,NumJobs=16,depth=1 Type=write,Size=64K,NumJobs=16,depth=1
write: IOPS=170k, BW=10.4GiB/s (11.2GB/s)(104GiB/10001msec) | write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10001msec)
slat (nsec): min=3811, max=82769, avg=5460.88, stdev=1987 | slat (nsec): min=4383, max=73755, avg=5607.60, stdev=1278
clat (usec): min=13, max=490, avg=87.97, stdev=11.39 | clat (usec): min=22, max=424, avg=85.24, stdev= 8.07
lat (usec): min=37, max=494, avg=93.49, stdev=11.57 | lat (usec): min=38, max=430, avg=90.90, stdev= 8.07
| 1.00th=[ 57], 5.00th=[ 72], 10.00th=[ 77], 20. | | 1.00th=[ 68], 5.00th=[ 76], 10.00th=[ 79], 20.
| 30.00th=[ 84], 40.00th=[ 86], 50.00th=[ 88], 60. | | 30.00th=[ 82], 40.00th=[ 83], 50.00th=[ 84], 60.
| 70.00th=[ 92], 80.00th=[ 96], 90.00th=[ 102], 95. | | 70.00th=[ 87], 80.00th=[ 91], 90.00th=[ 98], 95.
| 99.00th=[ 120], 99.50th=[ 126], 99.90th=[ 141], 99. | | 99.00th=[ 106], 99.50th=[ 112], 99.90th=[ 124], 99.
| 99.99th=[ 190] | | 99.99th=[ 137]
Type=write,Size=64K,NumJobs=16,depth=16 Type=write,Size=64K,NumJobs=16,depth=16
write: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10002msec) | write: IOPS=176k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec)
slat (nsec): min=3415, max=60314, avg=6662.15, stdev=1972 | slat (nsec): min=3611, max=70931, avg=6507.12, stdev=1389
clat (usec): min=94, max=3871, avg=1480.76, stdev=360.52 | clat (usec): min=109, max=3915, avg=1451.13, stdev=409.41
lat (usec): min=111, max=3880, avg=1487.48, stdev=360.80 | lat (usec): min=129, max=3920, avg=1457.68, stdev=409.50
| 1.00th=[ 1270], 5.00th=[ 1287], 10.00th=[ 1287], 20. | | 1.00th=[ 1254], 5.00th=[ 1270], 10.00th=[ 1270], 20.
| 30.00th=[ 1369], 40.00th=[ 1369], 50.00th=[ 1385], 60. | | 30.00th=[ 1270], 40.00th=[ 1270], 50.00th=[ 1287], 60.
| 70.00th=[ 1418], 80.00th=[ 1467], 90.00th=[ 1500], 95. | | 70.00th=[ 1352], 80.00th=[ 1369], 90.00th=[ 2540], 95.
| 99.00th=[ 2802], 99.50th=[ 2802], 99.90th=[ 2835], 99. | | 99.00th=[ 2737], 99.50th=[ 2737], 99.90th=[ 2769], 99.
| 99.99th=[ 2900] | | 99.99th=[ 2769]
Type=write,Size=64K,NumJobs=16,depth=64 Type=write,Size=64K,NumJobs=16,depth=64
write: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10005msec) | write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10004msec)
slat (usec): min=2, max=5927, avg=15.49, stdev=123.06 | slat (nsec): min=3042, max=96073, avg=6866.24, stdev=2170
clat (usec): min=114, max=16669, avg=5934.82, stdev=1800. | clat (usec): min=124, max=10103, avg=5850.40, stdev=798.1
lat (usec): min=122, max=17554, avg=5950.36, stdev=1825. | lat (usec): min=131, max=10109, avg=5857.31, stdev=798.1
| 1.00th=[ 4293], 5.00th=[ 4424], 10.00th=[ 4555], 20. | | 1.00th=[ 4752], 5.00th=[ 4883], 10.00th=[ 4948], 20.
| 30.00th=[ 4948], 40.00th=[ 5145], 50.00th=[ 5342], 60. | | 30.00th=[ 5276], 40.00th=[ 5407], 50.00th=[ 5669], 60.
| 70.00th=[ 5932], 80.00th=[ 6390], 90.00th=[ 9110], 95. | | 70.00th=[ 6259], 80.00th=[ 6587], 90.00th=[ 7046], 95.
| 99.00th=[11994], 99.50th=[12518], 99.90th=[13566], 99. | | 99.00th=[ 7767], 99.50th=[ 7898], 99.90th=[ 8455], 99.
| 99.99th=[15401] | | 99.99th=[ 8848]
Type=write,Size=64K,NumJobs=56,depth=1 Type=write,Size=64K,NumJobs=56,depth=1
write: IOPS=171k, BW=10.5GiB/s (11.2GB/s)(105GiB/10002msec) | write: IOPS=175k, BW=10.7GiB/s (11.4GB/s)(107GiB/10002msec)
slat (nsec): min=3796, max=64949, avg=5590.94, stdev=2033 | slat (nsec): min=4029, max=63780, avg=5771.34, stdev=1537
clat (usec): min=47, max=940, avg=320.48, stdev=68.32 | clat (usec): min=32, max=979, avg=314.28, stdev=57.25
lat (usec): min=53, max=947, avg=326.12, stdev=68.40 | lat (usec): min=40, max=985, avg=320.11, stdev=57.29
| 1.00th=[ 190], 5.00th=[ 202], 10.00th=[ 302], 20. | | 1.00th=[ 186], 5.00th=[ 293], 10.00th=[ 297], 20.
| 30.00th=[ 306], 40.00th=[ 310], 50.00th=[ 314], 60. | | 30.00th=[ 297], 40.00th=[ 302], 50.00th=[ 302], 60.
| 70.00th=[ 322], 80.00th=[ 326], 90.00th=[ 334], 95. | | 70.00th=[ 310], 80.00th=[ 310], 90.00th=[ 326], 95.
| 99.00th=[ 635], 99.50th=[ 644], 99.90th=[ 660], 99. | | 99.00th=[ 603], 99.50th=[ 611], 99.90th=[ 627], 99.
| 99.99th=[ 783] | | 99.99th=[ 750]
Type=write,Size=64K,NumJobs=56,depth=16 Type=write,Size=64K,NumJobs=56,depth=16
write: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10005msec) | write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10005msec)
slat (nsec): min=3155, max=79675, avg=7689.97, stdev=2581 | slat (nsec): min=3542, max=89310, avg=7585.26, stdev=2460
clat (usec): min=151, max=13705, avg=5207.44, stdev=985.7 | clat (usec): min=190, max=14367, avg=5116.17, stdev=1396.
lat (usec): min=175, max=13712, avg=5215.19, stdev=985.8 | lat (usec): min=198, max=14375, avg=5123.82, stdev=1396.
| 1.00th=[ 2573], 5.00th=[ 2606], 10.00th=[ 5014], 20. | | 1.00th=[ 2540], 5.00th=[ 2540], 10.00th=[ 2573], 20.
| 30.00th=[ 5145], 40.00th=[ 5211], 50.00th=[ 5211], 60. | | 30.00th=[ 5080], 40.00th=[ 5080], 50.00th=[ 5145], 60.
| 70.00th=[ 5211], 80.00th=[ 5276], 90.00th=[ 5866], 95. | | 70.00th=[ 5145], 80.00th=[ 5669], 90.00th=[ 6849], 95.
| 99.00th=[ 8455], 99.50th=[ 8848], 99.90th=[10028], 99. | | 99.00th=[ 8979], 99.50th=[ 9634], 99.90th=[11207], 99.
| 99.99th=[11600] | | 99.99th=[12780]
Type=write,Size=64K,NumJobs=56,depth=64 Type=write,Size=64K,NumJobs=56,depth=64
write: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10014msec) | write: IOPS=174k, BW=10.6GiB/s (11.4GB/s)(107GiB/10011msec)
slat (usec): min=2, max=45005, avg=184.96, stdev=907.62 | slat (usec): min=3, max=32839, avg=176.71, stdev=925.53
clat (usec): min=195, max=128349, avg=20590.58, stdev=799 | clat (usec): min=189, max=132076, avg=20366.61, stdev=728
lat (usec): min=229, max=128358, avg=20775.62, stdev=814 | lat (usec): min=218, max=132086, avg=20543.39, stdev=740
| 1.00th=[ 8455], 5.00th=[ 9503], 10.00th=[10552], 20. | | 1.00th=[ 7898], 5.00th=[ 9110], 10.00th=[10945], 20.
| 30.00th=[19006], 40.00th=[20055], 50.00th=[20317], 60. | | 30.00th=[17957], 40.00th=[19530], 50.00th=[20317], 60.
| 70.00th=[20841], 80.00th=[22152], 90.00th=[26608], 95. | | 70.00th=[21103], 80.00th=[23462], 90.00th=[28705], 95.
| 99.00th=[52691], 99.50th=[58459], 99.90th=[70779], 99. | | 99.00th=[45876], 99.50th=[50594], 99.90th=[63177], 99.
| 99.99th=[94897] | | 99.99th=[84411]
Target with DIM Without DIM
Type=read,Size=512B,NumJobs=1,depth=1 Type=read,Size=512B,NumJobs=1,depth=1
read: IOPS=57.6k, BW=28.1MiB/s (29.5MB/s)(281MiB/10001msec | read: IOPS=47.0k, BW=22.0MiB/s (24.1MB/s)(230MiB/10001msec
slat (nsec): min=2050, max=44230, avg=2445.95, stdev=1144 | slat (nsec): min=1981, max=53196, avg=3048.29, stdev=2391
clat (nsec): min=838, max=98356, avg=14486.18, stdev=4396 | clat (nsec): min=442, max=107156, avg=17658.37, stdev=647
lat (usec): min=12, max=116, avg=16.97, stdev= 4.73 | lat (usec): min=13, max=109, avg=20.78, stdev= 7.18
| 1.00th=[12864], 5.00th=[12864], 10.00th=[12992], 20. | | 1.00th=[13632], 5.00th=[13760], 10.00th=[13888], 20.
| 30.00th=[13120], 40.00th=[13120], 50.00th=[13120], 60. | | 30.00th=[14144], 40.00th=[14144], 50.00th=[14400], 60.
| 70.00th=[13376], 80.00th=[13504], 90.00th=[15552], 95. | | 70.00th=[17024], 80.00th=[19072], 90.00th=[27776], 95.
| 99.00th=[35072], 99.50th=[36096], 99.90th=[40704], 99. | | 99.00th=[40704], 99.50th=[45824], 99.90th=[51456], 99.
| 99.99th=[51968] | | 99.99th=[58624]
Type=read,Size=512B,NumJobs=1,depth=16 Type=read,Size=512B,NumJobs=1,depth=16
read: IOPS=348k, BW=170MiB/s (178MB/s)(1697MiB/10001msec) | read: IOPS=334k, BW=163MiB/s (171MB/s)(1629MiB/10001msec)
slat (nsec): min=1594, max=45285, avg=1933.31, stdev=410. | slat (nsec): min=1564, max=28780, avg=2162.48, stdev=588.
clat (usec): min=36, max=162, avg=43.80, stdev= 8.09 | clat (usec): min=17, max=118, avg=45.50, stdev= 6.30
lat (usec): min=37, max=164, avg=45.77, stdev= 8.11 | lat (usec): min=20, max=120, avg=47.70, stdev= 6.64
| 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20. | | 1.00th=[41728], 5.00th=[42752], 10.00th=[42752], 20.
| 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 42], 60. | | 30.00th=[43264], 40.00th=[43776], 50.00th=[43776], 60.
| 70.00th=[ 43], 80.00th=[ 44], 90.00th=[ 52], 95. | | 70.00th=[44288], 80.00th=[44288], 90.00th=[46336], 95.
| 99.00th=[ 81], 99.50th=[ 90], 99.90th=[ 112], 99. | | 99.00th=[72192], 99.50th=[73216], 99.90th=[77312], 99.
| 99.99th=[ 139] | | 99.99th=[86528]
Type=read,Size=512B,NumJobs=1,depth=64 Type=read,Size=512B,NumJobs=1,depth=64
read: IOPS=373k, BW=182MiB/s (191MB/s)(1822MiB/10001msec) | read: IOPS=344k, BW=168MiB/s (176MB/s)(1682MiB/10001msec)
slat (nsec): min=1581, max=43893, avg=1907.45, stdev=238. | slat (nsec): min=1607, max=30650, avg=2093.56, stdev=233.
clat (usec): min=58, max=295, avg=169.31, stdev= 2.74 | clat (usec): min=19, max=238, avg=183.42, stdev= 2.22
lat (usec): min=60, max=297, avg=171.25, stdev= 2.76 | lat (usec): min=21, max=241, avg=185.55, stdev= 2.23
| 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20. | | 1.00th=[ 180], 5.00th=[ 182], 10.00th=[ 182], 20.
| 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60. | | 30.00th=[ 182], 40.00th=[ 184], 50.00th=[ 184], 60.
| 70.00th=[ 169], 80.00th=[ 172], 90.00th=[ 172], 95. | | 70.00th=[ 184], 80.00th=[ 186], 90.00th=[ 186], 95.
| 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 210], 99. | | 99.00th=[ 190], 99.50th=[ 192], 99.90th=[ 196], 99.
| 99.99th=[ 231] | | 99.99th=[ 210]
Type=read,Size=512B,NumJobs=16,depth=1 Type=read,Size=512B,NumJobs=16,depth=1
read: IOPS=924k, BW=451MiB/s (473MB/s)(4510MiB/10001msec) | read: IOPS=907k, BW=443MiB/s (465MB/s)(4431MiB/10001msec)
slat (nsec): min=1865, max=66933, avg=2473.89, stdev=487. | slat (nsec): min=1925, max=69082, avg=2505.79, stdev=869.
clat (nsec): min=598, max=426580, avg=14400.84, stdev=125 | clat (nsec): min=476, max=112521, avg=14672.56, stdev=182
lat (usec): min=11, max=428, avg=16.92, stdev= 1.37 | lat (usec): min=12, max=131, avg=17.23, stdev= 2.08
| 1.00th=[11840], 5.00th=[13248], 10.00th=[13504], 20. | | 1.00th=[11584], 5.00th=[13248], 10.00th=[13376], 20.
| 30.00th=[13888], 40.00th=[14016], 50.00th=[14144], 60. | | 30.00th=[13888], 40.00th=[14016], 50.00th=[14272], 60.
| 70.00th=[14528], 80.00th=[14912], 90.00th=[15552], 95. | | 70.00th=[14784], 80.00th=[15296], 90.00th=[16512], 95.
| 99.00th=[18816], 99.50th=[20864], 99.90th=[24960], 99. | | 99.00th=[22400], 99.50th=[24448], 99.90th=[32384], 99.
| 99.99th=[32128] | | 99.99th=[35584]
Type=read,Size=512B,NumJobs=16,depth=16 Type=read,Size=512B,NumJobs=16,depth=16
read: IOPS=4616k, BW=2254MiB/s (2363MB/s)(22.0GiB/10001mse | read: IOPS=4055k, BW=1980MiB/s (2076MB/s)(19.3GiB/10003mse
slat (nsec): min=1571, max=101196, avg=2275.70, stdev=782 | slat (nsec): min=1572, max=239521, avg=2836.25, stdev=282
clat (usec): min=24, max=492, avg=52.83, stdev= 9.68 | clat (usec): min=9, max=616, avg=59.85, stdev=55.40
lat (usec): min=26, max=523, avg=55.15, stdev= 9.91 | lat (usec): min=15, max=630, avg=62.74, stdev=57.98
| 1.00th=[ 42], 5.00th=[ 43], 10.00th=[ 43], 20. | | 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 44], 20.
| 30.00th=[ 47], 40.00th=[ 50], 50.00th=[ 52], 60. | | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 46], 60.
| 70.00th=[ 57], 80.00th=[ 60], 90.00th=[ 65], 95. | | 70.00th=[ 48], 80.00th=[ 52], 90.00th=[ 60], 95.
| 99.00th=[ 79], 99.50th=[ 82], 99.90th=[ 88], 99. | | 99.00th=[ 326], 99.50th=[ 343], 99.90th=[ 367], 99.
| 99.99th=[ 306] | | 99.99th=[ 519]
Type=read,Size=512B,NumJobs=16,depth=64 Type=read,Size=512B,NumJobs=16,depth=64
read: IOPS=5208k, BW=2543MiB/s (2666MB/s)(24.8GiB/10002mse | read: IOPS=3451k, BW=1685MiB/s (1767MB/s)(16.5GiB/10001mse
slat (nsec): min=1575, max=61538, avg=2219.34, stdev=1268 | slat (nsec): min=1569, max=204726, avg=3322.74, stdev=274
clat (usec): min=42, max=378, avg=194.04, stdev=28.59 | clat (usec): min=5, max=1543, avg=292.82, stdev=220.59
lat (usec): min=44, max=380, avg=196.30, stdev=28.94 | lat (usec): min=14, max=1561, avg=296.20, stdev=223.03
| 1.00th=[ 172], 5.00th=[ 174], 10.00th=[ 174], 20. | | 1.00th=[ 176], 5.00th=[ 178], 10.00th=[ 180], 20.
| 30.00th=[ 176], 40.00th=[ 178], 50.00th=[ 178], 60. | | 30.00th=[ 188], 40.00th=[ 227], 50.00th=[ 243], 60.
| 70.00th=[ 208], 80.00th=[ 212], 90.00th=[ 233], 95. | | 70.00th=[ 249], 80.00th=[ 285], 90.00th=[ 685], 95.
| 99.00th=[ 306], 99.50th=[ 318], 99.90th=[ 330], 99. | | 99.00th=[ 1270], 99.50th=[ 1319], 99.90th=[ 1385], 99.
| 99.99th=[ 343] | | 99.99th=[ 1450]
Type=read,Size=512B,NumJobs=56,depth=1 Type=read,Size=512B,NumJobs=56,depth=1
read: IOPS=2294k, BW=1120MiB/s (1175MB/s)(10.9GiB/10003mse | read: IOPS=2351k, BW=1148MiB/s (1204MB/s)(11.2GiB/10002mse
slat (nsec): min=1826, max=842064, avg=3629.96, stdev=274 | slat (nsec): min=1744, max=502891, avg=3525.21, stdev=199
clat (nsec): min=468, max=3959.9k, avg=20048.74, stdev=10 | clat (nsec): min=456, max=781278, avg=19563.02, stdev=418
lat (usec): min=10, max=3962, avg=23.76, stdev=11.12 | lat (usec): min=11, max=785, avg=23.18, stdev= 4.79
| 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20. | | 1.00th=[13120], 5.00th=[14912], 10.00th=[15936], 20.
| 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 20], 60. | | 30.00th=[17792], 40.00th=[18304], 50.00th=[19072], 60.
| 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 25], 95. | | 70.00th=[20608], 80.00th=[21632], 90.00th=[23680], 95.
| 99.00th=[ 37], 99.50th=[ 41], 99.90th=[ 61], 99. | | 99.00th=[31872], 99.50th=[35072], 99.90th=[41728], 99.
| 99.99th=[ 529] | | 99.99th=[56064]
Type=read,Size=512B,NumJobs=56,depth=16 Type=read,Size=512B,NumJobs=56,depth=16
read: IOPS=8192k, BW=4000MiB/s (4194MB/s)(39.1GiB/10001mse | read: IOPS=7234k, BW=3532MiB/s (3704MB/s)(34.5GiB/10002mse
slat (nsec): min=1639, max=7118.6k, avg=5019.37, stdev=74 | slat (nsec): min=1588, max=13073k, avg=5296.03, stdev=602
clat (usec): min=5, max=7261, avg=103.57, stdev=52.37 | clat (usec): min=9, max=13311, avg=117.66, stdev=77.29
lat (usec): min=34, max=7264, avg=108.69, stdev=54.81 | lat (usec): min=18, max=13322, avg=123.07, stdev=80.61
| 1.00th=[ 64], 5.00th=[ 68], 10.00th=[ 69], 20. | | 1.00th=[ 63], 5.00th=[ 67], 10.00th=[ 69], 20.
| 30.00th=[ 73], 40.00th=[ 74], 50.00th=[ 77], 60. | | 30.00th=[ 72], 40.00th=[ 76], 50.00th=[ 81], 60.
| 70.00th=[ 94], 80.00th=[ 157], 90.00th=[ 182], 95. | | 70.00th=[ 108], 80.00th=[ 184], 90.00th=[ 247], 95.
| 99.00th=[ 247], 99.50th=[ 306], 99.90th=[ 412], 99. | | 99.00th=[ 338], 99.50th=[ 383], 99.90th=[ 494], 99.
| 99.99th=[ 947] | | 99.99th=[ 594]
Type=read,Size=512B,NumJobs=56,depth=64 Type=read,Size=512B,NumJobs=56,depth=64
read: IOPS=8312k, BW=4059MiB/s (4256MB/s)(39.6GiB/10002mse | read: IOPS=7769k, BW=3793MiB/s (3978MB/s)(37.1GiB/10002mse
slat (nsec): min=1629, max=669215, avg=5032.55, stdev=769 | slat (nsec): min=1583, max=3785.4k, avg=5319.20, stdev=81
clat (usec): min=43, max=1951, avg=425.25, stdev=208.07 | clat (usec): min=2, max=5212, avg=455.02, stdev=334.71
lat (usec): min=63, max=1968, avg=430.39, stdev=210.56 | lat (usec): min=17, max=5214, avg=460.45, stdev=338.64
| 1.00th=[ 273], 5.00th=[ 281], 10.00th=[ 289], 20. | | 1.00th=[ 265], 5.00th=[ 273], 10.00th=[ 281], 20.
| 30.00th=[ 302], 40.00th=[ 302], 50.00th=[ 306], 60. | | 30.00th=[ 293], 40.00th=[ 297], 50.00th=[ 302], 60.
| 70.00th=[ 322], 80.00th=[ 709], 90.00th=[ 750], 95. | | 70.00th=[ 310], 80.00th=[ 437], 90.00th=[ 1074], 95.
| 99.00th=[ 971], 99.50th=[ 1270], 99.90th=[ 1500], 99. | | 99.00th=[ 1401], 99.50th=[ 1991], 99.90th=[ 2245], 99.
| 99.99th=[ 1663] | | 99.99th=[ 2442]
Type=read,Size=4K,NumJobs=1,depth=1 Type=read,Size=4K,NumJobs=1,depth=1
read: IOPS=52.3k, BW=204MiB/s (214MB/s)(2042MiB/10001msec) | read: IOPS=54.4k, BW=213MiB/s (223MB/s)(2126MiB/10001msec)
slat (nsec): min=2073, max=41533, avg=2731.01, stdev=1940 | slat (nsec): min=2044, max=28781, avg=2556.03, stdev=1441
clat (nsec): min=647, max=121164, avg=15919.33, stdev=561 | clat (nsec): min=621, max=70566, avg=15359.54, stdev=5250
lat (usec): min=13, max=123, avg=18.70, stdev= 6.22 | lat (nsec): min=13302, max=73823, avg=17966.25, stdev=56
| 1.00th=[13376], 5.00th=[13504], 10.00th=[13504], 20. | | 1.00th=[13120], 5.00th=[13248], 10.00th=[13248], 20.
| 30.00th=[13632], 40.00th=[13760], 50.00th=[13888], 60. | | 30.00th=[13376], 40.00th=[13376], 50.00th=[13376], 60.
| 70.00th=[14144], 80.00th=[15040], 90.00th=[20096], 95. | | 70.00th=[13632], 80.00th=[14144], 90.00th=[19328], 95.
| 99.00th=[36608], 99.50th=[40192], 99.90th=[50944], 99. | | 99.00th=[35584], 99.50th=[36608], 99.90th=[45312], 99.
| 99.99th=[73216] | | 99.99th=[53504]
Type=read,Size=4K,NumJobs=1,depth=16 Type=read,Size=4K,NumJobs=1,depth=16
read: IOPS=321k, BW=1255MiB/s (1316MB/s)(12.3GiB/10001msec | read: IOPS=334k, BW=1306MiB/s (1370MB/s)(12.8GiB/10001msec
slat (nsec): min=1652, max=42407, avg=2045.49, stdev=500. | slat (nsec): min=1696, max=45538, avg=2168.76, stdev=208.
clat (usec): min=34, max=181, avg=47.45, stdev=10.18 | clat (usec): min=18, max=148, avg=45.39, stdev= 1.47
lat (usec): min=36, max=183, avg=49.54, stdev=10.20 | lat (usec): min=20, max=192, avg=47.59, stdev= 1.48
| 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20. | | 1.00th=[43776], 5.00th=[44288], 10.00th=[44288], 20.
| 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 44], 60. | | 30.00th=[44800], 40.00th=[45312], 50.00th=[45312], 60.
| 70.00th=[ 45], 80.00th=[ 53], 90.00th=[ 60], 95. | | 70.00th=[45312], 80.00th=[45824], 90.00th=[45824], 95.
| 99.00th=[ 92], 99.50th=[ 98], 99.90th=[ 122], 99. | | 99.00th=[49408], 99.50th=[54528], 99.90th=[64256], 99.
| 99.99th=[ 147] | | 99.99th=[70144]
Type=read,Size=4K,NumJobs=1,depth=64 Type=read,Size=4K,NumJobs=1,depth=64
read: IOPS=358k, BW=1398MiB/s (1466MB/s)(13.7GiB/10001msec | read: IOPS=338k, BW=1319MiB/s (1383MB/s)(12.9GiB/10001msec
slat (nsec): min=1586, max=39977, avg=2013.84, stdev=228. | slat (nsec): min=1621, max=52632, avg=2145.29, stdev=209.
clat (usec): min=33, max=277, avg=176.53, stdev= 3.73 | clat (usec): min=36, max=236, avg=187.02, stdev= 2.05
lat (usec): min=34, max=279, avg=178.58, stdev= 3.75 | lat (usec): min=38, max=238, avg=189.20, stdev= 2.06
| 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20. | | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 186], 20.
| 30.00th=[ 176], 40.00th=[ 176], 50.00th=[ 176], 60. | | 30.00th=[ 186], 40.00th=[ 188], 50.00th=[ 188], 60.
| 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 180], 95. | | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 190], 95.
| 99.00th=[ 192], 99.50th=[ 204], 99.90th=[ 217], 99. | | 99.00th=[ 192], 99.50th=[ 196], 99.90th=[ 200], 99.
| 99.99th=[ 260] | | 99.99th=[ 217]
Type=read,Size=4K,NumJobs=16,depth=1 Type=read,Size=4K,NumJobs=16,depth=1
read: IOPS=876k, BW=3422MiB/s (3588MB/s)(33.4GiB/10001msec | read: IOPS=869k, BW=3396MiB/s (3561MB/s)(33.2GiB/10002msec
slat (nsec): min=2018, max=54502, avg=2490.14, stdev=570. | slat (nsec): min=1890, max=70947, avg=2497.58, stdev=813.
clat (nsec): min=586, max=146434, avg=15319.20, stdev=175 | clat (nsec): min=590, max=79209, avg=15456.14, stdev=1968
lat (usec): min=11, max=148, avg=17.86, stdev= 1.86 | lat (usec): min=12, max=110, avg=18.01, stdev= 2.18
| 1.00th=[12864], 5.00th=[13888], 10.00th=[14144], 20. | | 1.00th=[12352], 5.00th=[13888], 10.00th=[14144], 20.
| 30.00th=[14656], 40.00th=[14784], 50.00th=[14912], 60. | | 30.00th=[14528], 40.00th=[14784], 50.00th=[15040], 60.
| 70.00th=[15424], 80.00th=[15936], 90.00th=[16768], 95. | | 70.00th=[15552], 80.00th=[16192], 90.00th=[17280], 95.
| 99.00th=[22400], 99.50th=[24192], 99.90th=[32640], 99. | | 99.00th=[23936], 99.50th=[25728], 99.90th=[33536], 99.
| 99.99th=[72192] | | 99.99th=[36096]
Type=read,Size=4K,NumJobs=16,depth=16 Type=read,Size=4K,NumJobs=16,depth=16
read: IOPS=2665k, BW=10.2GiB/s (10.9GB/s)(102GiB/10001msec | read: IOPS=1843k, BW=7201MiB/s (7551MB/s)(70.3GiB/10002mse
slat (nsec): min=1576, max=160409, avg=2277.16, stdev=105 | slat (nsec): min=1582, max=104435, avg=3170.92, stdev=170
clat (usec): min=21, max=895, avg=93.43, stdev=13.26 | clat (usec): min=40, max=554, avg=135.18, stdev=32.57
lat (usec): min=25, max=897, avg=95.75, stdev=13.22 | lat (usec): min=42, max=557, avg=138.42, stdev=32.59
| 1.00th=[ 74], 5.00th=[ 78], 10.00th=[ 80], 20. | | 1.00th=[ 68], 5.00th=[ 92], 10.00th=[ 104], 20.
| 30.00th=[ 88], 40.00th=[ 93], 50.00th=[ 96], 60. | | 30.00th=[ 125], 40.00th=[ 130], 50.00th=[ 135], 60.
| 70.00th=[ 99], 80.00th=[ 100], 90.00th=[ 102], 95. | | 70.00th=[ 141], 80.00th=[ 147], 90.00th=[ 161], 95.
| 99.00th=[ 167], 99.50th=[ 174], 99.90th=[ 186], 99. | | 99.00th=[ 262], 99.50th=[ 306], 99.90th=[ 392], 99.
| 99.99th=[ 258] | | 99.99th=[ 461]
Type=read,Size=4K,NumJobs=16,depth=64 Type=read,Size=4K,NumJobs=16,depth=64
read: IOPS=2670k, BW=10.2GiB/s (10.9GB/s)(102GiB/10002msec | read: IOPS=1917k, BW=7487MiB/s (7850MB/s)(73.1GiB/10001mse
slat (nsec): min=1559, max=332598, avg=2408.47, stdev=359 | slat (nsec): min=1566, max=653412, avg=3347.14, stdev=385
clat (usec): min=25, max=1243, avg=380.63, stdev=95.54 | clat (usec): min=43, max=1445, avg=530.39, stdev=127.84
lat (usec): min=30, max=1245, avg=383.08, stdev=96.26 | lat (usec): min=45, max=1619, avg=533.81, stdev=128.49
| 1.00th=[ 182], 5.00th=[ 247], 10.00th=[ 302], 20. | | 1.00th=[ 188], 5.00th=[ 351], 10.00th=[ 420], 20.
| 30.00th=[ 347], 40.00th=[ 359], 50.00th=[ 367], 60. | | 30.00th=[ 486], 40.00th=[ 502], 50.00th=[ 519], 60.
| 70.00th=[ 383], 80.00th=[ 408], 90.00th=[ 465], 95. | | 70.00th=[ 553], 80.00th=[ 578], 90.00th=[ 644], 95.
| 99.00th=[ 742], 99.50th=[ 766], 99.90th=[ 816], 99. | | 99.00th=[ 1012], 99.50th=[ 1037], 99.90th=[ 1090], 99.
| 99.99th=[ 873] | | 99.99th=[ 1205]
Type=read,Size=4K,NumJobs=56,depth=1 Type=read,Size=4K,NumJobs=56,depth=1
read: IOPS=1594k, BW=6227MiB/s (6530MB/s)(60.8GiB/10002mse | read: IOPS=1571k, BW=6138MiB/s (6436MB/s)(59.9GiB/10001mse
slat (nsec): min=1747, max=711701, avg=2984.71, stdev=222 | slat (nsec): min=1693, max=505053, avg=2771.65, stdev=110
clat (nsec): min=468, max=4376.0k, avg=31538.77, stdev=12 | clat (nsec): min=479, max=1348.1k, avg=32311.24, stdev=36
lat (usec): min=13, max=4378, avg=34.59, stdev=13.16 | lat (usec): min=15, max=1350, avg=35.15, stdev= 3.81
| 1.00th=[ 22], 5.00th=[ 26], 10.00th=[ 28], 20. | | 1.00th=[26752], 5.00th=[28800], 10.00th=[29568], 20.
| 30.00th=[ 30], 40.00th=[ 31], 50.00th=[ 31], 60. | | 30.00th=[30848], 40.00th=[31360], 50.00th=[31872], 60.
| 70.00th=[ 33], 80.00th=[ 34], 90.00th=[ 36], 95. | | 70.00th=[33024], 80.00th=[33536], 90.00th=[35584], 95.
| 99.00th=[ 45], 99.50th=[ 50], 99.90th=[ 243], 99. | | 99.00th=[42240], 99.50th=[43776], 99.90th=[49920], 99.
| 99.99th=[ 519] | | 99.99th=[59136]
Type=read,Size=4K,NumJobs=56,depth=16 Type=read,Size=4K,NumJobs=56,depth=16
read: IOPS=2652k, BW=10.1GiB/s (10.9GB/s)(101GiB/10002msec | read: IOPS=2009k, BW=7846MiB/s (8227MB/s)(76.6GiB/10001mse
slat (nsec): min=1571, max=937333, avg=3004.91, stdev=194 | slat (nsec): min=1566, max=379585, avg=3204.59, stdev=230
clat (usec): min=26, max=1529, avg=334.21, stdev=57.70 | clat (usec): min=44, max=1761, avg=442.22, stdev=244.27
lat (usec): min=28, max=1531, avg=337.28, stdev=57.71 | lat (usec): min=46, max=1764, avg=445.50, stdev=244.24
| 1.00th=[ 147], 5.00th=[ 210], 10.00th=[ 297], 20. | | 1.00th=[ 161], 5.00th=[ 194], 10.00th=[ 215], 20.
| 30.00th=[ 322], 40.00th=[ 330], 50.00th=[ 334], 60. | | 30.00th=[ 273], 40.00th=[ 306], 50.00th=[ 351], 60.
| 70.00th=[ 351], 80.00th=[ 359], 90.00th=[ 388], 95. | | 70.00th=[ 506], 80.00th=[ 644], 90.00th=[ 832], 95.
| 99.00th=[ 490], 99.50th=[ 529], 99.90th=[ 709], 99. | | 99.00th=[ 1156], 99.50th=[ 1221], 99.90th=[ 1336], 99.
| 99.99th=[ 1004] | | 99.99th=[ 1450]
Type=read,Size=4K,NumJobs=56,depth=64 Type=read,Size=4K,NumJobs=56,depth=64
read: IOPS=2659k, BW=10.1GiB/s (10.9GB/s)(101GiB/10003msec | read: IOPS=1801k, BW=7036MiB/s (7378MB/s)(68.7GiB/10002mse
slat (nsec): min=1576, max=1678.0k, avg=14535.12, stdev=5 | slat (nsec): min=1586, max=2410.2k, avg=20136.27, stdev=8
clat (usec): min=29, max=7242, avg=1332.35, stdev=730.40 | clat (usec): min=58, max=8215, avg=1968.75, stdev=947.22
lat (usec): min=32, max=7244, avg=1346.94, stdev=743.63 | lat (usec): min=60, max=8632, avg=1988.96, stdev=962.84
| 1.00th=[ 515], 5.00th=[ 619], 10.00th=[ 644], 20. | | 1.00th=[ 449], 5.00th=[ 734], 10.00th=[ 865], 20.
| 30.00th=[ 693], 40.00th=[ 807], 50.00th=[ 1303], 60. | | 30.00th=[ 1418], 40.00th=[ 1729], 50.00th=[ 1909], 60.
| 70.00th=[ 1418], 80.00th=[ 1975], 90.00th=[ 2442], 95. | | 70.00th=[ 2212], 80.00th=[ 2507], 90.00th=[ 3326], 95.
| 99.00th=[ 3392], 99.50th=[ 3621], 99.90th=[ 4047], 99. | | 99.00th=[ 4817], 99.50th=[ 5080], 99.90th=[ 5735], 99.
| 99.99th=[ 4752] | | 99.99th=[ 6587]
Type=read,Size=64K,NumJobs=1,depth=1 Type=read,Size=64K,NumJobs=1,depth=1
read: IOPS=29.7k, BW=1854MiB/s (1944MB/s)(18.1GiB/10001mse | read: IOPS=29.8k, BW=1864MiB/s (1955MB/s)(18.2GiB/10001mse
slat (nsec): min=4662, max=82948, avg=5935.55, stdev=2735 | slat (nsec): min=4768, max=95484, avg=6119.74, stdev=2779
clat (usec): min=3, max=136, avg=27.27, stdev= 7.95 | clat (nsec): min=1555, max=75556, avg=26881.73, stdev=781
lat (usec): min=25, max=141, avg=33.25, stdev= 9.18 | lat (usec): min=25, max=102, avg=33.05, stdev= 9.07
| 1.00th=[21888], 5.00th=[22144], 10.00th=[22144], 20. | | 1.00th=[21632], 5.00th=[21632], 10.00th=[21632], 20.
| 30.00th=[22400], 40.00th=[22400], 50.00th=[22656], 60. | | 30.00th=[21888], 40.00th=[22144], 50.00th=[22144], 60.
| 70.00th=[28288], 80.00th=[34560], 90.00th=[41216], 95. | | 70.00th=[28032], 80.00th=[34048], 90.00th=[40704], 95.
| 99.00th=[48384], 99.50th=[50944], 99.90th=[66048], 99. | | 99.00th=[47360], 99.50th=[49920], 99.90th=[59648], 99.
| 99.99th=[78336] | | 99.99th=[70144]
Type=read,Size=64K,NumJobs=1,depth=16 Type=read,Size=64K,NumJobs=1,depth=16
read: IOPS=168k, BW=10.3GiB/s (11.0GB/s)(103GiB/10001msec) | read: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10001msec)
slat (nsec): min=3463, max=75651, avg=4588.13, stdev=499. | slat (nsec): min=3756, max=65673, avg=4930.28, stdev=334.
clat (usec): min=67, max=271, avg=90.06, stdev= 7.31 | clat (usec): min=83, max=243, avg=87.86, stdev= 2.04
lat (usec): min=72, max=348, avg=94.68, stdev= 7.26 | lat (usec): min=87, max=310, avg=92.83, stdev= 2.05
| 1.00th=[ 81], 5.00th=[ 84], 10.00th=[ 85], 20. | | 1.00th=[ 86], 5.00th=[ 87], 10.00th=[ 87], 20.
| 30.00th=[ 87], 40.00th=[ 89], 50.00th=[ 90], 60. | | 30.00th=[ 87], 40.00th=[ 88], 50.00th=[ 88], 60.
| 70.00th=[ 92], 80.00th=[ 93], 90.00th=[ 96], 95. | | 70.00th=[ 88], 80.00th=[ 89], 90.00th=[ 91], 95.
| 99.00th=[ 127], 99.50th=[ 135], 99.90th=[ 172], 99. | | 99.00th=[ 96], 99.50th=[ 99], 99.90th=[ 108], 99.
| 99.99th=[ 194] | | 99.99th=[ 114]
Type=read,Size=64K,NumJobs=1,depth=64 Type=read,Size=64K,NumJobs=1,depth=64
read: IOPS=171k, BW=10.5GiB/s (11.2GB/s)(105GiB/10001msec) | read: IOPS=168k, BW=10.2GiB/s (10.0GB/s)(102GiB/10001msec)
slat (usec): min=2, max=419, avg= 3.53, stdev= 1.24 | slat (usec): min=2, max=390, avg= 4.15, stdev= 1.56
clat (usec): min=63, max=973, avg=369.73, stdev=13.21 | clat (usec): min=37, max=947, avg=376.95, stdev=15.70
lat (usec): min=69, max=1053, avg=373.30, stdev=13.09 | lat (usec): min=42, max=988, avg=381.15, stdev=16.25
| 1.00th=[ 338], 5.00th=[ 359], 10.00th=[ 359], 20. | | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 367], 20.
| 30.00th=[ 367], 40.00th=[ 367], 50.00th=[ 371], 60. | 30.00th=[ 367], 40.00th=[ 367], 50.00th=[ 371], 60.
| 70.00th=[ 371], 80.00th=[ 375], 90.00th=[ 379], 95. | | 70.00th=[ 388], 80.00th=[ 392], 90.00th=[ 400], 95.
| 99.00th=[ 404], 99.50th=[ 416], 99.90th=[ 461], 99. | | 99.00th=[ 412], 99.50th=[ 416], 99.90th=[ 429], 99.
| 99.99th=[ 652] | | 99.99th=[ 668]
Type=read,Size=64K,NumJobs=16,depth=1 Type=read,Size=64K,NumJobs=16,depth=1
read: IOPS=171k, BW=10.4GiB/s (11.2GB/s)(104GiB/10001msec) | read: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10002msec)
slat (nsec): min=3795, max=63452, avg=5333.62, stdev=2920 | slat (nsec): min=3480, max=74756, avg=5273.21, stdev=2961
clat (nsec): min=626, max=175700, avg=87848.01, stdev=183 | clat (nsec): min=616, max=453786, avg=87271.90, stdev=161
lat (usec): min=26, max=180, avg=93.23, stdev=18.52 | lat (usec): min=26, max=458, avg=92.60, stdev=16.20
| 1.00th=[ 31], 5.00th=[ 48], 10.00th=[ 62], 20. | | 1.00th=[ 37], 5.00th=[ 57], 10.00th=[ 68], 20.
| 30.00th=[ 85], 40.00th=[ 90], 50.00th=[ 93], 60. | | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 90], 60.
| 70.00th=[ 97], 80.00th=[ 100], 90.00th=[ 104], 95. | | 70.00th=[ 95], 80.00th=[ 98], 90.00th=[ 104], 95.
| 99.00th=[ 125], 99.50th=[ 130], 99.90th=[ 139], 99. | | 99.00th=[ 126], 99.50th=[ 130], 99.90th=[ 143], 99.
| 99.99th=[ 155] | | 99.99th=[ 161]
Type=read,Size=64K,NumJobs=16,depth=16 Type=read,Size=64K,NumJobs=16,depth=16
read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec)
slat (nsec): min=3878, max=65832, avg=5321.95, stdev=1310 | slat (nsec): min=3866, max=57337, avg=5204.47, stdev=1220
clat (usec): min=104, max=3983, avg=1458.24, stdev=313.83 | clat (usec): min=116, max=3985, avg=1453.77, stdev=349.92
lat (usec): min=111, max=3988, avg=1463.62, stdev=313.72 | lat (usec): min=128, max=3990, avg=1459.03, stdev=349.93
| 1.00th=[ 1352], 5.00th=[ 1352], 10.00th=[ 1352], 20. | | 1.00th=[ 1254], 5.00th=[ 1270], 10.00th=[ 1270], 20.
| 30.00th=[ 1352], 40.00th=[ 1369], 50.00th=[ 1369], 60. | | 30.00th=[ 1352], 40.00th=[ 1352], 50.00th=[ 1352], 60.
| 70.00th=[ 1385], 80.00th=[ 1450], 90.00th=[ 1467], 95. | | 70.00th=[ 1369], 80.00th=[ 1401], 90.00th=[ 1467], 95.
| 99.00th=[ 2769], 99.50th=[ 2769], 99.90th=[ 2802], 99. | | 99.00th=[ 2737], 99.50th=[ 2769], 99.90th=[ 2769], 99.
| 99.99th=[ 2802] | 99.99th=[ 2802]
Type=read,Size=64K,NumJobs=16,depth=64 Type=read,Size=64K,NumJobs=16,depth=64
read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec) read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec)
slat (usec): min=2, max=1287, avg=13.57, stdev=76.18 | slat (usec): min=2, max=1185, avg= 9.71, stdev=60.22
clat (usec): min=116, max=15982, avg=5828.20, stdev=1647. | clat (usec): min=116, max=16490, avg=5824.81, stdev=1345.
lat (usec): min=127, max=15986, avg=5841.81, stdev=1674. | lat (usec): min=133, max=16494, avg=5834.58, stdev=1367.
| 1.00th=[ 5080], 5.00th=[ 5080], 10.00th=[ 5080], 20. | | 1.00th=[ 5407], 5.00th=[ 5407], 10.00th=[ 5473], 20.
| 30.00th=[ 5080], 40.00th=[ 5145], 50.00th=[ 5145], 60. | | 30.00th=[ 5473], 40.00th=[ 5473], 50.00th=[ 5473], 60.
| 70.00th=[ 5473], 80.00th=[ 5473], 90.00th=[10028], 95. | | 70.00th=[ 5473], 80.00th=[ 5473], 90.00th=[ 5538], 95.
| 99.00th=[10814], 99.50th=[10945], 99.90th=[10945], 99. | | 99.00th=[10945], 99.50th=[10945], 99.90th=[11076], 99.
| 99.99th=[11469] | | 99.99th=[11338]
Type=read,Size=64K,NumJobs=56,depth=1 Type=read,Size=64K,NumJobs=56,depth=1
read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec)
slat (usec): min=3, max=155, avg= 4.93, stdev= 1.32 | slat (nsec): min=3801, max=63985, avg=4896.45, stdev=1489
clat (usec): min=39, max=624, avg=314.57, stdev=91.62 | clat (usec): min=29, max=550, avg=314.45, stdev=76.21
lat (usec): min=48, max=629, avg=319.55, stdev=91.63 | lat (usec): min=35, max=555, avg=319.41, stdev=76.15
| 1.00th=[ 172], 5.00th=[ 174], 10.00th=[ 176], 20. | | 1.00th=[ 174], 5.00th=[ 178], 10.00th=[ 194], 20.
| 30.00th=[ 302], 40.00th=[ 302], 50.00th=[ 306], 60. | | 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 306], 60.
| 70.00th=[ 310], 80.00th=[ 449], 90.00th=[ 457], 95. | | 70.00th=[ 310], 80.00th=[ 322], 90.00th=[ 457], 95.
| 99.00th=[ 478], 99.50th=[ 482], 99.90th=[ 494], 99. | | 99.00th=[ 474], 99.50th=[ 482], 99.90th=[ 490], 99.
| 99.99th=[ 594] | | 99.99th=[ 506]
Type=read,Size=64K,NumJobs=56,depth=16 Type=read,Size=64K,NumJobs=56,depth=16
read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec) read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec)
slat (usec): min=3, max=156, avg= 5.40, stdev= 2.21 | slat (usec): min=3, max=251, avg= 5.40, stdev= 2.34
clat (usec): min=186, max=13998, avg=5112.23, stdev=1678. | clat (usec): min=236, max=14486, avg=5106.35, stdev=1241.
lat (usec): min=201, max=14003, avg=5117.68, stdev=1678. | lat (usec): min=265, max=14491, avg=5111.80, stdev=1241.
| 1.00th=[ 2442], 5.00th=[ 2442], 10.00th=[ 2474], 20. | | 1.00th=[ 2442], 5.00th=[ 2540], 10.00th=[ 4883], 20.
| 30.00th=[ 4883], 40.00th=[ 4948], 50.00th=[ 4948], 60. | | 30.00th=[ 4948], 40.00th=[ 5080], 50.00th=[ 5080], 60.
| 70.00th=[ 5014], 80.00th=[ 7373], 90.00th=[ 7439], 95. | | 70.00th=[ 5080], 80.00th=[ 5145], 90.00th=[ 7373], 95.
| 99.00th=[ 9896], 99.50th=[ 9896], 99.90th=[10290], 99. | | 99.00th=[ 9896], 99.50th=[ 9896], 99.90th=[ 9896], 99.
| 99.99th=[10290] | | 99.99th=[10028]
Type=read,Size=64K,NumJobs=56,depth=64 Type=read,Size=64K,NumJobs=56,depth=64
read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10016msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10018msec)
slat (usec): min=2, max=9793, avg=214.10, stdev=778.24 | slat (usec): min=2, max=10184, avg=189.22, stdev=637.78
clat (usec): min=174, max=61025, avg=20233.29, stdev=9310 | clat (usec): min=347, max=61952, avg=20252.15, stdev=7056
lat (usec): min=178, max=63984, avg=20447.45, stdev=9493 | lat (usec): min=351, max=65141, avg=20441.44, stdev=7203
| 1.00th=[ 9110], 5.00th=[ 9110], 10.00th=[ 9503], 20. | | 1.00th=[ 9503], 5.00th=[ 9765], 10.00th=[ 9896], 20.
| 30.00th=[17957], 40.00th=[18744], 50.00th=[18744], 60. | | 30.00th=[19530], 40.00th=[19530], 50.00th=[19792], 60.
| 70.00th=[23987], 80.00th=[28443], 90.00th=[36439], 95. | | 70.00th=[20317], 80.00th=[20579], 90.00th=[29492], 95.
| 99.00th=[41157], 99.50th=[56886], 99.90th=[56886], 99. | | 99.00th=[41157], 99.50th=[58983], 99.90th=[59507], 99.
| 99.99th=[58459] | | 99.99th=[61080]
Type=write,Size=512B,NumJobs=1,depth=1 Type=write,Size=512B,NumJobs=1,depth=1
write: IOPS=57.8k, BW=28.2MiB/s (29.6MB/s)(282MiB/10001msec | write: IOPS=56.0k, BW=27.4MiB/s (28.7MB/s)(274MiB/10001msec
slat (nsec): min=2145, max=77559, avg=2556.80, stdev=1310 | slat (nsec): min=2112, max=43957, avg=2561.65, stdev=1423
clat (nsec): min=626, max=100854, avg=14269.28, stdev=471 | clat (nsec): min=633, max=60031, avg=14812.68, stdev=5638
lat (usec): min=12, max=107, avg=16.87, stdev= 5.13 | lat (nsec): min=12369, max=72124, avg=17416.39, stdev=60
| 1.00th=[12480], 5.00th=[12480], 10.00th=[12480], 20. | | 1.00th=[12352], 5.00th=[12352], 10.00th=[12480], 20.
| 30.00th=[12608], 40.00th=[12608], 50.00th=[12736], 60. | | 30.00th=[12480], 40.00th=[12608], 50.00th=[12608], 60.
| 70.00th=[12864], 80.00th=[13248], 90.00th=[16512], 95. | | 70.00th=[12864], 80.00th=[14656], 90.00th=[20096], 95.
| 99.00th=[34560], 99.50th=[35584], 99.90th=[43264], 99. | | 99.00th=[35072], 99.50th=[36608], 99.90th=[47360], 99.
| 99.99th=[53504] | | 99.99th=[54016]
Type=write,Size=512B,NumJobs=1,depth=16 Type=write,Size=512B,NumJobs=1,depth=16
write: IOPS=302k, BW=147MiB/s (154MB/s)(1472MiB/10001msec) | write: IOPS=328k, BW=160MiB/s (168MB/s)(1601MiB/10001msec)
slat (nsec): min=1637, max=35628, avg=1993.52, stdev=591. | slat (nsec): min=1641, max=32941, avg=2199.97, stdev=236.
clat (usec): min=24, max=194, avg=50.75, stdev=12.31 | clat (usec): min=14, max=113, avg=46.29, stdev= 1.27
lat (usec): min=25, max=198, avg=52.78, stdev=12.32 | lat (usec): min=16, max=115, avg=48.52, stdev= 1.28
| 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20. | | 1.00th=[43776], 5.00th=[44800], 10.00th=[45312], 20.
| 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 47], 60. | | 30.00th=[45824], 40.00th=[45824], 50.00th=[46336], 60.
| 70.00th=[ 54], 80.00th=[ 59], 90.00th=[ 70], 95. | | 70.00th=[46336], 80.00th=[46848], 90.00th=[47360], 95.
| 99.00th=[ 93], 99.50th=[ 102], 99.90th=[ 127], 99. | | 99.00th=[50432], 99.50th=[53504], 99.90th=[57600], 99.
| 99.99th=[ 145] | | 99.99th=[63744]
Type=write,Size=512B,NumJobs=1,depth=64 Type=write,Size=512B,NumJobs=1,depth=64
write: IOPS=362k, BW=177MiB/s (185MB/s)(1769MiB/10001msec) | write: IOPS=337k, BW=164MiB/s (172MB/s)(1644MiB/10001msec)
slat (nsec): min=1644, max=36941, avg=1976.26, stdev=244. | slat (nsec): min=1642, max=43437, avg=2146.30, stdev=234.
clat (usec): min=50, max=219, avg=174.38, stdev= 3.50 | clat (usec): min=16, max=235, avg=187.62, stdev= 2.46
lat (usec): min=52, max=221, avg=176.39, stdev= 3.54 | lat (usec): min=18, max=237, avg=189.80, stdev= 2.48
| 1.00th=[ 172], 5.00th=[ 172], 10.00th=[ 172], 20. | | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 186], 20.
| 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 174], 60. | | 30.00th=[ 186], 40.00th=[ 188], 50.00th=[ 188], 60.
| 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 178], 95. | | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 192], 95.
| 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 194], 99. | | 99.00th=[ 194], 99.50th=[ 196], 99.90th=[ 200], 99.
| 99.99th=[ 198] | | 99.99th=[ 204]
Type=write,Size=512B,NumJobs=16,depth=1 Type=write,Size=512B,NumJobs=16,depth=1
write: IOPS=928k, BW=453MiB/s (475MB/s)(4531MiB/10001msec) | write: IOPS=914k, BW=446MiB/s (468MB/s)(4462MiB/10001msec)
slat (nsec): min=2034, max=70704, avg=2534.91, stdev=709. | slat (nsec): min=1818, max=83376, avg=2566.34, stdev=824.
clat (nsec): min=588, max=855972, avg=14244.47, stdev=155 | clat (nsec): min=465, max=343253, avg=14459.07, stdev=181
lat (usec): min=12, max=858, avg=16.82, stdev= 1.78 | lat (usec): min=11, max=345, avg=17.08, stdev= 2.09
| 1.00th=[12480], 5.00th=[13120], 10.00th=[13248], 20. | | 1.00th=[12224], 5.00th=[12992], 10.00th=[13248], 20.
| 30.00th=[13632], 40.00th=[13760], 50.00th=[14016], 60. | 30.00th=[13632], 40.00th=[13760], 50.00th=[14016], 60.
| 70.00th=[14272], 80.00th=[14528], 90.00th=[15296], 95. | | 70.00th=[14528], 80.00th=[15040], 90.00th=[16320], 95.
| 99.00th=[20864], 99.50th=[22912], 99.90th=[30848], 99. | | 99.00th=[21888], 99.50th=[23936], 99.90th=[32384], 99.
| 99.99th=[34560] | | 99.99th=[35584]
Type=write,Size=512B,NumJobs=16,depth=16 Type=write,Size=512B,NumJobs=16,depth=16
write: IOPS=4532k, BW=2213MiB/s (2320MB/s)(21.6GiB/10001mse | write: IOPS=3218k, BW=1571MiB/s (1648MB/s)(15.3GiB/10001mse
slat (nsec): min=1608, max=191266, avg=2313.43, stdev=780 | slat (nsec): min=1621, max=154997, avg=3548.60, stdev=323
clat (usec): min=13, max=557, avg=53.80, stdev=10.94 | clat (usec): min=14, max=473, avg=75.44, stdev=65.86
lat (usec): min=15, max=574, avg=56.16, stdev=11.19 | lat (usec): min=17, max=489, avg=79.05, stdev=68.82
| 1.00th=[ 42], 5.00th=[ 43], 10.00th=[ 44], 20. | | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 46], 20.
| 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 53], 60. | | 30.00th=[ 47], 40.00th=[ 48], 50.00th=[ 52], 60.
| 70.00th=[ 57], 80.00th=[ 60], 90.00th=[ 65], 95. | | 70.00th=[ 61], 80.00th=[ 64], 90.00th=[ 178], 95.
| 99.00th=[ 85], 99.50th=[ 90], 99.90th=[ 109], 99. | | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 351], 99.
| 99.99th=[ 396] | | 99.99th=[ 367]
Type=write,Size=512B,NumJobs=16,depth=64 Type=write,Size=512B,NumJobs=16,depth=64
write: IOPS=5003k, BW=2443MiB/s (2562MB/s)(23.9GiB/10001mse | write: IOPS=3973k, BW=1940MiB/s (2034MB/s)(18.0GiB/10002mse
slat (nsec): min=1613, max=68192, avg=2314.76, stdev=1502 | slat (nsec): min=1627, max=171980, avg=2926.70, stdev=209
clat (usec): min=32, max=588, avg=201.99, stdev=31.05 | clat (usec): min=10, max=1606, avg=254.29, stdev=176.36
lat (usec): min=35, max=590, avg=204.35, stdev=31.42 | lat (usec): min=16, max=1615, avg=257.27, stdev=178.32
| 1.00th=[ 176], 5.00th=[ 178], 10.00th=[ 178], 20. | | 1.00th=[ 184], 5.00th=[ 186], 10.00th=[ 188], 20.
| 30.00th=[ 182], 40.00th=[ 182], 50.00th=[ 186], 60. | | 30.00th=[ 194], 40.00th=[ 202], 50.00th=[ 237], 60.
| 70.00th=[ 212], 80.00th=[ 217], 90.00th=[ 241], 95. | | 70.00th=[ 255], 80.00th=[ 260], 90.00th=[ 293], 95.
| 99.00th=[ 310], 99.50th=[ 318], 99.90th=[ 330], 99. | | 99.00th=[ 1385], 99.50th=[ 1418], 99.90th=[ 1467], 99.
| 99.99th=[ 351] | | 99.99th=[ 1500]
Type=write,Size=512B,NumJobs=56,depth=1 Type=write,Size=512B,NumJobs=56,depth=1
write: IOPS=2299k, BW=1123MiB/s (1177MB/s)(10.0GiB/10002mse | write: IOPS=2369k, BW=1157MiB/s (1213MB/s)(11.3GiB/10002mse
slat (nsec): min=1875, max=1144.8k, avg=3714.43, stdev=29 | slat (nsec): min=1823, max=501953, avg=3567.43, stdev=201
clat (nsec): min=471, max=1780.1k, avg=19859.06, stdev=11 | clat (nsec): min=449, max=1722.5k, avg=19318.77, stdev=42
lat (usec): min=10, max=1783, avg=23.66, stdev=12.06 | lat (usec): min=11, max=1724, avg=22.98, stdev= 4.83
| 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 16], 20. | | 1.00th=[12992], 5.00th=[14656], 10.00th=[15680], 20.
| 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 19], 60. | | 30.00th=[17536], 40.00th=[18048], 50.00th=[18816], 60.
| 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 25], 95. | | 70.00th=[20352], 80.00th=[21632], 90.00th=[23424], 95.
| 99.00th=[ 37], 99.50th=[ 41], 99.90th=[ 70], 99. | | 99.00th=[31872], 99.50th=[35584], 99.90th=[41728], 99.
| 99.99th=[ 553] | | 99.99th=[54016]
Type=write,Size=512B,NumJobs=56,depth=16 Type=write,Size=512B,NumJobs=56,depth=16
write: IOPS=8134k, BW=3972MiB/s (4165MB/s)(38.8GiB/10001mse | write: IOPS=6237k, BW=3046MiB/s (3193MB/s)(29.7GiB/10002mse
slat (nsec): min=1697, max=8922.2k, avg=5089.14, stdev=73 | slat (nsec): min=1640, max=2381.5k, avg=5462.05, stdev=67
clat (usec): min=6, max=9049, avg=104.25, stdev=53.53 | clat (usec): min=5, max=10435, avg=137.19, stdev=82.68
lat (usec): min=33, max=9053, avg=109.44, stdev=56.00 | lat (usec): min=15, max=10447, avg=142.77, stdev=85.58
| 1.00th=[ 65], 5.00th=[ 69], 10.00th=[ 70], 20. | | 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 70], 20.
| 30.00th=[ 74], 40.00th=[ 75], 50.00th=[ 77], 60. | | 30.00th=[ 85], 40.00th=[ 96], 50.00th=[ 104], 60.
| 70.00th=[ 92], 80.00th=[ 161], 90.00th=[ 186], 95. | | 70.00th=[ 149], 80.00th=[ 210], 90.00th=[ 265], 95.
| 99.00th=[ 255], 99.50th=[ 326], 99.90th=[ 424], 99. | | 99.00th=[ 363], 99.50th=[ 396], 99.90th=[ 553], 99.
| 99.99th=[ 775] | | 99.99th=[ 873]
Type=write,Size=512B,NumJobs=56,depth=64 Type=write,Size=512B,NumJobs=56,depth=64
write: IOPS=8214k, BW=4011MiB/s (4205MB/s)(39.2GiB/10002mse | write: IOPS=7488k, BW=3656MiB/s (3834MB/s)(35.7GiB/10002mse
slat (nsec): min=1699, max=946558, avg=5094.80, stdev=734 | slat (nsec): min=1666, max=7929.3k, avg=5382.53, stdev=12
clat (nsec): min=1952, max=1945.9k, avg=430363.82, stdev= | clat (usec): min=14, max=12517, avg=472.19, stdev=331.30
lat (usec): min=43, max=1949, avg=435.56, stdev=208.15 | lat (usec): min=17, max=12520, avg=477.70, stdev=334.84
| 1.00th=[ 281], 5.00th=[ 289], 10.00th=[ 297], 20. | | 1.00th=[ 265], 5.00th=[ 277], 10.00th=[ 289], 20.
| 30.00th=[ 306], 40.00th=[ 310], 50.00th=[ 310], 60. | | 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 314], 60.
| 70.00th=[ 330], 80.00th=[ 709], 90.00th=[ 750], 95. | | 70.00th=[ 347], 80.00th=[ 562], 90.00th=[ 1057], 95.
| 99.00th=[ 971], 99.50th=[ 1254], 99.90th=[ 1450], 99. | | 99.00th=[ 1434], 99.50th=[ 1696], 99.90th=[ 2040], 99.
| 99.99th=[ 1582] | | 99.99th=[ 2474]
Type=write,Size=4K,NumJobs=1,depth=1 Type=write,Size=4K,NumJobs=1,depth=1
write: IOPS=53.8k, BW=210MiB/s (220MB/s)(2100MiB/10001msec) | write: IOPS=55.8k, BW=218MiB/s (229MB/s)(2180MiB/10000msec)
slat (nsec): min=2141, max=79859, avg=2640.37, stdev=1677 | slat (nsec): min=2138, max=44613, avg=2552.61, stdev=1404
clat (nsec): min=616, max=110520, avg=15483.87, stdev=567 | clat (nsec): min=627, max=624863, avg=14910.38, stdev=517
lat (usec): min=13, max=116, avg=18.16, stdev= 6.15 | lat (usec): min=13, max=627, avg=17.50, stdev= 5.55
| 1.00th=[12864], 5.00th=[12992], 10.00th=[12992], 20. | | 1.00th=[12736], 5.00th=[12864], 10.00th=[12864], 20.
| 30.00th=[13120], 40.00th=[13248], 50.00th=[13248], 60. | | 30.00th=[12992], 40.00th=[12992], 50.00th=[12992], 60.
| 70.00th=[13504], 80.00th=[15552], 90.00th=[20864], 95. | | 70.00th=[13248], 80.00th=[13632], 90.00th=[20352], 95.
| 99.00th=[36096], 99.50th=[39168], 99.90th=[50432], 99. | | 99.00th=[35584], 99.50th=[37120], 99.90th=[44800], 99.
| 99.99th=[74240] | | 99.99th=[53504]
Type=write,Size=4K,NumJobs=1,depth=16 Type=write,Size=4K,NumJobs=1,depth=16
write: IOPS=280k, BW=1093MiB/s (1146MB/s)(10.7GiB/10001msec | write: IOPS=325k, BW=1271MiB/s (1332MB/s)(12.4GiB/10001msec
slat (nsec): min=1746, max=32381, avg=2137.66, stdev=633. | slat (nsec): min=1697, max=32986, avg=2232.67, stdev=205.
clat (usec): min=41, max=167, avg=54.70, stdev=12.58 | clat (nsec): min=14620, max=89670, avg=46638.22, stdev=11
lat (usec): min=43, max=169, avg=56.87, stdev=12.60 | lat (nsec): min=16814, max=91833, avg=48907.48, stdev=11
| 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 44], 20. | | 1.00th=[44800], 5.00th=[45312], 10.00th=[45824], 20.
| 30.00th=[ 45], 40.00th=[ 49], 50.00th=[ 52], 60. | | 30.00th=[46336], 40.00th=[46336], 50.00th=[46336], 60.
| 70.00th=[ 59], 80.00th=[ 64], 90.00th=[ 72], 95. | | 70.00th=[46848], 80.00th=[46848], 90.00th=[47360], 95.
| 99.00th=[ 96], 99.50th=[ 111], 99.90th=[ 130], 99. | | 99.00th=[50944], 99.50th=[53504], 99.90th=[58112], 99.
| 99.99th=[ 147] | | 99.99th=[63744]
Type=write,Size=4K,NumJobs=1,depth=64 Type=write,Size=4K,NumJobs=1,depth=64
write: IOPS=340k, BW=1329MiB/s (1394MB/s)(12.0GiB/10001msec | write: IOPS=322k, BW=1259MiB/s (1320MB/s)(12.3GiB/10001msec
slat (nsec): min=1728, max=34272, avg=2133.44, stdev=216. | slat (nsec): min=1697, max=33655, avg=2273.54, stdev=208.
clat (usec): min=76, max=257, avg=185.64, stdev= 1.60 | clat (usec): min=18, max=238, avg=195.95, stdev= 2.07
lat (usec): min=78, max=259, avg=187.81, stdev= 1.61 | lat (usec): min=20, max=241, avg=198.26, stdev= 2.08
| 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20. | | 1.00th=[ 192], 5.00th=[ 194], 10.00th=[ 194], 20.
| 30.00th=[ 186], 40.00th=[ 186], 50.00th=[ 186], 60. | | 30.00th=[ 196], 40.00th=[ 196], 50.00th=[ 196], 60.
| 70.00th=[ 186], 80.00th=[ 188], 90.00th=[ 188], 95. | | 70.00th=[ 198], 80.00th=[ 198], 90.00th=[ 198], 95.
| 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 194], 99. | | 99.00th=[ 202], 99.50th=[ 204], 99.90th=[ 208], 99.
| 99.99th=[ 200] | | 99.99th=[ 212]
Type=write,Size=4K,NumJobs=16,depth=1 Type=write,Size=4K,NumJobs=16,depth=1
write: IOPS=863k, BW=3371MiB/s (3535MB/s)(32.9GiB/10001msec | write: IOPS=855k, BW=3339MiB/s (3502MB/s)(32.6GiB/10001msec
slat (nsec): min=1980, max=66528, avg=2566.89, stdev=734. | slat (nsec): min=1832, max=65363, avg=2579.38, stdev=826.
clat (nsec): min=583, max=653122, avg=15498.34, stdev=199 | clat (nsec): min=472, max=134403, avg=15646.36, stdev=196
lat (usec): min=12, max=655, avg=18.11, stdev= 2.17 | lat (usec): min=12, max=136, avg=18.28, stdev= 2.21
| 1.00th=[12992], 5.00th=[13888], 10.00th=[14144], 20. | | 1.00th=[13120], 5.00th=[13760], 10.00th=[14016], 20.
| 30.00th=[14656], 40.00th=[14912], 50.00th=[15168], 60. | | 30.00th=[14656], 40.00th=[14912], 50.00th=[15168], 60.
| 70.00th=[15680], 80.00th=[16192], 90.00th=[17280], 95. | | 70.00th=[15936], 80.00th=[16512], 90.00th=[17536], 95.
| 99.00th=[23168], 99.50th=[25216], 99.90th=[33536], 99. | | 99.00th=[23680], 99.50th=[25728], 99.90th=[33536], 99.
| 99.99th=[72192] | | 99.99th=[37120]
Type=write,Size=4K,NumJobs=16,depth=16 Type=write,Size=4K,NumJobs=16,depth=16
write: IOPS=2631k, BW=10.0GiB/s (10.8GB/s)(100GiB/10002msec | write: IOPS=1826k, BW=7133MiB/s (7480MB/s)(69.7GiB/10001mse
slat (nsec): min=1629, max=70383, avg=2374.75, stdev=939. | slat (nsec): min=1639, max=113071, avg=3312.71, stdev=236
clat (usec): min=28, max=370, avg=94.53, stdev=12.94 | clat (usec): min=40, max=454, avg=136.35, stdev=35.22
lat (usec): min=31, max=387, avg=96.95, stdev=12.90 | lat (usec): min=43, max=456, avg=139.72, stdev=35.40
| 1.00th=[ 72], 5.00th=[ 78], 10.00th=[ 80], 20. | | 1.00th=[ 72], 5.00th=[ 91], 10.00th=[ 101], 20.
| 30.00th=[ 89], 40.00th=[ 94], 50.00th=[ 97], 60. | | 30.00th=[ 121], 40.00th=[ 127], 50.00th=[ 133], 60.
| 70.00th=[ 100], 80.00th=[ 102], 90.00th=[ 105], 95. | | 70.00th=[ 143], 80.00th=[ 149], 90.00th=[ 182], 95.
| 99.00th=[ 141], 99.50th=[ 172], 99.90th=[ 190], 99. | | 99.00th=[ 258], 99.50th=[ 273], 99.90th=[ 314], 99.
| 99.99th=[ 231] | | 99.99th=[ 367]
Type=write,Size=4K,NumJobs=16,depth=64 Type=write,Size=4K,NumJobs=16,depth=64
write: IOPS=2615k, BW=9.97GiB/s (10.7GB/s)(99.7GiB/10001mse | write: IOPS=1875k, BW=7325MiB/s (7680MB/s)(71.5GiB/10001mse
slat (nsec): min=1633, max=196327, avg=2501.76, stdev=263 | slat (nsec): min=1638, max=627057, avg=3232.91, stdev=324
clat (usec): min=18, max=1044, avg=388.76, stdev=64.99 | clat (usec): min=49, max=1357, avg=542.32, stdev=162.71
lat (usec): min=21, max=1052, avg=391.30, stdev=65.69 | lat (usec): min=52, max=1384, avg=545.61, stdev=162.80
| 1.00th=[ 318], 5.00th=[ 338], 10.00th=[ 347], 20. | | 1.00th=[ 202], 5.00th=[ 277], 10.00th=[ 330], 20.
| 30.00th=[ 367], 40.00th=[ 375], 50.00th=[ 383], 60. | | 30.00th=[ 453], 40.00th=[ 498], 50.00th=[ 537], 60.
| 70.00th=[ 392], 80.00th=[ 396], 90.00th=[ 412], 95. | | 70.00th=[ 635], 80.00th=[ 685], 90.00th=[ 758], 95.
| 99.00th=[ 734], 99.50th=[ 750], 99.90th=[ 775], 99. | | 99.00th=[ 922], 99.50th=[ 955], 99.90th=[ 1029], 99.
| 99.99th=[ 799] | | 99.99th=[ 1123]
Type=write,Size=4K,NumJobs=56,depth=1 Type=write,Size=4K,NumJobs=56,depth=1
write: IOPS=1419k, BW=5545MiB/s (5814MB/s)(54.2GiB/10002mse | write: IOPS=1416k, BW=5533MiB/s (5802MB/s)(54.0GiB/10002mse
slat (nsec): min=1813, max=624655, avg=3014.79, stdev=214 | slat (nsec): min=1846, max=533231, avg=2881.71, stdev=111
clat (nsec): min=498, max=1174.4k, avg=35813.70, stdev=12 | clat (nsec): min=498, max=4453.2k, avg=36057.87, stdev=38
lat (usec): min=13, max=1177, avg=38.89, stdev=12.35 | lat (usec): min=15, max=4456, avg=39.01, stdev= 4.01
| 1.00th=[ 27], 5.00th=[ 31], 10.00th=[ 32], 20. | | 1.00th=[29312], 5.00th=[32128], 10.00th=[33024], 20.
| 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 36], 60. | | 30.00th=[34560], 40.00th=[35072], 50.00th=[35584], 60.
| 70.00th=[ 37], 80.00th=[ 38], 90.00th=[ 40], 95. | | 70.00th=[37120], 80.00th=[37632], 90.00th=[38656], 95.
| 99.00th=[ 51], 99.50th=[ 56], 99.90th=[ 229], 99. | | 99.00th=[49408], 99.50th=[52480], 99.90th=[56576], 99.
| 99.99th=[ 498] | | 99.99th=[64768]
Type=write,Size=4K,NumJobs=56,depth=16 Type=write,Size=4K,NumJobs=56,depth=16
write: IOPS=2489k, BW=9724MiB/s (10.2GB/s)(94.0GiB/10002mse | write: IOPS=1616k, BW=6314MiB/s (6621MB/s)(61.7GiB/10001mse
slat (nsec): min=1637, max=742083, avg=3046.89, stdev=226 | slat (nsec): min=1649, max=738008, avg=3245.81, stdev=163
clat (usec): min=39, max=1654, avg=356.23, stdev=126.26 | clat (usec): min=44, max=5046, avg=550.36, stdev=143.07
lat (usec): min=44, max=1657, avg=359.34, stdev=126.24 | lat (usec): min=46, max=5049, avg=553.68, stdev=143.03
| 1.00th=[ 217], 5.00th=[ 237], 10.00th=[ 247], 20. | | 1.00th=[ 215], 5.00th=[ 343], 10.00th=[ 404], 20.
| 30.00th=[ 277], 40.00th=[ 285], 50.00th=[ 297], 60. | | 30.00th=[ 490], 40.00th=[ 510], 50.00th=[ 529], 60.
| 70.00th=[ 383], 80.00th=[ 474], 90.00th=[ 562], 95. | | 70.00th=[ 586], 80.00th=[ 660], 90.00th=[ 758], 95.
| 99.00th=[ 709], 99.50th=[ 742], 99.90th=[ 865], 99. | | 99.00th=[ 938], 99.50th=[ 996], 99.90th=[ 1106], 99.
| 99.99th=[ 1074] | | 99.99th=[ 1221]
Type=write,Size=4K,NumJobs=56,depth=64 Type=write,Size=4K,NumJobs=56,depth=64
write: IOPS=2216k, BW=8657MiB/s (9078MB/s)(84.6GiB/10003mse | write: IOPS=1625k, BW=6348MiB/s (6656MB/s)(62.0GiB/10003mse
slat (nsec): min=1658, max=1799.0k, avg=16827.79, stdev=6 | slat (nsec): min=1656, max=1933.3k, avg=21620.58, stdev=7
clat (usec): min=40, max=7314, avg=1599.52, stdev=743.15 | clat (usec): min=66, max=8663, avg=2182.95, stdev=879.90
lat (usec): min=43, max=7318, avg=1616.41, stdev=754.88 | lat (usec): min=69, max=8669, avg=2204.64, stdev=895.51
| 1.00th=[ 400], 5.00th=[ 644], 10.00th=[ 734], 20. | | 1.00th=[ 693], 5.00th=[ 971], 10.00th=[ 1074], 20.
| 30.00th=[ 1254], 40.00th=[ 1418], 50.00th=[ 1516], 60. | | 30.00th=[ 1942], 40.00th=[ 2114], 50.00th=[ 2212], 60.
| 70.00th=[ 1778], 80.00th=[ 2057], 90.00th=[ 2638], 95. | | 70.00th=[ 2343], 80.00th=[ 2507], 90.00th=[ 3392], 95.
| 99.00th=[ 3884], 99.50th=[ 4178], 99.90th=[ 4817], 99. | | 99.00th=[ 4883], 99.50th=[ 5211], 99.90th=[ 5800], 99.
| 99.99th=[ 5604] | | 99.99th=[ 6587]
Type=write,Size=64K,NumJobs=1,depth=1 Type=write,Size=64K,NumJobs=1,depth=1
write: IOPS=23.1k, BW=1444MiB/s (1515MB/s)(14.1GiB/10001mse | write: IOPS=23.3k, BW=1454MiB/s (1525MB/s)(14.2GiB/10001mse
slat (nsec): min=4822, max=98365, avg=6876.23, stdev=3203 | slat (nsec): min=4849, max=39702, avg=6880.02, stdev=3240
clat (usec): min=3, max=121, avg=35.83, stdev= 9.35 | clat (usec): min=17, max=133, avg=35.53, stdev= 9.39
lat (usec): min=33, max=133, avg=42.75, stdev=10.50 | lat (usec): min=33, max=146, avg=42.46, stdev=10.52
| 1.00th=[29568], 5.00th=[29568], 10.00th=[29568], 20. | | 1.00th=[29312], 5.00th=[29312], 10.00th=[29568], 20.
| 30.00th=[29824], 40.00th=[29824], 50.00th=[30080], 60. | | 30.00th=[29568], 40.00th=[29824], 50.00th=[29824], 60.
| 70.00th=[37120], 80.00th=[44800], 90.00th=[49408], 95. | | 70.00th=[36608], 80.00th=[43776], 90.00th=[49408], 95.
| 99.00th=[68096], 99.50th=[72192], 99.90th=[85504], 99. | | 99.00th=[68096], 99.50th=[72192], 99.90th=[85504], 99.
| 99.99th=[90624] | 99.99th=[90624]
Type=write,Size=64K,NumJobs=1,depth=16 Type=write,Size=64K,NumJobs=1,depth=16
write: IOPS=140k, BW=8743MiB/s (9167MB/s)(85.4GiB/10001msec | write: IOPS=146k, BW=9109MiB/s (9552MB/s)(88.0GiB/10001msec
slat (nsec): min=3861, max=47468, avg=5274.03, stdev=845. | slat (nsec): min=3787, max=44724, avg=5539.44, stdev=622.
clat (usec): min=82, max=297, avg=108.76, stdev=13.98 | clat (usec): min=76, max=228, avg=103.90, stdev=11.08
lat (usec): min=87, max=302, avg=114.07, stdev=13.98 | lat (usec): min=82, max=233, avg=109.48, stdev=11.10
| 1.00th=[ 92], 5.00th=[ 96], 10.00th=[ 98], 20. | | 1.00th=[ 90], 5.00th=[ 93], 10.00th=[ 95], 20.
| 30.00th=[ 102], 40.00th=[ 104], 50.00th=[ 105], 60. | | 30.00th=[ 98], 40.00th=[ 99], 50.00th=[ 101], 60.
| 70.00th=[ 111], 80.00th=[ 114], 90.00th=[ 123], 95. | | 70.00th=[ 105], 80.00th=[ 109], 90.00th=[ 120], 95.
| 99.00th=[ 167], 99.50th=[ 180], 99.90th=[ 206], 99. | | 99.00th=[ 141], 99.50th=[ 149], 99.90th=[ 182], 99.
| 99.99th=[ 239] | | 99.99th=[ 200]
Type=write,Size=64K,NumJobs=1,depth=64 Type=write,Size=64K,NumJobs=1,depth=64
write: IOPS=144k, BW=8977MiB/s (9413MB/s)(87.7GiB/10001msec | write: IOPS=139k, BW=8708MiB/s (9131MB/s)(85.0GiB/10001msec
slat (nsec): min=2697, max=72397, avg=4695.54, stdev=1467 | slat (nsec): min=2676, max=64177, avg=5388.79, stdev=1847
clat (usec): min=125, max=987, avg=440.52, stdev=70.72 | clat (usec): min=39, max=894, avg=453.54, stdev=56.76
lat (usec): min=135, max=993, avg=445.25, stdev=70.70 | lat (usec): min=45, max=898, avg=458.97, stdev=57.24
| 1.00th=[ 330], 5.00th=[ 338], 10.00th=[ 347], 20. | | 1.00th=[ 338], 5.00th=[ 355], 10.00th=[ 367], 20.
| 30.00th=[ 396], 40.00th=[ 412], 50.00th=[ 437], 60. | | 30.00th=[ 408], 40.00th=[ 457], 50.00th=[ 478], 60.
| 70.00th=[ 478], 80.00th=[ 498], 90.00th=[ 529], 95. | | 70.00th=[ 494], 80.00th=[ 498], 90.00th=[ 506], 95.
| 99.00th=[ 619], 99.50th=[ 668], 99.90th=[ 791], 99. | | 99.00th=[ 545], 99.50th=[ 562], 99.90th=[ 734], 99.
| 99.99th=[ 889] | | 99.99th=[ 848]
Type=write,Size=64K,NumJobs=16,depth=1 Type=write,Size=64K,NumJobs=16,depth=1
write: IOPS=174k, BW=10.6GiB/s (11.4GB/s)(106GiB/10001msec) | write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10001msec)
slat (nsec): min=4083, max=63053, avg=5530.24, stdev=1175 | slat (nsec): min=4383, max=73755, avg=5607.60, stdev=1278
clat (usec): min=17, max=167, avg=85.80, stdev= 7.67 | clat (usec): min=22, max=424, avg=85.24, stdev= 8.07
lat (usec): min=37, max=172, avg=91.37, stdev= 7.76 | lat (usec): min=38, max=430, avg=90.90, stdev= 8.07
| 1.00th=[ 60], 5.00th=[ 76], 10.00th=[ 80], 20. | | 1.00th=[ 68], 5.00th=[ 76], 10.00th=[ 79], 20.
| 30.00th=[ 84], 40.00th=[ 85], 50.00th=[ 86], 60. | | 30.00th=[ 82], 40.00th=[ 83], 50.00th=[ 84], 60.
| 70.00th=[ 88], 80.00th=[ 90], 90.00th=[ 94], 95. | | 70.00th=[ 87], 80.00th=[ 91], 90.00th=[ 98], 95.
| 99.00th=[ 109], 99.50th=[ 115], 99.90th=[ 128], 99. | | 99.00th=[ 106], 99.50th=[ 112], 99.90th=[ 124], 99.
| 99.99th=[ 143] | | 99.99th=[ 137]
Type=write,Size=64K,NumJobs=16,depth=16 Type=write,Size=64K,NumJobs=16,depth=16
write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) | write: IOPS=176k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec)
slat (nsec): min=3477, max=59587, avg=6630.17, stdev=1304 | slat (nsec): min=3611, max=70931, avg=6507.12, stdev=1389
clat (usec): min=125, max=5561, avg=1451.88, stdev=382.58 | clat (usec): min=109, max=3915, avg=1451.13, stdev=409.41
lat (usec): min=136, max=5567, avg=1458.55, stdev=382.73 | lat (usec): min=129, max=3920, avg=1457.68, stdev=409.50
| 1.00th=[ 1254], 5.00th=[ 1270], 10.00th=[ 1270], 20. | 1.00th=[ 1254], 5.00th=[ 1270], 10.00th=[ 1270], 20.
| 30.00th=[ 1352], 40.00th=[ 1352], 50.00th=[ 1352], 60. | | 30.00th=[ 1270], 40.00th=[ 1270], 50.00th=[ 1287], 60.
| 70.00th=[ 1369], 80.00th=[ 1369], 90.00th=[ 1401], 95. | | 70.00th=[ 1352], 80.00th=[ 1369], 90.00th=[ 2540], 95.
| 99.00th=[ 2737], 99.50th=[ 2737], 99.90th=[ 2769], 99. | 99.00th=[ 2737], 99.50th=[ 2737], 99.90th=[ 2769], 99.
| 99.99th=[ 2802] | | 99.99th=[ 2769]
Type=write,Size=64K,NumJobs=16,depth=64 Type=write,Size=64K,NumJobs=16,depth=64
write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10004msec) write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10004msec)
slat (usec): min=2, max=6867, avg=10.05, stdev=84.22 | slat (nsec): min=3042, max=96073, avg=6866.24, stdev=2170
clat (usec): min=114, max=17739, avg=5829.80, stdev=1403. | clat (usec): min=124, max=10103, avg=5850.40, stdev=798.1
lat (usec): min=121, max=17748, avg=5839.90, stdev=1417. | lat (usec): min=131, max=10109, avg=5857.31, stdev=798.1
| 1.00th=[ 4228], 5.00th=[ 4490], 10.00th=[ 4621], 20. | | 1.00th=[ 4752], 5.00th=[ 4883], 10.00th=[ 4948], 20.
| 30.00th=[ 4948], 40.00th=[ 5145], 50.00th=[ 5473], 60. | | 30.00th=[ 5276], 40.00th=[ 5407], 50.00th=[ 5669], 60.
| 70.00th=[ 6194], 80.00th=[ 6718], 90.00th=[ 7242], 95. | | 70.00th=[ 6259], 80.00th=[ 6587], 90.00th=[ 7046], 95.
| 99.00th=[11338], 99.50th=[12125], 99.90th=[14484], 99. | | 99.00th=[ 7767], 99.50th=[ 7898], 99.90th=[ 8455], 99.
| 99.99th=[16188] | | 99.99th=[ 8848]
Type=write,Size=64K,NumJobs=56,depth=1 Type=write,Size=64K,NumJobs=56,depth=1
write: IOPS=176k, BW=10.7GiB/s (11.5GB/s)(107GiB/10001msec) | write: IOPS=175k, BW=10.7GiB/s (11.4GB/s)(107GiB/10002msec)
slat (nsec): min=4056, max=63403, avg=6384.23, stdev=1521 | slat (nsec): min=4029, max=63780, avg=5771.34, stdev=1537
clat (usec): min=57, max=4543, avg=312.06, stdev=48.67 | clat (usec): min=32, max=979, avg=314.28, stdev=57.25
lat (usec): min=66, max=4549, avg=318.50, stdev=48.73 | lat (usec): min=40, max=985, avg=320.11, stdev=57.29
| 1.00th=[ 188], 5.00th=[ 289], 10.00th=[ 297], 20. | | 1.00th=[ 186], 5.00th=[ 293], 10.00th=[ 297], 20.
| 30.00th=[ 306], 40.00th=[ 306], 50.00th=[ 310], 60. | | 30.00th=[ 297], 40.00th=[ 302], 50.00th=[ 302], 60.
| 70.00th=[ 310], 80.00th=[ 314], 90.00th=[ 318], 95. | | 70.00th=[ 310], 80.00th=[ 310], 90.00th=[ 326], 95.
| 99.00th=[ 478], 99.50th=[ 611], 99.90th=[ 627], 99. | | 99.00th=[ 603], 99.50th=[ 611], 99.90th=[ 627], 99.
| 99.99th=[ 750] | 99.99th=[ 750]
Type=write,Size=64K,NumJobs=56,depth=16 Type=write,Size=64K,NumJobs=56,depth=16
write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10005msec) write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10005msec)
slat (usec): min=3, max=197, avg= 7.76, stdev= 2.34 | slat (nsec): min=3542, max=89310, avg=7585.26, stdev=2460
clat (usec): min=183, max=15544, avg=5102.42, stdev=1298. | clat (usec): min=190, max=14367, avg=5116.17, stdev=1396.
lat (usec): min=193, max=15551, avg=5110.24, stdev=1298. | lat (usec): min=198, max=14375, avg=5123.82, stdev=1396.
| 1.00th=[ 2442], 5.00th=[ 2540], 10.00th=[ 2573], 20. | | 1.00th=[ 2540], 5.00th=[ 2540], 10.00th=[ 2573], 20.
| 30.00th=[ 5080], 40.00th=[ 5080], 50.00th=[ 5080], 60. | | 30.00th=[ 5080], 40.00th=[ 5080], 50.00th=[ 5145], 60.
| 70.00th=[ 5145], 80.00th=[ 5538], 90.00th=[ 6718], 95. | | 70.00th=[ 5145], 80.00th=[ 5669], 90.00th=[ 6849], 95.
| 99.00th=[ 8717], 99.50th=[ 9241], 99.90th=[10552], 99. | | 99.00th=[ 8979], 99.50th=[ 9634], 99.90th=[11207], 99.
| 99.99th=[12125] | | 99.99th=[12780]
Type=write,Size=64K,NumJobs=56,depth=64 Type=write,Size=64K,NumJobs=56,depth=64
write: IOPS=175k, BW=10.7GiB/s (11.4GB/s)(107GiB/10013msec) | write: IOPS=174k, BW=10.6GiB/s (11.4GB/s)(107GiB/10011msec)
slat (usec): min=2, max=40799, avg=182.18, stdev=918.40 | slat (usec): min=3, max=32839, avg=176.71, stdev=925.53
clat (usec): min=178, max=107843, avg=20332.76, stdev=698 | clat (usec): min=189, max=132076, avg=20366.61, stdev=728
lat (usec): min=182, max=107849, avg=20515.01, stdev=709 | lat (usec): min=218, max=132086, avg=20543.39, stdev=740
| 1.00th=[ 7832], 5.00th=[ 9372], 10.00th=[11469], 20. | | 1.00th=[ 7898], 5.00th=[ 9110], 10.00th=[10945], 20.
| 30.00th=[18220], 40.00th=[19530], 50.00th=[20317], 60. | | 30.00th=[17957], 40.00th=[19530], 50.00th=[20317], 60.
| 70.00th=[20841], 80.00th=[22938], 90.00th=[27657], 95. | | 70.00th=[21103], 80.00th=[23462], 90.00th=[28705], 95.
| 99.00th=[45876], 99.50th=[51643], 99.90th=[64226], 99. | | 99.00th=[45876], 99.50th=[50594], 99.90th=[63177], 99.
| 99.99th=[84411] | 99.99th=[84411]
