Can't get RAID5/RAID6 NVMe randomread IOPS - AMD ROME what am I missing?????

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Sorry, this will be a long email with everything I find to be relevant, but I can get over 110GB/s of 4kB random reads from individual NVMe SSDs, but I'm at a loss why mdraid can only do a very small  fraction of it.   I'm at my "organizational world record" for sustained IOPS, but I need protected IOPS to do something useful.     This is everything I do to a server to make the I/O crank.....My role is that of a lab researcher/resident expert/consultant.   I'm just stumped why I can't do better.   If there is a fine manual that somebody can point me to, I'm happy to read it.......

I have tried both RAID5 and RAID6 trying to be highly cognizant of NUMAness.    The ROME is set to numas per socket to 1 and the BIOS is set to maximize infinity fabric performance and pcie performance via AMD's white papers.   NVMe drives are all Gen4 (I believe HPE rebadged SAMSUNG 1733a? - I can get the drives doing 1.45M 4KB random reads  each if I try hard.

Everything I can think to share:

[root@<server> <server>]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.4 (Ootpa)
[root@<server> <server>]# uname -r
4.18.0-305.el8.x86_64

root@<server> ~]# modinfo raid6
filename:       /lib/modules/4.18.0-305.el8.x86_64/kernel/drivers/md/raid456.ko.xz
alias:          raid6
alias:          raid5
alias:          md-level-6
alias:          md-raid6
alias:          md-personality-8
alias:          md-level-4
alias:          md-level-5
alias:          md-raid4
alias:          md-raid5
alias:          md-personality-4
description:    RAID4/5/6 (striping with parity) personality for MD
license:        GPL
rhelversion:    8.4
srcversion:     FE86A53E1C1CDAE8F972CBA
depends:        async_raid6_recov,async_pq,libcrc32c,raid6_pq,async_tx,async_memcpy,async_xor
intree:         Y
name:           raid456
vermagic:       4.18.0-305.el8.x86_64 SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         Red Hat Enterprise Linux kernel signing key

[root@<server> ~]# lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
nvme16n1     259:0    0   1.8T  0 disk  
├─nvme16n1p1 259:1    0   512M  0 part  /boot/efi
├─nvme16n1p2 259:2    0   512M  0 part  /boot
├─nvme16n1p3 259:3    0  49.4G  0 part  [SWAP]
└─nvme16n1p4 259:4    0   1.7T  0 part  /
nvme0n1      259:5    0    14T  0 disk  
└─md0          9:0    0 139.7T  0 raid5 
nvme1n1      259:6    0    14T  0 disk  
└─md0          9:0    0 139.7T  0 raid5 
nvme2n1      259:7    0    14T  0 disk  
└─md0          9:0    0 139.7T  0 raid5 
nvme3n1      259:8    0    14T  0 disk  
└─md0          9:0    0 139.7T  0 raid5 
nvme7n1      259:9    0    14T  0 disk  
└─md0          9:0    0 139.7T  0 raid5 
nvme11n1     259:10   0    14T  0 disk  
└─md0          9:0    0 139.7T  0 raid5 
nvme10n1     259:11   0    14T  0 disk  
└─md0          9:0    0 139.7T  0 raid5 
nvme14n1     259:12   0    14T  0 disk  
└─md1          9:1    0 139.7T  0 raid5 
nvme5n1      259:13   0    14T  0 disk  
└─md0          9:0    0 139.7T  0 raid5 
nvme8n1      259:14   0    14T  0 disk  
└─md0          9:0    0 139.7T  0 raid5 
nvme6n1      259:15   0    14T  0 disk  
└─md0          9:0    0 139.7T  0 raid5 
nvme9n1      259:16   0    14T  0 disk  
└─md0          9:0    0 139.7T  0 raid5 
nvme15n1     259:17   0    14T  0 disk  
└─md1          9:1    0 139.7T  0 raid5 
nvme20n1     259:18   0    14T  0 disk  
└─md1          9:1    0 139.7T  0 raid5 
nvme13n1     259:19   0    14T  0 disk  
└─md1          9:1    0 139.7T  0 raid5 
nvme18n1     259:20   0    14T  0 disk  
└─md1          9:1    0 139.7T  0 raid5 
nvme4n1      259:21   0    14T  0 disk  
└─md0          9:0    0 139.7T  0 raid5 
nvme21n1     259:22   0    14T  0 disk  
└─md1          9:1    0 139.7T  0 raid5 
nvme22n1     259:23   0    14T  0 disk  
└─md1          9:1    0 139.7T  0 raid5 
nvme24n1     259:24   0    14T  0 disk  
└─md1          9:1    0 139.7T  0 raid5 
nvme12n1     259:25   0    14T  0 disk  
└─md1          9:1    0 139.7T  0 raid5 
nvme17n1     259:26   0    14T  0 disk  
└─md1          9:1    0 139.7T  0 raid5
nvme19n1     259:27   0    14T  0 disk  
└─md1          9:1    0 139.7T  0 raid5 
nvme23n1     259:28   0    14T  0 disk  
└─md1          9:1    0 139.7T  0 raid5

[root@<server> ~]# lsblk -o KNAME,MODEL,VENDOR
KNAME      MODEL                                    VENDOR
nvme0n1    MZXL515THALA-000H3                       
nvme1n1    MZXL515THALA-000H3                       
nvme2n1    MZXL515THALA-000H3                       
nvme3n1    MZXL515THALA-000H3                       
nvme7n1    MZXL515THALA-000H3                       
nvme11n1   MZXL515THALA-000H3                       
nvme10n1   MZXL515THALA-000H3                       
nvme14n1   MZXL515THALA-000H3                       
nvme5n1    MZXL515THALA-000H3                       
nvme8n1    MZXL515THALA-000H3                       
nvme6n1    MZXL515THALA-000H3                       
nvme9n1    MZXL515THALA-000H3                       
nvme15n1   MZXL515THALA-000H3                       
nvme20n1   MZXL515THALA-000H3                       
nvme13n1   MZXL515THALA-000H3                       
nvme18n1   MZXL515THALA-000H3                       
nvme4n1    MZXL515THALA-000H3                       
nvme21n1   MZXL515THALA-000H3                       
nvme22n1   MZXL515THALA-000H3                       
nvme24n1   MZXL515THALA-000H3                       
nvme12n1   MZXL515THALA-000H3                       
nvme17n1   MZXL515THALA-000H3                       
nvme19n1   MZXL515THALA-000H3                       
nvme23n1   MZXL515THALA-000H3    

[root@<server> jim]# ./map_numa.sh       (16 is the boot drive 0-11 on numa 0, 12-16,17-24 on numa 1)
device: nvme8 numanode: 0
device: nvme9 numanode: 0
device: nvme10 numanode: 0
device: nvme11 numanode: 0
device: nvme4 numanode: 0
device: nvme5 numanode: 0
device: nvme6 numanode: 0
device: nvme7 numanode: 0
device: nvme2 numanode: 0
device: nvme3 numanode: 0
device: nvme0 numanode: 0
device: nvme1 numanode: 0
device: nvme21 numanode: 1
device: nvme22 numanode: 1
device: nvme23 numanode: 1
device: nvme24 numanode: 1
device: nvme16 numanode: 1
device: nvme17 numanode: 1
device: nvme18 numanode: 1
device: nvme19 numanode: 1
device: nvme20 numanode: 1
device: nvme14 numanode: 1
device: nvme15 numanode: 1
device: nvme12 numanode: 1
device: nvme13 numanode: 1

[root@<server> jim]# cat /etc/udev/rules.d/99-abj.nr_32.rules 
KERNEL=="nvme*[0-9]n*[0-9]",ATTRS{model}=="MZXL515THALA-000H3",ATTR{queue/io_poll}="1",ATTR{queue/io_poll_delay}="100000",ATTR{queue/nomerges}="2",ATTR{queue/nr_requests}="1023",ATTR{queue/rotational}="0",ATTR{queue/rq_affinity}="2",ATTR{queue/scheduler}="none",ATTR{queue/add_random}="0",ATTR{queue/max_sectors_kb}="4096",PROGRAM="/usr/sbin/nvme set-feature /dev/%k --feature-id 8 --value 522 "    {coalesce up to 10 interrupts per device}



SUBSYSTEM=="block", ACTION=="add|change", KERNEL=="md*", ATTR{md/sync_speed_max}="2000000",ATTR{md/group_thread_cnt}="64", ATTR{md/stripe_cache_size}="8192",ATTR{queue/io_poll}="1",ATTR{queue/io_poll_delay}="100000",ATTR{queue/nomerges}="2",ATTR{queue/nr_requests}="1023",ATTR{queue/rotational}="0",ATTR{queue/rq_affinity}="2",ATTR{queue/scheduler}="none",ATTR{queue/add_random}="0",ATTR{queue/max_sectors_kb}="4096"

(I know the 1023 doesn't work in the md, but there for reference) - we tune for max iops, not for latency, thus the going hard at rq_affinity, nomerges, etc.....

[root@<server> <server>]# cat /proc/mdstat  (128K chunk is just something Fusion IO told me way back when and never needed to change)
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 nvme11n1[11](S) nvme10n1[10] nvme9n1[9] nvme8n1[8] nvme7n1[7] nvme6n1[6] nvme5n1[5] nvme4n1[4] nvme3n1[3] nvme2n1[2] nvme1n1[1] nvme0n1[0]
      150007961600 blocks super 1.2 level 5, 128k chunk, algorithm 2 [11/11] [UUUUUUUUUUU]
      bitmap: 0/112 pages [0KB], 65536KB chunk

md1 : active raid5 nvme24n1[11](S) nvme23n1[10] nvme22n1[9] nvme21n1[8] nvme20n1[7] nvme19n1[6] nvme18n1[5] nvme17n1[4] nvme15n1[3] nvme14n1[2] nvme13n1[1] nvme12n1[0]
      150007961600 blocks super 1.2 level 5, 128k chunk, algorithm 2 [11/11] [UUUUUUUUUUU]
      bitmap: 0/112 pages [0KB], 65536KB chunk

unused devices: <none>

[root@<server> /]# grep raid /var/log/messages......What troubles me is if mdraid checked parity on read, I could somewhat understand, but I would think the reads are nearly a pass through....
Jul 27 00:00:02 <server> rpmlist_verification[12745]: libblockdev-mdraid 2.24 Thu 22 Jul 2021 02:58:37 PM GMT
Jul 27 18:00:28 <server> kernel: raid6: sse2x1   gen()  9792 MB/s
Jul 27 18:00:28 <server> kernel: raid6: sse2x1   xor()  6436 MB/s
Jul 27 18:00:28 <server> kernel: raid6: sse2x2   gen() 11198 MB/s
Jul 27 18:00:28 <server> kernel: raid6: sse2x2   xor()  9546 MB/s
Jul 27 18:00:28 <server> kernel: raid6: sse2x4   gen() 14271 MB/s
Jul 27 18:00:29 <server> kernel: raid6: sse2x4   xor()  6354 MB/s
Jul 27 18:00:29 <server> kernel: raid6: avx2x1   gen() 22838 MB/s
Jul 27 18:00:29 <server> kernel: raid6: avx2x1   xor() 14069 MB/s
Jul 27 18:00:29 <server> kernel: raid6: avx2x2   gen() 26973 MB/s
Jul 27 18:00:29 <server> kernel: raid6: avx2x2   xor() 18380 MB/s
Jul 27 18:00:29 <server> kernel: raid6: avx2x4   gen() 26601 MB/s
Jul 27 18:00:29 <server> kernel: raid6: avx2x4   xor()  7025 MB/s
Jul 27 18:00:29 <server> kernel: raid6: using algorithm avx2x2 gen() 26973 MB/s
Jul 27 18:00:29 <server> kernel: raid6: .... xor() 18380 MB/s, rmw enabled
Jul 27 18:00:29 <server> kernel: raid6: using avx2x2 recovery algorithm

[root@<server> <server>]# cat fiojim.hpdl385.nps1
 [global]
name=random
iodepth=128
ioengine=libaio
direct=1
norandommap
group_reporting
randrepeat=1
random_generator=tausworthe64
bs=4k
rw=randread
numjobs=64
runtime=60

[socket0]
new_group
numa_mem_policy=bind:0
numa_cpu_nodes=0
filename=/dev/nvme0n1
filename=/dev/nvme1n1
filename=/dev/nvme2n1
filename=/dev/nvme3n1
filename=/dev/nvme4n1
filename=/dev/nvme5n1
filename=/dev/nvme6n1
filename=/dev/nvme7n1
filename=/dev/nvme8n1
filename=/dev/nvme9n1
filename=/dev/nvme10n1
filename=/dev/nvme11n1
[socket1]
new_group
numa_mem_policy=bind:1
numa_cpu_nodes=1
filename=/dev/nvme12n1
filename=/dev/nvme13n1
filename=/dev/nvme14n1
filename=/dev/nvme15n1
filename=/dev/nvme17n1
filename=/dev/nvme18n1
filename=/dev/nvme19n1
filename=/dev/nvme20n1
filename=/dev/nvme21n1
filename=/dev/nvme22n1
filename=/dev/nvme23n1
filename=/dev/nvme24n1
[socket0-md]
stonewall
numa_mem_policy=bind:0
numa_cpu_nodes=0
filename=/dev/md0
[socket1-md]
new_group
numa_mem_policy=bind:1
numa_cpu_nodes=1
filename=/dev/md1

iostat -xkz 1 with the drives raw:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.32    0.00   38.30    0.00    0.00   53.39

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
nvme0n1       1317510.00    0.00 5270044.00      0.00     0.00     0.00   0.00   0.00    0.31    0.00 411.95     4.00     0.00   0.00 100.40
nvme1n1       1317548.00    0.00 5270192.00      0.00     0.00     0.00   0.00   0.00    0.32    0.00 417.38     4.00     0.00   0.00 100.00
nvme2n1       1317578.00    0.00 5270316.00      0.00     0.00     0.00   0.00   0.00    0.31    0.00 414.77     4.00     0.00   0.00 100.20
nvme3n1       1317554.00    0.00 5270216.00      0.00     0.00     0.00   0.00   0.00    0.31    0.00 413.25     4.00     0.00   0.00 100.40
nvme7n1       1317559.00    0.00 5270236.00      0.00     0.00     0.00   0.00   0.00    0.33    0.00 430.03     4.00     0.00   0.00 100.40
nvme11n1      1317502.00    0.00 5269996.00      0.00     0.00     0.00   0.00   0.00    0.73    0.00 964.85     4.00     0.00   0.00 100.40
nvme10n1      1317656.00    0.00 5270624.00      0.00     0.00     0.00   0.00   0.00    0.80    0.00 1050.05     4.00     0.00   0.00 100.40
nvme14n1      1107632.00    0.00 4430528.00      0.00     0.00     0.00   0.00   0.00    0.28    0.00 307.52     4.00     0.00   0.00 100.40
nvme5n1       1317583.00    0.00 5270332.00      0.00     0.00     0.00   0.00   0.00    0.33    0.00 430.47     4.00     0.00   0.00 100.00
nvme8n1       1317617.00    0.00 5270468.00      0.00     0.00     0.00   0.00   0.00    0.74    0.00 972.52     4.00     0.00   0.00 101.00
nvme6n1       1317535.00    0.00 5270144.00      0.00     0.00     0.00   0.00   0.00    0.33    0.00 432.48     4.00     0.00   0.00 100.60
nvme9n1       1317582.00    0.00 5270328.00      0.00     0.00     0.00   0.00   0.00    0.75    0.00 992.82     4.00     0.00   0.00 100.40
nvme15n1      1107703.00    0.00 4430816.00      0.00     0.00     0.00   0.00   0.00    0.28    0.00 305.93     4.00     0.00   0.00 100.60
nvme20n1      1107712.00    0.00 4430848.00      0.00     0.00     0.00   0.00   0.00    0.28    0.00 306.72     4.00     0.00   0.00 100.20
nvme13n1      1107714.00    0.00 4430852.00      0.00     0.00     0.00   0.00   0.00    0.28    0.00 307.10     4.00     0.00   0.00 101.40
nvme18n1      1107674.00    0.00 4430696.00      0.00     0.00     0.00   0.00   0.00    0.28    0.00 306.04     4.00     0.00   0.00 100.20
nvme4n1       1317521.00    0.00 5270076.00      0.00     0.00     0.00   0.00   0.00    0.33    0.00 431.63     4.00     0.00   0.00 100.20
nvme21n1      1107714.00    0.00 4430856.00      0.00     0.00     0.00   0.00   0.00    0.28    0.00 309.11     4.00     0.00   0.00 100.40
nvme22n1      1107711.00    0.00 4430840.00      0.00     0.00     0.00   0.00   0.00    0.28    0.00 308.52     4.00     0.00   0.00 100.60
nvme24n1      1107441.00    0.00 4429768.00      0.00     0.00     0.00   0.00   0.00    3.86    0.00 4271.29     4.00     0.00   0.00 100.20
nvme12n1      1107733.00    0.00 4430932.00      0.00     0.00     0.00   0.00   0.00    0.28    0.00 307.70     4.00     0.00   0.00 100.40
nvme17n1      1107858.00    0.00 4431436.00      0.00     0.00     0.00   0.00   0.00    0.28    0.00 307.95     4.00     0.00   0.00 100.60
nvme19n1      1107766.00    0.00 4431064.00      0.00     0.00     0.00   0.00   0.00    0.28    0.00 307.17     4.00     0.00   0.00 100.40
nvme23n1      1108033.00    0.00 4432132.00      0.00     0.00     0.00   0.00   0.00    0.31    0.00 340.62     4.00     0.00   0.00 100.00



iostat -xkz 1 with the md's
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.56    0.00   49.94    0.00    0.00   49.51

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
nvme0n1       114589.00    0.00 458356.00      0.00     0.00     0.00   0.00   0.00    0.29    0.00  33.54     4.00     0.00   0.01 100.00
nvme1n1       115284.00    0.00 461136.00      0.00     0.00     0.00   0.00   0.00    0.29    0.00  33.77     4.00     0.00   0.01 100.00
nvme2n1       114911.00    0.00 459644.00      0.00     0.00     0.00   0.00   0.00    0.29    0.00  33.61     4.00     0.00   0.01 100.00
nvme3n1       114538.00    0.00 458152.00      0.00     0.00     0.00   0.00   0.00    0.29    0.00  33.55     4.00     0.00   0.01 100.00
nvme7n1       114524.00    0.00 458096.00      0.00     0.00     0.00   0.00   0.00    0.29    0.00  33.53     4.00     0.00   0.01 100.00
nvme10n1      114934.00    0.00 459736.00      0.00     0.00     0.00   0.00   0.00    0.29    0.00  33.61     4.00     0.00   0.01 100.00
nvme14n1      97399.00    0.00 389596.00      0.00     0.00     0.00   0.00   0.00    0.30    0.00  29.41     4.00     0.00   0.01 100.00
nvme5n1       114929.00    0.00 459716.00      0.00     0.00     0.00   0.00   0.00    0.29    0.00  33.61     4.00     0.00   0.01 100.00
nvme8n1       114393.00    0.00 457572.00      0.00     0.00     0.00   0.00   0.00    0.29    0.00  33.40     4.00     0.00   0.01  99.90
nvme6n1       114731.00    0.00 458924.00      0.00     0.00     0.00   0.00   0.00    0.29    0.00  33.56     4.00     0.00   0.01  99.90
nvme9n1       114146.00    0.00 456584.00      0.00     0.00     0.00   0.00   0.00    0.29    0.00  33.37     4.00     0.00   0.01  99.90
nvme15n1      96960.00    0.00 387840.00      0.00     0.00     0.00   0.00   0.00    0.30    0.00  29.30     4.00     0.00   0.01 100.00
nvme20n1      97171.00    0.00 388684.00      0.00     0.00     0.00   0.00   0.00    0.30    0.00  29.36     4.00     0.00   0.01 100.00
nvme13n1      96874.00    0.00 387496.00      0.00     0.00     0.00   0.00   0.00    0.30    0.00  29.31     4.00     0.00   0.01 100.00
nvme18n1      96696.00    0.00 386784.00      0.00     0.00     0.00   0.00   0.00    0.30    0.00  29.16     4.00     0.00   0.01 100.00
nvme4n1       115220.00    0.00 460876.00      0.00     0.00     0.00   0.00   0.00    0.29    0.00  33.75     4.00     0.00   0.01 100.00
nvme21n1      96756.00    0.00 387024.00      0.00     0.00     0.00   0.00   0.00    0.30    0.00  29.24     4.00     0.00   0.01 100.00
nvme22n1      97352.00    0.00 389408.00      0.00     0.00     0.00   0.00   0.00    0.30    0.00  29.36     4.00     0.00   0.01 100.00
nvme12n1      96899.00    0.00 387596.00      0.00     0.00     0.00   0.00   0.00    0.30    0.00  29.22     4.00     0.00   0.01 100.20
nvme17n1      96748.00    0.00 386992.00      0.00     0.00     0.00   0.00   0.00    0.30    0.00  29.24     4.00     0.00   0.01 100.00
nvme19n1      97191.00    0.00 388764.00      0.00     0.00     0.00   0.00   0.00    0.30    0.00  29.30     4.00     0.00   0.01 100.00
nvme23n1      96787.00    0.00 387148.00      0.00     0.00     0.00   0.00   0.00    0.29    0.00  28.41     4.00     0.00   0.01  99.90
md1           1066812.00    0.00 4267248.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     4.00     0.00   0.00   0.00
md0           1262173.00    0.00 5048692.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     4.00     0.00   0.00   0.00

fio output:
socket0: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
socket1: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
socket0-md: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
socket1-md: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
fio-3.26
Starting 256 processes
Jobs: 128 (f=128): [_(128),r(128)][1.6%][r=9103MiB/s][r=2330k IOPS][eta 02h:08m:00s]        
socket0: (groupid=0, jobs=64): err= 0: pid=18344: Tue Jul 27 20:00:10 2021
  read: IOPS=16.0M, BW=60.8GiB/s (65.3GB/s)(3651GiB/60003msec)
    slat (nsec): min=1222, max=18033k, avg=2429.23, stdev=2975.48
    clat (usec): min=24, max=20221, avg=510.51, stdev=336.57
     lat (usec): min=30, max=20240, avg=513.01, stdev=336.58
    clat percentiles (usec):
     |  1.00th=[  147],  5.00th=[  194], 10.00th=[  229], 20.00th=[  281],
     | 30.00th=[  326], 40.00th=[  367], 50.00th=[  412], 60.00th=[  469],
     | 70.00th=[  553], 80.00th=[  676], 90.00th=[  914], 95.00th=[ 1156],
     | 99.00th=[ 1778], 99.50th=[ 2073], 99.90th=[ 2868], 99.95th=[ 3294],
     | 99.99th=[ 4424]
   bw (  MiB/s): min=52367, max=65429, per=32.81%, avg=62388.68, stdev=33.73, samples=7424
   iops        : min=13406054, max=16749890, avg=15971477.42, stdev=8635.86, samples=7424
  lat (usec)   : 50=0.01%, 100=0.02%, 250=13.89%, 500=50.33%, 750=19.72%
  lat (usec)   : 1000=8.24%
  lat (msec)   : 2=7.22%, 4=0.57%, 10=0.02%, 20=0.01%, 50=0.01%
  cpu          : usr=17.93%, sys=49.30%, ctx=21719222, majf=0, minf=9915
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=957111950,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128
socket1: (groupid=1, jobs=64): err= 0: pid=18408: Tue Jul 27 20:00:10 2021
  read: IOPS=13.5M, BW=51.4GiB/s (55.2GB/s)(3085GiB/60008msec)
    slat (nsec): min=1232, max=1696.9k, avg=2580.28, stdev=2841.95
    clat (usec): min=21, max=26808, avg=604.58, stdev=1211.79
     lat (usec): min=26, max=26810, avg=607.23, stdev=1211.80
    clat percentiles (usec):
     |  1.00th=[  124],  5.00th=[  157], 10.00th=[  184], 20.00th=[  225],
     | 30.00th=[  258], 40.00th=[  289], 50.00th=[  318], 60.00th=[  351],
     | 70.00th=[  388], 80.00th=[  437], 90.00th=[  586], 95.00th=[ 2769],
     | 99.00th=[ 6587], 99.50th=[ 9372], 99.90th=[12649], 99.95th=[13829],
     | 99.99th=[16712]
   bw (  MiB/s): min=32950, max=67704, per=20.46%, avg=52713.11, stdev=106.96, samples=7424
   iops        : min=8435402, max=17332350, avg=13494532.64, stdev=27383.02, samples=7424
  lat (usec)   : 50=0.01%, 100=0.16%, 250=27.38%, 500=59.09%, 750=4.93%
  lat (usec)   : 1000=0.30%
  lat (msec)   : 2=0.60%, 4=5.67%, 10=1.47%, 20=0.39%, 50=0.01%
  cpu          : usr=14.86%, sys=45.29%, ctx=36050249, majf=0, minf=10046
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=808781317,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128
socket0-md: (groupid=2, jobs=64): err= 0: pid=18479: Tue Jul 27 20:00:10 2021
  read: IOPS=1263k, BW=4934MiB/s (5174MB/s)(289GiB/60001msec)
    slat (nsec): min=1512, max=48037k, avg=49957.85, stdev=33615.19
    clat (usec): min=176, max=51614, avg=6432.56, stdev=410.54
     lat (usec): min=178, max=51639, avg=6482.58, stdev=412.23
    clat percentiles (usec):
     |  1.00th=[ 6128],  5.00th=[ 6259], 10.00th=[ 6325], 20.00th=[ 6325],
     | 30.00th=[ 6390], 40.00th=[ 6390], 50.00th=[ 6456], 60.00th=[ 6456],
     | 70.00th=[ 6521], 80.00th=[ 6521], 90.00th=[ 6587], 95.00th=[ 6587],
     | 99.00th=[ 6652], 99.50th=[ 6718], 99.90th=[ 7635], 99.95th=[16909],
     | 99.99th=[18220]
   bw (  MiB/s): min= 4582, max= 5934, per=100.00%, avg=4938.25, stdev= 2.07, samples=7616
   iops        : min=1173219, max=1519297, avg=1264175.97, stdev=528.77, samples=7616
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.34%, 10=99.57%, 20=0.08%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=1.23%, sys=95.69%, ctx=2557, majf=0, minf=9064
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=75789817,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128
socket1-md: (groupid=3, jobs=64): err= 0: pid=18543: Tue Jul 27 20:00:10 2021
  read: IOPS=1071k, BW=4183MiB/s (4386MB/s)(245GiB/60002msec)
    slat (nsec): min=1563, max=14080k, avg=59051.10, stdev=22401.39
    clat (usec): min=179, max=20799, avg=7588.23, stdev=303.92
     lat (usec): min=211, max=20853, avg=7647.34, stdev=305.26
    clat percentiles (usec):
     |  1.00th=[ 7111],  5.00th=[ 7373], 10.00th=[ 7439], 20.00th=[ 7504],
     | 30.00th=[ 7504], 40.00th=[ 7570], 50.00th=[ 7570], 60.00th=[ 7635],
     | 70.00th=[ 7635], 80.00th=[ 7701], 90.00th=[ 7767], 95.00th=[ 7767],
     | 99.00th=[ 7898], 99.50th=[ 7898], 99.90th=[ 8586], 99.95th=[13304],
     | 99.99th=[19006]
   bw (  MiB/s): min= 3955, max= 4642, per=100.00%, avg=4186.20, stdev= 0.98, samples=7616
   iops        : min=1012714, max=1188416, avg=1071653.68, stdev=251.68, samples=7616
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=99.94%, 20=0.05%, 50=0.01%
  cpu          : usr=1.06%, sys=95.70%, ctx=1980, majf=0, minf=9030
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=64246431,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=60.8GiB/s (65.3GB/s), 60.8GiB/s-60.8GiB/s (65.3GB/s-65.3GB/s), io=3651GiB (3920GB), run=60003-60003msec

Run status group 1 (all jobs):
   READ: bw=51.4GiB/s (55.2GB/s), 51.4GiB/s-51.4GiB/s (55.2GB/s-55.2GB/s), io=3085GiB (3313GB), run=60008-60008msec

Run status group 2 (all jobs):
   READ: bw=4934MiB/s (5174MB/s), 4934MiB/s-4934MiB/s (5174MB/s-5174MB/s), io=289GiB (310GB), run=60001-60001msec

Run status group 3 (all jobs):
   READ: bw=4183MiB/s (4386MB/s), 4183MiB/s-4183MiB/s (4386MB/s-4386MB/s), io=245GiB (263GB), run=60002-60002msec

Disk stats (read/write):
  nvme0n1: ios=79463384/0, merge=0/0, ticks=25148472/0, in_queue=25148472, util=98.78%
  nvme1n1: ios=79463574/0, merge=0/0, ticks=25224784/0, in_queue=25224784, util=98.87%
  nvme2n1: ios=79463699/0, merge=0/0, ticks=25305193/0, in_queue=25305193, util=98.96%
  nvme3n1: ios=79463925/0, merge=0/0, ticks=25234093/0, in_queue=25234093, util=99.00%
  nvme4n1: ios=79464135/0, merge=0/0, ticks=25396547/0, in_queue=25396547, util=99.06%
  nvme5n1: ios=79464346/0, merge=0/0, ticks=25393624/0, in_queue=25393624, util=99.10%
  nvme6n1: ios=79464535/0, merge=0/0, ticks=25330700/0, in_queue=25330700, util=99.19%
  nvme7n1: ios=79464721/0, merge=0/0, ticks=25349171/0, in_queue=25349171, util=99.24%
  nvme8n1: ios=79464029/0, merge=0/0, ticks=59063115/0, in_queue=59063115, util=99.32%
  nvme9n1: ios=79464120/0, merge=0/0, ticks=59023913/0, in_queue=59023913, util=99.33%
  nvme10n1: ios=79464799/0, merge=0/0, ticks=59136926/0, in_queue=59136927, util=99.39%
  nvme11n1: ios=79465392/0, merge=0/0, ticks=59091104/0, in_queue=59091104, util=99.51%
  nvme12n1: ios=67137057/0, merge=0/0, ticks=18685135/0, in_queue=18685136, util=99.60%
  nvme13n1: ios=67137217/0, merge=0/0, ticks=18638940/0, in_queue=18638940, util=99.76%
  nvme14n1: ios=67137341/0, merge=0/0, ticks=18663275/0, in_queue=18663275, util=99.70%
  nvme15n1: ios=67137620/0, merge=0/0, ticks=18629947/0, in_queue=18629948, util=99.77%
  nvme17n1: ios=67137778/0, merge=0/0, ticks=18709586/0, in_queue=18709585, util=99.80%
  nvme18n1: ios=67137952/0, merge=0/0, ticks=18591798/0, in_queue=18591797, util=99.72%
  nvme19n1: ios=67138199/0, merge=0/0, ticks=18669545/0, in_queue=18669545, util=99.86%
  nvme20n1: ios=67138378/0, merge=0/0, ticks=18600128/0, in_queue=18600128, util=99.89%
  nvme21n1: ios=67138562/0, merge=0/0, ticks=18720763/0, in_queue=18720763, util=100.00%
  nvme22n1: ios=67138772/0, merge=0/0, ticks=18659716/0, in_queue=18659716, util=100.00%
  nvme23n1: ios=67138982/0, merge=0/0, ticks=27862395/0, in_queue=27862395, util=100.00%
  nvme24n1: ios=67134934/0, merge=0/0, ticks=241977879/0, in_queue=241977879, util=100.00%
  md0: ios=75701982/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
  md1: ios=64175011/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%


I'm used to tuning interrupts, so here are the interrupts during the hero portion of the fio and the mdraid portion.....Without polling they are just well balanced irq's across the different nvme MQs
[root@<server> jim]# ./top-irq.pl -k 1
reporting top 10 every 6 secs subject to thresh=10 kernel=1

CAL             532284   CPU146   Function call interrupts
CAL             529615   CPU154   Function call interrupts
CAL             526198   CPU162   Function call interrupts
CAL             524012   CPU142   Function call interrupts
CAL             521467   CPU174   Function call interrupts
CAL             520821   CPU178   Function call interrupts
CAL             518798   CPU176   Function call interrupts
CAL             518244   CPU166   Function call interrupts
CAL             517524   CPU180   Function call interrupts
CAL             514563   CPU136   Function call interrupts

  reported top 10 (of 1885)
  reported interrupts = 5223526  870587.7 per sec    6.8% of all interrupts
^C
[root@<server> jim]# !!
./top-irq.pl -k 1
reporting top 10 every 6 secs subject to thresh=10 kernel=1

CAL              63759   CPU15    Function call interrupts
CAL              63664   CPU178   Function call interrupts
CAL              63428   CPU142   Function call interrupts
CAL              63382   CPU51    Function call interrupts
CAL              63285   CPU140   Function call interrupts
CAL              63068   CPU150   Function call interrupts
CAL              63017   CPU148   Function call interrupts
CAL              62984   CPU144   Function call interrupts
CAL              62842   CPU25    Function call interrupts
CAL              62835   CPU37    Function call interrupts

  reported top 10 (of 1885)
  reported interrupts = 632264  105377.3 per sec    4.0% of all interrupts


Lastly, I can't make md0 and md1 each get ~2M IOPS at the same time.   Sometimes the NUMA0 md is the fastest, sometimes the NUMA1 md is the fastest - I think there might some sort of bottleneck/race somewhere.   It stays that way until I stop them and reassemble.....and then it may switch.    I haven't troubleshooted enough to notice the pattern.

I have to work out with HPE why the socket0/socket1 difference in hero numbers 16.0M/13.5M is something I'll have to take up with HPE or maybe there is a card slowing down the drives in socket1.


Any help is greatly appreciated.  Criticism will be accepted and worst case, IF I HAVEN'T MISSED SOMETHING SO UTTERLY SILLY,  this becomes a defacto "where to start" for the base users like me before the kernel level experts get involved.

As an FYI - I have booted a 5.13 kernel and started using io_uring - no noticeable difference in md performance on a different server with GEN3 drives.....I can raise my "hero numbers" when I have time to play, but right now, my job is to get protected IOPS.


Jim Finlayson
U.S. Department of Defense






[Index of Archives]     [Linux RAID Wiki]     [ATA RAID]     [Linux SCSI Target Infrastructure]     [Linux Block]     [Linux IDE]     [Linux SCSI]     [Linux Hams]     [Device Mapper]     [Device Mapper Cryptographics]     [Kernel]     [Linux Admin]     [Linux Net]     [GFS]     [RPM]     [git]     [Yosemite Forum]


  Powered by Linux