Good day! 2nd part of the question, in relation of hardware/system from previous thread - "raid10, far layout initial sync slow + XFS question" https://www.spinics.net/lists/raid/msg74907.html - Ubuntu 20.04 with kernel "5.4.0-153-generic #170-Ubuntu" on Hetzner AX161 / AMD EPYC 7502P 32-Core Processor Gist: issuing the same load on RAID10 4 drives N2 16kb chunk is slower than running the same load on a single member of that RAID Question: is such kind of behavior normal and expected? Am I doing something terribly wrong? RAID10: 18.5k iops SINGLE DRIVE: 26k iops raw data: RAID config root@node2:/data# cat /proc/mdstat Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] md3 : active raid10 nvme5n1[3] nvme3n1[2] nvme4n1[1] nvme0n1[0] 7501212320 blocks super 1.2 16K chunks 2 near-copies [4/4] [UUUU] Single drive with: root@node2:/data# mdadm /dev/md3 --fail /dev/nvme5n1 --remove /dev/nvme5n1 mdadm: set /dev/nvme5n1 faulty in /dev/md3 mdadm: hot removed /dev/nvme5n1 from /dev/md3 mdadm --zero-superblock /dev/nvme5n1 TEST COMMANDS RADI10: fio --rw=write --ioengine=sync --fdatasync=1 --filename=/dev/md3 --size=8200m --bs=16k --name=mytest SINGLE DRIVE: fio --rw=write --ioengine=sync --fdatasync=1 --filename=/dev/nvme5n1 --size=8200m --bs=16k --name=mytest FIO output: RAID10: root@node2:/mnt# fio --rw=write --ioengine=sync --fdatasync=1 --filename=/dev/md3 --size=8200m --bs=16k --name=mytest mytest: (g=0): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=sync, iodepth=1 fio-3.16 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][w=298MiB/s][w=19.0k IOPS][eta 00m:00s] mytest: (groupid=0, jobs=1): err= 0: pid=2130392: Sat Sep 2 08:21:39 2023 write: IOPS=18.5k, BW=290MiB/s (304MB/s)(8200MiB/28321msec); 0 zone resets clat (usec): min=5, max=745, avg=12.12, stdev= 7.30 lat (usec): min=6, max=746, avg=12.47, stdev= 7.34 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 10], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 11], 60.00th=[ 11], | 70.00th=[ 12], 80.00th=[ 13], 90.00th=[ 16], 95.00th=[ 20], | 99.00th=[ 39], 99.50th=[ 55], 99.90th=[ 100], 99.95th=[ 116], | 99.99th=[ 147] bw ( KiB/s): min=276160, max=308672, per=99.96%, avg=296354.86, stdev=6624.06, samples=56 iops : min=17260, max=19292, avg=18522.18, stdev=414.00, samples=56 Run status group 0 (all jobs): WRITE: bw=290MiB/s (304MB/s), 290MiB/s-290MiB/s (304MB/s-304MB/s), io=8200MiB (8598MB), run=28321-28321msec Disk stats (read/write): md3: ios=0/2604727, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=25/262403, aggrmerge=0/787199, aggrticks=1/5563, aggrin_queue=0, aggrutil=98.10% nvme0n1: ios=40/262402, merge=1/787200, ticks=3/5092, in_queue=0, util=98.09% nvme3n1: ios=33/262404, merge=1/787198, ticks=2/5050, in_queue=0, util=98.08% nvme5n1: ios=15/262404, merge=0/787198, ticks=1/6061, in_queue=0, util=98.08% nvme4n1: ios=12/262402, merge=0/787200, ticks=1/6052, in_queue=0, util=98.10% SINGLE DRIVE: root@node2:/mnt# fio --rw=write --ioengine=sync --fdatasync=1 --filename=/dev/nvme5n1 --size=8200m --bs=16k --name=mytest mytest: (g=0): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=sync, iodepth=1 fio-3.16 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][w=414MiB/s][w=26.5k IOPS][eta 00m:00s] mytest: (groupid=0, jobs=1): err= 0: pid=2155313: Sat Sep 2 08:26:23 2023 write: IOPS=26.2k, BW=410MiB/s (430MB/s)(8200MiB/20000msec); 0 zone resets clat (usec): min=4, max=848, avg=11.25, stdev= 7.15 lat (usec): min=5, max=848, avg=11.50, stdev= 7.17 clat percentiles (usec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 10], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 11], 80.00th=[ 12], 90.00th=[ 15], 95.00th=[ 18], | 99.00th=[ 43], 99.50th=[ 62], 99.90th=[ 95], 99.95th=[ 108], | 99.99th=[ 133] bw ( KiB/s): min=395040, max=464480, per=99.90%, avg=419438.95, stdev=17496.05, samples=39 iops : min=24690, max=29030, avg=26214.92, stdev=1093.56, samples=39 Run status group 0 (all jobs): WRITE: bw=423MiB/s (444MB/s), 423MiB/s-423MiB/s (444MB/s-444MB/s), io=8200MiB (8598MB), run=19379-19379msec Disk stats (read/write): nvme5n1: ios=49/518250, merge=0/1554753, ticks=2/10629, in_queue=0, util=99.61% -- Best regards, [COOLCOLD-RIPN]
