On Mon, 2022-03-07 at 13:15 -0500, Larkin Lowrey wrote:
> I am seeing a 'check' speed regression between kernels 5.15 and 5.16.
> One host with a 20 drive array went from 170MB/s to 11MB/s. Another
> host
> with a 15 drive array went from 180MB/s to 43MB/s. In both cases the
> arrays are almost completely idle. I can flip between the two kernels
> with no other changes and observe the performance changes.
>
> Is this a known issue?
>
> --Larkin
I killed it in the end. The computer went from "slow" and "delayed"...
to taking an annoyingly long time to do anything.
It also gave me a chance to test using the other kernel. Booting to
5.15.0-3-amd64 and starting the "check" shows circa 400mins to complete
which is what it normally takes.
re-booting to 5.16.0-3-amd64 and starting the check shows circa
1000mins to complete.
I noticed on marc.info that Song had posted a request (hadn't filtered
to mail). This is the output of that for two of the arrays:
/dev/md8:
Version : 1.2
Creation Time : Fri Feb 14 08:38:30 2020
Raid Level : raid6
Array Size : 15073892352 (14.04 TiB 15.44 TB)
Used Dev Size : 3768473088 (3.51 TiB 3.86 TB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Tue Mar 8 10:28:00 2022
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : bitmap
Name : debianz97:8
UUID : 51cfc705:98c0ef75:d2b5c558:363f2fd0
Events : 159898
Number Major Minor RaidDevice State
0 8 88 0 active sync /dev/sdf8
1 8 40 1 active sync /dev/sdc8
2 8 72 2 active sync /dev/sde8
3 8 56 3 active sync /dev/sdd8
4 8 24 4 active sync /dev/sdb8
5 8 8 5 active sync /dev/sda8
/dev/md4:
Version : 1.2
Creation Time : Wed Feb 5 11:11:16 2020
Raid Level : raid10
Array Size : 71236608 (67.94 GiB 72.95 GB)
Used Dev Size : 71236608 (67.94 GiB 72.95 GB)
Raid Devices : 2
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Tue Mar 8 10:17:08 2022
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1
Layout : far=2
Chunk Size : 512K
Consistency Policy : resync
Name : BusterTR4:R10Swap
UUID : 3f2d098b:4b0df7a4:dfa23b05:0af8f480
Events : 144
Number Major Minor RaidDevice State
0 259 14 0 active sync /dev/nvme1n1p4
1 259 10 1 active sync /dev/nvme2n1p4
2 259 5 - spare /dev/nvme0n1p4
