On 5/10/20 2:07 PM, Wolfgang Denk wrote:
Hi,
I'm running raid6check on a 12 TB (8 x 2 TB harddisks)
RAID6 array and wonder why it is so extremely slow...
It seems to be reading the disks only a about 400 kB/s,
which results in an estimated time of some 57 days!!!
to complete checking the array. The system is basically idle, there
is neither any significant CPU load nor any other I/o (no to the
tested array, nor to any other storage on this system).
Am I doing something wrong?
The command I'm running is simply:
# raid6check /dev/md0 0 0
This is with mdadm-4.1 on a Fedora 32 system (mdadm-4.1-4.fc32.x86_64).
The array data:
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Nov 7 19:30:03 2013
Raid Level : raid6
Array Size : 11720301024 (11177.35 GiB 12001.59 GB)
Used Dev Size : 1953383504 (1862.89 GiB 2000.26 GB)
Raid Devices : 8
Total Devices : 8
Persistence : Superblock is persistent
Update Time : Mon May 4 22:12:02 2020
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 16K
Consistency Policy : resync
Name : atlas.denx.de:0 (local to host atlas.denx.de)
UUID : 4df90724:87913791:1700bb31:773735d0
Events : 181544
Number Major Minor RaidDevice State
12 8 64 0 active sync /dev/sde
11 8 80 1 active sync /dev/sdf
13 8 112 2 active sync /dev/sdh
8 8 128 3 active sync /dev/sdi
9 8 144 4 active sync /dev/sdj
10 8 160 5 active sync /dev/sdk
14 8 176 6 active sync /dev/sdl
15 8 192 7 active sync /dev/sdm
# iostat /dev/sd[efhijklm]
Linux 5.6.8-300.fc32.x86_64 (atlas.denx.de) 2020-05-07 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.18 0.01 1.11 0.21 0.00 98.49
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
sde 19.23 388.93 0.09 0.00 158440224 35218 0
sdf 19.20 388.94 0.09 0.00 158447574 34894 0
sdh 19.23 388.89 0.08 0.00 158425596 34178 0
sdi 19.23 388.99 0.09 0.00 158466326 34690 0
sdj 20.18 388.93 0.09 0.00 158439780 34766 0
sdk 19.23 388.88 0.09 0.00 158419988 35366 0
sdl 19.20 388.97 0.08 0.00 158457352 34426 0
sdm 19.21 388.92 0.08 0.00 158435748 34566 0
top - 09:08:19 up 4 days, 17:10, 3 users, load average: 1.00, 1.00, 1.00
Tasks: 243 total, 1 running, 242 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.2 us, 0.5 sy, 0.0 ni, 98.5 id, 0.1 wa, 0.6 hi, 0.1 si, 0.0 st
MiB Mem : 24034.6 total, 11198.4 free, 1871.8 used, 10964.3 buff/cache
MiB Swap: 7828.5 total, 7828.5 free, 0.0 used. 21767.6 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19719 root 20 0 2852 2820 2020 D 5.1 0.0 285:40.07 raid6check
Seems raid6check is in 'D' state, what are the output of 'cat
/proc/19719/stack' and /proc/mdstat?
1123 root 20 0 0 0 0 S 0.3 0.0 25:47.54 md0_raid6
37816 root 20 0 0 0 0 I 0.3 0.0 0:00.08 kworker/3:1-events
37903 root 20 0 219680 4540 3716 R 0.3 0.0 0:00.02 top
...
HDD in use:
/dev/sde : ST2000NM0033-9ZM175
/dev/sdf : ST2000NM0033-9ZM175
/dev/sdh : ST2000NM0033-9ZM175
/dev/sdi : ST2000NM0033-9ZM175
/dev/sdj : ST2000NM0033-9ZM175
/dev/sdk : ST2000NM0033-9ZM175
/dev/sdl : ST2000NM0033-9ZM175
/dev/sdm : ST2000NM0008-2F3100
3 days later:
Is raid6check still in 'D' state as before?
Thanks,
Guoqing
