> 7 disks raid6 array(*1). boot, root and swap on a separate > SSD. [...] One disk was removed recently and sent for > replacement. The system felt OK but a few days later I noticed > an issue... I started copying the full dataset (260GB 8,911k > files). [...] 4GB (system limit) dirty blocks were created, > which took 12h to clear. The MD RAID set is performing well as designed. The achievable speed is very low except on special workloads, but that low speed is still good performance for that design, which is close to optimal for maximizing wait times on your workload. Maximizing storage wait times is a common optimization in many IT places. https://www.sabi.co.uk/blog/17-one.html?170610#170610 https://www.sabi.co.uk/blog/15-one.html?150305#150305 > [...] The copy completed fast but the kthread took about 1.5 > hours at 100%CPU to clear the dirty blocks. > - When copying more files (3-5GB) the rsync was consuming > 100%CPU and started pausing every few files (then killed). A > - 'dd if=/dev/zero of=/data1/no-backup/old-backups/100GB' > completed quickly, no issues. [...] [...] > + 100.00% 1.60% [kernel] [k] ext4_mb_regular_allocator > + 67.08% 5.93% [kernel] [k] ext4_mb_find_good_group_avg_frag_lists > + 62.47% 42.34% [kernel] [k] ext4_mb_good_group > + 22.51% 11.36% [kernel] [k] ext4_get_group_info > + 19.70% 10.80% [kernel] [k] ext4_mb_scan_aligned My guess is that the filesystem residing on that RAID set is nearly full, has lots of fragmentation, has lots of small files and likely two out of three or even all three. Those are also common optimizations used in many IT places to maximize storage wait times. https://www.techrepublic.com/forums/discussions/low-disk-performance-after-reching-75-of-disk-space/ https://www.spinics.net/lists/linux-ext4/msg26470.html
