On 31/10/2023 14.21, Carlos Carvalho wrote:
Roger Heflin (rogerheflin@xxxxxxxxx) wrote on Mon, Oct 30, 2023 at 01:14:49PM -03:
look at SAR -d output for all the disks in the raid6. It may be a
disk issue (though I suspect not given the 100% cpu show in raid).
Clearly something very expensive/deadlockish is happening because of
the raid having to rebuild the data from the missing disk, not sure
what could be wrong with it.
This is very similar to what I complained some 3 months ago. For me it happens
with an array in normal state. sar shows no disk activity yet there are no
writes to the array (reads happen normally) and the flushd thread uses 100%
cpu.
For the latest 6.5.* I can reliably reproduce it with
% xzcat linux-6.5.tar.xz | tar x -f -
This leaves the machine with ~1.5GB of dirty pages (as reported by
/proc/meminfo) that it never manages to write to the array. I've waited for
several hours to no avail. After a reboot the kernel tree had only about 220MB
instead of ~1.5GB...
More evidence that the problem relates to the cache not flushed to disk.
If I run 'rsync --fsync ...' it slows it down as the writing is flushed to disk for each file.
But it also evicts it from the cache, so nothing accumulates.
The result is a slower than otherwise copying but it streams with no pauses.
It seems that the array is slow to sync files somehow. Mythtv has no problems because it write
only a few large files. rsync copies a very large number of small files which somehow triggers
the problem.
This is why my 'dd if=/dev/zero of=file-on-array' goes fast without problems.
Just my guess.
BTW I ran fsck on the fs (on the array) and it found no fault.
--
Eyal at Home (eyal@xxxxxxxxxxxxxx)
