Hi Stefan! On Tue 16-08-22 22:45:48, Stefan Wahren wrote: > Am 16.08.22 um 11:34 schrieb Jan Kara: > > Hi Stefan! > > So this is interesting. We can see the card is 100% busy. The IO submitted > > to the card is formed by small requests - 18-38 KB per request - and each > > request takes 0.3-0.5s to complete. So the resulting throughput is horrible > > - only tens of KB/s. Also we can see there are many IOs queued for the > > device in parallel (aqu-sz columnt). This does not look like load I would > > expect to be generated by download of a large file from the web. > > > > You have mentioned in previous emails that with dd(1) you can do couple > > MB/s writing to this card which is far more than these tens of KB/s. So the > > file download must be doing something which really destroys the IO pattern > > (and with mb_optimize_scan=0 ext4 happened to be better dealing with it and > > generating better IO pattern). Can you perhaps strace the process doing the > > download (or perhaps strace -f the whole rpi-update process) so that we can > > see how does the load generated on the filesystem look like? Thanks! > > i didn't create the strace yet, but i looked at the source of rpi-update. At > the end the download phase is a curl call to download a tar archive and pipe > it directly to tar. > > You can find the content list of the tar file here: > > https://raw.githubusercontent.com/lategoodbye/mb_optimize_scan_regress/main/rpi-firmware-tar-content-list.txt Thanks for the details! This is indeed even better. Looking at the tar archive I can see it consists of a lot of small files big part of them is even below 10k. So this very much matches the workload I was examining with reaim where I saw regression (although only ~8%) even on normal rotating drive on x86 machine. In that case I have pretty much confirmed that the problem is due to mb_optimize_scan=1 spreading small allocated files more which is likely also harmful for the SD card because it requires touching more erase blocks. Thanks for help with debugging this, I will implement some of the heuristics we discussed with other ext4 developers to avoid this behavior and will send you patch for testing. Honza -- Jan Kara <jack@xxxxxxxx> SUSE Labs, CR
