Re: btrfs system slow down with 100GB file

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, Mar 23, 2021 at 8:39 AM Richard Shaw <hobbes1069@xxxxxxxxx> wrote:
>
> I'm getting significant iowait while writing to a 100GB file.

High iowait means the system is under load and not CPU bound but IO
bound. It sounds like the drive is writing as fast as it can.   What's
the workload? Reproduce the GUI stalls and capture all of the
following:

sudo iostat -x -d -m 5

This is part of sysstat package (you can disable the service and timer
units it installs). Probably best to copy/paste into a plaint text
file and put it up in a file share service, most anything else is
going to wrap it, making it hard to read. A minute of capture while
the workload is proceeding is enough. Also capture a few

grep -R . /proc/pressure

And each of these (workload doesn't need to be running)

lsblk -o NAME,FSTYPE,SIZE,FSUSE%,MOUNTPOINT,UUID,MIN-IO,SCHED,DISC-GRAN,MODEL
uname -r
mount | grep btrfs

>I have already made it nocow by copying it to another directory, marking the director nocow (+C) and using cat <oldfile> <newfile> to re-create it from scratch.
>
> I was under the impression that this should fix the problem.

It depends on the workload for this file. Was the 100G file fallocated
or created as a sparse file? File format?

>
> On a tangent, it took about 30 minutes to delete the old file... My system is a Ryzen 5 3600 w/ 16GB or memory but it is a spinning disk. I use an NVME for the system and the spinning disk for /home.

filefrag 100G.file
What's the path to the file?

>
> Currently I'm getting random GUI freezes due to the iowait problem and my HDD indicator light basically stays on solid for over an hour now.
>

Have sysrq+t ready in a shell but don't issue it. Reproduce this
problem (the GUI freezes) and then issue the sysrq+t. Depending on how
many processes, this could exceed both the kernel message buffer and
the journald rate limiter. Either use log-buf-len=8M boot parameter,
and then dmesg will have the whole sysrq+t. The other option is
temporarily turn off journald rate limiting in journald.conf

#RateLimitIntervalSec=30s
#RateLimitBurst=10000

Add a 0 should work. Restart journald. Issue sysrq+t. Output to a file
by 'journalctl -k -o short-monotonic --no-hostname > journal.log'

I suggest opening a bug against the kernel, and post the URL here so I
can tag it. Attach the iostat output and dmesg/journalctl output as
files to that bug, and everything else can just go in the description.

Also note any other customizations to /proc or /sys that differ from
Fedora defaults.


-- 
Chris Murphy
_______________________________________________
users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure



[Index of Archives]     [Older Fedora Users]     [Fedora Announce]     [Fedora Package Announce]     [EPEL Announce]     [EPEL Devel]     [Fedora Magazine]     [Fedora Summer Coding]     [Fedora Laptop]     [Fedora Cloud]     [Fedora Advisory Board]     [Fedora Education]     [Fedora Security]     [Fedora Scitech]     [Fedora Robotics]     [Fedora Infrastructure]     [Fedora Websites]     [Anaconda Devel]     [Fedora Devel Java]     [Fedora Desktop]     [Fedora Fonts]     [Fedora Marketing]     [Fedora Management Tools]     [Fedora Mentors]     [Fedora Package Review]     [Fedora R Devel]     [Fedora PHP Devel]     [Kickstart]     [Fedora Music]     [Fedora Packaging]     [Fedora SELinux]     [Fedora Legal]     [Fedora Kernel]     [Fedora OCaml]     [Coolkey]     [Virtualization Tools]     [ET Management Tools]     [Yum Users]     [Yosemite News]     [Gnome Users]     [KDE Users]     [Fedora Art]     [Fedora Docs]     [Fedora Sparc]     [Libvirt Users]     [Fedora ARM]

  Powered by Linux