On Mon, Mar 30, 2015 at 12:58 PM, Dave Johansen <davejohansen@xxxxxxxxx> wrote: > I noticed on RHEL 6 that when a large amount of disk I/O is happening that > CPU bound tasks "slow down". I have been able to reproduce it in Fedora 21 > as well and here are the instructions of how I can reproduce it with a > simple test: > > 1) Build the disk_test.cc (the "CPU bound task") and run it. > 2) Create a large file to copy ( fallocate -l 10G junk ). > 3) Copy that file with a one minute delay between copies ( while true; do cp > junk junk2; sleep 60; done ) > > If you direct the output of disk_test.cc to a file, then you can plot the > results in gnuplot with the following commands to see the change in the mean > time between "finishing the work cycle" when the file is being copied: > set xdata time > set timefmt "%s" > plot "out.txt" using 1:3 with lines > > You can also notice that the load average is also going up, so it seems like > something in the kernel/scheduler is getting some sort of exclusive lock in > the disk I/O process and that's causing the CPU bound task to not be able to > execute when it should. Any ideas? > > Thanks, > Dave > > -- > devel mailing list > devel@xxxxxxxxxxxxxxxxxxxxxxx > https://admin.fedoraproject.org/mailman/listinfo/devel > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct This has been an issue in Linux for a long time. It was old news in 2008 when I dug into it. Basically, the response you will get when you say "It hurts when I do that" is "Don't do that!" ;-) As a well-known performance engineer once said, "The only good I/O is a dead I/O". Cache as if your life depended on it, use SSDs when you can afford them, etc. If you *must* build on top of rotating disks, there are tools to tune the I/O schedulers - blktrace and systemtap are the best ones at the low level. By the way, the Linux "load average" counts both CPU-bound tasks ("R" state) and I/O-bound tasks ("D" state). That's why the load average is going up. But the cure is to not design I/O-bound applications. ;-) -- OSJourno: Robust Power Tools for Digital Journalists http://www.znmeb.mobi/stories/osjourno-robust-power-tools-for-digital-journalists Remember, if you're traveling to Bactria, Hump Day is Tuesday and Thursday. -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
