Here’s a /proc/<pid>/stack from when the issue is happening: [<0>] balance_dirty_pages_ratelimited+0x2ca/0x3b0 [<0>] __handle_mm_fault+0xe6e/0x1280 [<0>] handle_mm_fault+0xbe/0x1d0 [<0>] __do_page_fault+0x249/0x4f0 [<0>] page_fault+0x1e/0x30 How can I obtain the numbers for the next step? Thanks, On Tue, Mar 3, 2020 at 2:04 PM Tejun Heo <tj@xxxxxxxxxx> wrote: > > Hello, > > Sorry about the delay. > > On Wed, Feb 26, 2020 at 08:59:55PM +0100, Miklos Szeredi wrote: > > - apparently memcpy is copying downwards (from largest address to > > smallest address). Not sure why, when I run the reproducer, it copies > > upwards. > > - there's a slow batch of reads of the first ~4MB of data, then a > > quick writeback > > - there's a quick read of the rest (~95MB) of data, then a quick > > writeback of the same > > > > Plots of the whole and closeups of slow and quick segments attached. > > X axis is time, Y axis is offset. > > > > Tejun, could this behavior be attributed to dirty throttling? What > > would be the best way to trace this? > > Yeah, seems likely. Can you please try offcputime (or just sample > /proc/PID/stack) and see whether it's in balance dirty pages? > > https://github.com/iovisor/bcc/blob/master/tools/offcputime.py > > If it's dirty throttling, the next step would be watching the bdp > tracepoints to find out what kind of numbers it's getting. > > Thanks. > > -- > tejun
