Re: RT/ext4/jbd2 circular dependency

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 10/29/2014 12:05 PM, Thomas Gleixner wrote:
On Mon, 27 Oct 2014, Chris Friesen wrote:
There are details (stack traces, etc.) in the first message in the thread:
http://www.spinics.net/lists/linux-rt-users/msg12261.html


Originally we had thought that nfsd might have been implicated somehow, but it
seems like it was just a trigger (possibly by increasing the rate of sync
I/O).

In the interest of full disclosure I should point out that we're using a
modified kernel so there is a chance that we have introduced the problem
ourselves.  That said, we have not made significant changes to either ext4 or
jbd2.  (Just a couple of minor cherry-picked bugfixes.)

I don't think it's an ext4/jdb2 problem.

If we turn off journalling in ext4 we can't reproduce the problem. Not conclusive, I'll admit...but interesting.

The relevant code paths are:

Journal commit.  The important thing here is that we set the PG_writeback on a
page, put the jbd2 journal head on BJ_Shadow list, then sleep waiting for page
writeback complete.  If the page writeback never completes, then the journal
head never comes off the BJ_Shadow list.

And that's what you need to investigate.

The rest of the threads being stuck waiting for the journal writeback
or inode->sem are just the consequence of it and have nothing to do
with the root cause of the problem.

ftrace with the block/writeback/jdb/ext4/sched tracepoints enabled
should provide a first insight into the issue.

It seems plausible that the reason why page writeback never completes is that it's blocking trying to take inode->i_data_sem for reading, as seen in the following stack trace (from a hung system):

[<ffffffff8109cd0c>] rt_down_read+0x2c/0x40
[<ffffffff8120ac91>] ext4_map_blocks+0x41/0x270
[<ffffffff8120f0dc>] mpage_da_map_and_submit+0xac/0x4c0
[<ffffffff8120f9c9>] write_cache_pages_da+0x3f9/0x420
[<ffffffff8120fd30>] ext4_da_writepages+0x340/0x720
[<ffffffff8111a5f4>] do_writepages+0x24/0x40
[<ffffffff81191b71>] writeback_single_inode+0x181/0x4b0
[<ffffffff811922a2>] writeback_sb_inodes+0x1b2/0x290
[<ffffffff8119241e>] __writeback_inodes_wb+0x9e/0xd0
[<ffffffff811928e3>] wb_writeback+0x223/0x3f0
[<ffffffff81192b4f>] wb_check_old_data_flush+0x9f/0xb0
[<ffffffff8119403f>] wb_do_writeback+0x12f/0x250
[<ffffffff811941f4>] bdi_writeback_thread+0x94/0x320

I have ftrace logs for two of the three components that we think are involved. I don't have ftrace logs for the above writeback case. My instrumentation was set up to end tracing when someone blocked for 5 seconds trying to get inode->i_data_sem, and it happened to be an nfsd task instead of the page writeback code. I could conceivably modify the instrumentation to only get triggered by page writeback blocking.


For what it's worth, I'm currently testing a backport of commit b34090e from mainline (which in turn required backporting commits e5a120a and f5113ef). It switches from using the BJ_Shadow list to using the BH_Shadow flag on the buffer head. More interestingly, waiters now get woken up from journal_end_buffer_io_sync() instead of from jbd2_journal_commit_transaction().

So far this seems to be helping a lot. It's lasted about 15x as long under stress as without the patches.

Chris
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Reiser Filesystem Development]     [Ceph FS]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Linux FS]     [Yosemite National Park]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Device Mapper]     [Linux Media]

  Powered by Linux