On Wed, May 08, 2019 at 10:08:59AM -0400, Rik van Riel wrote: > On Wed, 2019-05-08 at 07:22 +1000, Dave Chinner wrote: > > On Tue, May 07, 2019 at 01:05:28PM -0400, Rik van Riel wrote: > > > The code in xlog_wait uses the spinlock to make adding the task to > > > the wait queue, and setting the task state to UNINTERRUPTIBLE > > > atomic > > > with respect to the waker. > > > > > > Doing the wakeup after releasing the spinlock opens up the > > > following > > > race condition: > > > > > > - add task to wait queue > > > > > > - wake up task > > > > > > - set task state to UNINTERRUPTIBLE > > > > > > Simply moving the spin_unlock to after the wake_up_all results > > > in the waker not being able to see a task on the waitqueue before > > > it has set its state to UNINTERRUPTIBLE. > > > > Yup, seems like an issue. Good find, Rik. > > > > So, what problem is this actually fixing? Was it noticed by > > inspection, or is it actually manifesting on production machines? > > If it is manifesting IRL, what are the symptoms (e.g. hang running > > out of log space?) and do you have a test case or any way to > > exercise it easily? > > Chris spotted a hung kworker task, in UNINTERRUPTIBLE > state, but with an empty wait queue. This does not seem > like something that is easily reproducible. Yeah, I just read that, not something we can trigger with a regression test :P > > And, FWIW, did you check all the other xlog_wait() users for the > > same problem? > > I did not, but am looking now. The xlog_wait code itself > is fine, but it seems there are a few other wakers that > are doing the wakeup after releasing the lock. > > It looks like xfs_log_force_umount() and the other wakeup > in xlog_state_do_callback() suffer from the same issue. Hmmm, the first wakeup in xsdc is this one, right: /* wake up threads waiting in xfs_log_force() */ wake_up_all(&iclog->ic_force_wait); At the end of the iclog iteration loop? That one is under the ic_loglock - the lock is dropped to run callbacks, then picked up again once the callbacks are done and before the ic_callback_lock is dropped (about 10 lines above the wakeup). So unless I'm missing something (like enough coffee!) that one look fine. ..... > I am not sure about xfs_log_force_umount(). Could the unlock > be moved to after the wake_up_all, or does that create lock > ordering issues with the xlog_grant_head_wake_all calls? > Could a simple lock + unlock of log->l_icloglock around the > wake_up_all do the trick, or is there some other state that > also needs to stay locked? Need to be careful which lock is used with which wait queue :) This one is waking the the xc_commit_wait queue (CIL push commit sequencing wait queue), which is protected by the log->l_cilp->xc_push_lock. That should nest jsut fine inside any locks we are holding at this point, so you should just be able to wrap it. It's not a common code path, though, it'll only hit this code when the filesystem is already considered to be in an unrecoverable state. Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx
