On Fri, Aug 30, 2019 at 09:08:17AM +1000, Dave Chinner wrote:
> On Thu, Aug 29, 2019 at 10:51:59AM +0530, Chandan Rajendra wrote:
> > 786576: kworker/4:1H-kb 1825 [004] 217.041079: xfs:xfs_log_assign_tail_lsn: dev 7:1 new tail lsn 2/19333, old lsn 2/19330, last sync 3/18501
>
> 200ms later the tail has moved, and last_sync_lsn is now 3/18501.
> i.e. the iclog writes have made it to disk, and the items have been
> moved into the AIL. I don't know where that came from, but I'm
> assuming it's an IO completion based on it being run from a
> kworker context that doesn't have an "xfs-" name prefix(*).
>
> As the tail has moved, this should have woken the anything sleeping
> on the log tail in xlog_grant_head_wait() via a call to
> xfs_log_space_wake(). The first waiter should wake, see that there
> still isn't room in the log (only 3 sectors were freed in the log,
> we need at least 60). That woken process should then run
> xlog_grant_push_ail() again and go back to sleep.
Actually, it doesn't get woken because xlog_grant_head_wake() checks
how much space is available before waking waiters, and there clearly
isn't enough here. So that's one likely vector. Can you try this
patch?
-Dave.
--
Dave Chinner
david@xxxxxxxxxxxxx
xfs: push the AIL in xlog_grant_head_wake
From: Dave Chinner <dchinner@xxxxxxxxxx>
In the situation where the log is full and the CIL has not recently
flushed, the AIL push threshold is throttled back to the where the
last write of the head of the log was completed. This is stored in
log->l_last_sync_lsn. Hence if the CIL holds > 25% of the log space
pinned by flushes and/or aggregation in progress, we can get the
situation where the head of the log lags a long way behind the
reservation grant head.
When this happens, the AIL push target is trimmed back from where
the reservation grant head wants to push the log tail to, back to
where the head of the log currently is. This means the push target
doesn't reach far enough into the log to actually move the tail
before the transaction reservation goes to sleep.
When the CIL push completes, it moves the log head forward such that
the AIL push target can now be moved, but that has no mechanism for
puhsing the log tail. Further, if the next tail movement of the log
is not large enough wake the waiter (i.e. still not enough space for
it to have a reservation granted), we don't wake anything up, and
hence we do not update the AIL push target to take into account the
head of the log moving and allowing the push target to be moved
forwards.
To avoid this particular condition, if we fail to wake the first
waiter on the grant head because we don't have enough space,
push on the AIL again. This will pick up any movement of the log
head and allow the push target to move forward due to completion of
CIL pushing.
XXX: this does not address the situation where there are no tail
updates after the log head moves forward. There is currently no
infrastructure for a log head update to trigger an AIL tail push if
necessary.
Signed-off-by: Dave Chinner <dchinner@xxxxxxxxxx>
---
fs/xfs/xfs_log.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/fs/xfs/xfs_log.c b/fs/xfs/xfs_log.c
index b159a9e9fef0..941f10ff99d9 100644
--- a/fs/xfs/xfs_log.c
+++ b/fs/xfs/xfs_log.c
@@ -214,15 +214,20 @@ xlog_grant_head_wake(
{
struct xlog_ticket *tic;
int need_bytes;
+ bool woken_task = false;
list_for_each_entry(tic, &head->waiters, t_queue) {
need_bytes = xlog_ticket_reservation(log, head, tic);
- if (*free_bytes < need_bytes)
+ if (*free_bytes < need_bytes) {
+ if (!woken_task)
+ xlog_grant_push_ail(log, need_bytes);
return false;
+ }
*free_bytes -= need_bytes;
trace_xfs_log_grant_wake_up(log, tic);
wake_up_process(tic->t_task);
+ woken_task = true;
}
return true;
