On Sat, Jun 06, 2020 at 07:32:10AM +1000, Dave Chinner wrote:
> On Fri, Jun 05, 2020 at 02:27:22PM -0400, Brian Foster wrote:
> > On Thu, Jun 04, 2020 at 05:46:00PM +1000, Dave Chinner wrote:
> > > From: Dave Chinner <dchinner@xxxxxxxxxx>
> > >
> > > Once we have inodes pinning the cluster buffer and attached whenever
> > > they are dirty, we no longer have a guarantee that the items are
> > > flush locked when we lock the cluster buffer. Hence we cannot just
> > > walk the buffer log item list and modify the attached inodes.
> > >
> > > If the inode is not flush locked, we have to ILOCK it first and then
> > > flush lock it to do all the prerequisite checks needed to avoid
> > > races with other code. This is already handled by
> > > xfs_ifree_get_one_inode(), so rework the inode iteration loop and
> > > function to update all inodes in cache whether they are attached to
> > > the buffer or not.
> > >
> > > Note: we also remove the copying of the log item lsn to the
> > > ili_flush_lsn as xfs_iflush_done() now uses the XFS_ISTALE flag to
> > > trigger aborts and so flush lsn matching is not needed in IO
> > > completion for processing freed inodes.
> > >
> > > Signed-off-by: Dave Chinner <dchinner@xxxxxxxxxx>
> > > Reviewed-by: Darrick J. Wong <darrick.wong@xxxxxxxxxx>
> > > ---
> > > fs/xfs/xfs_inode.c | 158 ++++++++++++++++++---------------------------
> > > 1 file changed, 62 insertions(+), 96 deletions(-)
> > >
> > > diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
> > > index 272b54cf97000..fb4c614c64fda 100644
> > > --- a/fs/xfs/xfs_inode.c
> > > +++ b/fs/xfs/xfs_inode.c
> > ...
> > > @@ -2559,43 +2563,53 @@ xfs_ifree_get_one_inode(
> > > */
> > > if (ip != free_ip) {
> > > if (!xfs_ilock_nowait(ip, XFS_ILOCK_EXCL)) {
> > > + spin_unlock(&ip->i_flags_lock);
> > > rcu_read_unlock();
> > > delay(1);
> > > goto retry;
> > > }
> > > -
> > > - /*
> > > - * Check the inode number again in case we're racing with
> > > - * freeing in xfs_reclaim_inode(). See the comments in that
> > > - * function for more information as to why the initial check is
> > > - * not sufficient.
> > > - */
> > > - if (ip->i_ino != inum) {
> > > - xfs_iunlock(ip, XFS_ILOCK_EXCL);
> > > - goto out_rcu_unlock;
> > > - }
> >
> > Why is the recheck under ILOCK_EXCL no longer necessary? It looks like
> > reclaim decides whether to proceed or not under the ilock and doesn't
> > acquire the spinlock until it decides to reclaim. Hm?
>
> Because we now take the ILOCK while still holding the i_flags_lock
> instead of dropping the spin lock then trying to get the ILOCK.
> Hence with this change, if we get the ILOCK we are guaranteed that
> the inode number has not changed and don't need to recheck it.
>
> This is guaranteed by xfs_reclaim_inode() because it locks in
> the order of ILOCK -> i_flags_lock and it zeroes the ip->i_ino
> while holding both these locks. Hence if we've got the i_flags_lock
> and we try to get the ILOCK, the inode is either going to be valid and
> reclaim will skip the inode (because we hold locks) or the inode
> will already be in reclaim and the ip->i_ino will be zero....
>
Got it, thanks.
>
> > > }
> > > + ip->i_flags |= XFS_ISTALE;
> > > + spin_unlock(&ip->i_flags_lock);
> > > rcu_read_unlock();
> > >
> > > - xfs_iflock(ip);
> > > - xfs_iflags_set(ip, XFS_ISTALE);
> > > + /*
> > > + * If we can't get the flush lock, the inode is already attached. All
> > > + * we needed to do here is mark the inode stale so buffer IO completion
> > > + * will remove it from the AIL.
> > > + */
> >
> > To make sure I'm following this correctly, we can assume the inode is
> > attached based on an iflock_nowait() failure because we hold the ilock,
> > right?
>
> Actually, because we hold the buffer lock. We only flush the inode
> to the buffer when we are holding the buffer lock, so all flush
> locking shold nest inside the buffer lock. So for taking the flock,
> the lock order is bp->b_sema -> ILOCK_EXCL -> iflock. We drop the
> flush lock before we drop the buffer lock in IO completion, and
> hence if we hold the buffer lock, nothing else can actually unlock
> the inode flush lock.
>
I was thinking about xfs_reclaim_inode() again where we trylock the
flush lock before checking clean state. There's no buffer involved in
that path (as of this patch, at least), so afaict the ILOCK is what
protects us in that general scenario.
> > IOW, any other task doing a similar iflock check would have to do
> > so under ilock and release the flush lock first if the inode didn't end
> > up flushed, for whatever reason.
>
> Yes, anything taking the flush lock needs to first hold the ILOCK -
> that's always been the case and we've always done that because the
> ILOCK is needed to provides serialisation against a) other
> modifications while we are accessing/flushing the inode, and b)
> inode reclaim.
>
Right, Ok.
> /me checks.
>
> After this patchset nothing calls xfs_iflock() at all - everything
> uses xfs_iflock_nowait(), so it might be time to turn this back into
> a plain status flag and get rid of the iflock stuff altogether as
> it's just a state flag now...
>
Yeah, I think that would be a nice follow on cleanup. The whole "if
trylock fails, then assume we own the lock" type logic is really obscure
and confusing to read IMO. Treating it as the "inode is flushed" state
the lock fundamentally represents would be far more readable.
That addresses both my concerns, so with the nit below fixed:
Reviewed-by: Brian Foster <bfoster@xxxxxxxxxx>
Brian
> > > + ASSERT(iip->ili_fields);
> > > + spin_lock(&iip->ili_lock);
> > > + iip->ili_last_fields = iip->ili_fields;
> > > + iip->ili_fields = 0;
> > > + iip->ili_fsync_fields = 0;
> > > + spin_unlock(&iip->ili_lock);
> > > + list_add_tail(&iip->ili_item.li_bio_list, &bp->b_li_list);
> > > + ASSERT(iip->ili_last_fields);
> >
> > We already asserted ->ili_fields and assigned ->ili_fields to
> > ->ili_last_fields, so this assert seems spurious.
>
> Ah, the first ASSERT goes away in the next patch, I think. It was
> debug, and I may have removed it from the wrong patch...
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@xxxxxxxxxxxxx
>
