On Fri, 2022-05-20 at 21:52 +0000, Chuck Lever III wrote:
>
>
> > On May 20, 2022, at 12:40 PM, Trond Myklebust
> > <trondmy@xxxxxxxxxxxxxxx> wrote:
> >
> > On Fri, 2022-05-20 at 15:36 +0000, Chuck Lever III wrote:
> > >
> > >
> > > > On May 11, 2022, at 10:36 AM, Chuck Lever III
> > > > <chuck.lever@xxxxxxxxxx> wrote:
> > > >
> > > >
> > > >
> > > > > On May 11, 2022, at 10:23 AM, Greg KH
> > > > > <gregkh@xxxxxxxxxxxxxxxxxxx> wrote:
> > > > >
> > > > > On Wed, May 11, 2022 at 02:16:19PM +0000, Chuck Lever III
> > > > > wrote:
> > > > > >
> > > > > >
> > > > > > > On May 11, 2022, at 8:38 AM, Greg KH
> > > > > > > <gregkh@xxxxxxxxxxxxxxxxxxx> wrote:
> > > > > > >
> > > > > > > On Wed, May 11, 2022 at 12:03:13PM +0200, Wolfgang Walter
> > > > > > > wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > starting with 5.4.188 wie see a massive performance
> > > > > > > > regression on our
> > > > > > > > nfs-server. It basically is serving requests very very
> > > > > > > > slowly with cpu
> > > > > > > > utilization of 100% (with 5.4.187 and earlier it is
> > > > > > > > 10%) so
> > > > > > > > that it is
> > > > > > > > unusable as a fileserver.
> > > > > > > >
> > > > > > > > The culprit are commits (or one of it):
> > > > > > > >
> > > > > > > > c32f1041382a88b17da5736886da4a492353a1bb "nfsd: cleanup
> > > > > > > > nfsd_file_lru_dispose()"
> > > > > > > > 628adfa21815f74c04724abc85847f24b5dd1645 "nfsd:
> > > > > > > > Containerise filecache
> > > > > > > > laundrette"
> > > > > > > >
> > > > > > > > (upstream 36ebbdb96b694dd9c6b25ad98f2bbd263d022b63 and
> > > > > > > > 9542e6a643fc69d528dfb3303f145719c61d3050)
> > > > > > > >
> > > > > > > > If I revert them in v5.4.192 the kernel works as before
> > > > > > > > and
> > > > > > > > performance is
> > > > > > > > ok again.
> > > > > > > >
> > > > > > > > I did not try to revert them one by one as any
> > > > > > > > disruption
> > > > > > > > of our nfs-server
> > > > > > > > is a severe problem for us and I'm not sure if they are
> > > > > > > > related.
> > > > > > > >
> > > > > > > > 5.10 and 5.15 both always performed very badly on our
> > > > > > > > nfs-
> > > > > > > > server in a
> > > > > > > > similar way so we were stuck with 5.4.
> > > > > > > >
> > > > > > > > I now think this is because of
> > > > > > > > 36ebbdb96b694dd9c6b25ad98f2bbd263d022b63
> > > > > > > > and/or 9542e6a643fc69d528dfb3303f145719c61d3050 though
> > > > > > > > I
> > > > > > > > didn't tried to
> > > > > > > > revert them in 5.15 yet.
> > > > > > >
> > > > > > > Odds are 5.18-rc6 is also a problem?
> > > > > >
> > > > > > We believe that
> > > > > >
> > > > > > 6b8a94332ee4 ("nfsd: Fix a write performance regression")
> > > > > >
> > > > > > addresses the performance regression. It was merged into
> > > > > > 5.18-
> > > > > > rc.
> > > > >
> > > > > And into 5.17.4 if someone wants to try that release.
> > > >
> > > > I don't have a lot of time to backport this one myself, so
> > > > I welcome anyone who wants to apply that commit to their
> > > > favorite LTS kernel and test it for us.
> > > >
> > > >
> > > > > > > If so, I'll just wait for the fix to get into Linus's
> > > > > > > tree as
> > > > > > > this does
> > > > > > > not seem to be a stable-tree-only issue.
> > > > > >
> > > > > > Unfortunately I've received a recent report that the fix
> > > > > > introduces
> > > > > > a "sleep while spinlock is held" for NFSv4.0 in rare cases.
> > > > >
> > > > > Ick, not good, any potential fixes for that?
> > > >
> > > > Not yet. I was at LSF last week, so I've just started digging
> > > > into this one. I've confirmed that the report is a real bug,
> > > > but we still don't know how hard it is to hit it with real
> > > > workloads.
> > >
> > > We believe the following, which should be part of the first
> > > NFSD pull request for 5.19, will properly address the splat.
> > >
> > > https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/commit/?h=for-next&id=556082f5e5d7ecfd0ee45c3641e2b364bff9ee44
> > >
> > >
> > Uh... What happens if you have 2 simultaneous calls to
> > nfsd4_release_lockowner() for the same file? i.e. 2 separate
> > processes
> > owned by the same user, both locking the same file.
> >
> > Can't that cause the 'putlist' to get corrupted when both callers
> > add
> > the same nf->nf_putfile to two separate lists?
>
> IIUC, cl_lock serializes the two RELEASE_LOCKOWNER calls.
>
> The first call finds the lockowner in cl_ownerstr_hashtbl and
> unhashes it before releasing cl_lock.
>
> Then the second cannot find that lockowner, thus it can't
> requeue it for bulk_put.
>
> Am I missing something?
In the example I quoted, there are 2 separate processes running on the
client. Those processes could share the same open owner + open stateid,
and hence the same struct nfs4_file, since that depends only on the
process credentials matching. However they will not normally share a
lock owner, since POSIX does not expect different processes to share
locks.
IOW: The point is that one can relatively easily create 2 different
lock owners with different lock stateids that share the same underlying
struct nfs4_file.
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@xxxxxxxxxxxxxxx
