I had re-run the tests. So, indeed, if I wait log enough, then client falls back to IO through MDS and issues a new LAYOUTGET after the first successful READ operation. the capture available at https://sas.desy.de/index.php/s/StGeijXTGysdHa2 Best regards, Tigran. ----- Original Message ----- > From: "trondmy" <trondmy@xxxxxxxxxxxxxxx> > To: "Tigran Mkrtchyan" <tigran.mkrtchyan@xxxxxxx>, "Olga Kornievskaia" <olga.kornievskaia@xxxxxxxxx> > Cc: "linux-nfs" <linux-nfs@xxxxxxxxxxxxxxx>, "Anna Schumaker" <anna.schumaker@xxxxxxxxxx> > Sent: Wednesday, 1 June, 2022 17:50:53 > Subject: Re: [PATCH] pNFS: fix IO thread starvation problem during LAYOUTUNAVAILABLE error > On Wed, 2022-06-01 at 09:27 +0200, Mkrtchyan, Tigran wrote: >> Hi Olga, >> >> ----- Original Message ----- >> > From: "Olga Kornievskaia" <olga.kornievskaia@xxxxxxxxx> >> > To: "trondmy" <trondmy@xxxxxxxxxxxxxxx> >> > Cc: "Anna Schumaker" <anna.schumaker@xxxxxxxxxx>, "linux-nfs" >> > <linux-nfs@xxxxxxxxxxxxxxx> >> > Sent: Tuesday, 31 May, 2022 18:03:34 >> > Subject: Re: [PATCH] pNFS: fix IO thread starvation problem during >> > LAYOUTUNAVAILABLE error >> >> > > <snip> > >> > To clarify, can you confirm that LAYOUTUNAVAILABLE would only turn >> > off >> > the inode permanently but for a period of time? >> > >> > It looks to me that for the LAYOUTTRYLATER, the client would face >> > the >> > same starvation problem in the same situation. I don't see anything >> > marking the segment failed for such error? I believe the client >> > returns nolayout for that error, falls back to MDS but allows >> > asking >> > for the layout for a period of time, having again the queue of >> > waiters >> >> Your assumption doesn't match to our observation. For files that off- >> line >> (DS down or file is on tape) we return LAYOUTTRYLATER. Usually, >> client keep >> re-trying LAYOUTGET until file is available again. We use flexfile >> layout >> as nfs4_file has less predictable behavior. For files that should be >> served >> by MDS we return LAYOUTUNAVAILABLE. Typically, those files are quite >> small >> and served with a single READ request, so I haven't observe >> repetitive LAYOUTGET >> calls. > > Right. If you're only doing READ, and this is a small file, then you > are unlikely to see any repetition of the LAYOUTGET calls like Olga > describes, because the client page cache will take care of serialising > the initial I/O (by means of the folio lock/page lock) and will cache > the results so that no further I/O is needed. > > The main problems with NFS4ERR_LAYOUTUNAVAILABLE in current pNFS > implementation will occur when reading large files and/or when writing > to the file. In those cases, we will send a LAYOUTGET each time we need > to schedule more I/O because we don't cache the negative result of the > previous attempt. > > -- > Trond Myklebust > Linux NFS client maintainer, Hammerspace > trond.myklebust@xxxxxxxxxxxxxxx
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
