On Tue, 2024-07-23 at 10:15 -0400, Brian Cowan wrote: > I am responsible for supporting an application that opens LOTS of > files over NFS from a given host, and potentially a few files/host > from a LOT of clients. We've run into some "interesting" limitations > from other OS's when it comes to NFSv4... > > Solaris, for example, "only" allows 10K or so files per NFS export to > be opened over NFSv4. When you have 2500+ client hosts opening files > over NFSv4, the Solaris NFS server stops responding to "open" requests > until an entry in its state table is freed up by a file close. Which > causes single threaded client processes trying to open said files to > hang... Luckily we convinced that customer to move the clients back to > using NFSv3 since they didn't need the additional features of V4. > > We're also seeing a potential issue with NetApp filers where opening > too many files from a single host seems to have issues. We're being > told that DataOnTAP has a per-client-host limit on the number of files > in the Openstate pool (and not being told what that limit is...) I say > "potential" since the only report is from things falling apart after > moving from AIX 7.2 to 7.3 (meaning there is a non-zero chance that > this is actually an AIX NFS issue). In this case, NFSv3 is not an > option since NFSv4 ACLs are required... > > Anyway, as a result, I'm trying to find out if the Linux NFSv4 server > has a limit on either total number of files, total number of files per > export, or total files per host. > An interesting question. There is a practical limit: Each on-the-wire OPEN takes a stateid, and when we allocate a new one, we generate its id with an idr hash. That's limited to 2^32 values, so once you hit that many active stateids you won't be able to create more. Note that that includes all stateful objects (open, lock, delegation and layout stateids), so you may not be able to do that many OPENs depending on what else is out there. In principle, we might be able to extend that too, but we'd need some evidence that someone's workload was hitting that limit. Beyond that, I can't think of any other limits we might have on being able to create new stateful objects. I'll chime in again though if I think of others though. -- Jeff Layton <jlayton@xxxxxxxxxx>
