On Tue, 25 Jan 2022 at 14:00, J. Bruce Fields <bfields@xxxxxxxxxxxx> wrote: > > On Tue, Jan 25, 2022 at 12:52:46PM +0000, Daire Byrne wrote: > > Yea, it does seem like the server is the ultimate arbitrar and the > > fact that multiple clients can achieve much higher rates of > > parallelism does suggest that the VFS locking per client is somewhat > > redundant and limiting (in this super niche case). > > It doesn't seem *so* weird to have a server with fast storage a long > round-trip time away, in which case the client-side operation could take > several orders of magnitude longer than the server. Yea, I'm fine with the speed of light constraints for a single process far away. But the best way to achieve aggregate performance in such environments is to have multiple parallel streams in flight at once (preferably bulk transfers). Because I am writing through a single re-export server, I just so happen to be killing any parallelism for single directory files creates even though it works reasonably well for opens, reads, writes and stat (and anything cacheable) which all retain a certain amount of useful parallelism over a high latency network (nconnect helps too). Each of our batch jobs all read 95% of the same files each time and they all tend to run within the same short (hour) time periods so highly cacheable. > Though even if the client locking wasn't a factor, you might still have > to do some work to take advantage of that. (E.g. if your workload is > just a single "untar"--it still waits for one create before doing the > next one). Yep. Again I'm okay with each client of a re-export server doing 3 creates per second, my problem is that all N instances together do 3 creates per second aggregate (in a single directory). But I guess for this kind of workload, win some lose some. I just need to figure out if I can engineer it to be less of a loser... Of course, Hammerspace have a different approach to this kind of problem with their global namespace and replicated MDT servers. And that is probably a much more sensible way of going about this kind of thing. Daire
