On Mon, 3 Mar 2014 13:22:29 -0500 Trond Myklebust <trond.myklebust@xxxxxxxxxxxxxxx> wrote: > > On Mar 3, 2014, at 11:41, Jeff Layton <jlayton@xxxxxxxxxx> wrote: > > > On Mon, 3 Mar 2014 10:46:37 -0500 > > Trond Myklebust <trond.myklebust@xxxxxxxxxxxxxxx> wrote: > > > >> > >> On Mar 3, 2014, at 10:43, Jeff Layton <jlayton@xxxxxxxxxx> wrote: > >> > >>> On Mon, 03 Mar 2014 06:47:52 +0100 > >>> Dennis Jacobfeuerborn <dennisml@xxxxxxxxxxxx> wrote: > >>> > >>>> Hi, > >>>> I'm experimenting with using NFSv4 as storage for web servers and while > >>>> regular file access seems to work fine as soon as I bring flock() into > >>>> play things become more problematic. > >>>> I've create a tiny test php script that basically opens a file, locks it > >>>> using flock(), writes that fact into a log file (on a local filesystem), > >>>> performs a usleep(1000), writes into the log that it is about to unlock > >>>> the file and finally unlocks it. > >>>> I invoke that script using ab with a concurrency of 20 for a few > >>>> thousand requests. > >>>> > >>> > >>> Is all the activity from a single client, or are multiple clients > >>> contending for the lock? > >>> > >>>> The result is that while 99% of the request respond quickly a few > >>>> request seem to hang for up to 30 seconds. According to the log file > >>>> they must eventually succeed since I see all expected entries and the > >>>> locking seems to work as well since all entries are in the expected order. > >>>> > >>>> Is it expected that these long delays happen? When I comment the locking > >>>> function out these hangs disappear. > >>>> Are there some knobs to tune NFS and make it behave better in these > >>>> situations? > >>>> > >>> > >>> NFSv4 locking is inherently unfair. If you're doing a blocking lock, > >>> then the client is expected to poll for it. So, long delays are > >>> possible if you just happen to be unlucky and keep missing the lock. > >>> > >>> There's no knob to tune, but there probably is room for improvement in > >>> this code. In principle we could try to be more aggressive about > >>> getting the lock by trying to wake up one or more blocked tasks whenever > >>> a lock is released. You might still end up with delays, but it could > >>> help improve responsiveness. > >> > >> …or you could implement the NFSv4.1 lock callback functionality. That would scale better than more aggressive polling. > > > > I had forgotten about those. I wonder what servers actually implement > > them? I don't think Linux' knfsd does yet. > > > > I wasn't really suggesting more aggressive polling. The timer semantics > > seem fine as they are, but we could short circuit it when we know that > > a lock on the inode has just become free. > > How do we “know” that the lock is free? We already track all the locks that our client holds, and wait for those to be released. I can’t see what else there is to do. > Right, we do that, but tasks that are polling for the lock don't get woken up when a task releases a lock. They currently just wait until the timeout occurs and then attempt to acquire the lock. The pessimal case is that: - try to acquire the lock and be denied - task goes to sleep for 30s - just after that, another task releases the lock The first task will wait for 30s before retrying when it could have gotten the lock soon afterward. The idea would be to go ahead and wake up all the blocked waiters on an inode when a task releases a lock. They'd then just re-attempt acquiring the lock immediately instead of waiting on the timeout. On a highly contended lock, most of the waiters would just go back to sleep after being denied again, but one might end up getting the lock and keeping things moving. We could also try to be clever and only wake up tasks that are blocked on the range being released, but in Dennis' case, he's using flock() so that wouldn't really buy him anything. -- Jeff Layton <jlayton@xxxxxxxxxx> -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
