My expectation is that programs shouldn't mix classic and file-private locks, but Glenn Skinner pointed out to me that that may occur at times even if the programmer isn't aware. Suppose we have a program that uses file-private locks. That program then links in a library that uses classic POSIX locks. If those locks end up conflicting and one is using blocking locks, then the program could end up deadlocked. Try to catch this situation in posix_locks_deadlock by looking for the case where the blocking lock was set by the same process but has a different type, and have the kernel return EDEADLK if that occurs. This check is not perfect. You could (in principle) have a threaded process that is using classic locks in one thread and file-private locks in another. That's not necessarily a deadlockable situation but this check would cause an EDEADLK return in that case. By the same token, you could also have a file-private lock that was inherited across a fork(). If the inheriting process ends up blocking on that while trying to set a classic POSIX lock then this check would miss it and the program would deadlock. Signed-off-by: Jeff Layton <jlayton@xxxxxxxxxx> --- fs/locks.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/fs/locks.c b/fs/locks.c index 6fdf26a79cc8..19c0c5c24b93 100644 --- a/fs/locks.c +++ b/fs/locks.c @@ -790,7 +790,17 @@ static int posix_locks_deadlock(struct file_lock *caller_fl, int i = 0; /* - * This deadlock detector can't reasonably detect deadlocks with + * If one lock is file-private and the other one isn't, and these are + * owned by the same process, then we may be in a situation where + * a library is attempting to use a different locking flavor than the + * original program. + */ + if (caller_fl->fl_pid == block_fl->fl_pid && + IS_FILE_PVT(caller_fl) != IS_FILE_PVT(block_fl)) + return 1; + + /* + * This deadlock detector can't reasonably detect cyclic deadlocks with * FL_FILE_PVT locks, since they aren't owned by a process, per-se. */ if (IS_FILE_PVT(caller_fl)) -- 1.8.5.3 -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
