The patch titled
Subject: epoll: eliminate unnecessary lock for zero timeout
has been added to the -mm tree. Its filename is
epoll-eliminate-unnecessary-lock-for-zero-timeout.patch
This patch should soon appear at
https://ozlabs.org/~akpm/mmots/broken-out/epoll-eliminate-unnecessary-lock-for-zero-timeout.patch
and later at
https://ozlabs.org/~akpm/mmotm/broken-out/epoll-eliminate-unnecessary-lock-for-zero-timeout.patch
Before you just go and hit "reply", please:
a) Consider who else should be cc'ed
b) Prefer to cc a suitable mailing list as well
c) Ideally: find the original patch on the mailing list and do a
reply-to-all to that, adding suitable additional cc's
*** Remember to use Documentation/process/submit-checklist.rst when testing your code ***
The -mm tree is included into linux-next and is updated
there every 3-4 working days
------------------------------------------------------
From: Soheil Hassas Yeganeh <soheil@xxxxxxxxxx>
Subject: epoll: eliminate unnecessary lock for zero timeout
We call ep_events_available() under lock when timeout is 0, and then call
it without locks in the loop for the other cases.
Instead, call ep_events_available() without lock for all cases. For
non-zero timeouts, we will recheck after adding the thread to the wait
queue. For zero timeout cases, by definition, user is opportunistically
polling and will have to call epoll_wait again in the future.
Note that this lock was kept in c5a282e9635e9 because the whole loop was
historically under lock.
This patch results in a 1% CPU/RPC reduction in RPC benchmarks.
Link: https://lkml.kernel.org/r/20201106231635.3528496-9-soheil.kdev@xxxxxxxxx
Signed-off-by: Soheil Hassas Yeganeh <soheil@xxxxxxxxxx>
Suggested-by: Eric Dumazet <edumazet@xxxxxxxxxx>
Reviewed-by: Eric Dumazet <edumazet@xxxxxxxxxx>
Reviewed-by: Willem de Bruijn <willemb@xxxxxxxxxx>
Reviewed-by: Khazhismel Kumykov <khazhy@xxxxxxxxxx>
Cc: Guantao Liu <guantaol@xxxxxxxxxx>
Cc: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
---
fs/eventpoll.c | 25 ++++++++++++-------------
1 file changed, 12 insertions(+), 13 deletions(-)
--- a/fs/eventpoll.c~epoll-eliminate-unnecessary-lock-for-zero-timeout
+++ a/fs/eventpoll.c
@@ -1830,7 +1830,7 @@ static inline struct timespec64 ep_set_m
static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
int maxevents, long timeout)
{
- int res, eavail = 0, timed_out = 0;
+ int res, eavail, timed_out = 0;
u64 slack = 0;
wait_queue_entry_t wait;
ktime_t expires, *to = NULL;
@@ -1846,18 +1846,21 @@ static int ep_poll(struct eventpoll *ep,
} else if (timeout == 0) {
/*
* Avoid the unnecessary trip to the wait queue loop, if the
- * caller specified a non blocking operation. We still need
- * lock because we could race and not see an epi being added
- * to the ready list while in irq callback. Thus incorrectly
- * returning 0 back to userspace.
+ * caller specified a non blocking operation.
*/
timed_out = 1;
-
- write_lock_irq(&ep->lock);
- eavail = ep_events_available(ep);
- write_unlock_irq(&ep->lock);
}
+ /*
+ * This call is racy: We may or may not see events that are being added
+ * to the ready list under the lock (e.g., in IRQ callbacks). For, cases
+ * with a non-zero timeout, this thread will check the ready list under
+ * lock and will added to the wait queue. For, cases with a zero
+ * timeout, the user by definition should not care and will have to
+ * recheck again.
+ */
+ eavail = ep_events_available(ep);
+
while (1) {
if (eavail) {
/*
@@ -1873,10 +1876,6 @@ static int ep_poll(struct eventpoll *ep,
if (timed_out)
return 0;
- eavail = ep_events_available(ep);
- if (eavail)
- continue;
-
eavail = ep_busy_loop(ep, timed_out);
if (eavail)
continue;
_
Patches currently in -mm which might be from soheil@xxxxxxxxxx are
epoll-check-for-events-when-removing-a-timed-out-thread-from-the-wait-queue.patch
epoll-simplify-signal-handling.patch
epoll-pull-fatal-signal-checks-into-ep_send_events.patch
epoll-move-eavail-next-to-the-list_empty_careful-check.patch
epoll-simplify-and-optimize-busy-loop-logic.patch
epoll-pull-all-code-between-fetch_events-and-send_event-into-the-loop.patch
epoll-replace-gotos-with-a-proper-loop.patch
epoll-eliminate-unnecessary-lock-for-zero-timeout.patch
