PROBLEM: epoll_wait() does not return events when running in multiple threads

[Sergey Nikitin <nikitins@xxxxxxxxxxxx>]

Hi!

epoll does not report an event to all the threads running epoll_wait() 
on the same epoll descriptor.
The behavior appeared in recent kernel versions starting with 5.6 probably.

How to reproduce:
- create a pair of sockets
- create epoll instance
- register the socket on the epoll instance, listen for EPOLLIN events
- start 2 threads running epoll_wait()
- send some data to the socket
- see that epoll_wait() within one of the threads reported an event, 
unlike another.

I attached a python script reproducing the issue.
Here's the output on my environment:
1. Fail case
  $ cat /proc/version
  Linux version 5.7.9-200.fc32.x86_64 
(mockbuild@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx) (gcc version 10.1.1 
20200507 (Red Hat 10.1.1-1) (GCC), GNU ld version 2.34-3.fc32) #1 SMP 
Fri Jul 17 16:23:37 UTC 2020
  $ ./multiple_same_epfd.py
  MainThread: created epfd5
  Thread-1 epfd5: start polling
  Thread-2 epfd5: start polling
  MainThread: Send some data
  Thread-2 epfd5: got events: 1
  Thread-1 epfd5: got events: 0
2. Pass case
  $ cat /proc/version
  Linux version 5.4.17-200.fc31.x86_64 
(mockbuild@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx) (gcc version 9.2.1 20190827 
(Red Hat 9.2.1-1) (GCC)) #1 SMP Sat Feb 1 19:00:13 UTC 2020
  $ ./multiple_same_epfd.py
  MainThread: created epfd5
  Thread-1 epfd5: start polling
  Thread-2 epfd5: start polling
  MainThread: Send some data
  Thread-2 epfd5: got events: 1
  Thread-1 epfd5: got events: 1

I created a Bugzilla bug also:
https://bugzilla.kernel.org/show_bug.cgi?id=208943

--
Best regards,
Sergey Nikitin

#!/usr/bin/python3

import select
import socket
import threading

# Mutex to print messages from multiple threads
lock = threading.Lock()


def epoll_wait_thread(epfd):
    lock.acquire()
    print(threading.currentThread().getName(), " epfd", epfd.fileno(), ": start polling", sep='')
    lock.release()
    events = epfd.poll(3)
    lock.acquire()
    print(threading.currentThread().getName(), " epfd", epfd.fileno(), ": got events: ", len(events), sep='')
    lock.release()


# Create a connection
s1, s2 = socket.socketpair(socket.AF_UNIX)

# Create epoll descriptor and register a socket
epfd = select.epoll()
epfd.register(s1.fileno(), select.EPOLLIN)
print(threading.currentThread().getName(), ": created epfd", epfd.fileno(), sep='')

# Start 2 threads with epoll_wait() routine
threads = []
for i in range(2):
    thread = threading.Thread(target=epoll_wait_thread, args=(epfd,))
    thread.start()
    threads.append(thread)

# Send some data to unblock epoll_wait() threads
lock.acquire()
print(threading.currentThread().getName(), ": Send some data", sep='')
lock.release()
s2.sendall(b'qwerty')

# Cleanup
for thread in threads:
    thread.join()
epfd.close()
s1.close()
s2.close()