Re: disappearing listen()ed SO_REUSEPORT sockets across fork() when using epoll

[Shawn Landden <shawnlandden@xxxxxxxxx>]

On Mon, Nov 25, 2013 at 10:05 AM, Jason Baron <jbaron@xxxxxxxxxx> wrote:
> On 11/22/2013 12:53 PM, Shawn Landden wrote:
>> Hello, when running the attached program on 3.12 child processes
>> are missing a socket fd opened, set with SO_REUSEPORT, listen()ed to,
>> and added to epoll_ctl().
>>
>> This is the output I get when pointing "wget http://localhost:5555/";
>> at the attached program:
>>
>> main PID 31591
>> PID 31634 started
>> PID 31634 accept()ed connection
>> PID 31635 started
>> PID 31636 started
>> PID 31635 accept() failed: Bad file descriptor
>> PID 31636 accept() failed: Bad file descriptor
>> PID 31634 accept()ed connection
>> PID 31634 accept()ed connection
>> PID 31634 accept()ed connection
>> PID 31634 accept()ed connection
>>
>>
>> While I would expect something like:
>>
>> main PID 31591
>> PID 31634 started
>> PID 31634 accept()ed connection
>> PID 31635 started
>> PID 31636 started
>> PID 31635 accept()ed connection
>> PID 31636 accept()ed connection
>>
>> -more new processes, but inversely proportional to number of listening processes
>> -accept() always returns successfully
>>
>>
>
> The 'close(sockfd);' looks to be racing with the accept() calls. Removing seems
> to get the result you are looking for.
Interesting. That works, but it shouldn't. The close() is operating in
the parent, so it shouldn't affect the child,
there is a leak here of process separation.

New version with pid set to volatile, and making sure that we are in the parent.
>
> Thanks,
>
> -Jason
>



-- 

---
Shawn Landden
+1 360 389 3001 (SMS preferred)

#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <stdlib.h>
#include <string.h>
#include <sys/epoll.h>

int main( int argc, char *argv[]) {
        int sockfd, epollfd, acceptfd, portno;
        struct epoll_event event = {EPOLLIN, NULL}, gotevent;
        char buffer[256];
        struct sockaddr_in serv_addr;
        int  n;
	volatile pid_t pid;

        printf("main PID %d\n", getpid());

        memset((char *) &serv_addr, 0, sizeof(serv_addr));
        portno = 5555;
        serv_addr.sin_family = AF_INET;
        serv_addr.sin_addr.s_addr = INADDR_ANY;
        serv_addr.sin_port = htons(portno);

        if ((sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0)
		return printf("socket() failed: %m\n");

        int optval = 1;
        if ((setsockopt(sockfd, SOL_SOCKET, SO_REUSEPORT, &optval, sizeof(optval))) < 0)
                return printf("setsockopt() failed: %m");

        if (bind(sockfd, (struct sockaddr *) &serv_addr,
                          sizeof(serv_addr)) < 0)
		return printf("bind() failed: %m\n");

        if (listen(sockfd, SOMAXCONN) < 0)
		return printf("listen() failed: %m\n");

	if ((epollfd = epoll_create1(0)) < 0)
	        return printf("epoll_create1() failed: %m\n");

        if (epoll_ctl(epollfd,  EPOLL_CTL_ADD, sockfd, &event) < 0)
                return printf("epoll_ctl() failed: %m\n");

        while (1) {
                if (epoll_wait(epollfd, &gotevent, 1, -1) != 1)
                        return printf("epoll_wait() failed: %m\n");

                pid = fork();
                if (pid == 0) {
                        printf("PID %d started\n", getpid());
                        while(1) {
                                struct sockaddr_in cli_addr;
                                socklen_t cli_size = sizeof cli_addr;

                                if ((acceptfd = accept(sockfd, (struct sockaddr *)&cli_addr, 
                                                        &cli_size)) < 0) {
	                                printf("PID %d accept() failed: %m\n", getpid());
	                                sleep(60);
	                                return 1;
                                }

                                printf("PID %d accept()ed connection\n", getpid());

                                if (close(acceptfd) < 0)
                                        return printf("close() failed: %m\n");
                        }
                } else if (pid > 0)
                	close(sockfd);
        }
}