2012/4/10, Bogdan Cristea <cristeab@xxxxxxxxx>: > On Tuesday 10 April 2012 01:46:45 you wrote: >> Hello, >> >> I'm not sure about the mailing list I should ask this. I would try >> linux-net if it was still alive. >> >> I spent a few days searching about a weird bug I had. >> It is actually similar to this one: >> http://marc.info/?l=linux-net&m=127651583824851&w=2 . >> >> Summary for lazy people ^^: calling close(2) on a socket with a >> non-empty receive kernel-buffer cause the connection to be ReSeT and >> the send buffer discarded and not sent. >> >> >> I have a server program (code included below) that wait for a client. >> When a client connect, the server: >> - sent "Greet\n" >> - read one char >> - send "Hello " >> - send "World\n" >> - send "Quit\n" >> - close the socket >> >> And it happens that sometimes (quite often actually) that the client >> does not recieve anything after "Hello " and the connection is just >> closed. >> Actually, after I dumped the TCP trafic with wireshark, I saw that >> "World\n" and "Quit\n" were NOT sent, despite the send(2) succeded. >> The server just send a packet with the RST flag to interrupt the >> connection. >> Even more strange: This only occurs when the client send more data to >> the server than what was expected. >> i.e.: the server read one byte, the client send one, the last messages >> arrives just fine. The client send two bytes, the last messages never >> arrives! >> >> From what I intuited and understood from the Linux kernel code: >> http://lxr.linux.no/linux+v3.3.1/net/ipv4/tcp.c#L1893 >> When the sever call close(2) on the socket file descriptor, the >> connection is "reset" if the receive buffer was not empty. And in that >> case, the output buffer is never sent to the client (whenever >> SO_LINGER is set or not). >> >> >> The workaround I found is to call shutdown(2) before calling close. >> When the outgoing direction of the socket is shutdown the buffer is >> flushed and sent, and it initiate a gentle connection ending. (While >> the close still send a RST because of the non-empty input buffer.) >> >> Therefore I have two questions: >> 1) Is this a standard behavior? Doesn't the RFC state that every >> pending data is sent when the connection is closed? >> 2) Shouldn't that behavior be documented somewhere? I didn't found any >> information about that anywhere. I looked at the man close(2), >> shutdown(2), socket(7), tcp(7). >> >> From this I deduce that shutdown must be called everytime we want to >> close a socket. But this is not taught anywhere. :p >> >> >> >> Here is the code of the server for those who want to try it. And since >> it seems time related I also provide a client that exhibit the bug (at >> least on my machine). >> >> #include <stdio.h> >> #include <stdlib.h> >> #include <unistd.h> >> #include <string.h> >> #include <sys/types.h> >> #include <sys/socket.h> >> #include <netdb.h> >> >> >> >> #define STRING_GREET "Greet\n" >> #define STRING_HELLO "Hello " >> #define STRING_WORLD "World" >> #define STRING_QUIT "Quit\n" >> >> >> >> >> int create_socket(void) { >> struct addrinfo hints, *res, *rp; >> int sockfd = -1; >> int err; >> >> >> memset(&hints, 0, sizeof(hints)); >> hints.ai_family = AF_UNSPEC; >> hints.ai_socktype = SOCK_STREAM; >> hints.ai_protocol = 0; >> hints.ai_flags = AI_V4MAPPED | AI_PASSIVE; >> >> err = getaddrinfo("127.0.0.1", "1337", &hints, &res); >> if (err) { >> fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(err)); >> exit(EXIT_FAILURE); >> } >> >> for (rp = res; rp; rp = rp->ai_next) { >> int optval; >> >> sockfd = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol); >> if (sockfd == -1) { >> perror("socket"); >> continue; >> } >> >> optval = 1; >> err = setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &optval, >> sizeof(optval)); if (err == -1) { >> perror("setsockopt"); >> close(sockfd); >> sockfd = -1; >> continue; >> } >> >> err = bind(sockfd, rp->ai_addr, rp->ai_addrlen); >> if (err == -1) { >> perror("bind"); >> close(sockfd); >> sockfd = -1; >> continue; >> } >> >> break; >> } >> >> freeaddrinfo(res); >> >> if (sockfd == -1) { >> fprintf(stderr, "can't bind at all\n"); >> exit(EXIT_FAILURE); >> } >> >> err = listen(sockfd, 5); >> if (err == -1) { >> perror("listen"); >> close(sockfd); >> exit(EXIT_FAILURE); >> } >> >> return sockfd; >> } >> >> >> >> void handle_client(int sockfd, int s) { >> char c; >> int err; >> >> err = send(s, STRING_GREET, strlen(STRING_GREET), MSG_NOSIGNAL); >> if (err == -1) { >> perror("send"); >> close(s); >> close(sockfd); >> exit(EXIT_FAILURE); >> } >> fprintf(stderr, "send(STRING_GREET) = %d\n", err); >> >> err = recv(s, &c, sizeof(c), 0); >> if (err == -1) { >> perror("recv"); >> close(s); >> close(sockfd); >> exit(EXIT_FAILURE); >> } >> >> err = send(s, STRING_HELLO, strlen(STRING_HELLO), MSG_NOSIGNAL); >> if (err == -1) { >> perror("send"); >> close(s); >> close(sockfd); >> exit(EXIT_FAILURE); >> } >> fprintf(stderr, "send(STRING_HELLO) = %d\n", err); >> >> err = send(s, STRING_WORLD, strlen(STRING_WORLD), MSG_NOSIGNAL); >> if (err == -1) { >> perror("send"); >> close(s); >> close(sockfd); >> exit(EXIT_FAILURE); >> } >> fprintf(stderr, "send(STRING_WORLD) = %d\n", err); >> >> err = send(s, "\n", 1, 0); >> if (err == -1) { >> perror("send"); >> close(s); >> close(sockfd); >> exit(EXIT_FAILURE); >> } >> fprintf(stderr, "send(\\n) = %d\n", err); >> >> err = send(s, STRING_QUIT, strlen(STRING_QUIT), MSG_NOSIGNAL); >> if (err == -1) { >> perror("send"); >> close(s); >> close(sockfd); >> exit(EXIT_FAILURE); >> } >> fprintf(stderr, "send(STRING_QUIT) = %d\n", err); >> >> /*err = shutdown(s, SHUT_RDWR); >> if (err == -1) { >> perror("shutdown"); >> close(sockfd); >> exit(EXIT_FAILURE); >> }*/ >> >> err = close(s); >> if (err == -1) { >> perror("close"); >> close(sockfd); >> exit(EXIT_FAILURE); >> } >> } >> >> >> >> int main(void) { >> struct linger lin; >> int sockfd; >> int err; >> >> sockfd = create_socket(); >> >> lin.l_onoff = 1; >> lin.l_linger = 1; >> err = setsockopt(sockfd, SOL_SOCKET, SO_LINGER, &lin, sizeof(lin)); >> if (err == -1) { >> perror("setsockopt(SO_LINGER)"); >> close(sockfd); >> exit(EXIT_FAILURE); >> } >> >> >> while (1) { >> int s; >> >> s = accept(sockfd, NULL, NULL); >> if (s == -1) { >> perror("accept"); >> close(sockfd); >> exit(EXIT_FAILURE); >> } >> >> handle_client(sockfd, s); >> } >> >> err = close(sockfd); >> if (err == -1) { >> perror("close"); >> exit(EXIT_FAILURE); >> } >> >> return EXIT_SUCCESS; >> } >> >> >> >> >> >> >> >> >> The client: >> >> #include <stdio.h> >> #include <stdlib.h> >> #include <unistd.h> >> #include <string.h> >> #include <sys/types.h> >> #include <sys/socket.h> >> #include <netdb.h> >> >> >> #define MIN(a, b) (((a) < (b)) ? (a) : (b)) >> >> >> size_t readline(int fd, unsigned char *out, size_t out_size) { >> static unsigned char buff[8 * 1024]; >> static size_t data_buff = 0; >> size_t retval = 0; >> ssize_t nr; >> >> while (1) { >> unsigned char *nl; >> >> >> /* Is there already a \n in the buffer? */ >> nl = memchr(buff, '\n', data_buff); >> if (nl) { >> size_t nc = MIN(out_size, nl - buff + 1UL); >> memcpy(out, buff, nc); >> retval += nc; >> memmove(buff, buff + nc, data_buff - nc); >> data_buff -= nc; >> break; >> } else { >> /* No \n found */ >> if (data_buff >= out_size) { >> /* No space left in the out buffer */ >> memcpy(out, buff, out_size); >> retval += out_size; >> memmove(buff, buff + out_size, data_buff - out_size); >> data_buff -= out_size; >> break; >> } else { >> /* No \n and some space left in the out buffer. >> * copy _ALL_ the buffer! */ >> memcpy(out, buff, data_buff); >> retval += data_buff; >> out += data_buff; >> out_size -= data_buff; >> data_buff = 0; >> } >> } >> >> >> nr = recv(fd, buff, sizeof(buff), 0); >> >> /* No matter the errors, we have data! */ >> if (nr == -1 && retval != 0) >> break; >> >> if (nr == -1) { >> perror("recv"); >> close(fd); >> exit(EXIT_FAILURE); >> } >> >> if (nr == 0) >> break; >> >> data_buff = nr; >> } >> >> return retval; >> } >> >> >> >> int main(void) { >> struct addrinfo hints, *res, *rp; >> int sockfd = -1; >> int err; >> unsigned char buff[8 * 1024]; >> size_t line_size = 0; >> >> >> memset(&hints, 0, sizeof(hints)); >> hints.ai_family = AF_INET; >> hints.ai_socktype = SOCK_STREAM; >> hints.ai_protocol = 0; >> hints.ai_flags = AI_V4MAPPED; >> >> err = getaddrinfo("127.0.0.1", "1337", &hints, &res); >> if (err) { >> fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(err)); >> exit(EXIT_FAILURE); >> } >> >> for (rp = res; rp; rp = rp->ai_next) { >> sockfd = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol); >> if (sockfd == -1) { >> perror("socket"); >> continue; >> } >> >> err = connect(sockfd, rp->ai_addr, rp->ai_addrlen); >> if (err == -1) { >> perror("connect"); >> close(sockfd); >> sockfd = -1; >> continue; >> } >> >> break; >> } >> >> freeaddrinfo(res); >> >> if (sockfd == -1) { >> fprintf(stderr, "can't connect at all\n"); >> exit(EXIT_FAILURE); >> } >> >> line_size = readline(sockfd, buff, sizeof(buff)); >> write(STDOUT_FILENO, buff, line_size); >> >> err = send(sockfd, "XX", 2, 0); >> if (err == -1) { >> perror("send"); >> close(sockfd); >> exit(EXIT_FAILURE); >> } >> >> line_size = readline(sockfd, buff, sizeof(buff)); >> write(STDOUT_FILENO, buff, line_size); >> >> line_size = readline(sockfd, buff, sizeof(buff)); >> write(STDOUT_FILENO, buff, line_size); >> >> err = close(sockfd); >> if (err == -1) { >> perror("close"); >> exit(EXIT_FAILURE); >> } >> >> return EXIT_SUCCESS; >> } >> >> >> Thanks for reading that whole long message. >> >> Celelibi >> -- >> To unsubscribe from this list: send the line "unsubscribe >> linux-c-programming" in the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > Hi > > It seems indeed a little strange that with SO_LINGER option you cannot send > everything. Anyway, at the first look it seems that you have several > programming errors: > - socket ops for the server should be set before starting to listen on the > socket > -in the main function you never exit from the while loop, so why bother to > close the socket after this loop ? > > Have a closer look at your code, there might be a bug in your program. > > regards > -- > Bogdan > Hello, thanks for reading my long message. By the way, I'm not sure I successfully subscribed to the mailing list since your mail hadn't a List-Id header, and I do not have a majordomo signature added. My bad for the setsockopt. But wherever I put it, I see no change. And I notice that everywhere I read some information about SO_LINGER, it says that it is a way to LOSE data, and that by default the TCP will try its best to deliver the data. I tried to move the setsockopt to just after the socket() call. Changed nothing. The close after the infinite loop is here because at first the server exited after one client only. That's just a toy program to show the case. And as most part of my student had this bug last year, I guess this is not a programming error. I think the bug is most likely in: - My understanding of the RFC + a lack of documentation - Linux kernel - Something I didn't think about. But don't you agree this is a strange behavior that when send and receive buffer are non-empty when I call close, the send buffer is not actually sent? Celelibi -- To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
