In some of our hottest network services, fget_light + fput overhead can represent 1-2% of the processes' total CPU usage. I'd like to discuss ways to reduce this overhead. One proposal we have been testing is removing the refcount increment and decrement, and using some sort of safe memory reclamation instead. The hottest callers include recvmsg, sendmsg, epoll_wait, etc - mostly networking calls, often used on non-blocking sockets. Often we are only incrementing and decrementing the refcount for a very short period of time, ideally we wouldn't adjust the refcount unless we know we are going to block. We could use RCU, but we would have to be particularly careful that none of these calls ever block, or ensure that we increment the refcount at the blocking locations. As an alternative to RCU, hazard pointers have similar overhead to SRCU, and could work equally well on blocking or nonblocking syscalls without additional changes. (There were also recent related discussions on SCM_RIGHTS refcount cycle issues, which is the other half of a file* gc) There might also be ways to rearrange the file* struct or fd table so that we're not taking so many cache misses for sockfd_lookup_light, since for sockets we don't use most of the file* struct at all.
