Jason Baron <jbaron@xxxxxxxxxx> wrote:
> On 01/18/2015 10:50 PM, Eric Wong wrote:
> > Jason Baron <jbaron@xxxxxxxxxx> wrote:
> >> In addition, we've observed some short comings from our production systems.
> >> Because wait queues are woken up in order, we can get unfair loading
> >> across threads,
> > Are you referring to LIFO scheduling of epoll_wait calling
> > __add_wait_queue_exclusive? I always thought LIFO was preferable
> > since it kept the same threads active to reduce cache misses.
>
> So the specific case I had in mind was where you have an epfd
> per-thread that is attached to a single listen socket. When a
> POLLIN occurs on the listen socket, all threads in epoll_wait will
> be woken up in the order they were added. Then, network
> traffic ends up being processed on the thread which does the
> accept(). This tends to result in an unbalanced load across the
> threads.
Ah, yes. Non-blocking listen sockets is a problem with epoll.
Have you tried using a dedicated thread for blocking accept() and
EPOLL_CTL_ADD?
Other threads will work off entries via epoll_wait and do non-blocking
I/O like a normal epoll server. I do this for cmogstored:
git clone git://bogomips.org/cmogstored
(see accept_loop.c, works best w/o TCP_DEFER_ACCEPT)
One extra side effect of blocking accept() is it ends up balancing
fairly across multiple processes, too, so cmogstored's undocumented
multi-process option can distribute FDs more evenly.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
