From: Kuniyuki Iwashima <kuniyu@xxxxxxxxxxxx> Date: Wed, 9 Dec 2020 17:05:09 +0900 > From: Martin KaFai Lau <kafai@xxxxxx> > Date: Tue, 8 Dec 2020 19:09:03 -0800 > > On Tue, Dec 08, 2020 at 05:17:48PM +0900, Kuniyuki Iwashima wrote: > > > From: Martin KaFai Lau <kafai@xxxxxx> > > > Date: Mon, 7 Dec 2020 23:34:41 -0800 > > > > On Tue, Dec 08, 2020 at 03:31:34PM +0900, Kuniyuki Iwashima wrote: > > > > > From: Martin KaFai Lau <kafai@xxxxxx> > > > > > Date: Mon, 7 Dec 2020 12:33:15 -0800 > > > > > > On Thu, Dec 03, 2020 at 11:14:24PM +0900, Kuniyuki Iwashima wrote: > > > > > > > From: Eric Dumazet <eric.dumazet@xxxxxxxxx> > > > > > > > Date: Tue, 1 Dec 2020 16:25:51 +0100 > > > > > > > > On 12/1/20 3:44 PM, Kuniyuki Iwashima wrote: > > > > > > > > > This patch lets reuseport_detach_sock() return a pointer of struct sock, > > > > > > > > > which is used only by inet_unhash(). If it is not NULL, > > > > > > > > > inet_csk_reqsk_queue_migrate() migrates TCP_ESTABLISHED/TCP_SYN_RECV > > > > > > > > > sockets from the closing listener to the selected one. > > > > > > > > > > > > > > > > > > Listening sockets hold incoming connections as a linked list of struct > > > > > > > > > request_sock in the accept queue, and each request has reference to a full > > > > > > > > > socket and its listener. In inet_csk_reqsk_queue_migrate(), we only unlink > > > > > > > > > the requests from the closing listener's queue and relink them to the head > > > > > > > > > of the new listener's queue. We do not process each request and its > > > > > > > > > reference to the listener, so the migration completes in O(1) time > > > > > > > > > complexity. However, in the case of TCP_SYN_RECV sockets, we take special > > > > > > > > > care in the next commit. > > > > > > > > > > > > > > > > > > By default, the kernel selects a new listener randomly. In order to pick > > > > > > > > > out a different socket every time, we select the last element of socks[] as > > > > > > > > > the new listener. This behaviour is based on how the kernel moves sockets > > > > > > > > > in socks[]. (See also [1]) > > > > > > > > > > > > > > > > > > Basically, in order to redistribute sockets evenly, we have to use an eBPF > > > > > > > > > program called in the later commit, but as the side effect of such default > > > > > > > > > selection, the kernel can redistribute old requests evenly to new listeners > > > > > > > > > for a specific case where the application replaces listeners by > > > > > > > > > generations. > > > > > > > > > > > > > > > > > > For example, we call listen() for four sockets (A, B, C, D), and close the > > > > > > > > > first two by turns. The sockets move in socks[] like below. > > > > > > > > > > > > > > > > > > socks[0] : A <-. socks[0] : D socks[0] : D > > > > > > > > > socks[1] : B | => socks[1] : B <-. => socks[1] : C > > > > > > > > > socks[2] : C | socks[2] : C --' > > > > > > > > > socks[3] : D --' > > > > > > > > > > > > > > > > > > Then, if C and D have newer settings than A and B, and each socket has a > > > > > > > > > request (a, b, c, d) in their accept queue, we can redistribute old > > > > > > > > > requests evenly to new listeners. > > > > > > > > > > > > > > > > > > socks[0] : A (a) <-. socks[0] : D (a + d) socks[0] : D (a + d) > > > > > > > > > socks[1] : B (b) | => socks[1] : B (b) <-. => socks[1] : C (b + c) > > > > > > > > > socks[2] : C (c) | socks[2] : C (c) --' > > > > > > > > > socks[3] : D (d) --' > > > > > > > > > > > > > > > > > > Here, (A, D) or (B, C) can have different application settings, but they > > > > > > > > > MUST have the same settings at the socket API level; otherwise, unexpected > > > > > > > > > error may happen. For instance, if only the new listeners have > > > > > > > > > TCP_SAVE_SYN, old requests do not have SYN data, so the application will > > > > > > > > > face inconsistency and cause an error. > > > > > > > > > > > > > > > > > > Therefore, if there are different kinds of sockets, we must attach an eBPF > > > > > > > > > program described in later commits. > > > > > > > > > > > > > > > > > > Link: https://lore.kernel.org/netdev/CAEfhGiyG8Y_amDZ2C8dQoQqjZJMHjTY76b=KBkTKcBtA=dhdGQ@xxxxxxxxxxxxxx/ > > > > > > > > > Reviewed-by: Benjamin Herrenschmidt <benh@xxxxxxxxxx> > > > > > > > > > Signed-off-by: Kuniyuki Iwashima <kuniyu@xxxxxxxxxxxx> > > > > > > > > > --- > > > > > > > > > include/net/inet_connection_sock.h | 1 + > > > > > > > > > include/net/sock_reuseport.h | 2 +- > > > > > > > > > net/core/sock_reuseport.c | 10 +++++++++- > > > > > > > > > net/ipv4/inet_connection_sock.c | 30 ++++++++++++++++++++++++++++++ > > > > > > > > > net/ipv4/inet_hashtables.c | 9 +++++++-- > > > > > > > > > 5 files changed, 48 insertions(+), 4 deletions(-) > > > > > > > > > > > > > > > > > > diff --git a/include/net/inet_connection_sock.h b/include/net/inet_connection_sock.h > > > > > > > > > index 7338b3865a2a..2ea2d743f8fc 100644 > > > > > > > > > --- a/include/net/inet_connection_sock.h > > > > > > > > > +++ b/include/net/inet_connection_sock.h > > > > > > > > > @@ -260,6 +260,7 @@ struct dst_entry *inet_csk_route_child_sock(const struct sock *sk, > > > > > > > > > struct sock *inet_csk_reqsk_queue_add(struct sock *sk, > > > > > > > > > struct request_sock *req, > > > > > > > > > struct sock *child); > > > > > > > > > +void inet_csk_reqsk_queue_migrate(struct sock *sk, struct sock *nsk); > > > > > > > > > void inet_csk_reqsk_queue_hash_add(struct sock *sk, struct request_sock *req, > > > > > > > > > unsigned long timeout); > > > > > > > > > struct sock *inet_csk_complete_hashdance(struct sock *sk, struct sock *child, > > > > > > > > > diff --git a/include/net/sock_reuseport.h b/include/net/sock_reuseport.h > > > > > > > > > index 0e558ca7afbf..09a1b1539d4c 100644 > > > > > > > > > --- a/include/net/sock_reuseport.h > > > > > > > > > +++ b/include/net/sock_reuseport.h > > > > > > > > > @@ -31,7 +31,7 @@ struct sock_reuseport { > > > > > > > > > extern int reuseport_alloc(struct sock *sk, bool bind_inany); > > > > > > > > > extern int reuseport_add_sock(struct sock *sk, struct sock *sk2, > > > > > > > > > bool bind_inany); > > > > > > > > > -extern void reuseport_detach_sock(struct sock *sk); > > > > > > > > > +extern struct sock *reuseport_detach_sock(struct sock *sk); > > > > > > > > > extern struct sock *reuseport_select_sock(struct sock *sk, > > > > > > > > > u32 hash, > > > > > > > > > struct sk_buff *skb, > > > > > > > > > diff --git a/net/core/sock_reuseport.c b/net/core/sock_reuseport.c > > > > > > > > > index fd133516ac0e..60d7c1f28809 100644 > > > > > > > > > --- a/net/core/sock_reuseport.c > > > > > > > > > +++ b/net/core/sock_reuseport.c > > > > > > > > > @@ -216,9 +216,11 @@ int reuseport_add_sock(struct sock *sk, struct sock *sk2, bool bind_inany) > > > > > > > > > } > > > > > > > > > EXPORT_SYMBOL(reuseport_add_sock); > > > > > > > > > > > > > > > > > > -void reuseport_detach_sock(struct sock *sk) > > > > > > > > > +struct sock *reuseport_detach_sock(struct sock *sk) > > > > > > > > > { > > > > > > > > > struct sock_reuseport *reuse; > > > > > > > > > + struct bpf_prog *prog; > > > > > > > > > + struct sock *nsk = NULL; > > > > > > > > > int i; > > > > > > > > > > > > > > > > > > spin_lock_bh(&reuseport_lock); > > > > > > > > > @@ -242,8 +244,12 @@ void reuseport_detach_sock(struct sock *sk) > > > > > > > > > > > > > > > > > > reuse->num_socks--; > > > > > > > > > reuse->socks[i] = reuse->socks[reuse->num_socks]; > > > > > > > > > + prog = rcu_dereference(reuse->prog); > > > > > > > > > > > > > > > > > > if (sk->sk_protocol == IPPROTO_TCP) { > > > > > > > > > + if (reuse->num_socks && !prog) > > > > > > > > > + nsk = i == reuse->num_socks ? reuse->socks[i - 1] : reuse->socks[i]; > > > > > > > > > + > > > > > > > > > reuse->num_closed_socks++; > > > > > > > > > reuse->socks[reuse->max_socks - reuse->num_closed_socks] = sk; > > > > > > > > > } else { > > > > > > > > > @@ -264,6 +270,8 @@ void reuseport_detach_sock(struct sock *sk) > > > > > > > > > call_rcu(&reuse->rcu, reuseport_free_rcu); > > > > > > > > > out: > > > > > > > > > spin_unlock_bh(&reuseport_lock); > > > > > > > > > + > > > > > > > > > + return nsk; > > > > > > > > > } > > > > > > > > > EXPORT_SYMBOL(reuseport_detach_sock); > > > > > > > > > > > > > > > > > > diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c > > > > > > > > > index 1451aa9712b0..b27241ea96bd 100644 > > > > > > > > > --- a/net/ipv4/inet_connection_sock.c > > > > > > > > > +++ b/net/ipv4/inet_connection_sock.c > > > > > > > > > @@ -992,6 +992,36 @@ struct sock *inet_csk_reqsk_queue_add(struct sock *sk, > > > > > > > > > } > > > > > > > > > EXPORT_SYMBOL(inet_csk_reqsk_queue_add); > > > > > > > > > > > > > > > > > > +void inet_csk_reqsk_queue_migrate(struct sock *sk, struct sock *nsk) > > > > > > > > > +{ > > > > > > > > > + struct request_sock_queue *old_accept_queue, *new_accept_queue; > > > > > > > > > + > > > > > > > > > + old_accept_queue = &inet_csk(sk)->icsk_accept_queue; > > > > > > > > > + new_accept_queue = &inet_csk(nsk)->icsk_accept_queue; > > > > > > > > > + > > > > > > > > > + spin_lock(&old_accept_queue->rskq_lock); > > > > > > > > > + spin_lock(&new_accept_queue->rskq_lock); > > > > > > > > > > > > > > > > Are you sure lockdep is happy with this ? > > > > > > > > > > > > > > > > I would guess it should complain, because : > > > > > > > > > > > > > > > > lock(A); > > > > > > > > lock(B); > > > > > > > > ... > > > > > > > > unlock(B); > > > > > > > > unlock(A); > > > > > > > > > > > > > > > > will fail when the opposite action happens eventually > > > > > > > > > > > > > > > > lock(B); > > > > > > > > lock(A); > > > > > > > > ... > > > > > > > > unlock(A); > > > > > > > > unlock(B); > > > > > > > > > > > > > > I enabled lockdep and did not see warnings of lockdep. > > > > > > > > > > > > > > Also, the inversion deadlock does not happen in this case. > > > > > > > In reuseport_detach_sock(), sk is moved backward in socks[] and poped out > > > > > > > from the eBPF map, so the old listener will not be selected as the new > > > > > > > listener. > > > > > > > > > > > > > > > > > > > > > > > + > > > > > > > > > + if (old_accept_queue->rskq_accept_head) { > > > > > > > > > + if (new_accept_queue->rskq_accept_head) > > > > > > > > > + old_accept_queue->rskq_accept_tail->dl_next = > > > > > > > > > + new_accept_queue->rskq_accept_head; > > > > > > > > > + else > > > > > > > > > + new_accept_queue->rskq_accept_tail = old_accept_queue->rskq_accept_tail; > > > > > > > > > + > > > > > > > > > + new_accept_queue->rskq_accept_head = old_accept_queue->rskq_accept_head; > > > > > > > > > + old_accept_queue->rskq_accept_head = NULL; > > > > > > > > > + old_accept_queue->rskq_accept_tail = NULL; > > > > > > > > > + > > > > > > > > > + WRITE_ONCE(nsk->sk_ack_backlog, nsk->sk_ack_backlog + sk->sk_ack_backlog); > > > > > > > > > + WRITE_ONCE(sk->sk_ack_backlog, 0); > > > > > > > > > + } > > > > > > > > > + > > > > > > > > > + spin_unlock(&new_accept_queue->rskq_lock); > > > > > > > > > + spin_unlock(&old_accept_queue->rskq_lock); > > > > > > > > > +} > > > > > > > > > +EXPORT_SYMBOL(inet_csk_reqsk_queue_migrate); > > > > > > > > > > > > > > > > I fail to understand how the kernel can run fine right after this patch, before following patches are merged. > > > > > > > > > > > > > > I will squash the two or reorganize them into definition part and migration > > > > > > > part. > > > > > > > > > > > > > > > > > > > > > > All request sockets in the socket accept queue MUST have their rsk_listener set to the listener, > > > > > > > > this is how we designed things (each request socket has a reference taken on the listener) > > > > > > > > > > > > > > > > We might even have some "BUG_ON(sk != req->rsk_listener);" in some places. > > > > > > > > > > > > > > > > Since you splice list from old listener to the new one, without changing req->rsk_listener, bad things will happen. > > > > > > I also have similar concern on the inconsistency in req->rsk_listener. > > > > > > > > > > > > The fix-up in req->rsk_listener for the TFO req in patch 4 > > > > > > makes it clear that req->rsk_listener should be updated during > > > > > > the migration instead of asking a much later code path > > > > > > to accommodate this inconsistent req->rsk_listener pointer. > > > > > > > > > > When I started this patchset, I read this thread and misunderstood that I > > > > > had to migrate sockets in O(1) for scalability. So, I selected the fix-up > > > > > approach and checked rsk_listener is not used except for TFO. > > > > > > > > > > ---8<--- > > > > > Whole point of BPF was to avoid iterate through all sockets [1], > > > > > and let user space use whatever selection logic it needs. > > > > > > > > > > [1] This was okay with up to 16 sockets. But with 128 it does not scale. > > > > > ---&<--- > > > > > https://lore.kernel.org/netdev/1458837191.12033.4.camel@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/ > > > > > > > > > > > > > > > However, I've read it again, and this was about iterating over listeners > > > > > to select a new listener, not about iterating over requests... > > > > > In this patchset, we can select a listener in O(1) and it is enough. > > > > > > > > > > > > > > > > The current inet_csk_listen_stop() is already iterating > > > > > > the icsk_accept_queue and fastopenq. The extra cost > > > > > > in updating rsk_listener may be just noise? > > > > > > > > > > Exactly. > > > > > If we end up iterating requests, it is better to migrate than close. I will > > > > > update each rsk_listener in inet_csk_reqsk_queue_migrate() in v3 patchset. > > > > To be clear, I meant to do migration in inet_csk_listen_stop() instead > > > > of doing it in the new inet_csk_reqsk_queue_migrate() which reqires a > > > > double lock and then need to re-bring in the whole spin_lock_bh_nested > > > > patch in the patch 3 of v2. > > > > > > > > e.g. in the first while loop in inet_csk_listen_stop(), > > > > if there is a target to migrate to, it can do > > > > something similar to inet_csk_reqsk_queue_add(target_sk, ...) > > > > instead of doing the current inet_child_forget(). > > > > It probably needs something different from > > > > inet_csk_reqsk_queue_add(), e.g. also update rsk_listener, > > > > but the idea should be similar. > > > > > > > > Since the rsk_listener has to be updated one by one, there is > > > > really no point to do the list splicing which requires > > > > the double lock. > > > > > > I think it is a bit complex to pass the new listener from > > > reuseport_detach_sock() to inet_csk_listen_stop(). > > > > > > __tcp_close/tcp_disconnect/tcp_abort > > > |-tcp_set_state > > > | |-unhash > > > | |-reuseport_detach_sock (return nsk) > > > |-inet_csk_listen_stop > > Picking the new listener does not have to be done in > > reuseport_detach_sock(). > > > > IIUC, it is done there only because it prefers to pick > > the last sk from socks[] when bpf prog is not attached. > > This seems to get into the way of exploring other potential > > implementation options. > > Yes. > This is just idea, but we can reserve the last index of socks[] to hold the > last 'moved' socket in reuseport_detach_sock() and use it in > inet_csk_listen_stop(). > > > > Merging the discussion on the last socks[] pick from another thread: > > > > > > I think most applications start new listeners before closing listeners, in > > > this case, selecting the moved socket as the new listener works well. > > > > > > > > > > That said, if it is still desired to do a random pick by kernel when > > > > there is no bpf prog, it probably makes sense to guard it in a sysctl as > > > > suggested in another reply. To keep it simple, I would also keep this > > > > kernel-pick consistent instead of request socket is doing something > > > > different from the unhash path. > > > > > > Then, is this way better to keep kernel-pick consistent? > > > > > > 1. call reuseport_select_migrated_sock() without sk_hash from any path > > > 2. generate a random number in reuseport_select_migrated_sock() > > > 3. pass it to __reuseport_select_sock() only for select-by-hash > > > (4. pass 0 as sk_hash to bpf_run_sk_reuseport not to use it) > > > 5. do migration per queue in inet_csk_listen_stop() or per request in > > > receive path. > > > > > > I understand it is beautiful to keep consistensy, but also think > > > the kernel-pick with heuristic performs better than random-pick. > > I think discussing the best kernel pick without explicit user input > > is going to be a dead end. There is always a case that > > makes this heuristic (or guess) fail. e.g. what if multiple > > sk(s) being closed are always the last one in the socks[]? > > all their child sk(s) will then be piled up at one listen sk > > because the last socks[] is always picked? > > There can be such a case, but it means the newly listened sockets are > closed earlier than old ones. > > > > Lets assume the last socks[] is indeed the best for all cases. Then why > > the in-progress req don't pick it this way? I feel the implementation > > is doing what is convenient at that point. And that is fine, I think > > In this patchset, I originally assumed four things: > > migration should be done > (i) from old to new > (ii) to redistribute requests evenly as possible > (iii) to keep the order of requests in the queue > (resulting in splicing queues) > (iv) in O(1) for scalability > (resulting in fix-up rsk_listener approach) > > I selected the last socket in unhash path to satisfy above four because the > last socket changes at every close() syscall if application closes from > older socket. > > But in receiving ACK or retransmitting SYN+ACK, we cannot get the last > 'moved' socket. Even if we reserve the last 'moved' socket in the last > index by the idea above, we cannot sure the last socket is changed after > close() for each req->listener. For example, we have listeners A, B, C, and > D, and then call close(A) and close(B), and receive the final ACKs for A > and B, then both of them are assigned to C. In this case, A for D and B for > C is desired. So, selecting the last socket in socks[] for incoming > requests cannnot realize (ii). > > This is why I selected the last moved socket in unhash path and a random > listener in receive path. > > > > for kernel-pick, it should just go for simplicity and stay with > > the random(/hash) pick instead of pretending the kernel knows the > > application must operate in a certain way. It is fine > > that the pick was wrong, the kernel will eventually move the > > childs/reqs to the survived listen sk. > > Exactly. Also the heuristic way is not fair for every application. > > After reading below idea (migrated_sk), I think random-pick is better > at simplicity and passing each sk. > > > > [ I still think the kernel should not even pick if > > there is no bpf prog to instruct how to pick > > but I am fine as long as there is a sysctl to > > guard this. ] > > Unless different applications listen on the same port, random-pick can save > connections which would be aborted. So, I would add a sysctl to do > migration when no eBPF prog is attached. > > > > I would rather focus on ensuring the bpf prog getting what it > > needs to make the migration pick. A few things > > I would like to discuss and explore: > > > If we splice requests like this, we do not need double lock? > > > > > > 1. lock the accept queue of the old listener > > > 2. unlink all requests and decrement refcount > > > 3. unlock > > > 4. update all requests with new listener > > I guess updating rsk_listener can be done without acquiring > > the lock in (5) below is because it is done under the > > listening_hash's bucket lock (and also the global reuseport_lock) so > > that the new listener will stay in TCP_LISTEN state? > > If we do migration in inet_unhash(), the lock is held, but it is not held > in inet_csk_listen_stop(). > > > > I am not sure iterating the queue under these > > locks is a very good thing to do though. The queue may not be > > very long in usual setup but still let see > > if that can be avoided. > > I agree, lock should not be held long. > > > > Do you think the iteration can be done without holding > > bucket lock and the global reuseport_lock? inet_csk_reqsk_queue_add() > > is taking the rskq_lock and then check for TCP_LISTEN. May be > > something similar can be done also? > > I think either one is necessary at least, so if the sk_state of selected > listener is TCP_CLOSE (this is mostly by random-pick of kernel), then we > have to fall back to call inet_child_forget(). > > > > While doing BPF_SK_REUSEPORT_MIGRATE_REQUEST, > > the bpf prog can pick per req and have the sk_hash. > > However, while doing BPF_SK_REUSEPORT_MIGRATE_QUEUE, > > the bpf prog currently does not have a chance to > > pick individually for each req/child on the queue. > > Since it is iterating the queue anyway, does it make > > sense to also call the bpf to pick for each req/child > > in the queue? It then can pass sk_hash (from child->sk_hash?) > > to the bpf prog also instead of current 0. The cost of calling > > bpf prog is not really that much / signficant at the > > migration code path. If the queue is somehow > > unusally long, there is already an existing > > cond_resched() in inet_csk_listen_stop(). > > > > Then, instead of adding sk_reuseport_md->migration, > > it can then add sk_reuseport_md->migrate_sk. > > "migrate_sk = req" for in-progress req and "migrate_sk = child" > > for iterating acceptq. The bpf_prog can then tell what sk (req or child) > > it is migrating by reading migrate_sk->state. It can then also > > learn the 4 tuples src/dst ip/port while skb is missing. > > The sk_reuseport_md->sk can still point to the closed sk > > such that the bpf prog can learn the cookie. > > > > I suspect a few things between BPF_SK_REUSEPORT_MIGRATE_REQUEST > > and BPF_SK_REUSEPORT_MIGRATE_QUEUE can be folded together > > by doing the above. It also gives a more consistent > > interface for the bpf prog, no more MIGRATE_QUEUE vs MIGRATE_REQUEST. > > I think this is really nice idea. Also, I tried to implement random-pick > one by one in inet_csk_listen_stop() yesterday, I found a concern about how > to handle requests in TFO queue. > > The request can be already accepted, so passing it to eBPF prog is > confusing? But, redistributing randomly can affect all listeners > unnecessary. How should we handle such requests? I've implemented one-by-one migration only for the accept queue for now. In addition to the concern about TFO queue, I want to discuss which should we pass NULL or request_sock to eBPF program as migrate_sk when selecting a listener for SYN ? ---8<--- diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c index a82fd4c912be..d0ddd3cb988b 100644 --- a/net/ipv4/inet_connection_sock.c +++ b/net/ipv4/inet_connection_sock.c @@ -1001,6 +1001,29 @@ struct sock *inet_csk_reqsk_queue_add(struct sock *sk, } EXPORT_SYMBOL(inet_csk_reqsk_queue_add); +static bool inet_csk_reqsk_queue_migrate(struct sock *sk, struct sock *nsk, struct request_sock *req) +{ + struct request_sock_queue *queue = &inet_csk(nsk)->icsk_accept_queue; + bool migrated = false; + + spin_lock(&queue->rskq_lock); + if (likely(nsk->sk_state == TCP_LISTEN)) { + migrated = true; + + req->dl_next = NULL; + if (queue->rskq_accept_head == NULL) + WRITE_ONCE(queue->rskq_accept_head, req); + else + queue->rskq_accept_tail->dl_next = req; + queue->rskq_accept_tail = req; + sk_acceptq_added(nsk); + inet_csk_reqsk_queue_migrated(sk, nsk, req); + } + spin_unlock(&queue->rskq_lock); + + return migrated; +} + struct sock *inet_csk_complete_hashdance(struct sock *sk, struct sock *child, struct request_sock *req, bool own_req) { @@ -1023,9 +1046,11 @@ EXPORT_SYMBOL(inet_csk_complete_hashdance); */ void inet_csk_listen_stop(struct sock *sk) { + struct sock_reuseport *reuseport_cb = rcu_access_pointer(sk->sk_reuseport_cb); struct inet_connection_sock *icsk = inet_csk(sk); struct request_sock_queue *queue = &icsk->icsk_accept_queue; struct request_sock *next, *req; + struct sock *nsk; /* Following specs, it would be better either to send FIN * (and enter FIN-WAIT-1, it is normal close) @@ -1043,8 +1068,19 @@ void inet_csk_listen_stop(struct sock *sk) WARN_ON(sock_owned_by_user(child)); sock_hold(child); + if (reuseport_cb) { + nsk = reuseport_select_migrated_sock(sk, req_to_sk(req)->sk_hash, NULL); + if (nsk) { + if (inet_csk_reqsk_queue_migrate(sk, nsk, req)) + goto unlock_sock; + else + sock_put(nsk); + } + } + inet_child_forget(sk, req, child); reqsk_put(req); +unlock_sock: bh_unlock_sock(child); local_bh_enable(); sock_put(child); ---8<--- > > > 5. lock the accept queue of the new listener > > > 6. splice requests and increment refcount > > > 7. unlock > > > > > > Also, I think splicing is better to keep the order of requests. Adding one > > > by one reverses it. > > It can keep the order but I think it is orthogonal here.
