thanks Julian yes, I know that the one-second delay issue has been fixed by commit f0a5e4d7a594e0fe237d3dfafb069bb82f80f42f if we set conn_reuse_mode to 1 BUT it's still NOT what we expected with sysctl settings (conn_reuse_mode == 0 && expire_nodest_conn == 1). We run kubernetes in extremely diverse environments and this issue happens a lot. On Tue, Oct 26, 2021 at 1:44 PM Julian Anastasov <ja@xxxxxx> wrote: > > > Hello, > > On Tue, 26 Oct 2021, yangxingwu wrote: > > > thanks julian > > > > What happens in this situation is that if we set the wait of the > > realserver to 0 and do NOT remove the weight zero realserver with > > sysctl settings (conn_reuse_mode == 0 && expire_nodest_conn == 1), and > > the client reuses its source ports, the kernel will constantly > > reuse connections and send the traffic to the weight 0 realserver. > > Yes, this is expected when conn_reuse_mode=0. > > > you may check the details from > > https://github.com/kubernetes/kubernetes/issues/81775 > > What happens if you try conn_reuse_mode=1? The > one-second delay in previous kernels should be corrected with > > commit f0a5e4d7a594e0fe237d3dfafb069bb82f80f42f > Date: Wed Jul 1 18:17:19 2020 +0300 > > ipvs: allow connection reuse for unconfirmed conntrack > > > On Tue, Oct 26, 2021 at 2:12 AM Julian Anastasov <ja@xxxxxx> wrote: > > > > > > On Mon, 25 Oct 2021, yangxingwu wrote: > > > > > > > Since commit dc7b3eb900aa ("ipvs: Fix reuse connection if real server is > > > > dead"), new connections to dead servers are redistributed immediately to > > > > new servers. > > > > > > > > Then commit d752c3645717 ("ipvs: allow rescheduling of new connections when > > > > port reuse is detected") disable expire_nodest_conn if conn_reuse_mode is > > > > 0. And new connection may be distributed to a real server with weight 0. > > > > > > Your change does not look correct to me. At the time > > > expire_nodest_conn was created, it was not checked when > > > weight is 0. At different places different terms are used > > > but in short, we have two independent states for real server: > > > > > > - inhibited: weight=0 and no new connections should be served, > > > packets for existing connections can be routed to server > > > if it is still available and packets are not dropped > > > by expire_nodest_conn. > > > The new feature is that port reuse detection can > > > redirect the new TCP connection into a new IPVS conn and > > > to expire the existing cp/ct. > > > > > > - unavailable (!IP_VS_DEST_F_AVAILABLE): server is removed, > > > can be temporary, drop traffic for existing connections > > > but on expire_nodest_conn we can select different server > > > > > > The new conn_reuse_mode flag allows port reuse to > > > be detected. Only then expire_nodest_conn has the > > > opportunity with commit dc7b3eb900aa to check weight=0 > > > and to consider the old traffic as finished. If a new > > > server is selected, any retrans from previous connection > > > would be considered as part from the new connection. It > > > is a rapid way to switch server without checking with > > > is_new_conn_expected() because we can not have many > > > conns/conntracks to different servers. > > Regards > > -- > Julian Anastasov <ja@xxxxxx>
