Some mistake are printed on the working example, here is the fixed one: root@dev-lvstest-01 ~ ❱ tail -f /var/log/kern.20211021 | grep "172.16.42.170" Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.322832] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.322834] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.322836] IPVS: lookup service: fwm 0 TCP 172.16.42.170:2222 hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.322851] IPVS: Bind-dest TCP c:10.0.0.6:49677 v:172.16.42.170:2222 d:172.16.42.168:2222 fwd:R s:53052 conn->flags:183 conn->refcnt:1 dest->refcnt:2048 Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.322855] IPVS: Schedule fwd:R c:10.0.0.6:49677 v:172.16.42.170:2222 d:172.16.42.168:2222 conn->flags:1C3 conn->refcnt:2 Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.322858] IPVS: TCP input [S...] c:10.0.0.6:49677 v:172.16.42.170:2222 d:172.16.42.168:2222 state: NONE->SYN_RECV conn->refcnt:2 Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.324470] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.324473] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.324475] IPVS: TCP input [..A.] c:10.0.0.6:49677 v:172.16.42.170:2222 d:172.16.42.168:2222 state: SYN_RECV->ESTABLISHED conn->refcnt:2 Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.324998] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.325002] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.334977] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.334979] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.336872] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.336875] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.342537] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.342539] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.345179] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.345181] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.350319] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.350322] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.374611] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.374614] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.419041] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.419045] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.421754] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.421756] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.421763] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.421764] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.432266] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.432268] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.432289] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.432291] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.434754] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.434757] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:33 dev-lvstest-01 kernel: [ 135.041063] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:33 dev-lvstest-01 kernel: [ 135.041065] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:33 dev-lvstest-01 kernel: [ 135.053383] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:33 dev-lvstest-01 kernel: [ 135.053385] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:33 dev-lvstest-01 kernel: [ 135.053398] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:33 dev-lvstest-01 kernel: [ 135.053400] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.706476] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.706479] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.708135] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.708137] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.708208] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.708210] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711111] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711115] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711130] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711133] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711155] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711156] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711183] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711184] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711192] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711194] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711202] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711203] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711219] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711220] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711228] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711230] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711237] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711239] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711255] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711256] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.712512] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.712515] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.712526] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.712528] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 136.098170] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:34 dev-lvstest-01 kernel: [ 136.098173] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.563477] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.563482] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566119] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566123] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566150] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566152] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566215] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566218] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566238] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566240] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566252] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566254] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566255] IPVS: TCP input [.FA.] c:10.0.0.6:49677 v:172.16.42.170:2222 d:172.16.42.168:2222 state: ESTABLISHED->FIN_WAIT conn->refcnt:2 Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.570330] IPVS: lookup/out TCP 10.0.0.6:49677->172.16.42.170:2222 not hit Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.570335] IPVS: lookup/in TCP 10.0.0.6:49677->172.16.42.170:2222 hit Regards, Alexis Vachette. On Thu, 21 Oct 2021 at 11:54, Alexis Vachette <avachette@xxxxxxxxxx> wrote: > > Hello everyone, > > If this is not the correct list to address my issue feel free to tell me. > > We are heavily using Netfilter/IPVS with maglev scheduler algorithm > and it's working great on most use case. > > The only case when we face issue is when inhibit_on_failure option is enabled. > > Here is our setup: > > - 2 LVS director in active/passive mode > - 2 backend node > > Exposed service is SSH using port 2222. > > You can find below the LVS state when issue is occurring: > > root@dev-lvstest-01 ~ ❱ ipvsadm -L -n > IP Virtual Server version 1.2.1 (size=1048576) > Prot LocalAddress:Port Scheduler Flags > -> RemoteAddress:Port Forward Weight ActiveConn InActConn > TCP 172.16.42.170:2222 mh (mh-port) > -> 172.16.42.168:2222 Route 100 0 0 > -> 172.16.42.169:2222 Route 0 0 0 > > Here is a working example output: > > root@dev-lvstest-01 ~ ❱ tail -f /var/log/kern.20211021 | grep "172.16.42.170" > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.322832] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.322834] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.322836] IPVS: lookup > service: fwm 0 TCP 172.16.42.170:2222 hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.322851] IPVS: Bind-dest > TCP c:10.0.0.6:49677 v:172.16.42.170:2222 d:172.31.0.168:2222 fwd:R > s:53052 conn->flags:183 conn->refcnt:1 dest->refcnt:2048 > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.322855] IPVS: Schedule > fwd:R c:10.0.0.6:49677 v:172.16.42.170:2222 d:172.31.0.168:2222 > conn->flags:1C3 conn->refcnt:2 > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.322858] IPVS: TCP input > [S...] c:10.0.0.6:49677 v:172.16.42.170:2222 d:172.31.0.168:2222 > state: NONE->SYN_RECV conn->refcnt:2 > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.324470] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.324473] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.324475] IPVS: TCP input > [..A.] c:10.0.0.6:49677 v:172.16.42.170:2222 d:172.31.0.168:2222 > state: SYN_RECV->ESTABLISHED conn->refcnt:2 > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.324998] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.325002] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.334977] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.334979] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.336872] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.336875] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.342537] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.342539] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.345179] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.345181] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.350319] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.350322] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.374611] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.374614] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.419041] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.419045] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.421754] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.421756] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.421763] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.421764] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.432266] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.432268] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.432289] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.432291] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.434754] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:32 dev-lvstest-01 kernel: [ 134.434757] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:33 dev-lvstest-01 kernel: [ 135.041063] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:33 dev-lvstest-01 kernel: [ 135.041065] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:33 dev-lvstest-01 kernel: [ 135.053383] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:33 dev-lvstest-01 kernel: [ 135.053385] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:33 dev-lvstest-01 kernel: [ 135.053398] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:33 dev-lvstest-01 kernel: [ 135.053400] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.706476] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.706479] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.708135] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.708137] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.708208] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.708210] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711111] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711115] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711130] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711133] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711155] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711156] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711183] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711184] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711192] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711194] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711202] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711203] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711219] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711220] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711228] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711230] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711237] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711239] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711255] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.711256] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.712512] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.712515] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.712526] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 135.712528] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 136.098170] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:34 dev-lvstest-01 kernel: [ 136.098173] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.563477] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.563482] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566119] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566123] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566150] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566152] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566215] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566218] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566238] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566240] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566252] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566254] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.566255] IPVS: TCP input > [.FA.] c:10.0.0.6:49677 v:172.16.42.170:2222 d:172.31.0.168:2222 > state: ESTABLISHED->FIN_WAIT conn->refcnt:2 > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.570330] IPVS: lookup/out > TCP 10.0.0.6:49677->172.16.42.170:2222 not hit > Oct 21 09:18:38 dev-lvstest-01 kernel: [ 139.570335] IPVS: lookup/in > TCP 10.0.0.6:49677->172.16.42.170:2222 hit > > Here is the output when we face the issue described above: > > root@dev-lvstest-01 ~ ☠ ❱ tail -f /var/log/kern.20211021 | grep "172.16.42.170" > Oct 21 09:17:34 dev-lvstest-01 kernel: [ 75.590627] IPVS: lookup/out > TCP 10.0.0.6:49673->172.16.42.170:2222 not hit > Oct 21 09:17:34 dev-lvstest-01 kernel: [ 75.590630] IPVS: lookup/in > TCP 10.0.0.6:49673->172.16.42.170:2222 not hit > Oct 21 09:17:34 dev-lvstest-01 kernel: [ 75.590633] IPVS: lookup > service: fwm 0 TCP 172.16.42.170:2222 hit > Oct 21 09:17:34 dev-lvstest-01 kernel: [ 75.590637] IPVS: mh: TCP > 172.16.42.170:2222 - no destination available > Oct 21 09:17:34 dev-lvstest-01 kernel: [ 75.593020] IPVS: lookup/out > TCP 172.16.42.170:2222->10.0.0.6:49673 not hit > Oct 21 09:17:34 dev-lvstest-01 kernel: [ 75.593022] IPVS: lookup/in > TCP 172.16.42.170:2222->10.0.0.6:49673 not hit > > I'm wondering if in some case the IPVS stack is using the node with > inhibit_on_failure aka weight 0. > > I'm available to do more testing or anything else that can help > troubleshoot this specific issue. > > Regards, > Alexis Vachette.
