Hello,
I had a scp failure problem here. I analyze the code, and the reasons may be as follows:
>From commit e7a5f1f1cd00 ("bpf/sockmap: Read psock ingress_msg before
sk_receive_queue", if we use sockops (BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB
and BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB) to enable socket's sockmap
function, and don't enable strparse and verdict function, the out-of-order
problem may occur in the following process.
client SK server SK
--------------------------------------------------------------------------
tcp_rcv_synsent_state_process
tcp_finish_connect
tcp_init_transfer
tcp_set_state(sk, TCP_ESTABLISHED);
// insert SK to sockmap
wake up waitter
tcp_send_ack
tcp_bpf_sendmsg(msgA)
// msgA will go tcp stack
tcp_rcv_state_process
tcp_init_transfer
//insert SK to sockmap
tcp_set_state(sk,
TCP_ESTABLISHED)
wake up waitter
tcp_bpf_sendmsg(msgB)
// msgB go sockmap
tcp_bpf_recvmsg
//msgB, out-of-order
tcp_bpf_recvmsg
//msgA, out-of-order
Even if msgA arrives earlier than msgB (in most cases), tcp_bpf_recvmsg receives msg from the psock queue first.
The worst case is that msgA waits for serverSK to change to TCP_ESTABLISHED in the protocol stack. msgA may arrive at the serverSK receive queue later than msgB.
If msgA befor than msgB,
If the ACK packets of the three-way TCP handshake are dropped for a period of time, the OOO problem is easily reproduced.
iptables -A INPUT -p tcp -m tcp --dport 5006 --tcp-flags SYN,RST,ACK,FIN ACK -j DROP
...
iptables -D INPUT -p tcp -m tcp --dport 5006 --tcp-flags SYN,RST,ACK,FIN ACK -j DROP
Best Wishes
Liu Jian
