On 9/29/20 7:35 PM, Stephen Hemminger wrote: > > > Begin forwarded message: > > Date: Tue, 29 Sep 2020 17:15:23 +0000 > From: bugzilla-daemon@xxxxxxxxxxxxxxxxxxx > To: stephen@xxxxxxxxxxxxxxxxxx > Subject: [Bug 209427] New: Incorrect timestamp cause packet to be dropped > > > https://bugzilla.kernel.org/show_bug.cgi?id=209427 > > Bug ID: 209427 > Summary: Incorrect timestamp cause packet to be dropped > Product: Networking > Version: 2.5 > Kernel Version: 5.8.10 > Hardware: All > OS: Linux > Tree: Mainline > Status: NEW > Severity: normal > Priority: P1 > Component: Other > Assignee: stephen@xxxxxxxxxxxxxxxxxx > Reporter: abt-admin@xxxxxxx > Regression: No > > After upgrading from my 3.10 to 5.8.10 I found out some of my packets are > getting dropped by ipvlan interface (I'm using qdisk fq). Debugging session led > me to the place where this happens > > net/sched/sch_fq.c:464 > ... > if (fq_packet_beyond_horizon(skb, q)) { > if (q->horizon_drop) { > q->stat_horizon_drops++; > return qdisc_drop(skb, sch, to_free); > } > q->stat_horizon_caps++; > skb->tstamp = q->ktime_cache + q->horizon; > } > ... > > then I noticed that in some cases skb->tstamp is equal to real ts whereas in > the regular cases where a packet pass through it's time since kernel boot. This > doesn't make any sense for me as this condition is satisfied constantly > > net/sched/sch_fq.c:439 > static bool fq_packet_beyond_horizon(const struct sk_buff *skb, > const struct fq_sched_data *q) > { > return unlikely((s64)skb->tstamp > (s64)(q->ktime_cache + q->horizon)); > } > > Any ideas on what it can be? > > some outputs: > [Tue Sep 29 14:59:06 2020] DBG: TIME: trid: -1081131982. all: > q->ktime_cache 328453964122, skb->tstamp 1601391546982793177 << > [Tue Sep 29 14:59:06 2020] DBG: TIME: trid: -1485308564. all: > q->ktime_cache 334998110463, skb->tstamp 335012588783 > ... > [Tue Sep 29 14:59:06 2020] DBG: TIME: trid: -1010372082. all: > q->ktime_cache 335873778729, skb->tstamp 335741726080 > [Tue Sep 29 14:59:06 2020] DBG: TIME: trid: 192888327. all: q->ktime_cache > 335860696387, skb->tstamp 335870531339 > [Tue Sep 29 14:59:07 2020] DBG: TIME: trid: -1463571809. all: > q->ktime_cache 335305774517, skb->tstamp 1601391548109319017 << > > perf trace: > curl 32613 [014] 1546.957467: skb:kfree_skb: skbaddr=0xffff888f57159ae0 > protocol=2048 location=0xffffffff817bfdad > 9bfd56 kfree_skb (/lib/modules/5.8.10/build/vmlinux) > 9bfd56 kfree_skb (/lib/modules/5.8.10/build/vmlinux) > 9bfdad kfree_skb_list (/lib/modules/5.8.10/build/vmlinux) > 9da2b1 __dev_queue_xmit (/lib/modules/5.8.10/build/vmlinux) > 9da380 dev_queue_xmit (/lib/modules/5.8.10/build/vmlinux) > 1f39 vlan_dev_open ([8021q]) > 9d9851 dev_hard_start_xmit > (/lib/modules/5.8.10/build/vmlinux) > 9d9d05 __dev_queue_xmit (/lib/modules/5.8.10/build/vmlinux) > 9da380 dev_queue_xmit (/lib/modules/5.8.10/build/vmlinux) > 9d9d05 __dev_queue_xmit (/lib/modules/5.8.10/build/vmlinux) > 9da380 dev_queue_xmit (/lib/modules/5.8.10/build/vmlinux) > 9e32c3 neigh_connected_output > (/lib/modules/5.8.10/build/vmlinux) > a68cdb ip_finish_output2 (/lib/modules/5.8.10/build/vmlinux) > a69b77 __ip_finish_output (/lib/modules/5.8.10/build/vmlinux) > a69cad ip_finish_output (/lib/modules/5.8.10/build/vmlinux) > a6b65a ip_output (/lib/modules/5.8.10/build/vmlinux) > f2e4 ip_vs_tunnel_xmit ([ip_vs]) > 536a ip_vs_nat_icmp ([ip_vs]) > 5913 ip_vs_out_icmp ([ip_vs]) > 59c6 ip_vs_out_icmp ([ip_vs]) > a58255 nf_hook_slow (/lib/modules/5.8.10/build/vmlinux) > a656f1 ip_local_deliver (/lib/modules/5.8.10/build/vmlinux) > a64e07 ip_rcv_finish (/lib/modules/5.8.10/build/vmlinux) > b717bb ip_sabotage_in (/lib/modules/5.8.10/build/vmlinux) > a58255 nf_hook_slow (/lib/modules/5.8.10/build/vmlinux) > a657a0 ip_rcv (/lib/modules/5.8.10/build/vmlinux) > 9db608 __netif_receive_skb_one_core > (/lib/modules/5.8.10/build/vmlinux) > 9db658 __netif_receive_skb > (/lib/modules/5.8.10/build/vmlinux) > 9db6e9 netif_receive_skb (/lib/modules/5.8.10/build/vmlinux) > b5c5b6 br_netif_receive_skb > (/lib/modules/5.8.10/build/vmlinux) > b5c7d3 br_pass_frame_up (/lib/modules/5.8.10/build/vmlinux) > b5c94d br_handle_frame_finish > (/lib/modules/5.8.10/build/vmlinux) > b7253b br_nf_hook_thresh (/lib/modules/5.8.10/build/vmlinux) > b72fb0 br_nf_pre_routing_finish > (/lib/modules/5.8.10/build/vmlinux) > b733bc br_nf_pre_routing (/lib/modules/5.8.10/build/vmlinux) > b5cdf1 br_handle_frame (/lib/modules/5.8.10/build/vmlinux) > 9da8c8 __netif_receive_skb_core > (/lib/modules/5.8.10/build/vmlinux) > 9db5bf __netif_receive_skb_one_core > (/lib/modules/5.8.10/build/vmlinux) > 9db658 __netif_receive_skb > (/lib/modules/5.8.10/build/vmlinux) > 9db8a9 process_backlog (/lib/modules/5.8.10/build/vmlinux) > 9dd337 net_rx_action (/lib/modules/5.8.10/build/vmlinux) > e000e1 __do_softirq (/lib/modules/5.8.10/build/vmlinux) > c010c2 asm_call_on_stack (/lib/modules/5.8.10/build/vmlinux) > 235bef do_softirq_own_stack > (/lib/modules/5.8.10/build/vmlinux) > 297f86 do_softirq.part.0 (/lib/modules/5.8.10/build/vmlinux) > 297fe0 __local_bh_enable_ip > (/lib/modules/5.8.10/build/vmlinux) > a68cef ip_finish_output2 (/lib/modules/5.8.10/build/vmlinux) > a69b77 __ip_finish_output (/lib/modules/5.8.10/build/vmlinux) > a69cad ip_finish_output (/lib/modules/5.8.10/build/vmlinux) > a6b65a ip_output (/lib/modules/5.8.10/build/vmlinux) > a6ad4d ip_local_out (/lib/modules/5.8.10/build/vmlinux) > a6b097 __ip_queue_xmit (/lib/modules/5.8.10/build/vmlinux) > a8f740 ip_queue_xmit (/lib/modules/5.8.10/build/vmlinux) > a89746 __tcp_transmit_skb (/lib/modules/5.8.10/build/vmlinux) > a8a5de tcp_connect (/lib/modules/5.8.10/build/vmlinux) > a90f73 tcp_v4_connect (/lib/modules/5.8.10/build/vmlinux) > aad59a __inet_stream_connect > (/lib/modules/5.8.10/build/vmlinux) > aad88b inet_stream_connect > (/lib/modules/5.8.10/build/vmlinux) > 9b3e3f __sys_connect_file (/lib/modules/5.8.10/build/vmlinux) > 9b3ef1 __sys_connect (/lib/modules/5.8.10/build/vmlinux) > 9b3f3a __x64_sys_connect (/lib/modules/5.8.10/build/vmlinux) > b896b2 do_syscall_64 (/lib/modules/5.8.10/build/vmlinux) > c0008c entry_SYSCALL_64 (/lib/modules/5.8.10/build/vmlinux) > 53878 [unknown] (/lib/ld-musl-x86_64.so.1) > 0 [unknown] ([unknown]) > 0 [unknown] ([unknown]) > Thanks for the detailed report ! I suspect ipvs or bridge code needs something similar to the fixes done in https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=de20900fbe1c4fd36de25a7a5a43223254ecf0d0 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=41d1c8839e5f8cb781cc635f12791decee8271b7 The reason for that is that skb->tstamp can get a timestamp in input path, with a base which is not CLOCK_MONOTONIC, unfortunately. Whenever a packet is forwarded, its tstamp must be cleared. Can you try : diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c index b00866d777fe0e9ed8018087ebc664c56f29b5c9..11e8ccdae358a89067046efa62ed40308b9e06f9 100644 --- a/net/netfilter/ipvs/ip_vs_xmit.c +++ b/net/netfilter/ipvs/ip_vs_xmit.c @@ -952,6 +952,8 @@ ip_vs_prepare_tunneled_skb(struct sk_buff *skb, int skb_af, ip_vs_drop_early_demux_sk(skb); + skb->tstamp = 0; + if (skb_headroom(skb) < max_headroom || skb_cloned(skb)) { new_skb = skb_realloc_headroom(skb, max_headroom); if (!new_skb)
