Answering myself ...
I've seen the frame processing sometimes freezes for one second when
stressing the isotp_rcv() from multiple sources. This finally freezes
the entire softirq which is either not good and not needed as we only
need to fix this race for stress tests - and not for real world usage
that does not create this case.
Therefore I created a V2 patch which uses the spin_trylock() to simply
drop the incomming frame in the race condition.
https://lore.kernel.org/linux-can/20220128074327.52229-1-socketcan@xxxxxxxxxxxx/T/
Please take a look, if it also fixes the issue in your test setup.
Thanks & best regards,
Oliver
On 27.01.22 20:44, Oliver Hartkopp wrote:
Hello Ziyang Xuan,
On 21.01.22 02:50, Ziyang Xuan (William) wrote:
On 20.01.22 12:28, Ziyang Xuan (William) wrote:
On 20.01.22 07:24, Ziyang Xuan (William) wrote:
I have reproduced the syz problem with Marc's commit, the commit
can not fix the panic problem.
So I tried to find the root cause for panic and gave my solution.
Marc's commit just fix the condition that packet size bigger than
INT_MAX which trigger
tpcon::{idx,len} integer overflow, but the packet size is 4096 in
the syz problem.
so->rx.len is 0 after the following logic in isotp_rcv_ff():
/* get the FF_DL */
so->rx.len = (cf->data[ae] & 0x0F) << 8;
so->rx.len += cf->data[ae + 1];
so->rx.len is 4096 after the following logic in isotp_rcv_ff():
/* FF_DL = 0 => get real length from next 4 bytes */
so->rx.len = cf->data[ae + 2] << 24;
so->rx.len += cf->data[ae + 3] << 16;
so->rx.len += cf->data[ae + 4] << 8;
so->rx.len += cf->data[ae + 5];
In these cases the values 0 could be the minimum value in
so->rx.len - but e.g. the value 0 can not show up in isotp_rcv_cf()
as this function requires so->rx.state to be ISOTP_WAIT_DATA.
Consider the scenario that isotp_rcv_cf() and isotp_rcv_cf() are
concurrent for the same isotp_sock as following sequence:
o_O
Sorry but the receive path is not designed to handle concurrent
receptions that would run isotp_rcv_cf() and isotp_rcv_ff()
simultaneously.
isotp_rcv_cf()
if (so->rx.state != ISOTP_WAIT_DATA) [false]
isotp_rcv_ff()
so->rx.state = ISOTP_IDLE
/* get the FF_DL */ [so->rx.len == 0]
alloc_skb() [so->rx.len == 0]
/* FF_DL = 0 => get real length from next 4
bytes */ [so->rx.len == 4096]
skb_put(nskb, so->rx.len) [so->rx.len == 4096]
skb_over_panic()
Even though this case is not possible with a real CAN bus due to the
CAN frame transmission times we could introduce some locking (or
dropping of concurrent CAN frames) in isotp_rcv() - but this code
runs in net softirq context ...
As discussed off-list I added a spin_lock() in isotp_rcv() as
https://www.kernel.org/doc/htmldocs/kernel-locking/lock-softirqs.html
suggests.
Please give this patch[1] a try in your test setup.
Many thanks,
Oliver
[1]:
https://lore.kernel.org/linux-can/20220127192429.336335-1-socketcan@xxxxxxxxxxxx/T/
