On Thu, Nov 3, 2022 at 5:53 AM Caowangbao <caowangbao@xxxxxxxxxx> wrote:
>
> I have reduced the recurrence conditions and can reproduce the problem by running the following statement:
>
> 18:00:56 executing program 0:
> r0 = socket$inet6_sctp(0xa, 0x1, 0x84)
> setsockopt$inet_sctp6_SCTP_SOCKOPT_BINDX_ADD(r0, 0x84, 0x64, &(0x7f00000001c0)=[@in={0x2, 0x4e20, @empty}], 0x10) (async)
> getsockopt$inet_sctp6_SCTP_SOCKOPT_CONNECTX3(r0, 0x84, 0x6f, &(0x7f0000000580)={<r1=>0x0, 0x10, &(0x7f0000000540)=[@in={0x2, 0x4e20, @local}]}, &(0x7f0000000600)=0x10) (async)
> r2 = dup2(r0, r0)
> setsockopt$inet_sctp6_SCTP_DEFAULT_PRINFO(r2, 0x84, 0x72, &(0x7f0000000000)={0x0, 0x6, 0x30}, 0xc) (async)
Thanks for the statements.
The crash seems related to SCTP_PR_SCTP_PRIO.
When there is not enough buffer for the out msg, it will prune the
out_chunk_list according to the priority set by SCTP_DEFAULT_PRINFO in
sctp_prsctp_prune_unsent(). However, it doesn't clear
asoc->stream.out_curr if all frag chunks of current msg are pruned.
Can you apply this patch to your kernel and give it try?
diff --git a/net/sctp/outqueue.c b/net/sctp/outqueue.c
index e213aaf45d67..41b8065cfe65 100644
--- a/net/sctp/outqueue.c
+++ b/net/sctp/outqueue.c
@@ -384,6 +384,7 @@ static int sctp_prsctp_prune_unsent(struct
sctp_association *asoc,
{
struct sctp_outq *q = &asoc->outqueue;
struct sctp_chunk *chk, *temp;
+ struct sctp_stream_out *sout;
q->sched->unsched_all(&asoc->stream);
@@ -398,12 +399,12 @@ static int sctp_prsctp_prune_unsent(struct
sctp_association *asoc,
sctp_sched_dequeue_common(q, chk);
asoc->sent_cnt_removable--;
asoc->abandoned_unsent[SCTP_PR_INDEX(PRIO)]++;
- if (chk->sinfo.sinfo_stream < asoc->stream.outcnt) {
- struct sctp_stream_out *streamout =
- SCTP_SO(&asoc->stream, chk->sinfo.sinfo_stream);
- streamout->ext->abandoned_unsent[SCTP_PR_INDEX(PRIO)]++;
- }
+ sout = SCTP_SO(&asoc->stream, chk->sinfo.sinfo_stream);
+ sout->ext->abandoned_unsent[SCTP_PR_INDEX(PRIO)]++;
+ if (asoc->stream.out_curr == sout &&
+ list_is_last(&chk->frag_list, &chk->msg->chunks))
+ asoc->stream.out_curr = NULL; /* clear
out_curr if all frag chunks are pruned */
msg_len -= chk->skb->truesize + sizeof(struct sctp_chunk);
sctp_chunk_free(chk);
Thanks.
> sendmmsg$sock(r2, ...) (async)
> r3 = socket$inet6_sctp(0xa, 0x1, 0x84)
> r4 = dup2(r2, r3)
> setsockopt$inet_sctp6_SCTP_DEFAULT_PRINFO(r4, 0x84, 0x72, &(0x7f0000000040), 0xc)
> sendmsg$alg(r4, &(0x7f0000002700)={0x0, 0x0, &(0x7f0000002500)=[{&(0x7f0000004100)='@', 0x1}], 0x1}, 0x0) (async)
> setsockopt$inet_sctp6_SCTP_DEFAULT_SNDINFO(r2, 0x84, 0x22, &(0x7f0000002480)={0x5, 0x202, 0x9, 0x10001, r1}, 0x10) (async)
> write$tun(r4, &(0x7f0000002640)=ANY=[@ANYBLOB="000000000000000000000000000006000000aaaaaaaaaabb467219e67b1d64441845437b38c12ddeb986e59e82bd4247f1ed8a05309a31b9494bda521ffd4b68bf072b030d7ef04cc219c73572fac79f47369d49ae19df01641921e3af34cb84766ede45e4fa9a14460fae51557f643d108ba54f7cb8440ce5aa0e60d7c2c4da"], 0x1e) (async)
> write(r0, &(0x7f0000000080)="e4", 0x1)
>
>
>
> -----邮件原件-----
> 发件人: Caowangbao
> 发送时间: 2022年11月3日 10:13
> 收件人: 'Xin Long' <lucien.xin@xxxxxxxxx>; Chenzhen(EulerOS) <chenzhen126@xxxxxxxxxx>
> 抄送: 'vyasevich@xxxxxxxxx' <vyasevich@xxxxxxxxx>; 'nhorman@xxxxxxxxxxxxx' <nhorman@xxxxxxxxxxxxx>; 'marcelo.leitner@xxxxxxxxx' <marcelo.leitner@xxxxxxxxx>; 'linux-sctp@xxxxxxxxxxxxxxx' <linux-sctp@xxxxxxxxxxxxxxx>; 'davem@xxxxxxxxxxxxx' <davem@xxxxxxxxxxxxx>; 'edumazet@xxxxxxxxxx' <edumazet@xxxxxxxxxx>; 'kuba@xxxxxxxxxx' <kuba@xxxxxxxxxx>; 'pabeni@xxxxxxxxxx' <pabeni@xxxxxxxxxx>; 'netdev@xxxxxxxxxxxxxxx' <netdev@xxxxxxxxxxxxxxx>; Yanan (Euler) <yanan@xxxxxxxxxx>
> 主题: 答复: BUG: kernel NULL pointer dereference in sctp_sched_dequeue_common
>
> It can be reproduce by the command " ./syz-execprog -procs=16 -repeat=0 sctp_sched_dequeue_common" with the attachments.
>
> void sctp_sched_dequeue_common(struct sctp_outq *q, struct sctp_chunk *ch) {
> list_del_init(&ch->list);
> list_del_init(&ch->stream_list);
> q->out_qlen -= ch->skb->len; // ch->skb is null in the VMCore
> }
>
> The kernel log records：
> [23411.786575] list_del corruption, ffffa035ddf01c18->next is NULL
> [23411.787780] WARNING: CPU: 1 PID: 250682 at lib/list_debug.c:49 __list_del_entry_valid+0x59/0xe0
> ******
> [23411.830256] Call Trace:
> [23411.830863] sctp_sched_dequeue_common+0x17/0x70 [sctp]
> [23411.831940] sctp_sched_fcfs_dequeue+0x37/0x50 [sctp]
> [23411.832967] sctp_outq_flush_data+0x85/0x360 [sctp] It means "ch->list" has no element.
>
> And in VMCore , there are many calls like:
> #2 [ffffaf7d84f6bbb8] __lock_sock at ffffffff8ac74ef9
> #3 [ffffaf7d84f6bc08] lock_sock_nested at ffffffff8ac74f92
> #4 [ffffaf7d84f6bc20] sctp_wait_for_sndbuf at ffffffffc0c8d9d2 [sctp]
> #5 [ffffaf7d84f6bc98] sctp_sendmsg_to_asoc at ffffffffc0c8dd1e [sctp]
> #6 [ffffaf7d84f6bd08] sctp_sendmsg at ffffffffc0c95f55 [sctp]
> #7 [ffffaf7d84f6bdb8] sock_sendmsg at ffffffff8ac6fd0b
> #8 [ffffaf7d84f6bdd0] sock_write_iter at ffffffff8ac6fdb7
> #9 [ffffaf7d84f6be48] new_sync_write at ffffffff8a784021
> #10 [ffffaf7d84f6bed0] vfs_write at ffffffff8a784d07
> #11 [ffffaf7d84f6bf08] ksys_write at ffffffff8a78719b
> #12 [ffffaf7d84f6bf40] do_syscall_64 at ffffffff8ae9a8b3 It may have something to do with these concurrent invocations.
>
> -----邮件原件-----
> 发件人: Xin Long [mailto:lucien.xin@xxxxxxxxx]
> 发送时间: 2022年11月3日 9:20
> 收件人: Chenzhen(EulerOS) <chenzhen126@xxxxxxxxxx>
> 抄送: vyasevich@xxxxxxxxx; nhorman@xxxxxxxxxxxxx; marcelo.leitner@xxxxxxxxx; linux-sctp@xxxxxxxxxxxxxxx; davem@xxxxxxxxxxxxx; edumazet@xxxxxxxxxx; kuba@xxxxxxxxxx; pabeni@xxxxxxxxxx; netdev@xxxxxxxxxxxxxxx; Caowangbao <caowangbao@xxxxxxxxxx>; Yanan (Euler) <yanan@xxxxxxxxxx>
> 主题: Re: BUG: kernel NULL pointer dereference in sctp_sched_dequeue_common
>
> On Wed, Nov 2, 2022 at 10:29 AM Zhen Chen <chenzhen126@xxxxxxxxxx> wrote:
> >
> > Hi,all
> >
> > We found the following crash when running fuzz tests on stable-5.10.
> >
> > ------------[ cut here ]------------
> > list_del corruption, ffffa035ddf01c18->next is NULL
> > WARNING: CPU: 1 PID: 250682 at lib/list_debug.c:49 __list_del_entry_valid+0x59/0xe0
> > CPU: 1 PID: 250682 Comm: syz-executor.7 Kdump: loaded Tainted: G O
> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> > rel-1.10.2-0-g5f4c7b1-20181220_000000-szxrtosci10000 04/01/2014
> > RIP: 0010:__list_del_entry_valid+0x59/0xe0
> > Code: c0 74 5a 4d 8b 00 49 39 f0 75 6a 48 8b 52 08 4c 39 c2 75 79 b8
> > 01 00 00 00 c3 cc cc cc cc 48 c7 c7 68 ae 78 8b e8 d2 3d 4e 00 <0f> 0b
> > 31 c0 c3 cc cc cc cc 48 c7 c7 90 ae 78 8b e8 bd 3d 4e 00 0f
> > RSP: 0018:ffffaf7d84a57930 EFLAGS: 00010286
> > RAX: 0000000000000000 RBX: ffffa035ddf01c18 RCX: 0000000000000000
> > RDX: ffffa035facb0820 RSI: ffffa035faca0410 RDI: ffffa035faca0410
> > RBP: ffffa035dddff6f8 R08: 0000000000000000 R09: ffffaf7d84a57770
> > R10: ffffaf7d84a57768 R11: ffffffff8bddc248 R12: ffffa035ddf01c18
> > R13: ffffaf7d84a57af8 R14: ffffaf7d84a57c28 R15: 0000000000000000
> > FS: 00007fb7353ae700(0000) GS:ffffa035fac80000(0000)
> > knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 00007f509a3d0ee8 CR3: 000000010f7c2001 CR4: 00000000001706e0 Call
> > Trace:
> > sctp_sched_dequeue_common+0x17/0x70 [sctp]
> > sctp_sched_fcfs_dequeue+0x37/0x50 [sctp]
> > sctp_outq_flush_data+0x85/0x360 [sctp]
> > sctp_outq_uncork+0x77/0xa0 [sctp]
> > sctp_cmd_interpreter.constprop.0+0x164/0x1450 [sctp]
> > sctp_side_effects+0x37/0xe0 [sctp]
> > sctp_do_sm+0xd0/0x230 [sctp]
> > sctp_primitive_SEND+0x2f/0x40 [sctp]
> > sctp_sendmsg_to_asoc+0x3fa/0x5c0 [sctp]
> > sctp_sendmsg+0x3d5/0x440 [sctp]
> > sock_sendmsg+0x5b/0x70
> > sock_write_iter+0x97/0x100
> > new_sync_write+0x1a1/0x1b0
> > vfs_write+0x1b7/0x250
> > ksys_write+0xab/0xe0
> > do_syscall_64+0x33/0x40
> > entry_SYSCALL_64_after_hwframe+0x61/0xc6
> > RIP: 0033:0x461e3d
> > Code: 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 89 f8 48
> > 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d
> > 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
> > RSP: 002b:00007fb7353adc08 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> > RAX: ffffffffffffffda RBX: 000000000058c1d0 RCX: 0000000000461e3d
> > RDX: 000000000000001e RSI: 0000000020002640 RDI: 0000000000000004
> > RBP: 000000000058c1d0 R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> > R13: 00007fb7353ae700 R14: 00007ffc4c20ce00 R15: 0000000000000fff ---[
> > end trace 332cf75246d5ba68 ]---
> > BUG: kernel NULL pointer dereference, address: 0000000000000070
> > #PF: supervisor read access in kernel mode
> > #PF: error_code(0x0000) - not-present page PGD 800000010c0d4067 P4D
> > 800000010c0d4067 PUD 10f275067 PMD 0
> > Oops: 0000 [#1] SMP PTI
> > CPU: 1 PID: 250682 Comm: syz-executor.7 Kdump: loaded Tainted: G W O
> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> > rel-1.10.2-0-g5f4c7b1-20181220_000000-szxrtosci10000 04/01/2014
> > RIP: 0010:sctp_sched_dequeue_common+0x5c/0x70 [sctp]
> > Code: 5b 08 4c 89 e7 e8 44 c5 cc c9 84 c0 74 0f 48 8b 53 18 48 8b 43
> > 20 48 89 42 08 48 89 10 48 8b 43 38 4c 89 63 18 4c 89 63 20 5b <8b> 40
> > 70 29 45 20 5d 41 5c c3 cc cc cc cc 66 0f 1f 44 00 00 0f 1f
> > RSP: 0018:ffffaf7d84a57940 EFLAGS: 00010202
> > RAX: 0000000000000000 RBX: ffffaf7d84a579a0 RCX: 0000000000000000
> > RDX: ffffa035ddf01c30 RSI: ffffa035ddf01c30 RDI: ffffa035ddf01c30
> > RBP: ffffa035dddff6f8 R08: ffffa035ddf01c30 R09: ffffaf7d84a57770
> > R10: ffffaf7d84a57768 R11: ffffffff8bddc248 R12: ffffa035ddf01c30
> > R13: ffffaf7d84a57af8 R14: ffffaf7d84a57c28 R15: 0000000000000000
> > FS: 00007fb7353ae700(0000) GS:ffffa035fac80000(0000)
> > knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 0000000000000070 CR3: 000000010f7c2001 CR4: 00000000001706e0 Call
> > Trace:
> > sctp_sched_fcfs_dequeue+0x37/0x50 [sctp]
> > sctp_outq_flush_data+0x85/0x360 [sctp]
> > sctp_outq_uncork+0x77/0xa0 [sctp]
> > sctp_cmd_interpreter.constprop.0+0x164/0x1450 [sctp]
> > sctp_side_effects+0x37/0xe0 [sctp]
> > sctp_do_sm+0xd0/0x230 [sctp]
> > sctp_primitive_SEND+0x2f/0x40 [sctp]
> > sctp_sendmsg_to_asoc+0x3fa/0x5c0 [sctp]
> > sctp_sendmsg+0x3d5/0x440 [sctp]
> > sock_sendmsg+0x5b/0x70
> > sock_write_iter+0x97/0x100
> > new_sync_write+0x1a1/0x1b0
> > vfs_write+0x1b7/0x250
> > ksys_write+0xab/0xe0
> > do_syscall_64+0x33/0x40
> > entry_SYSCALL_64_after_hwframe+0x61/0xc6
> > RIP: 0033:0x461e3d
> > Code: 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 89 f8 48
> > 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d
> > 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
> > RSP: 002b:00007fb7353adc08 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> > RAX: ffffffffffffffda RBX: 000000000058c1d0 RCX: 0000000000461e3d
> > RDX: 000000000000001e RSI: 0000000020002640 RDI: 0000000000000004
> > RBP: 000000000058c1d0 R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> > R13: 00007fb7353ae700 R14: 00007ffc4c20ce00 R15: 0000000000000fff
> >
> >
> > It is quite similar to the issue (See
> > https://lore.kernel.org/all/CAO4mrfcB0d+qbwtfndzqcrL+QEQgfOmJYQMAdzwxR
> > ePmP8TY1A@xxxxxxxxxxxxxx/ ) , which was addressed by 181d8d2066c0
> > (sctp: leave the err path free in sctp_stream_init to
> > sctp_stream_free), but unfortunately the patch do not work with this
> > bug :(
> >
> So this issue is reproducible in your env?
> Can you show what it does in your test or the reproducer if there is one?
>
> Thanks.
