On 10/3/24 10:22 PM, Yonghong Song wrote:
On 10/3/24 3:32 PM, Alexei Starovoitov wrote:
On Thu, Oct 3, 2024 at 1:44 PM Yonghong Song
<yonghong.song@xxxxxxxxx> wrote:
Looks like the idea needs more thought.
in_task_stack() won't recognize the private stack,
so it will look like stack overflow and double fault.
do you have CONFIG_VMAP_STACK ?
Yes, my above test runs fine withCONFIG_VMAP_STACK. Let me guard
private stack support with
CONFIG_VMAP_STACK for now. Not sure whether distributions enable
CONFIG_VMAP_STACK or not.
Good! but I'm surprised it makes a difference.
That only for the test case I tried. Now I tried the whole bpf selftests
with CONFIG_VMAP_STACK on. There are still some failures. Some of them
due to stack protector. I disabled stack protector and then those stack
protector error gone. But some other errors show up like below:
[ 27.186581] kernel tried to execute NX-protected page - exploit
attempt? (uid: 0)
[ 27.187480] BUG: unable to handle page fault for address:
ffff888109572800
[ 27.188299] #PF: supervisor instruction fetch in kernel mode
[ 27.189085] #PF: error_code(0x0011) - permissions violation
or
[ 27.736844] BUG: unable to handle page fault for address:
0000000080000000
[ 27.737759] #PF: supervisor instruction fetch in kernel mode
[ 27.738631] #PF: error_code(0x0010) - not-present page
[ 27.739455] PGD 0 P4D 0
[ 27.739818] Oops: Oops: 0010 [#1] PREEMPT SMP PTI
...
Some further investigations are needed.
I found one failure case (with stackprotector disabled):
[ 20.032611] traps: PANIC: double fault, error_code: 0x0
[ 20.032615] Oops: double fault: 0000 [#1] PREEMPT SMP PTI
[ 20.032619] CPU: 0 UID: 0 PID: 1959 Comm: test_progs Tainted: G OE 6.11.0-10576-g17baa0096769-dirty #1006
[ 20.032623] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[ 20.032624] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 20.032626] RIP: 0010:error_entry+0x17/0x140
[ 20.032633] Code: ff 0f 01 f8 e9 56 fe ff ff 90 90 90 90 90 90 90 90 90 90 56 48 8b 74 24 08 48 89 7c 24 08 52 51 50 41 50 41 51 41 52 49
[ 20.032635] RSP: 0018:ffffe8ffff400000 EFLAGS: 00010093
[ 20.032637] RAX: ffffe8ffff4000a8 RBX: ffffe8ffff4000a8 RCX: ffffffff82201737
[ 20.032639] RDX: 0000000000000000 RSI: ffffffff8220128d RDI: ffffe8ffff4000a8
[ 20.032640] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 20.032641] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 20.032642] R13: 0000000000000000 R14: 000000000002ed80 R15: 0000000000000000
[ 20.032643] FS: 00007f8a3a2006c0(0000) GS:ffff888237c00000(0000) knlGS:ffff888237c00000
[ 20.032645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 20.032646] CR2: ffffe8ffff3ffff8 CR3: 0000000103580002 CR4: 0000000000370ef0
[ 20.032649] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 20.032650] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 20.032651] Call Trace:
[ 20.032660] <#DF>
[ 20.032664] ? __die_body+0xaf/0xc0
[ 20.032667] ? die+0x2f/0x50
[ 20.032670] ? exc_double_fault+0xbf/0xd0
[ 20.032674] ? asm_exc_double_fault+0x23/0x30
[ 20.032678] ? restore_regs_and_return_to_kernel+0x1b/0x1b
[ 20.032681] ? asm_exc_page_fault+0xd/0x30
[ 20.032684] ? error_entry+0x17/0x140
[ 20.032687] </#DF>
The private stack for cpu 0:
priv_stack_ptr cpu 0 = [ffffe8ffff434000, ffffe8ffff438000] (total 16KB)
That is, the top stack is ffffe8ffff438000 and the bottom stack is ffffe8ffff434000.
During bpf execution, a softirq may happen, at that point,
stack pointer becomes:
RSP: 0018:ffffe8ffff400000 (see above)
and there is a read/write (mostly write) to address
CR2: ffffe8ffff3ffff8
And this may cause a fault.
After this fault, there are some further access and probably because
of invalid stack, double fault happens.
So the quesiton is why RSP is reset to ffffe8ffff400000?
I have not figured out which code changed this? Maybe somebody can help?
Please still root cause the crash without VMAP_STACK.
Sure. Let me investigate cases with VMAP_STACK first and
then will try to look at it without VMAP_STACK.
We need to do a lot more homework here before proceeding.
Look at arch/x86/kernel/dumpstack_64.c
At least we need new stack_type for priv stack.
stack_type_unknown doesn't inspire confidence.
Need to make sure stack trace is still reliable with priv stack.
Though it may look appealing from performance pov.
We may need to go back to r9 approach with push/pop around calls,
since that is surely keeping unwinder happy
while this approach will have to teach unwinder.
Good point.
