On 2/13/19 10:33 AM, Sebastian Andrzej Siewior wrote:
> On 2019-02-12 12:47:22 [+0000], Bernd Edlinger wrote:
>> The harden_branch_predictor function uses smp_processor_id,
>> which complains about being used in a preemptible context:
>>
>> BUG: using smp_processor_id() in preemptible [00000000] code: trie_map_rand.e/4938
>> caller is debug_smp_processor_id+0x18/0x24
>> CPU: 0 PID: 4938 Comm: trie_map_rand.e Not tainted 4.14.93-rt53-g74e4d87 #1
>> Hardware name: Altera SOCFPGA
>> [<8011162c>] (unwind_backtrace) from [<8010bfd4>] (show_stack+0x20/0x24)
>> [<8010bfd4>] (show_stack) from [<806732e8>] (dump_stack+0x78/0x94)
>> [<806732e8>] (dump_stack) from [<8041c254>] (check_preemption_disabled+0xec/0x128)
>> [<8041c254>] (check_preemption_disabled) from [<8041c2a8>] (debug_smp_processor_id+0x18/0x24)
>> [<8041c2a8>] (debug_smp_processor_id) from [<8011518c>] (__do_user_fault+0x34/0x100)
>> [<8011518c>] (__do_user_fault) from [<801155d0>] (do_page_fault+0x2f4/0x328)
>> [<801155d0>] (do_page_fault) from [<80101268>] (do_DataAbort+0x48/0xc8)
>> [<80101268>] (do_DataAbort) from [<8010d300>] (__dabt_usr+0x40/0x60)
>> Exception stack(0xae2fdfb0 to 0xae2fdff8)
>> dfa0: c31b0fc6 76c8f7d8 00000000 00000001
>> dfc0: 00000001 00c79780 0004f7f4 00c79780 7ea2e550 7ea2e408 7ea2e58c 3fc99999
>> dfe0: 00000100 7ea2e3d0 00033d24 00018edc 600c0010 ffffffff
>>
>> Work around the issue by disabling preemption while clearing
>> the branch predictor table.
>
> Can you verify if this only happens on RT or also on !RT? From what it
> looks like to me, it should also happen on non-RT.
>
The above stack trace and the following two slightly different ones were observed
on the RT linux target system (running gcc test suite).
BUG: using smp_processor_id() in preemptible [00000000] code: basic.exe/14229
caller is debug_smp_processor_id+0x18/0x24
CPU: 1 PID: 14229 Comm: basic.exe Not tainted 4.14.93-rt53-g74e4d87 #1
Hardware name: Altera SOCFPGA
[<8011162c>] (unwind_backtrace) from [<8010bfd4>] (show_stack+0x20/0x24)
[<8010bfd4>] (show_stack) from [<806732e8>] (dump_stack+0x78/0x94)
[<806732e8>] (dump_stack) from [<8041c254>] (check_preemption_disabled+0xec/0x128)
[<8041c254>] (check_preemption_disabled) from [<8041c2a8>] (debug_smp_processor_id+0x18/0x24)
[<8041c2a8>] (debug_smp_processor_id) from [<8011518c>] (__do_user_fault+0x34/0x100)
[<8011518c>] (__do_user_fault) from [<80115660>] (do_bad_area+0x5c/0x8c)
[<80115660>] (do_bad_area) from [<80115780>] (do_translation_fault+0xc0/0xcc)
[<80115780>] (do_translation_fault) from [<80101268>] (do_DataAbort+0x48/0xc8)
[<80101268>] (do_DataAbort) from [<8010d300>] (__dabt_usr+0x40/0x60)
Exception stack(0xb260ffb0 to 0xb260fff8)
ffa0: 7ec3351c 00925cb0 00000011 00000007
ffc0: 00031b4c 00925c98 00031b70 000310d4 00031b54 e12fff1e 7ec3351c 00031b6c
ffe0: 00000001 7ec334e0 00929868 00018c0c 200b0010 ffffffff
BUG: using smp_processor_id() in preemptible [00000000] code: 20040709-1.exe/23203
caller is debug_smp_processor_id+0x18/0x24
CPU: 1 PID: 23203 Comm: 20040709-1.exe Not tainted 4.14.93-rt53-g74e4d87 #1
Hardware name: Altera SOCFPGA
[<8011162c>] (unwind_backtrace) from [<8010bfd4>] (show_stack+0x20/0x24)
[<8010bfd4>] (show_stack) from [<806732e8>] (dump_stack+0x78/0x94)
[<806732e8>] (dump_stack) from [<8041c254>] (check_preemption_disabled+0xec/0x128)
[<8041c254>] (check_preemption_disabled) from [<8041c2a8>] (debug_smp_processor_id+0x18/0x24)
[<8041c2a8>] (debug_smp_processor_id) from [<8011518c>] (__do_user_fault+0x34/0x100)
[<8011518c>] (__do_user_fault) from [<80115660>] (do_bad_area+0x5c/0x8c)
[<80115660>] (do_bad_area) from [<801156b8>] (do_sect_fault+0x28/0x30)
[<801156b8>] (do_sect_fault) from [<80101268>] (do_DataAbort+0x48/0xc8)
[<80101268>] (do_DataAbort) from [<8010d300>] (__dabt_usr+0x40/0x60)
Exception stack(0xbf41dfb0 to 0xbf41dff8)
dfa0: 00000000 40150000 4cbce604 806871b0
dfc0: 00023038 000004cb 96830421 00000683 00023168 41c64e6d 00023170 99999cdc
dfe0: 00000b4e 7ec62aa0 0000b4ec 0001264c 60060010 ffffffff
I have previously executed the test suite on v4.18 (w/o rt-branch), and was unable to
reproduce this specific failure mode, but I think that is just because the non-rt system gets
OOM errors like the one below:
kswapd0: page allocation failure: order:0, mode:0x480020(GFP_ATOMIC), nodemask=(null)
kswapd0 cpuset=/ mems_allowed=0
CPU: 0 PID: 334 Comm: kswapd0 Not tainted 4.18.0-00005-g7523d7f #1
Hardware name: Altera SOCFPGA
[<80112e68>] (unwind_backtrace) from [<8010da28>] (show_stack+0x20/0x24)
[<8010da28>] (show_stack) from [<806bcaec>] (dump_stack+0x78/0x94)
[<806bcaec>] (dump_stack) from [<80209340>] (warn_alloc+0xac/0x168)
[<80209340>] (warn_alloc) from [<80209c38>] (__alloc_pages_nodemask+0x798/0xf7c)
[<80209c38>] (__alloc_pages_nodemask) from [<8020a54c>] (page_frag_alloc+0x68/0x180)
[<8020a54c>] (page_frag_alloc) from [<805381d8>] (__netdev_alloc_skb+0xa0/0x11c)
[<805381d8>] (__netdev_alloc_skb) from [<804dd374>] (stmmac_poll+0x398/0x990)
[<804dd374>] (stmmac_poll) from [<8054ed90>] (net_rx_action+0x130/0x3dc)
[<8054ed90>] (net_rx_action) from [<80102c30>] (__do_softirq+0x268/0x3e4)
[<80102c30>] (__do_softirq) from [<8012807c>] (irq_exit+0x88/0xa8)
[<8012807c>] (irq_exit) from [<8016ecbc>] (__handle_domain_irq+0xa4/0xf0)
[<8016ecbc>] (__handle_domain_irq) from [<80102990>] (gic_handle_irq+0x68/0xa0)
[<80102990>] (gic_handle_irq) from [<801020cc>] (__irq_svc+0x6c/0xa8)
Exception stack(0xbf2cdd80 to 0xbf2cddc8)
dd80: bf2cde04 00000046 00a20840 00000000 bf00ca00 bf2cdf00 0000016b 00000000
dda0: bf011800 00000004 0003cf65 bf2cdde4 bf2cdde8 bf2cddd4 8021a2c0 80408be8
ddc0: 000f0113 ffffffff
[<801020cc>] (__irq_svc) from [<80408be8>] (blk_start_plug+0x10/0x58)
[<80408be8>] (blk_start_plug) from [<8021a2c0>] (shrink_node_memcg+0x1ac/0x68c)
[<8021a2c0>] (shrink_node_memcg) from [<8021a874>] (shrink_node+0xd4/0x4c4)
[<8021a874>] (shrink_node) from [<8021bb00>] (kswapd+0x38c/0x7b8)
[<8021bb00>] (kswapd) from [<80141cc0>] (kthread+0x150/0x168)
[<80141cc0>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c)
Exception stack(0xbf2cdfb0 to 0xbf2cdff8)
dfa0: 00000000 00000000 00000000 00000000
dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
Mem-Info:
active_anon:187559 inactive_anon:54309 isolated_anon:0
active_file:4567 inactive_file:3317 isolated_file:0
unevictable:0 dirty:0 writeback:1610 unstable:0
slab_reclaimable:1212 slab_unreclaimable:2662
mapped:6112 shmem:236 pagetables:671 bounce:0
free:1045 free_pcp:167 free_cma:0
Node 0 active_anon:750236kB inactive_anon:217236kB active_file:18268kB inactive_file:13268kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:24448kB dirty:0kB writeback:6440kB shmem:944kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Normal free:4180kB min:4056kB low:5084kB high:6112kB active_anon:750620kB inactive_anon:216888kB active_file:18080kB inactive_file:13576kB unevictable:0kB writepending:6064kB present:1048576kB managed:1029596kB mlocked:0kB kernel_stack:696kB pagetables:2684kB bounce:0kB free_pcp:668kB local_pcp:668kB free_cma:0kB
lowmem_reserve[]: 0 0
Normal: 34*4kB (H) 39*8kB (H) 29*16kB (H) 28*32kB (H) 18*64kB (H) 2*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3472kB
9953 total pagecache pages
1849 pages in swap cache
Swap cache stats: add 2675265, delete 2673347, find 2126072/2558992
Free swap = 1731324kB
Total swap = 1774076kB
262144 pages RAM
0 pages HighMem/MovableOnly
4745 pages reserved
socfpga-dwmac ff700000.ethernet: fail to alloc skb entry 223
ksoftirqd/0: page allocation failure: order:0, mode:0x480020(GFP_ATOMIC), nodemask=(null)
ksoftirqd/0 cpuset=/ mems_allowed=0
This kind of errors I have never seen with the RT branch, so I assume those just prevent the
test to reach the other error condition.
Since I was unsuccessful to reproduce with v4.18 I have now started to test
4.14.93-rt53 without RT:
@@ -503,12 +509,11 @@
CONFIG_PREEMPT=y
CONFIG_PREEMPT_RT_BASE=y
CONFIG_HAVE_PREEMPT_LAZY=y
-CONFIG_PREEMPT_LAZY=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT__LL is not set
-# CONFIG_PREEMPT_RTB is not set
-CONFIG_PREEMPT_RT_FULL=y
+CONFIG_PREEMPT_RTB=y
+# CONFIG_PREEMPT_RT_FULL is not set
CONFIG_PREEMPT_COUNT=y
CONFIG_HZ_FIXED=0
CONFIG_HZ_100=y
But now I think I have an explanation, why the error got at least more severe
in the rt-branch, as follows:
$ git diff master...v4.14-rt arch/arm/mm/fault.c
diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
index 42f5853..b261967 100644
--- a/arch/arm/mm/fault.c
+++ b/arch/arm/mm/fault.c
@@ -164,6 +164,9 @@ __do_user_fault(struct task_struct *tsk, unsigned long addr,
{
struct siginfo si;
+ if (addr > TASK_SIZE)
+ harden_branch_predictor();
+
#ifdef CONFIG_DEBUG_USER
if (((user_debug & UDBG_SEGV) && (sig == SIGSEGV)) ||
((user_debug & UDBG_BUS) && (sig == SIGBUS))) {
@@ -434,6 +437,9 @@ do_translation_fault(unsigned long addr, unsigned int fsr,
if (addr < TASK_SIZE)
return do_page_fault(addr, fsr, regs);
+ if (interrupts_enabled(regs))
+ local_irq_enable();
+
if (user_mode(regs))
goto bad_area;
@@ -501,6 +507,9 @@ do_translation_fault(unsigned long addr, unsigned int fsr,
static int
do_sect_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
{
+ if (interrupts_enabled(regs))
+ local_irq_enable();
+
do_bad_area(addr, fsr, regs);
return 0;
}
Now, this explains why the two other stack dumps are only possible in the v4.14-rt
branch, since the local_irq_enable is a necessary condition for the failure.
git blame reveals this very interesting commit:
commit cf8928d1021902a72ff2cc5853067a724e52fc44
Author: Yadi.hu <yadi.hu@xxxxxxxxxxxxx>
Date: Wed Dec 10 10:32:09 2014 +0800
ARM: enable irq in translation/section permission fault handlers
Probably happens on all ARM, with
CONFIG_PREEMPT_RT_FULL
CONFIG_DEBUG_ATOMIC_SLEEP
This simple program....
int main() {
*((char*)0xc0001000) = 0;
};
[ 512.742724] BUG: sleeping function called from invalid context at kernel/rtmutex.c:658
[ 512.743000] in_atomic(): 0, irqs_disabled(): 128, pid: 994, name: a
[ 512.743217] INFO: lockdep is turned off.
[ 512.743360] irq event stamp: 0
[ 512.743482] hardirqs last enabled at (0): [< (null)>] (null)
[ 512.743714] hardirqs last disabled at (0): [<c0426370>] copy_process+0x3b0/0x11c0
[ 512.744013] softirqs last enabled at (0): [<c0426370>] copy_process+0x3b0/0x11c0
[ 512.744303] softirqs last disabled at (0): [< (null)>] (null)
[ 512.744631] [<c041872c>] (unwind_backtrace+0x0/0x104)
[ 512.745001] [<c09af0c4>] (dump_stack+0x20/0x24)
[ 512.745355] [<c0462490>] (__might_sleep+0x1dc/0x1e0)
[ 512.745717] [<c09b6770>] (rt_spin_lock+0x34/0x6c)
[ 512.746073] [<c0441bf0>] (do_force_sig_info+0x34/0xf0)
[ 512.746457] [<c0442668>] (force_sig_info+0x18/0x1c)
[ 512.746829] [<c041d880>] (__do_user_fault+0x9c/0xd8)
[ 512.747185] [<c041d938>] (do_bad_area+0x7c/0x94)
[ 512.747536] [<c041d990>] (do_sect_fault+0x40/0x48)
[ 512.747898] [<c040841c>] (do_DataAbort+0x40/0xa0)
[ 512.748181] Exception stack(0xecaa1fb0 to 0xecaa1ff8)
Oxc0000000 belongs to kernel address space, user task can not be
allowed to access it. For above condition, correct result is that
test case should receive a “segment fault” and exits but not stacks.
the root cause is commit 02fe2845d6a8 ("avoid enabling interrupts in
prefetch/data abort handlers"),it deletes irq enable block in Data
abort assemble code and move them into page/breakpiont/alignment fault
handlers instead. But author does not enable irq in translation/section
permission fault handlers. ARM disables irq when it enters exception/
interrupt mode, if kernel doesn't enable irq, it would be still disabled
during translation/section permission fault.
We see the above splat because do_force_sig_info is still called with
IRQs off, and that code eventually does a:
spin_lock_irqsave(&t->sighand->siglock, flags);
As this is architecture independent code, and we've not seen any other
need for other arch to have the siglock converted to raw lock, we can
conclude that we should enable irq for ARM translation/section
permission exception.
Signed-off-by: Yadi.hu <yadi.hu@xxxxxxxxxxxxx>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@xxxxxxxxxxxxx>
diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
index 42f5853..93d2ecc 100644
--- a/arch/arm/mm/fault.c
+++ b/arch/arm/mm/fault.c
@@ -434,6 +434,9 @@ do_translation_fault(unsigned long addr, unsigned int fsr,
if (addr < TASK_SIZE)
return do_page_fault(addr, fsr, regs);
+ if (interrupts_enabled(regs))
+ local_irq_enable();
+
if (user_mode(regs))
goto bad_area;
@@ -501,6 +504,9 @@ do_translation_fault(unsigned long addr, unsigned int fsr,
static int
do_sect_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
{
+ if (interrupts_enabled(regs))
+ local_irq_enable();
+
do_bad_area(addr, fsr, regs);
return 0;
}
And indeed the mentioned test program is able to reproduce the following
fault in v4.14-rt with !RT:
$ ./a.out
BUG: using smp_processor_id() in preemptible [00000000] code: a.out/9410
caller is debug_smp_processor_id+0x18/0x24
CPU: 0 PID: 9410 Comm: a.out Not tainted 4.14.93-rt53-g74e4d87 #1
Hardware name: Altera SOCFPGA
[<8011193c>] (unwind_backtrace) from [<8010c330>] (show_stack+0x20/0x24)
[<8010c330>] (show_stack) from [<806884b8>] (dump_stack+0x78/0x94)
[<806884b8>] (dump_stack) from [<8042d6e8>] (check_preemption_disabled+0xec/0x128)
[<8042d6e8>] (check_preemption_disabled) from [<8042d73c>] (debug_smp_processor_id+0x18/0x24)
[<8042d73c>] (debug_smp_processor_id) from [<8011548c>] (__do_user_fault+0x34/0x100)
[<8011548c>] (__do_user_fault) from [<80115960>] (do_bad_area+0x5c/0x8c)
[<80115960>] (do_bad_area) from [<80115a80>] (do_translation_fault+0xc0/0xcc)
[<80115a80>] (do_translation_fault) from [<80101268>] (do_DataAbort+0x48/0xc8)
[<80101268>] (do_DataAbort) from [<8010d640>] (__dabt_usr+0x40/0x60)
Exception stack(0xbf56dfb0 to 0xbf56dff8)
dfa0: 00000000 7eacbd94 7eacbd9c c0001000
dfc0: 000103e4 00000000 000102f4 00000000 00000000 00000000 76f3c000 00000000
dfe0: 00000000 7eacbc40 76ddabbc 000102ec 60060010 ffffffff
Segmentation fault
but nothing happens with this patch applied.
Thanks
Bernd.
