On 11/03/2014 05:34 PM, Joshua Kinard wrote:
On 11/03/2014 20:23, David Daney wrote:
On 11/03/2014 05:08 PM, Joshua Kinard wrote:
On 11/03/2014 13:52, David Daney wrote:
On 11/02/2014 02:53 AM, Joshua Kinard wrote:
So I have been testing the Onyx2 I have out the last few days with the IOC3
metadriver used on Octane, and I can get it to boot, but if
CONFIG_TRANSPARENT_HUGEPAGE is enabled in the kernel, bus errors can happen.
If I use CONFIG_PAGE_SIZE_4KB, I get bus errors rather frequently -- running
Gentoo's 'emerge' command can produce one. Switch to CONFIG_PAGE_SIZE_16KB,
and the bus errors are far less frequent. I suspect CONFIG_PAGE_SIZE_64KB
will
be even less.
Disable CONFIG_TRANSPARENT_HUGEPAGE, and the machine works pretty good. It's
been up for almost 8 hours compiling, and not a single bus error yet. It's
got
2x node board with dual R12K/400MHz CPUs per node.
I'm not really sure what CONFIG_TRANSPARENT_HUGEPAGE is enabling that's
causing
R12K CPUs on the IP27 such a headache (and on Octane, really screws up R14K
CPUs). I tried getting a core dump on one of the bus errors, but that
produces a
truncated or corrupted core file that actually crashed GDB, plus I get a nice
oops message in dmesg:
Well, as its name implies, if you enable CONFIG_TRANSPARENT_HUGEPAGE, huge
pages will be created and used in the background transparently to the userspace
application.
With 4KB base page size, the huge pages will be 2MB in size.. I don't know
much about the R10K/R12K/R14K CPUs, but it is possible that either their TLBs
cannot handle such pages, or that the TLB Exception handlers don't contain
proper code for these CPUs.
For each doubling of the base PAGE_SIZE, the huge page size will increase by a
factor of 4. So with 16KB base pages the huge page size would be 32MB, since
there are many fewer opportunities to transparently use a 32MB page, I would
expect any errors related to huge pages to be correspondingly less frequent.
With 64KB PAGE_SIZE the huge page size is 512MB, and It is likely that that
could never be used by normal userspace programs.
I checked the R10K/R12K manual, and the PageMask register there has bits 24:13
open for setting a mask value. It looks like these CPUs only support a page
size from 4KB to 16MB (so a 2MB page size should work w/ transparent
hugepages). I assume that the R14K on the Octane might be the same (but I
don't have a manual specific to the R14k, so I don't know). All of the
remaining bits in that register read 0 and must have 0's written back.
I guess I could find a way to have the kernel trigger a non-fatal oops/dump the
registers on a bus error and get a look at the cause register to see if that
sheds any light on things. Doesn't a SIGBUS on MIPS typically mean that an
address wasn't aligned on a 32-bit boundary? Or could it also mean other
things?
I believe that the R10K is largely compatible with the R4K-style TLB setup, but
Ralf or someone else more knowledge in that area will have to verify. Maybe
the R10k-family CPUs need their own TLB routines, or what currently exists
needs modifications? I have not tried to understand the whole TLB thing in
MIPS yet, so that's a bit of voodoo to me.
I haven't checked, but there may be workarounds required in the TLB management
code that are not in place for the huge page case. When the huge TLB code was
developed, we didn't do any testing on R10K. Somebody should dump the
exception handlers and carefully look at the rest of the huge TLB management
code, and check to see that any required workarounds are in place.
How does one dump the exception handlers? Is it a debug switch somewhere?
Add as the very first line of tlbex.c "#define DEBUG 1"
Then rebuild, and pass "debug" on the kernel command line.
The output can be fed though gas, and then disassembled with objdump -d
--J
David.
--J
[ 1302.260000] CPU: 0 PID: 1179 Comm: emerge Not tainted
3.17.1-mipsgit-20141006 #57
[ 1302.260000] task: a8000000ffbbf288 ti: a8000000fa6f0000 task.ti:
a8000000fa6f0000
[ 1302.260000] $ 0 : 0000000000000000 0000000000000001 0000000000000000
a8000000ff5ad800
[ 1302.260000] $ 4 : a8000000006d5480 00000000000f9c00 00000001f380173f
a800000001000000
[ 1302.260000] $ 8 : 00000001f380173f 0000000000100077 a8000000fe77a000
0000000000000000
[ 1302.260000] $12 : 0000000000660000 0000000000000000 0000000000000000
776bc40c00000004
[ 1302.260000] $16 : 0000000000e00000 0000000000000000 00000000018ee000
6db6db6db6db6db7
[ 1302.260000] $20 : 00000000000000ca a8000000006d5480 a8000000ff65fa68
0000000000001000
[ 1302.260000] $24 : 0000000000000000 a8000000000469c0
[ 1302.260000] $28 : a8000000fa6f0000 a8000000fa6f3a00 0000000000e00000
a800000000046720
[ 1302.260000] Hi : 00000000002ed400
[ 1302.260000] Lo : 00000000000f9c00
[ 1302.260000] epc : a8000000000467e4 r4k_flush_cache_page+0x104/0x2e0
[ 1302.260000] Not tainted
[ 1302.260000] ra : a800000000046720 r4k_flush_cache_page+0x40/0x2e0
[ 1302.260000] Status: 90001ce3 KX SX UX KERNEL EXL IE
[ 1302.260000] Cause : 0000c010
[ 1302.260000] BadVA : 00000001f380173f
[ 1302.260000] PrId : 00000e35 (R12000)
[ 1302.260000] Process emerge (pid: 1179, threadinfo=a8000000fa6f0000,
task=a8000000ffbbf288, tls=00000000778d2490)
[ 1302.260000] Stack : a8000000ff65fa68 0000000000e00000 00000000000f9c00
a8000000006d5480
a8000000ff65fa68 0000000000001000 0000000000e00000
a80000000010cb00
a8000000046a2000 a8000000ff65fa68 00000000018ee000
6db6db6db6db6db7
a8000000fe7fdce0 a8000000000375ec a8000000ff4e5800
a8000000005fbd90
0000000300000080 a8000000ff668580 a8000000005fbd90
5349474900000080
a8000000fa6f3ad8 a8000000005fbd90 0000000600000088
a8000000ff5ad928
a8000000005fbd90 46494c4500002bf9 c000000000101000
0000000a00000080
0000000000000000 0000000000000000 0000000000000000
0000000000000000
0000000000000000 0000000000000000 0000000000000000
0000000000000000
0000000000000000 0000000000000000 0000000000000000
0000000000000000
...
[ 1302.260000] Call Trace:
[ 1302.260000] [<a8000000000467e4>] r4k_flush_cache_page+0x104/0x2e0
[ 1302.260000] [<a80000000010cb00>] get_dump_page+0xc8/0xe8
[ 1302.260000] [<a8000000000375ec>] elf_core_dump+0x1294/0x14d8
[ 1302.260000] [<a8000000001b41e4>] do_coredump+0x5e4/0x1048
[ 1302.260000] [<a80000000005c0b8>] get_signal+0x1b8/0x710
[ 1302.260000] [<a8000000000299c0>] do_signal+0x18/0x240
[ 1302.260000] [<a80000000002a4c8>] do_notify_resume+0x70/0x88
[ 1302.260000] [<a8000000000255ac>] work_notifysig+0x10/0x18
[ 1302.260000]
[ 1302.260000]
Code: 0010327a 30c60ff8 00c8302d <dcc60000> 30c80001 1100003e 00000000
bfb40000 df880000
[ 1305.340000] ---[ end trace c7649a6433db8d18 ]---
Thoughts?
