On Mon, May 23, 2016 at 3:08 PM, Joshua Kinard <kumba@xxxxxxxxxx> wrote:
On 05/23/2016 14:52, Aaro Koskinen wrote:
> On Mon, May 23, 2016 at 09:21:22AM -0700, David Daney wrote:
>> On 05/23/2016 08:20 AM, Ralf Baechle wrote:
>>> On Mon, May 23, 2016 at 06:13:46PM +0300, Aaro Koskinen wrote:
>>>> I'm getting kernel crashes (see below) reliably when building Perl in
>>>> parallel (make -j16) on OCTEON EBH5600 board (8 cores, 4 GB RAM) with
>>>> Linux 4.6.
>>>>
>>>> It seems that CONFIG_TRANSPARENT_HUGEPAGE has something to do with the
>>>> issue - disabling it makes build go through fine.
>>>>
>>>> Any ideas?
>>>
>>> I thought it was working except on SGI Origin 200/2000 aka IP27 where
>>> Joshua Kinard (added to cc) was hitting issues as well.
>>>
>>> Joshua, does that similar to the issues you were hitting?
>>
>> There is nothing OCTEON specific in the THP code, or huge pages in general.
>>
>> That said, we have seen other THP related failures, and have never been able
>> to find the cause.
>>
>> If someone can come up with a reproducible test case that triggers quickly,
>> we can run it in our simulator and easily find the problem.
>
> Trying to build Perl is a reliable reproducer. Is that too heavyweight
> for your simulator?
>
> I was able to reproduce this also on EdgeRouter Pro, but there the kernel
> does not fail, only compiler dies with SIGBUS:
>
> [ 315.095264] Data bus error, epc == 0000000000a801c4, ra == 0000000000a80624
>
> And without THP the build is fine.
>
> I also tried CN68XX board with 16 GB RAM and also there I get SIGBUS failure
> instead of Machine Check.
SIGBUS is closer to what I was seeing on IP30/IP27, but there's two different
SIGBUS errors in MIPS, a Data Bus Error (DBE) and Instruction Bus Error (IBE).
I've only seen IBEs result from using THP on Octane/IP30 and Origin/Onyx2/IP27.
Also CC'ing Alastair Bridgewater (nyef), who was working on bringing up the
IP35 hardware (Origin 300/350), as he had been working on tracing down some
possible issues in the TLB code. He had a small test case at the below address
(use annotation #2), but I don't know if he got any further on debugging it.
http://paste.lisp.org/display/305809#2
I still haven't gotten anywhere with this. FWIW, we confirmed that it affects at least some of the R1x000 CPUs, and it doesn't seem to affect OCTEON (tested on an ERlite-3). And, at least for any IP35 testing (R14000 / R16000), my kernel configuration does not include CONFIG_TRANSPARENT_HUGEPAGE. One of the scary parts of this one is the cases where it DOESN'T fault. It manages to read SOMETHING, but it's not the right data (typically zeros, but it's still scary, and what if it was a write instead of a read?). I don't really have any other systems to test with, so I don't know if it's R10k-specific or somewhat more general.
-- Alastair
