I've been successfully running 2.6.12 on the sibyte bcm1250 for over a year and have recently been trying to move forward to a more recent kernel. I've got 2.6.18 from linux-mips.org's git tree at the 'linux-2.6.18' TAG built and almost booting. While usually I'd run SMP+PREEMPT, I've turned those off to simplify the kernel. I'm running n64 kernel with o32 userspace. It will run all the way through kernel startup, but once it starts userspace (glibc + sysvinit) things go down hill fast. I replaced init with a statically linked test program that does a few syscalls and then spins to try to track down the issue. When things go wrong the symptom is usually a SIGSEGV or SIGBUS to the process very shortly after it starts running. How far the test program gets varies, but it usually looks like the cpu starts executing incorrect code (at the right address) after returning to userspace from an interrupt/exception. I have two variants of the test 'init' program: 1) once into main() print hello world then branch to self. This program usually works reliably. If the program makes it all the way to the branch to self instruction things are good. The kernel schedules it just fine, taking timer interrupts as expected. 2) once into main() print hello world then call a function that consists of 2MB worth of 'addiu $8,$8,1' instructions then branch to self. Running this program _always_ fails part way through the adds. The program executes through the add instruction and every time it crosses a page boundary it causes a page fault and the kernel loads in the next page from the filesystem as expected. On startup, the program faults in various text, data, and stack pages, and prints Hello World. After this it starts linearly executing add instructions starting at about address 0x00401000. In the below case after about 100KB of add instructions the program takes a SEGV for no apparent reason at 0x00419140! The entire 0x00419000 - 0x00419FFF page should be full of add instructions none of which should cause a SEGV! I enabled some printk's in the page fault handler and I see this: Cpu0[init:1:0000000010004624:1:ffffffff8025f438] Cpu0[init:1:0000000000400190:0:0000000000400190] Cpu0[init:1:0000000010003f70:0:00000000004001e0] Cpu0[init:1:0000000000604670:0:0000000000604670] Cpu0[init:1:00000000100004f4:0:00000000006046f4] Cpu0[init:1:0000000010000550:1:0000000000604718] Cpu0[init:1:000000000060e380:0:000000000060e380] Cpu0[init:1:0000000000619890:0:0000000000619890] Cpu0[init:1:000000000062c028:0:000000000062c028] Cpu0[init:1:000000000061fd70:0:000000000061fd70] Cpu0[init:1:0000000010002f80:0:000000000061998c] Cpu0[init:1:0000000000618e10:0:0000000000618e10] Cpu0[init:1:0000000000620c50:0:0000000000620c50] Cpu0[init:1:000000000062ad20:0:000000000062ad20] Cpu0[init:1:0000000000679b74:0:000000000062ad78] Cpu0[init:1:000000000060fb40:0:000000000060fb40] Cpu0[init:1:0000000000606b24:0:0000000000606b24] Cpu0[init:1:0000000000605d20:0:0000000000605d20] Cpu0[init:1:0000000000607010:0:0000000000607010] Cpu0[init:1:000000000060c7f0:0:000000000060c7f0] Cpu0[init:1:0000000010006004:1:0000000000607894] Cpu0[init:1:0000000000610528:0:0000000000610528] Cpu0[init:1:000000000062e490:0:000000000062e490] Cpu0[init:1:0000000000678c30:0:0000000000678c30] Hello World! Cpu0[init:1:0000000000401000:0:0000000000401000] Cpu0[init:1:0000000000402000:0:0000000000402000] Cpu0[init:1:0000000000403000:0:0000000000403000] Cpu0[init:1:0000000000404000:0:0000000000404000] Cpu0[init:1:0000000000405000:0:0000000000405000] Cpu0[init:1:0000000000406000:0:0000000000406000] Cpu0[init:1:0000000000407000:0:0000000000407000] Cpu0[init:1:0000000000408000:0:0000000000408000] Cpu0[init:1:0000000000409000:0:0000000000409000] Cpu0[init:1:000000000040a000:0:000000000040a000] Cpu0[init:1:000000000040b000:0:000000000040b000] Cpu0[init:1:000000000040c000:0:000000000040c000] Cpu0[init:1:000000000040d000:0:000000000040d000] Cpu0[init:1:000000000040e000:0:000000000040e000] Cpu0[init:1:000000000040f000:0:000000000040f000] Cpu0[init:1:0000000000410000:0:0000000000410000] Cpu0[init:1:0000000000411000:0:0000000000411000] Cpu0[init:1:0000000000412000:0:0000000000412000] Cpu0[init:1:0000000000413000:0:0000000000413000] Cpu0[init:1:0000000000414000:0:0000000000414000] Cpu0[init:1:0000000000415000:0:0000000000415000] Cpu0[init:1:0000000000416000:0:0000000000416000] Cpu0[init:1:0000000000417000:0:0000000000417000] Cpu0[init:1:0000000000418000:0:0000000000418000] Cpu0[init:1:0000000000419000:0:0000000000419000] Cpu0[init:1:0000000000000098:1:0000000000419140] do_page_fault() #2: sending SIGSEGV to init for invalid write access to 0000000000000098 (epc == 0000000000419140, ra == 00000000006045b8) Cpu0[init:1:0000000000000098:1:0000000000419140] do_page_fault() #2: sending SIGSEGV to init for invalid write access to 0000000000000098 (epc == 0000000000419140, ra == 00000000006045b8) Cpu0[init:1:0000000000000098:1:0000000000419140] do_page_fault() #2: sending SIGSEGV to init for invalid write access to 0000000000000098 (epc == 0000000000419140, ra == 00000000006045b8) Cpu0[init:1:0000000000000098:1:0000000000419140] do_page_fault() #2: sending SIGSEGV to init for invalid write access to 0000000000000098 (epc == 0000000000419140, ra == 00000000006045b8) Cpu0[init:1:0000000000000098:1:0000000000419140] do_page_fault() #2: sending SIGSEGV to init for invalid write access to 0000000000000098 (epc == 0000000000419140, ra == 00000000006045b8) Cpu0[init:1:0000000000000098:1:0000000000419140] I've carefully gone through syscall and interrupt/exception entry/exit with a jtag debugger to make sure registers are saved/restored correctly and everything looks fine at least on the few times I walked through it. After taking the fault, I also examined the page that took the fault and verified it is full of 'addiu $8,$8,1' including the instruction that the kernel thinks a SEGV occurred on. Since the page contains correct data, I tried adding gratuitous icache flushes after each page fault before returning to userspace to rule out any issues there, but with no help. Has issues like this been seen before? If not, does anyone have ideas that I could try next? -- Dave Johnson Starent Networks
