On 09/08/2014 05:44, Joshua Kinard wrote: > On 09/08/2014 04:11, Miod Vallat wrote: >>> Disassembling a statically-built copy of the "sln" binary generated by >>> glibc's compile phase, there are slight differences in how gcc-4.7 and >>> gcc-4.8 are compiling the __lll_lock_wait_private function. The key >>> differences in the output asm are >>> this: >> >> [...] >> >>> gcc-4.8: >>> x+4 <START> >>> ... >>> x+24 bne v1,v0,<x+56> >>> ... >>> x+32 0x7c03e83b /* rdhwr */ >>> x+36 li a2,2 >>> x+40 lw a1,-29832(v1) >>> x+44 move a3,zero >>> x+48 li v0,4238 >>> x+52 syscall >>> * x+56 ll v0,0(s0) >>> * x+60 li at,2 >>> * x+64 sc at,0 >> >> Note how the sc address is no longer 0(s0). Since the address does >> not match the address used in the ll instruction, sc will always >> fail on the R10k. > > That would be a typo on my part. I typed that out by hand and just missed it. It should read: > > gcc-4.8: > x+4 <START> > ... > x+24 bne v1,v0,<x+56> > ... > x+32 0x7c03e83b /* rdhwr */ > x+36 li a2,2 > x+40 lw a1,-29832(v1) > x+44 move a3,zero > x+48 li v0,4238 > x+52 syscall > * x+56 ll v0,0(s0) > * x+60 li at,2 > * x+64 sc at,0(s0) > x+68 beqzl at,<x+56> > x+72 nop > x+76 sync > x+80 bnez v0,<x+32> I did some more tracing. It seems the issue with glibc itself stems from the addition of __atomic_* builtins added generally in gcc-4.7 and MIPS-specifically in gcc-4.8: >From ports/sysdeps/mips/bits/atomic.h (for 2.19) or sysdeps/mips/bits/atomic.h (for 2.20): /* The __atomic_* builtins are available in GCC 4.7 and later, but MIPS support for their efficient implementation was added only in GCC 4.8. We still want to use them even with GCC 4.7 for MIPS16 code where we have no assembly alternative available and want to avoid the __sync_* builtins if at all possible. */ #if __GNUC_PREREQ (4, 8) || (defined __mips16 && __GNUC_PREREQ (4, 7)) [snip] This is why the assembly is different between the two gcc versions. This same code is in the kernel's atomic.h copy under arch/mips/include/asm/ as well. I tested by removing the top part of the #if macro and basically forcing the inline versions only, then rebuilt glibc-2.20 with gcc-4.9.2 (20140921 prerelease), and lo and behold, sln executes and returns its usage information. When using the gcc internal builtins, a futex gets used, which is why I wasn't seeing futexes in 4.7-built copies of sln, only in 4.8 or greater-built copies. This means that the gcc internal __atomic_* builtins may be somewhat to blame for this problem on R1x000 systems. I traced the kernel side of the problem out and figured out that when the futex is taken by sln, the process gets frozen by the scheduler via a call to freezable_schedule() in function futex_wait_queue_me in kernel/futex.c. I added two printk statements, one before freezable_schedule() and one after, and the first statement executes (verified by dumping /proc/kmsg directly because dmesg itself generates futexes), but not the printk after. The printk after freezable_schedule() only executes when I ctrl+C the frozen process and it exits out of the futex code. I visually checked through include/linux/freezer.h and noticed that freezable_schedule eventually calls freezing(), which executes an atomic_read() on system_freezing_cnt. In the mips code, that just comes out as a pointer dereference of a volatile variable. I'm not certain, though, if in gcc's case, the use of volatile means it tries to use its builtin __atomic_ functions again, and tries to take another futex /while it's trying to take a futex/. Chicken and egg? So, could still very well be a gcc issue, or maybe it's something really subtle in the kernel code. I am not sure which. I at least know of a specific gcc commit that enables/disables the problem, and that's pointing the finger at gcc here. Ideas? -- Joshua Kinard Gentoo/MIPS kumba@xxxxxxxxxx 4096R/D25D95E3 2011-03-28 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic
