On 2011-02-24 02:29, Ryousei Takano wrote:
> Hi Jan,
>
> Thanks for your help.
>
> On Thu, Feb 24, 2011 at 2:26 AM, Jan Kiszka <jan.kiszka@xxxxxxxxxxx> wrote:
>> On 2011-02-23 16:53, Ryousei Takano wrote:
>>> Hi Jan,
>>>
>>> On Wed, Feb 23, 2011 at 11:47 PM, Jan Kiszka <jan.kiszka@xxxxxxxxxxx> wrote:
>>>> On 2011-02-23 15:38, Ryousei Takano wrote:
>>>>> This reverts commit c65827770b5e7282b5fb9da29c1de85ed08e816e
>>>>
>>>> Reasoning would belong here, not below the separator.
>>>>
>>>>>
>>>>> Signed-off-by: Ryousei Takano <takano-ryousei@xxxxxxxxxx>
>>>>> ---
>>>>> I am trying to run OpenMP (i.e., multi-threaded) programs using Intel
>>>>> Math Kernel Library (MKL) on a guest OS. In the qemu-kvm-0.12.5,
>>>>> my program works fine. In the qemu-kvm-0.13.0, however, it does not.
>>>>
>>>> What about 0.14?
>>>>
>>> 0.14-rc1 also has problem.
>>
>> Not automagically healed - what a pity.
>>
>>>
>>>>> The Linux kernel is 2.6.32.28. The outputs of 'ps -L' are shown as follows:
>>>>>
>>>>> qemu-kvm-0.12.5:
>>>>> PID LWP TTY TIME CMD
>>>>> 2731 2731 pts/0 00:00:00 bash
>>>>> 2799 2799 pts/0 00:00:19 a.out
>>>>> 2799 2803 pts/0 00:00:00 a.out
>>>>> 2799 2804 pts/0 00:00:04 a.out <-
>>>>> 2799 2805 pts/0 00:00:03 a.out <- these threads are working properly.
>>>>> 2799 2806 pts/0 00:00:04 a.out <-
>>>>> 2807 2807 pts/0 00:00:00 ps
>>>>>
>>>>> qemu-0.13.0:
>>>>> PID LWP TTY TIME CMD
>>>>> 2725 2725 pts/0 00:00:00 bash
>>>>> 2748 2748 pts/0 00:00:19 a.out
>>>>> 2748 2752 pts/0 00:00:00 a.out
>>>>> 2748 2753 pts/0 00:00:00 a.out <-
>>>>> 2748 2754 pts/0 00:00:00 a.out <- these threads are blocked by
>>>>> pthread_cond_wait(3).
>>>>> 2748 2755 pts/0 00:00:00 a.out <-
>>>>> 2756 2756 pts/0 00:00:00 ps
>>>>>
>>>>> This problem seems to be introduced by the commit
>>>>> c65827770b5e7282b5fb9da29c1de85ed08e816e.
>>>>> The attached revert patch solves it. But I do not understand what is
>>>>> happened;-)
>>>>> Any comments and suggestions will be appreciated.
>>>>>
>>>>> Regards,
>>>>> Ryousei
>>>>>
>>>>> hw/pc.c | 6 ++++++
>>>>> target-i386/helper.c | 2 --
>>>>> 2 files changed, 6 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/hw/pc.c b/hw/pc.c
>>>>> index ec97181..577abe7 100644
>>>>> --- a/hw/pc.c
>>>>> +++ b/hw/pc.c
>>>>> @@ -937,6 +937,12 @@ CPUState *pc_new_cpu(const char *cpu_model)
>>>>> }
>>>>> qemu_register_reset(pc_cpu_reset, env);
>>>>> pc_cpu_reset(env);
>>>>> +
>>>>> + /* kvm needs this to run after the apic is initialized. Otherwise,
>>>>> + * it can access invalid state and crash.
>>>>> + */
>>>>> + qemu_init_vcpu(env);
>>>>> +
>>>>> return env;
>>>>> }
>>>>>
>>>>> diff --git a/target-i386/helper.c b/target-i386/helper.c
>>>>> index 86202e6..69cda5e 100644
>>>>> --- a/target-i386/helper.c
>>>>> +++ b/target-i386/helper.c
>>>>> @@ -1224,8 +1224,6 @@ CPUX86State *cpu_x86_init(const char *cpu_model)
>>>>> }
>>>>> mce_init(env);
>>>>>
>>>>> - qemu_init_vcpu(env);
>>>>> -
>>>>> return env;
>>>>> }
>>>>>
>>>>
>>>> I don't believe yet that the patch is the reason, specifically as it
>>>> targets the setup phase (except for CPU hotplug, but that's broken for
>>>> ages), not the operational phase of the VM.
>>>>
>>>> What's your qemu-kvm command line?
>>>>
>>> Here is my command line.
>>>
>>> sudo qemu-system-x86_64 \
>>> -cpu host \
>>> -smp 8 \
>>> -m 46080 \
>>> -net nic,model=virtio,macaddr=00:16:3e:1d:ff:00 \
>>> -net tap,ifname=tap0,script=no,downscript=no \
>>> -monitor telnet::5963,server,nowait \
>>> -daemonize \
>>> disk.img
>>
>> How many cores are available on the host? >= 8 or less?
>>
> 8 cores. All cores are allocated to the VM.
> If 7 cores are allocated, the problem is reproduced.
>
>>>
>>>> Also, please attach gdb to the stuck VM and print the backtraces of all
>>>> threads ("thread apply all bt").
>>>>
>>> (gdb) thread apply all bt
>>> Thread 9 (Thread 0x41129940 (LWP 8118)):
>>> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
>>> #1 0x000000000042e56d in pthread_attr_setdetachstate ()
>>> #2 0x000000000042ea99 in pthread_attr_setdetachstate ()
>>> #3 0x000000000042ecf0 in pthread_attr_setdetachstate ()
>>> #4 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>>>
>>> Thread 8 (Thread 0x41ca8940 (LWP 8119)):
>>> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
>>> #1 0x000000000042e56d in pthread_attr_setdetachstate ()
>>> #2 0x000000000042ea99 in pthread_attr_setdetachstate ()
>>> #3 0x000000000042ecf0 in pthread_attr_setdetachstate ()
>>> #4 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>>>
>>> Thread 7 (Thread 0x425f2940 (LWP 8120)):
>>> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
>>> #1 0x000000000042e56d in pthread_attr_setdetachstate ()
>>> #2 0x000000000042ea99 in pthread_attr_setdetachstate ()
>>> #3 0x000000000042ecf0 in pthread_attr_setdetachstate ()
>>> #4 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>>>
>>> Thread 6 (Thread 0x42df3940 (LWP 8121)):
>>> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
>>> #1 0x000000000042e56d in pthread_attr_setdetachstate ()
>>> #2 0x000000000042ea99 in pthread_attr_setdetachstate ()
>>> #3 0x000000000042ecf0 in pthread_attr_setdetachstate ()
>>> #4 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>>>
>>> Thread 5 (Thread 0x435f4940 (LWP 8122)):
>>> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
>>> #1 0x000000000042e56d in pthread_attr_setdetachstate ()
>>> #2 0x000000000042ea99 in pthread_attr_setdetachstate ()
>>> #3 0x000000000042ecf0 in pthread_attr_setdetachstate ()
>>> #4 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>>>
>>> Thread 4 (Thread 0x43df5940 (LWP 8123)):
>>> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
>>> #1 0x000000000042e56d in pthread_attr_setdetachstate ()
>>> #2 0x000000000042ea99 in pthread_attr_setdetachstate ()
>>> #3 0x000000000042ecf0 in pthread_attr_setdetachstate ()
>>> #4 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>>>
>>> Thread 3 (Thread 0x445f6940 (LWP 8124)):
>>> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
>>> #1 0x000000000042e56d in pthread_attr_setdetachstate ()
>>> #2 0x000000000042ea99 in pthread_attr_setdetachstate ()
>>> #3 0x000000000042ecf0 in pthread_attr_setdetachstate ()
>>> #4 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>>>
>>> Thread 2 (Thread 0x44df7940 (LWP 8125)):
>>> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
>>> #1 0x000000000042e56d in pthread_attr_setdetachstate ()
>>> #2 0x000000000042ea99 in pthread_attr_setdetachstate ()
>>> #3 0x000000000042ecf0 in pthread_attr_setdetachstate ()
>>> #4 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>>>
>>> Thread 1 (Thread 0x7fe796ec1740 (LWP 8117)):
>>> #0 0x0000003dd38ccfc2 in select () from /lib64/libc.so.6
>>> #1 0x0000000000419620 in pthread_attr_setdetachstate ()
>>> #2 0x000000000042be07 in pthread_attr_setdetachstate ()
>>> #3 0x000000000041ae20 in pthread_attr_setdetachstate ()
>>> #4 0x0000003dd381d994 in __libc_start_main () from /lib64/libc.so.6
>>> #5 0x00000000004063d9 in pthread_attr_setdetachstate ()
>>> #6 0x00007fff8050c988 in ?? ()
>>> #7 0x0000000000000000 in ?? ()
>>>
>>
>> Can you retry with --enable-debug set during qemu-kvm configure? We need
>> the symbols of the qemu functions. Despite the lacking symbols, this
>> rather looks like we are stuck in kernel space, not on some condition
>> variable (ie. futex).
>>
> Here is the result with --enable-debug.
>
> Thread 9 (Thread 0x410c5940 (LWP 19303)):
> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
> #1 0x000000000043b85b in kvm_run (env=0x284c7c0) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:582
> #2 0x000000000043cf7f in kvm_cpu_exec (env=0x284c7c0) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1233
> #3 0x000000000043d73e in kvm_main_loop_cpu (env=0x284c7c0) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1419
> #4 0x000000000043d8ac in ap_main_loop (_env=0x284c7c0) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1466
> #5 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
> #6 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>
> Thread 8 (Thread 0x418c6940 (LWP 19304)):
> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
> #1 0x000000000043b85b in kvm_run (env=0x2868180) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:582
> #2 0x000000000043cf7f in kvm_cpu_exec (env=0x2868180) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1233
> #3 0x000000000043d73e in kvm_main_loop_cpu (env=0x2868180) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1419
> #4 0x000000000043d8ac in ap_main_loop (_env=0x2868180) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1466
> #5 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
> #6 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>
> Thread 7 (Thread 0x420c7940 (LWP 19305)):
> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
> #1 0x000000000043b85b in kvm_run (env=0x2877c70) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:582
> #2 0x000000000043cf7f in kvm_cpu_exec (env=0x2877c70) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1233
> #3 0x000000000043d73e in kvm_main_loop_cpu (env=0x2877c70) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1419
> #4 0x000000000043d8ac in ap_main_loop (_env=0x2877c70) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1466
> #5 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
> #6 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>
> Thread 6 (Thread 0x428c8940 (LWP 19306)):
> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
> #1 0x000000000043b85b in kvm_run (env=0x2887740) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:582
> #2 0x000000000043cf7f in kvm_cpu_exec (env=0x2887740) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1233
> #3 0x000000000043d73e in kvm_main_loop_cpu (env=0x2887740) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1419
> #4 0x000000000043d8ac in ap_main_loop (_env=0x2887740) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1466
> #5 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
> #6 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>
> Thread 5 (Thread 0x430c9940 (LWP 19307)):
> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
> #1 0x000000000043b85b in kvm_run (env=0x2897210) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:582
> #2 0x000000000043cf7f in kvm_cpu_exec (env=0x2897210) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1233
> #3 0x000000000043d73e in kvm_main_loop_cpu (env=0x2897210) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1419
> #4 0x000000000043d8ac in ap_main_loop (_env=0x2897210) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1466
> #5 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
> #6 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>
> Thread 4 (Thread 0x438ca940 (LWP 19308)):
> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
> #1 0x000000000043b85b in kvm_run (env=0x28a6ce0) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:582
> #2 0x000000000043cf7f in kvm_cpu_exec (env=0x28a6ce0) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1233
> #3 0x000000000043d73e in kvm_main_loop_cpu (env=0x28a6ce0) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1419
> #4 0x000000000043d8ac in ap_main_loop (_env=0x28a6ce0) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1466
> #5 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
> #6 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>
> Thread 3 (Thread 0x440cb940 (LWP 19309)):
> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
> #1 0x000000000043b85b in kvm_run (env=0x28b67b0) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:582
> #2 0x000000000043cf7f in kvm_cpu_exec (env=0x28b67b0) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1233
> #3 0x000000000043d73e in kvm_main_loop_cpu (env=0x28b67b0) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1419
> #4 0x000000000043d8ac in ap_main_loop (_env=0x28b67b0) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1466
> #5 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
> #6 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>
> Thread 2 (Thread 0x448cc940 (LWP 19310)):
> #0 0x0000003dd38cc647 in ioctl () from /lib64/libc.so.6
> #1 0x000000000043b85b in kvm_run (env=0x28c6280) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:582
> #2 0x000000000043cf7f in kvm_cpu_exec (env=0x28c6280) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1233
> #3 0x000000000043d73e in kvm_main_loop_cpu (env=0x28c6280) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1419
> #4 0x000000000043d8ac in ap_main_loop (_env=0x28c6280) at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1466
> #5 0x0000003dd400673d in start_thread () from /lib64/libpthread.so.0
> #6 0x0000003dd38d3d1d in clone () from /lib64/libc.so.6
>
> Thread 1 (Thread 0x7fd303b9c740 (LWP 19302)):
> #0 0x0000003dd38ccfc2 in select () from /lib64/libc.so.6
> #1 0x00000000004207a5 in main_loop_wait (nonblocking=0) at
> /home/takano/qemu-kvm-0.14.0-rc1/vl.c:1376
> #2 0x000000000043dd48 in kvm_main_loop () at
> /home/takano/qemu-kvm-0.14.0-rc1/qemu-kvm.c:1589
> #3 0x0000000000420ad2 in main_loop () at
> /home/takano/qemu-kvm-0.14.0-rc1/vl.c:1429
> #4 0x0000000000424b39 in main (argc=15, argv=0x7fffeb76db58,
> envp=0x7fffeb76dbd8) at /home/takano/qemu-kvm-0.14.0-rc1/vl.c:3201
OK, this finally makes me realize that we are neither talking about a
locked up qemu-kvm nor guest kernel but rather some deadlock the guest
user space application is seeing.
Did you already debugged this to more details? If the guest app is stuck
on some cond variable, is the associated condition true now? Did the
related cond_signal/broadcast took place?
>
>> Then probably also interesting would be a trace of the host kernel
>> (trace-cmd record -e kvm, see e.g. [1]).
>>
> OK. I will try trace as soon as possible.
Focus tracing (ftrace or even just strace) first of all on the
application from inside the guest. That provides more details. Also,
compare the activities with working environments (older qemu or even
native).
Jan
--
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
