On 07/26/2012 12:43 PM, Avi Kivity wrote:
On 07/26/2012 01:34 PM, Mian M. Hamayun wrote:On 07/26/2012 12:06 PM, Avi Kivity wrote:On 07/26/2012 12:38 PM, Mian M. Hamayun wrote: <snip>This mechanism 'seems' to work fine when both vcpu threads are in User Mode. But when booting an SMP Guest, the boot processor (BSP) initially executes the bootstrap code while the non-boot processors (APs) are waiting for initial INIT-SIPI-SIPI messages. What I fail to understand is if an AP is currently waiting for an INIT signal, and we call the "run_on_cpu" function above for this cpu, it blocks the whole system, as the AP is in Guest mode and cannot call the "flush_queued_work" and the BSP is waiting for this to happen. How can we resolve this deadlock ? Is there a way to force the AP to quit the Guest Mode, by using some specific mechanism from the User mode ?When a vcpu is waiting for an INIT, it still responds to signals and will return to userspace if a signal is received. Did you observe something different?Hi Avi, So it means that when we execute the following: err = pthread_kill(env->thread, SIG_IPI); then this VCPU thread should wake-up and force the VCPU to quit the guest mode ?Yes.But I am not getting this behavior as the VCPU thread remains blocked in Guest Mode. What could be wrong here ?Perhaps signals are blocked?
No, thats not the case.
Actually its based on kvm-tool and I have integrated some code from qemu-kvm to add debug support to kvm-tool.Can you share your reproducer?
I don't have a smaller example that could reproduce the same problem.
Many Thanks, Hamayun P.S. Please see the following trace; As it might help understanding the problem. (thread 2 is the BSP and thread 3 is for AP) run_on_cpu: Kicked CPU#1 ... Waiting on qemu_work_cond Program received signal SIGUSR1, User defined signal 1. ^C Program received signal SIGINT, Interrupt. 0xb7fdd424 in __kernel_vsyscall () (gdb) info threads Id Target Id Frame 3 Thread 0xaea17b40 (LWP 2843) "arch.x" 0xb7fdd424 in __kernel_vsyscall () 2 Thread 0xaf218b40 (LWP 2842) "arch.x" 0xb7fdd424 in __kernel_vsyscall () * 1 Thread 0xb7359940 (LWP 2822) "arch.x" 0xb7fdd424 in __kernel_vsyscall () (gdb) thread 2 [Switching to thread 2 (Thread 0xaf218b40 (LWP 2842))] #0 0xb7fdd424 in __kernel_vsyscall () (gdb) bt #0 0xb7fdd424 in __kernel_vsyscall () #1 0xb7f7096b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0xb7fc5715 in qemu_cond_wait (cond=0xb7fda640, lock=0xb7fda670) at gdb_srv_arch.c:578 #3 0xb7fc5981 in run_on_cpu (env=0x8465818, func=0xb7fc5417 <kvm_invoke_set_guest_debug>, data=0xaf20fe50) at gdb_srv_arch.c:782 #4 0xb7fc5b90 in kvm_update_guest_debug (env=0x8465818, reinject_trap=0) at gdb_srv_arch.c:863 #5 0xb7fc5ef1 in kvm_remove_all_breakpoints (current_env=0x8465418) at gdb_srv_arch.c:983 #6 0xb7fc2e0a in gdb_breakpoint_remove_all (env=0x8465418) at gdb_srv.c:369 #7 0xb7fc3139 in gdb_handle_packet (s=0x80ac3e0, line_buf=0x80ac3fc "?") at gdb_srv.c:497 #8 0xb7fc439c in gdb_read_byte (s=0x80ac3e0, ch=102) at gdb_srv.c:830 #9 0xb7fc4522 in gdb_loop (env=0x8465418) at gdb_srv.c:878 #10 0xb7fc4621 in gdb_srv_handle_debug (env=0x8465418) at gdb_srv.c:924 #11 0xb7fc5265 in kvm_arch_handle_debug (env=0x8465418) at gdb_srv_arch.c:424 #12 0xb7fac3a8 in kvm_cpu__start (cpu=0x8465418) at kvm-cpu.c:588 #13 0xb7fc6521 in kvm_cpu_thread (arg=0x8465418) at libkvm-main.c:276 #14 0xb7f6cd4c in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #15 0xb7d7bace in clone () from /lib/i386-linux-gnu/libc.so.6 (gdb) thread 3 [Switching to thread 3 (Thread 0xaea17b40 (LWP 2843))] #0 0xb7fdd424 in __kernel_vsyscall () (gdb) bt #0 0xb7fdd424 in __kernel_vsyscall () #1 0xb7d73869 in ioctl () from /lib/i386-linux-gnu/libc.so.6 #2 0xb7fabe9c in kvm_cpu__run (vcpu=0x8465818) at kvm-cpu.c:429 #3 0xb7fac376 in kvm_cpu__start (cpu=0x8465818) at kvm-cpu.c:578 #4 0xb7fc6521 in kvm_cpu_thread (arg=0x8465818) at libkvm-main.c:276 #5 0xb7f6cd4c in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #6 0xb7d7bace in clone () from /lib/i386-linux-gnu/libc.so.6Are you sure thread 3 did not receive the signal?
The thread 3 does actually receives the signal, but the order is not right.As the BSP (Thread 2) starts, it locks the "qemu_global_mutex" and releases it only when it calls the "run_on_cpu" function and starts waiting on "qemu_work_cond". The AP (Thread 3) wakes-up due to the SIG_IPI signal from thread 2, acquires the lock on "qemu_global_mutex" and enters the guest mode. (This is the deadlock case)
If we do not lock the "qemu_global_mutex" in each cpu thread at the beginning, and only lock it when we quit the guest mode, the problem seems to go away, as now we get the SIG_IPI when the Thread 3 is actually in the guest mode and it quits to user mode. But I am not sure if this is the right way to do it, as in qemu-kvm we _always_ start each cpu thread by locking the "qemu_global_mutex".
i.e.
static void *qemu_kvm_cpu_thread_fn(void *arg)
{
CPUArchState *env = arg;
int r;
qemu_mutex_lock(&qemu_global_mutex);
qemu_thread_get_self(env->thread);
env->thread_id = qemu_get_thread_id();
cpu_single_env = env;
r = kvm_init_vcpu(env);
if (r < 0) {
fprintf(stderr, "kvm_init_vcpu failed: %s\n", strerror(-r));
exit(1);
}
qemu_kvm_init_cpu_signals(env);
/* signal CPU creation */
env->created = 1;
qemu_cond_signal(&qemu_cpu_cond);
while (1) {
if (cpu_can_run(env)) {
r = kvm_cpu_exec(env);
if (r == EXCP_DEBUG) {
cpu_handle_guest_debug(env);
}
}
qemu_kvm_wait_io_event(env);
}
return NULL;
}
Try stracing the run.
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
