Can we force a KVM VCPU in Guest Mode to Exit to User Mode From User Mode ?

["Mian M. Hamayun" <mian-muhammad.hamayun@xxxxxxx>]

Hi Everyone,

I want to know if we can force a VMExit on a KVM VCPU currently in Guest 
Mode from the User Mode ?

As an example, I want to execute an IOCTL on a KVM VCPU which is 
currently in Guest Mode, say it is waiting to execute some guest 
instructions.
As far as I know, the VCPU should not be running for an IOCTL to 
complete on it.

To give you a concrete example, please see the following code segments 
(taken from qemu-kvm):

void run_on_cpu(CPUArchState *env, void (*func)(void *data), void *data)
{
    struct qemu_work_item wi;

    if (qemu_cpu_is_self(env)) {
        func(data);
        return;
    }

    wi.func = func;
    wi.data = data;
    if (!env->queued_work_first) {
        env->queued_work_first = &wi;
    } else {
        env->queued_work_last->next = &wi;
    }
    env->queued_work_last = &wi;
    wi.next = NULL;
    wi.done = false;

    qemu_cpu_kick(env);
    while (!wi.done) {
        CPUArchState *self_env = cpu_single_env;

        qemu_cond_wait(&qemu_work_cond, &qemu_global_mutex);
        cpu_single_env = self_env;
    }
}

The "run_on_cpu" function in qemu-kvm invokes "qemu_cpu_kick" to 
schedule the appropriate VCPU thread and blocks itself on the 
"qemu_work_cond".

void qemu_cpu_kick(void *_env)
{
    CPUArchState *env = _env;

    qemu_cond_broadcast(env->halt_cond);
    if (!tcg_enabled() && !env->thread_kicked) {
        qemu_cpu_kick_thread(env);
        env->thread_kicked = true;
    }
}

A "SIG_IPI" is sent to the vcpu thread, which wakes up the target VCPU 
thread.

static void qemu_cpu_kick_thread(CPUArchState *env)
{
    int err;

    err = pthread_kill(env->thread->thread, SIG_IPI);
    if (err) {
        fprintf(stderr, "qemu:%s: %s", __func__, strerror(err));
        exit(1);
    }
}

And this thread flushes the queued work using the following function and 
unblocks the original thread waiting on "qemu_work_cond".

static void flush_queued_work(CPUArchState *env)
{
    struct qemu_work_item *wi;

    if (!env->queued_work_first) {
        return;
    }

    while ((wi = env->queued_work_first)) {
        env->queued_work_first = wi->next;
        wi->func(wi->data);
        wi->done = true;
    }
    env->queued_work_last = NULL;
    qemu_cond_broadcast(&qemu_work_cond);
}

This mechanism 'seems' to work fine when both vcpu threads are in User 
Mode. But when booting an SMP Guest, the boot processor (BSP) initially 
executes the bootstrap code while the non-boot processors (APs) are 
waiting for initial INIT-SIPI-SIPI messages.

What I fail to understand is if an AP is currently waiting for an INIT 
signal, and we call the "run_on_cpu" function above for this cpu, it 
blocks the whole system, as the AP is in Guest mode and cannot call the 
"flush_queued_work" and the BSP is waiting for this to happen.

How can we resolve this deadlock ? Is there a way to force the AP to 
quit the Guest Mode, by using some specific mechanism from the User mode ?

I hope I was able to explain the problem.

Best Regards,
Hamayun

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature