Re: [PATCH V2 1/2] kvm tools: Add initial SPAPR PPC64 architecture support

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 15.12.2011, at 02:27, Matt Evans wrote:

> Heya Alex,
> 
> On 13/12/11 19:23, Alexander Graf wrote:
>> 
>> On 13.12.2011, at 08:00, Matt Evans <matt@xxxxxxxxxx> wrote:
>> 
>>> This patch adds a new arch directory, powerpc, basic file structure, register
>>> setup and where necessary stubs out arch-specific functions (e.g. interrupts,
>>> runloop exits) that later patches will provide.  The target is an
>>> SPAPR-compliant PPC64 machine (i.e. pSeries); there is no support for PPC32 or
>>> 'bare metal' PPC64 guests as yet.  Subsequent patches implement the hcalls and
>>> RTAS required to boot SPAPR pSeries kernels.
>>> 
>>> Memory is mapped from hugetlbfs (as that is currently required by upstream PPC64
>>> HV-mode KVM).  The mapping of a VRMA region is yet to be implemented; this is
>>> only necessary on processors that don't support VRMA, e.g. <= P6.  Work is
>>> therefore needed to get this going on pre-P7 CPUs.
>>> 
>>> Processor state is set up as a guest kernel would expect (both primary and
>>> secondaries), and SMP is fully supported.
>>> 
>>> Finally, support is added for simply loading flat binary kernels (plus initrd).
>>> (bzImages are not used on PPC, and this series does not add zImage support or an
>>> ELF loader.)  The intention is to later support loading firmware such as SLOF.
>>> 
>>> Signed-off-by: Matt Evans <matt@xxxxxxxxxx>
>>> ---
>>> tools/kvm/Makefile                           |   10 +
>>> tools/kvm/kvm.c                              |    3 +
>>> tools/kvm/powerpc/include/kvm/barrier.h      |    6 +
>>> tools/kvm/powerpc/include/kvm/kvm-arch.h     |   72 ++++++++
>>> tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h |   66 ++++++++
>>> tools/kvm/powerpc/ioport.c                   |   18 ++
>>> tools/kvm/powerpc/irq.c                      |   40 +++++
>>> tools/kvm/powerpc/kvm-cpu.c                  |  233 ++++++++++++++++++++++++++
>>> tools/kvm/powerpc/kvm.c                      |  187 +++++++++++++++++++++
>>> 9 files changed, 635 insertions(+), 0 deletions(-)
>>> create mode 100644 tools/kvm/powerpc/include/kvm/barrier.h
>>> create mode 100644 tools/kvm/powerpc/include/kvm/kvm-arch.h
>>> create mode 100644 tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h
>>> create mode 100644 tools/kvm/powerpc/ioport.c
>>> create mode 100644 tools/kvm/powerpc/irq.c
>>> create mode 100644 tools/kvm/powerpc/kvm-cpu.c
>>> create mode 100644 tools/kvm/powerpc/kvm.c
>>> 
>>> diff --git a/tools/kvm/Makefile b/tools/kvm/Makefile
>>> index 2bf70c9..3f1e84a 100644
>>> --- a/tools/kvm/Makefile
>>> +++ b/tools/kvm/Makefile
>>> @@ -124,6 +124,16 @@ ifeq ($(ARCH),x86)
>>>   OTHEROBJS    += x86/bios/bios-rom.o
>>>   ARCH_INCLUDE := x86/include
>>> endif
>>> +# POWER/ppc:  Actually only support ppc64 currently.
>> 
>> Why? I usually run ppc32 user land. Doesn't that expose 'ppc' here?
> 
> Not quite sure what you mean here; do you mean 32bit distro?  (Will still get 'ppc64' from a 64-bit kernel.)

Eh. Yes. Sorry, my bad.

> There is clearly some work required here to determine what to build for when we
> eventually support PPC32 guests/hosts though I'm not sure how that will look
> yet.  This is designed to break if you build on a 32bit kernel, as if it DID
> build, it wouldn't run anyway.  (It's building -m64 too...

Yeah, running -M pseries on PPC32 hosts doesn't make sense really.

> 
>>> +ifeq ($(uname_M), ppc64)
>>> +    DEFINES += -DCONFIG_PPC
>>> +    OBJS    += powerpc/ioport.o
>>> +    OBJS    += powerpc/irq.o
>>> +    OBJS    += powerpc/kvm.o
>>> +    OBJS    += powerpc/kvm-cpu.o
>>> +    ARCH_INCLUDE := powerpc/include
>>> +    CFLAGS += -m64
> 
> ...here.)
> 
>>> +endif
>>> 
>>> ###
>>> 
>>> diff --git a/tools/kvm/kvm.c b/tools/kvm/kvm.c
>>> index 35ca2c5..3fb46f6 100644
>>> --- a/tools/kvm/kvm.c
>>> +++ b/tools/kvm/kvm.c
>>> @@ -49,6 +49,9 @@ const char *kvm_exit_reasons[] = {
>>>   DEFINE_KVM_EXIT_REASON(KVM_EXIT_DCR),
>>>   DEFINE_KVM_EXIT_REASON(KVM_EXIT_NMI),
>>>   DEFINE_KVM_EXIT_REASON(KVM_EXIT_INTERNAL_ERROR),
>>> +#ifdef CONFIG_PPC64
>>> +    DEFINE_KVM_EXIT_REASON(KVM_EXIT_PAPR_HCALL),
>>> +#endif
>>> };
>>> 
>>> extern struct kvm *kvm;
>>> diff --git a/tools/kvm/powerpc/include/kvm/barrier.h b/tools/kvm/powerpc/include/kvm/barrier.h
>>> new file mode 100644
>>> index 0000000..bc7d179
>>> --- /dev/null
>>> +++ b/tools/kvm/powerpc/include/kvm/barrier.h
>>> @@ -0,0 +1,6 @@
>>> +#ifndef _KVM_BARRIER_H_
>>> +#define _KVM_BARRIER_H_
>>> +
>>> +#include <asm/system.h>
>>> +
>>> +#endif /* _KVM_BARRIER_H_ */
>>> diff --git a/tools/kvm/powerpc/include/kvm/kvm-arch.h b/tools/kvm/powerpc/include/kvm/kvm-arch.h
>>> new file mode 100644
>>> index 0000000..da61774
>>> --- /dev/null
>>> +++ b/tools/kvm/powerpc/include/kvm/kvm-arch.h
>>> @@ -0,0 +1,72 @@
>>> +/*
>>> + * PPC64 architecture-specific definitions
>>> + *
>>> + * Copyright 2011 Matt Evans <matt@xxxxxxxxxx>, IBM Corporation.
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify it
>>> + * under the terms of the GNU General Public License version 2 as published
>>> + * by the Free Software Foundation.
>>> + */
>>> +
>>> +#ifndef KVM__KVM_ARCH_H
>>> +#define KVM__KVM_ARCH_H
>>> +
>>> +#include <stdbool.h>
>>> +#include <linux/types.h>
>>> +#include <time.h>
>>> +
>>> +#define KVM_NR_CPUS            (255)
>> 
>> Why?
> 
> Good question; that's arbitrary & cut-paste I missed. :-)
> 
> I'll make this 1024, to match the max sensible NR_CPUS in the PPC64 kernel
> (which in turn limits KVM_MAX_VCPUS).

I thought Sasha converted this to a queryable interface?

> 
>>> +
>>> +/*
>>> + * MMIO lives after RAM, but it'd be nice if it didn't constantly move.
>>> + * Choose a suitably high address, e.g. 63T...  This limits RAM size.
>>> + */
>>> +#define PPC_MMIO_START            0x3F0000000000UL
>>> +#define PPC_MMIO_SIZE            0x010000000000UL
>>> +
>>> +#define KERNEL_LOAD_ADDR            0x0000000000000000
>>> +#define KERNEL_START_ADDR           0x0000000000000000
>>> +#define KERNEL_SECONDARY_START_ADDR     0x0000000000000060
>>> +#define INITRD_LOAD_ADDR            0x0000000002800000
>>> +
>>> +#define FDT_MAX_SIZE                0x10000
>>> +#define RTAS_MAX_SIZE               0x10000
>>> +
>>> +#define TIMEBASE_FREQ               512000000ULL
>>> +
>>> +#define KVM_MMIO_START            PPC_MMIO_START
>>> +
>>> +/*
>>> + * This is the address that pci_get_io_space_block() starts allocating
>>> + * from.  Note that this is a PCI bus address.
>>> + */
>>> +#define KVM_PCI_MMIO_AREA        0x1000000
>>> +
>>> +struct kvm {
>>> +    int            sys_fd;        /* For system ioctls(), i.e. /dev/kvm */
>>> +    int            vm_fd;        /* For VM ioctls() */
>>> +    timer_t            timerid;    /* Posix timer for interrupts */
>>> +
>>> +    int            nrcpus;        /* Number of cpus to run */
>>> +
>>> +    u32            mem_slots;    /* for KVM_SET_USER_MEMORY_REGION */
>>> +
>>> +    u64            ram_size;
>>> +    void            *ram_start;
>>> +
>>> +    bool            nmi_disabled;
>>> +
>>> +    bool            single_step;
>>> +
>>> +    const char        *vmlinux;
>>> +    struct disk_image       **disks;
>>> +    int                     nr_disks;
>>> +    unsigned long        rtas_gra;
>>> +    unsigned long        rtas_size;
>>> +    unsigned long        fdt_gra;
>>> +    unsigned long        initrd_gra;
>>> +    unsigned long        initrd_size;
>>> +    const char        *name;
>>> +};
>>> +
>>> +#endif /* KVM__KVM_ARCH_H */
>>> diff --git a/tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h b/tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h
>>> new file mode 100644
>>> index 0000000..64e4510
>>> --- /dev/null
>>> +++ b/tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h
>>> @@ -0,0 +1,66 @@
>>> +/*
>>> + * PPC64 cpu-specific definitions
>>> + *
>>> + * Copyright 2011 Matt Evans <matt@xxxxxxxxxx>, IBM Corporation.
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify it
>>> + * under the terms of the GNU General Public License version 2 as published
>>> + * by the Free Software Foundation.
>>> + */
>>> +
>>> +#ifndef KVM__KVM_CPU_ARCH_H
>>> +#define KVM__KVM_CPU_ARCH_H
>>> +
>>> +/* Architecture-specific kvm_cpu definitions. */
>>> +
>>> +#include <linux/kvm.h>    /* for struct kvm_regs */
>>> +
>>> +#include <pthread.h>
>>> +
>>> +#define MSR_SF        (1UL<<63)
>>> +#define MSR_HV        (1UL<<60)
>>> +#define MSR_VEC        (1UL<<25)
>>> +#define MSR_VSX        (1UL<<23)
>>> +#define MSR_POW        (1UL<<18)
>>> +#define MSR_EE        (1UL<<15)
>>> +#define MSR_PR        (1UL<<14)
>>> +#define MSR_FP        (1UL<<13)
>>> +#define MSR_ME        (1UL<<12)
>>> +#define MSR_FE0        (1UL<<11)
>>> +#define MSR_SE        (1UL<<10)
>>> +#define MSR_BE        (1UL<<9)
>>> +#define MSR_FE1        (1UL<<8)
>>> +#define MSR_IR        (1UL<<5)
>>> +#define MSR_DR        (1UL<<4)
>>> +#define MSR_PMM        (1UL<<2)
>>> +#define MSR_RI        (1UL<<1)
>>> +#define MSR_LE        (1UL<<0)
>>> +
>>> +struct kvm;
>>> +
>>> +struct kvm_cpu {
>>> +    pthread_t        thread;        /* VCPU thread */
>>> +
>>> +    unsigned long        cpu_id;
>>> +
>>> +    struct kvm        *kvm;        /* parent KVM */
>>> +    int            vcpu_fd;    /* For VCPU ioctls() */
>>> +    struct kvm_run        *kvm_run;
>>> +
>>> +    struct kvm_regs        regs;
>>> +    struct kvm_sregs    sregs;
>>> +    struct kvm_fpu        fpu;
>>> +
>>> +    u8            is_running;
>>> +    u8            paused;
>>> +    u8            needs_nmi;
>>> +    /*
>>> +     * Although PPC KVM doesn't yet support coalesced MMIO, generic code
>>> +     * needs this in our kvm_cpu:
>>> +     */
>>> +    struct kvm_coalesced_mmio_ring  *ring;
>>> +};
>>> +
>>> +void kvm_cpu__irq(struct kvm_cpu *vcpu, int pin, int level);
>>> +
>>> +#endif /* KVM__KVM_CPU_ARCH_H */
>>> diff --git a/tools/kvm/powerpc/ioport.c b/tools/kvm/powerpc/ioport.c
>>> new file mode 100644
>>> index 0000000..a8e4dc3
>>> --- /dev/null
>>> +++ b/tools/kvm/powerpc/ioport.c
>>> @@ -0,0 +1,18 @@
>>> +/*
>>> + * PPC64 ioport platform setup.  There isn't any! :-)
>>> + *
>>> + * Copyright 2011 Matt Evans <matt@xxxxxxxxxx>, IBM Corporation.
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify it
>>> + * under the terms of the GNU General Public License version 2 as published
>>> + * by the Free Software Foundation.
>>> + */
>>> +
>>> +#include "kvm/ioport.h"
>>> +
>>> +#include <stdlib.h>
>>> +
>>> +void ioport__setup_arch(void)
>>> +{
>>> +    /* PPC has no legacy ioports to set up */
>>> +}
>>> diff --git a/tools/kvm/powerpc/irq.c b/tools/kvm/powerpc/irq.c
>>> new file mode 100644
>>> index 0000000..46aa64f
>>> --- /dev/null
>>> +++ b/tools/kvm/powerpc/irq.c
>>> @@ -0,0 +1,40 @@
>>> +/*
>>> + * PPC64 IRQ routines
>>> + *
>>> + * Copyright 2011 Matt Evans <matt@xxxxxxxxxx>, IBM Corporation.
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify it
>>> + * under the terms of the GNU General Public License version 2 as published
>>> + * by the Free Software Foundation.
>>> + */
>>> +
>>> +#include "kvm/irq.h"
>>> +#include "kvm/kvm.h"
>>> +#include "kvm/util.h"
>>> +
>>> +#include <linux/types.h>
>>> +#include <linux/rbtree.h>
>>> +#include <linux/list.h>
>>> +#include <linux/kvm.h>
>>> +#include <sys/ioctl.h>
>>> +
>>> +#include <stddef.h>
>>> +#include <stdlib.h>
>>> +
>>> +int irq__register_device(u32 dev, u8 *num, u8 *pin, u8 *line)
>>> +{
>>> +    fprintf(stderr, "irq__register_device(%d, [%d], [%d], [%d]\n",
>>> +        dev, *num, *pin, *line);
>>> +    return 0;
>>> +}
>>> +
>>> +void irq__init(struct kvm *kvm)
>>> +{
>>> +    fprintf(stderr, __func__);
>>> +}
>>> +
>>> +int irq__add_msix_route(struct kvm *kvm, struct msi_msg *msg)
>>> +{
>>> +    die(__FUNCTION__);
>>> +    return 0;
>>> +}
>>> diff --git a/tools/kvm/powerpc/kvm-cpu.c b/tools/kvm/powerpc/kvm-cpu.c
>>> new file mode 100644
>>> index 0000000..ea99666
>>> --- /dev/null
>>> +++ b/tools/kvm/powerpc/kvm-cpu.c
>>> @@ -0,0 +1,233 @@
>>> +/*
>>> + * PPC64 processor support
>>> + *
>>> + * Copyright 2011 Matt Evans <matt@xxxxxxxxxx>, IBM Corporation.
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify it
>>> + * under the terms of the GNU General Public License version 2 as published
>>> + * by the Free Software Foundation.
>>> + */
>>> +
>>> +#include "kvm/kvm-cpu.h"
>>> +
>>> +#include "kvm/symbol.h"
>>> +#include "kvm/util.h"
>>> +#include "kvm/kvm.h"
>>> +
>>> +#include <sys/ioctl.h>
>>> +#include <sys/mman.h>
>>> +#include <signal.h>
>>> +#include <stdlib.h>
>>> +#include <string.h>
>>> +#include <errno.h>
>>> +#include <stdio.h>
>>> +
>>> +static int debug_fd;
>>> +
>>> +void kvm_cpu__set_debug_fd(int fd)
>>> +{
>>> +    debug_fd = fd;
>>> +}
>>> +
>>> +int kvm_cpu__get_debug_fd(void)
>>> +{
>>> +    return debug_fd;
>>> +}
>>> +
>>> +static struct kvm_cpu *kvm_cpu__new(struct kvm *kvm)
>>> +{
>>> +    struct kvm_cpu *vcpu;
>>> +
>>> +    vcpu        = calloc(1, sizeof *vcpu);
>>> +    if (!vcpu)
>>> +        return NULL;
>>> +
>>> +    vcpu->kvm    = kvm;
>>> +
>>> +    return vcpu;
>>> +}
>>> +
>>> +void kvm_cpu__delete(struct kvm_cpu *vcpu)
>>> +{
>>> +    free(vcpu);
>>> +}
>>> +
>>> +struct kvm_cpu *kvm_cpu__init(struct kvm *kvm, unsigned long cpu_id)
>>> +{
>>> +    struct kvm_cpu *vcpu;
>>> +    int mmap_size;
>>> +    struct kvm_enable_cap papr_cap = { .cap = KVM_CAP_PPC_PAPR };
>>> +
>>> +    vcpu        = kvm_cpu__new(kvm);
>>> +    if (!vcpu)
>>> +        return NULL;
>>> +
>>> +    vcpu->cpu_id    = cpu_id;
>>> +
>>> +    vcpu->vcpu_fd = ioctl(vcpu->kvm->vm_fd, KVM_CREATE_VCPU, cpu_id);
>>> +    if (vcpu->vcpu_fd < 0)
>>> +        die_perror("KVM_CREATE_VCPU ioctl");
>>> +
>>> +    mmap_size = ioctl(vcpu->kvm->sys_fd, KVM_GET_VCPU_MMAP_SIZE, 0);
>>> +    if (mmap_size < 0)
>>> +        die_perror("KVM_GET_VCPU_MMAP_SIZE ioctl");
>>> +
>>> +    vcpu->kvm_run = mmap(NULL, mmap_size, PROT_RW, MAP_SHARED, vcpu->vcpu_fd, 0);
>>> +    if (vcpu->kvm_run == MAP_FAILED)
>>> +        die("unable to mmap vcpu fd");
>>> +
>>> +    ioctl(vcpu->vcpu_fd, KVM_ENABLE_CAP, &papr_cap);
>> 
>> Have you tried running this on PR KVM? That should also need HIOR synchronization.
> 
> I have, but only briefly and I admit I built it from a random tree I had lying
> around, certainly stale, and immediately hit some "can't emulate MMIO" errors on
> some stdu instructions.  I will give it another go with your tree and see if I
> can get it working with kvmtool, it would be very cool for that to work.

Yup, it would also make your work executable to people outside of IBM :)


Alex

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [KVM ARM]     [KVM ia64]     [KVM ppc]     [Virtualization Tools]     [Spice Development]     [Libvirt]     [Libvirt Users]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite Questions]     [Linux Kernel]     [Linux SCSI]     [XFree86]
  Powered by Linux