On 15.12.2011, at 02:27, Matt Evans wrote: > Heya Alex, > > On 13/12/11 19:23, Alexander Graf wrote: >> >> On 13.12.2011, at 08:00, Matt Evans <matt@xxxxxxxxxx> wrote: >> >>> This patch adds a new arch directory, powerpc, basic file structure, register >>> setup and where necessary stubs out arch-specific functions (e.g. interrupts, >>> runloop exits) that later patches will provide. The target is an >>> SPAPR-compliant PPC64 machine (i.e. pSeries); there is no support for PPC32 or >>> 'bare metal' PPC64 guests as yet. Subsequent patches implement the hcalls and >>> RTAS required to boot SPAPR pSeries kernels. >>> >>> Memory is mapped from hugetlbfs (as that is currently required by upstream PPC64 >>> HV-mode KVM). The mapping of a VRMA region is yet to be implemented; this is >>> only necessary on processors that don't support VRMA, e.g. <= P6. Work is >>> therefore needed to get this going on pre-P7 CPUs. >>> >>> Processor state is set up as a guest kernel would expect (both primary and >>> secondaries), and SMP is fully supported. >>> >>> Finally, support is added for simply loading flat binary kernels (plus initrd). >>> (bzImages are not used on PPC, and this series does not add zImage support or an >>> ELF loader.) The intention is to later support loading firmware such as SLOF. >>> >>> Signed-off-by: Matt Evans <matt@xxxxxxxxxx> >>> --- >>> tools/kvm/Makefile | 10 + >>> tools/kvm/kvm.c | 3 + >>> tools/kvm/powerpc/include/kvm/barrier.h | 6 + >>> tools/kvm/powerpc/include/kvm/kvm-arch.h | 72 ++++++++ >>> tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h | 66 ++++++++ >>> tools/kvm/powerpc/ioport.c | 18 ++ >>> tools/kvm/powerpc/irq.c | 40 +++++ >>> tools/kvm/powerpc/kvm-cpu.c | 233 ++++++++++++++++++++++++++ >>> tools/kvm/powerpc/kvm.c | 187 +++++++++++++++++++++ >>> 9 files changed, 635 insertions(+), 0 deletions(-) >>> create mode 100644 tools/kvm/powerpc/include/kvm/barrier.h >>> create mode 100644 tools/kvm/powerpc/include/kvm/kvm-arch.h >>> create mode 100644 tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h >>> create mode 100644 tools/kvm/powerpc/ioport.c >>> create mode 100644 tools/kvm/powerpc/irq.c >>> create mode 100644 tools/kvm/powerpc/kvm-cpu.c >>> create mode 100644 tools/kvm/powerpc/kvm.c >>> >>> diff --git a/tools/kvm/Makefile b/tools/kvm/Makefile >>> index 2bf70c9..3f1e84a 100644 >>> --- a/tools/kvm/Makefile >>> +++ b/tools/kvm/Makefile >>> @@ -124,6 +124,16 @@ ifeq ($(ARCH),x86) >>> OTHEROBJS += x86/bios/bios-rom.o >>> ARCH_INCLUDE := x86/include >>> endif >>> +# POWER/ppc: Actually only support ppc64 currently. >> >> Why? I usually run ppc32 user land. Doesn't that expose 'ppc' here? > > Not quite sure what you mean here; do you mean 32bit distro? (Will still get 'ppc64' from a 64-bit kernel.) Eh. Yes. Sorry, my bad. > There is clearly some work required here to determine what to build for when we > eventually support PPC32 guests/hosts though I'm not sure how that will look > yet. This is designed to break if you build on a 32bit kernel, as if it DID > build, it wouldn't run anyway. (It's building -m64 too... Yeah, running -M pseries on PPC32 hosts doesn't make sense really. > >>> +ifeq ($(uname_M), ppc64) >>> + DEFINES += -DCONFIG_PPC >>> + OBJS += powerpc/ioport.o >>> + OBJS += powerpc/irq.o >>> + OBJS += powerpc/kvm.o >>> + OBJS += powerpc/kvm-cpu.o >>> + ARCH_INCLUDE := powerpc/include >>> + CFLAGS += -m64 > > ...here.) > >>> +endif >>> >>> ### >>> >>> diff --git a/tools/kvm/kvm.c b/tools/kvm/kvm.c >>> index 35ca2c5..3fb46f6 100644 >>> --- a/tools/kvm/kvm.c >>> +++ b/tools/kvm/kvm.c >>> @@ -49,6 +49,9 @@ const char *kvm_exit_reasons[] = { >>> DEFINE_KVM_EXIT_REASON(KVM_EXIT_DCR), >>> DEFINE_KVM_EXIT_REASON(KVM_EXIT_NMI), >>> DEFINE_KVM_EXIT_REASON(KVM_EXIT_INTERNAL_ERROR), >>> +#ifdef CONFIG_PPC64 >>> + DEFINE_KVM_EXIT_REASON(KVM_EXIT_PAPR_HCALL), >>> +#endif >>> }; >>> >>> extern struct kvm *kvm; >>> diff --git a/tools/kvm/powerpc/include/kvm/barrier.h b/tools/kvm/powerpc/include/kvm/barrier.h >>> new file mode 100644 >>> index 0000000..bc7d179 >>> --- /dev/null >>> +++ b/tools/kvm/powerpc/include/kvm/barrier.h >>> @@ -0,0 +1,6 @@ >>> +#ifndef _KVM_BARRIER_H_ >>> +#define _KVM_BARRIER_H_ >>> + >>> +#include <asm/system.h> >>> + >>> +#endif /* _KVM_BARRIER_H_ */ >>> diff --git a/tools/kvm/powerpc/include/kvm/kvm-arch.h b/tools/kvm/powerpc/include/kvm/kvm-arch.h >>> new file mode 100644 >>> index 0000000..da61774 >>> --- /dev/null >>> +++ b/tools/kvm/powerpc/include/kvm/kvm-arch.h >>> @@ -0,0 +1,72 @@ >>> +/* >>> + * PPC64 architecture-specific definitions >>> + * >>> + * Copyright 2011 Matt Evans <matt@xxxxxxxxxx>, IBM Corporation. >>> + * >>> + * This program is free software; you can redistribute it and/or modify it >>> + * under the terms of the GNU General Public License version 2 as published >>> + * by the Free Software Foundation. >>> + */ >>> + >>> +#ifndef KVM__KVM_ARCH_H >>> +#define KVM__KVM_ARCH_H >>> + >>> +#include <stdbool.h> >>> +#include <linux/types.h> >>> +#include <time.h> >>> + >>> +#define KVM_NR_CPUS (255) >> >> Why? > > Good question; that's arbitrary & cut-paste I missed. :-) > > I'll make this 1024, to match the max sensible NR_CPUS in the PPC64 kernel > (which in turn limits KVM_MAX_VCPUS). I thought Sasha converted this to a queryable interface? > >>> + >>> +/* >>> + * MMIO lives after RAM, but it'd be nice if it didn't constantly move. >>> + * Choose a suitably high address, e.g. 63T... This limits RAM size. >>> + */ >>> +#define PPC_MMIO_START 0x3F0000000000UL >>> +#define PPC_MMIO_SIZE 0x010000000000UL >>> + >>> +#define KERNEL_LOAD_ADDR 0x0000000000000000 >>> +#define KERNEL_START_ADDR 0x0000000000000000 >>> +#define KERNEL_SECONDARY_START_ADDR 0x0000000000000060 >>> +#define INITRD_LOAD_ADDR 0x0000000002800000 >>> + >>> +#define FDT_MAX_SIZE 0x10000 >>> +#define RTAS_MAX_SIZE 0x10000 >>> + >>> +#define TIMEBASE_FREQ 512000000ULL >>> + >>> +#define KVM_MMIO_START PPC_MMIO_START >>> + >>> +/* >>> + * This is the address that pci_get_io_space_block() starts allocating >>> + * from. Note that this is a PCI bus address. >>> + */ >>> +#define KVM_PCI_MMIO_AREA 0x1000000 >>> + >>> +struct kvm { >>> + int sys_fd; /* For system ioctls(), i.e. /dev/kvm */ >>> + int vm_fd; /* For VM ioctls() */ >>> + timer_t timerid; /* Posix timer for interrupts */ >>> + >>> + int nrcpus; /* Number of cpus to run */ >>> + >>> + u32 mem_slots; /* for KVM_SET_USER_MEMORY_REGION */ >>> + >>> + u64 ram_size; >>> + void *ram_start; >>> + >>> + bool nmi_disabled; >>> + >>> + bool single_step; >>> + >>> + const char *vmlinux; >>> + struct disk_image **disks; >>> + int nr_disks; >>> + unsigned long rtas_gra; >>> + unsigned long rtas_size; >>> + unsigned long fdt_gra; >>> + unsigned long initrd_gra; >>> + unsigned long initrd_size; >>> + const char *name; >>> +}; >>> + >>> +#endif /* KVM__KVM_ARCH_H */ >>> diff --git a/tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h b/tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h >>> new file mode 100644 >>> index 0000000..64e4510 >>> --- /dev/null >>> +++ b/tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h >>> @@ -0,0 +1,66 @@ >>> +/* >>> + * PPC64 cpu-specific definitions >>> + * >>> + * Copyright 2011 Matt Evans <matt@xxxxxxxxxx>, IBM Corporation. >>> + * >>> + * This program is free software; you can redistribute it and/or modify it >>> + * under the terms of the GNU General Public License version 2 as published >>> + * by the Free Software Foundation. >>> + */ >>> + >>> +#ifndef KVM__KVM_CPU_ARCH_H >>> +#define KVM__KVM_CPU_ARCH_H >>> + >>> +/* Architecture-specific kvm_cpu definitions. */ >>> + >>> +#include <linux/kvm.h> /* for struct kvm_regs */ >>> + >>> +#include <pthread.h> >>> + >>> +#define MSR_SF (1UL<<63) >>> +#define MSR_HV (1UL<<60) >>> +#define MSR_VEC (1UL<<25) >>> +#define MSR_VSX (1UL<<23) >>> +#define MSR_POW (1UL<<18) >>> +#define MSR_EE (1UL<<15) >>> +#define MSR_PR (1UL<<14) >>> +#define MSR_FP (1UL<<13) >>> +#define MSR_ME (1UL<<12) >>> +#define MSR_FE0 (1UL<<11) >>> +#define MSR_SE (1UL<<10) >>> +#define MSR_BE (1UL<<9) >>> +#define MSR_FE1 (1UL<<8) >>> +#define MSR_IR (1UL<<5) >>> +#define MSR_DR (1UL<<4) >>> +#define MSR_PMM (1UL<<2) >>> +#define MSR_RI (1UL<<1) >>> +#define MSR_LE (1UL<<0) >>> + >>> +struct kvm; >>> + >>> +struct kvm_cpu { >>> + pthread_t thread; /* VCPU thread */ >>> + >>> + unsigned long cpu_id; >>> + >>> + struct kvm *kvm; /* parent KVM */ >>> + int vcpu_fd; /* For VCPU ioctls() */ >>> + struct kvm_run *kvm_run; >>> + >>> + struct kvm_regs regs; >>> + struct kvm_sregs sregs; >>> + struct kvm_fpu fpu; >>> + >>> + u8 is_running; >>> + u8 paused; >>> + u8 needs_nmi; >>> + /* >>> + * Although PPC KVM doesn't yet support coalesced MMIO, generic code >>> + * needs this in our kvm_cpu: >>> + */ >>> + struct kvm_coalesced_mmio_ring *ring; >>> +}; >>> + >>> +void kvm_cpu__irq(struct kvm_cpu *vcpu, int pin, int level); >>> + >>> +#endif /* KVM__KVM_CPU_ARCH_H */ >>> diff --git a/tools/kvm/powerpc/ioport.c b/tools/kvm/powerpc/ioport.c >>> new file mode 100644 >>> index 0000000..a8e4dc3 >>> --- /dev/null >>> +++ b/tools/kvm/powerpc/ioport.c >>> @@ -0,0 +1,18 @@ >>> +/* >>> + * PPC64 ioport platform setup. There isn't any! :-) >>> + * >>> + * Copyright 2011 Matt Evans <matt@xxxxxxxxxx>, IBM Corporation. >>> + * >>> + * This program is free software; you can redistribute it and/or modify it >>> + * under the terms of the GNU General Public License version 2 as published >>> + * by the Free Software Foundation. >>> + */ >>> + >>> +#include "kvm/ioport.h" >>> + >>> +#include <stdlib.h> >>> + >>> +void ioport__setup_arch(void) >>> +{ >>> + /* PPC has no legacy ioports to set up */ >>> +} >>> diff --git a/tools/kvm/powerpc/irq.c b/tools/kvm/powerpc/irq.c >>> new file mode 100644 >>> index 0000000..46aa64f >>> --- /dev/null >>> +++ b/tools/kvm/powerpc/irq.c >>> @@ -0,0 +1,40 @@ >>> +/* >>> + * PPC64 IRQ routines >>> + * >>> + * Copyright 2011 Matt Evans <matt@xxxxxxxxxx>, IBM Corporation. >>> + * >>> + * This program is free software; you can redistribute it and/or modify it >>> + * under the terms of the GNU General Public License version 2 as published >>> + * by the Free Software Foundation. >>> + */ >>> + >>> +#include "kvm/irq.h" >>> +#include "kvm/kvm.h" >>> +#include "kvm/util.h" >>> + >>> +#include <linux/types.h> >>> +#include <linux/rbtree.h> >>> +#include <linux/list.h> >>> +#include <linux/kvm.h> >>> +#include <sys/ioctl.h> >>> + >>> +#include <stddef.h> >>> +#include <stdlib.h> >>> + >>> +int irq__register_device(u32 dev, u8 *num, u8 *pin, u8 *line) >>> +{ >>> + fprintf(stderr, "irq__register_device(%d, [%d], [%d], [%d]\n", >>> + dev, *num, *pin, *line); >>> + return 0; >>> +} >>> + >>> +void irq__init(struct kvm *kvm) >>> +{ >>> + fprintf(stderr, __func__); >>> +} >>> + >>> +int irq__add_msix_route(struct kvm *kvm, struct msi_msg *msg) >>> +{ >>> + die(__FUNCTION__); >>> + return 0; >>> +} >>> diff --git a/tools/kvm/powerpc/kvm-cpu.c b/tools/kvm/powerpc/kvm-cpu.c >>> new file mode 100644 >>> index 0000000..ea99666 >>> --- /dev/null >>> +++ b/tools/kvm/powerpc/kvm-cpu.c >>> @@ -0,0 +1,233 @@ >>> +/* >>> + * PPC64 processor support >>> + * >>> + * Copyright 2011 Matt Evans <matt@xxxxxxxxxx>, IBM Corporation. >>> + * >>> + * This program is free software; you can redistribute it and/or modify it >>> + * under the terms of the GNU General Public License version 2 as published >>> + * by the Free Software Foundation. >>> + */ >>> + >>> +#include "kvm/kvm-cpu.h" >>> + >>> +#include "kvm/symbol.h" >>> +#include "kvm/util.h" >>> +#include "kvm/kvm.h" >>> + >>> +#include <sys/ioctl.h> >>> +#include <sys/mman.h> >>> +#include <signal.h> >>> +#include <stdlib.h> >>> +#include <string.h> >>> +#include <errno.h> >>> +#include <stdio.h> >>> + >>> +static int debug_fd; >>> + >>> +void kvm_cpu__set_debug_fd(int fd) >>> +{ >>> + debug_fd = fd; >>> +} >>> + >>> +int kvm_cpu__get_debug_fd(void) >>> +{ >>> + return debug_fd; >>> +} >>> + >>> +static struct kvm_cpu *kvm_cpu__new(struct kvm *kvm) >>> +{ >>> + struct kvm_cpu *vcpu; >>> + >>> + vcpu = calloc(1, sizeof *vcpu); >>> + if (!vcpu) >>> + return NULL; >>> + >>> + vcpu->kvm = kvm; >>> + >>> + return vcpu; >>> +} >>> + >>> +void kvm_cpu__delete(struct kvm_cpu *vcpu) >>> +{ >>> + free(vcpu); >>> +} >>> + >>> +struct kvm_cpu *kvm_cpu__init(struct kvm *kvm, unsigned long cpu_id) >>> +{ >>> + struct kvm_cpu *vcpu; >>> + int mmap_size; >>> + struct kvm_enable_cap papr_cap = { .cap = KVM_CAP_PPC_PAPR }; >>> + >>> + vcpu = kvm_cpu__new(kvm); >>> + if (!vcpu) >>> + return NULL; >>> + >>> + vcpu->cpu_id = cpu_id; >>> + >>> + vcpu->vcpu_fd = ioctl(vcpu->kvm->vm_fd, KVM_CREATE_VCPU, cpu_id); >>> + if (vcpu->vcpu_fd < 0) >>> + die_perror("KVM_CREATE_VCPU ioctl"); >>> + >>> + mmap_size = ioctl(vcpu->kvm->sys_fd, KVM_GET_VCPU_MMAP_SIZE, 0); >>> + if (mmap_size < 0) >>> + die_perror("KVM_GET_VCPU_MMAP_SIZE ioctl"); >>> + >>> + vcpu->kvm_run = mmap(NULL, mmap_size, PROT_RW, MAP_SHARED, vcpu->vcpu_fd, 0); >>> + if (vcpu->kvm_run == MAP_FAILED) >>> + die("unable to mmap vcpu fd"); >>> + >>> + ioctl(vcpu->vcpu_fd, KVM_ENABLE_CAP, &papr_cap); >> >> Have you tried running this on PR KVM? That should also need HIOR synchronization. > > I have, but only briefly and I admit I built it from a random tree I had lying > around, certainly stale, and immediately hit some "can't emulate MMIO" errors on > some stdu instructions. I will give it another go with your tree and see if I > can get it working with kvmtool, it would be very cool for that to work. Yup, it would also make your work executable to people outside of IBM :) Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html