Hi, On 04/05/2017 15:23, Christoffer Dall wrote: > On Thu, May 04, 2017 at 01:44:21PM +0200, Eric Auger wrote: >> Add description for how to access ITS registers and how to save/restore >> ITS tables into/from memory. >> >> Signed-off-by: Eric Auger <eric.auger@xxxxxxxxxx> >> >> --- >> v5 -> v6: >> - add restoration ordering >> - 256B -> 256 Byte aligned >> - DTE Size is number of bits for the EVENTID >> - s/GITS_READR/GITS_CREADR >> >> v4 -> v5: >> - take into account Christoffer's comments >> - pending table save on GICV3 side now >> >> v3 -> v4: >> - take into account Peter's comments: >> - typos >> - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0 >> - add a validity bit in DTE >> - document all fields in CTE and ITE >> - document ABI revision >> - take into account Andre's comments: >> - document restrictions about GITS_CREADR writing and GITS_IIDR >> - document -EBUSY error if one or more VCPUS are runnning >> - document 64b registers only can be accessed with 64b access >> - itt_addr field matches bits [51:8] of the itt_addr >> >> v1 -> v2: >> - DTE and ITE now are 8 bytes >> - DTE and ITE now indexed by deviceid/eventid >> - use ITE name instead of ITTE >> - mentions ITT_addr matches bits [51:8] of the actual address >> - mentions LE layout >> --- >> Documentation/virtual/kvm/devices/arm-vgic-its.txt | 117 +++++++++++++++++++++ >> 1 file changed, 117 insertions(+) >> >> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt >> index 6081a5b..863c359 100644 >> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt >> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt >> @@ -32,7 +32,124 @@ Groups: >> KVM_DEV_ARM_VGIC_CTRL_INIT >> request the initialization of the ITS, no additional parameter in >> kvm_device_attr.addr. >> + >> + KVM_DEV_ARM_ITS_SAVE_TABLES >> + save the ITS table data into guest RAM, at the location provisioned >> + by the guest in corresponding registers/table entries. >> + >> + The layout of the tables in guest memory defines an ABI. The entries >> + are laid out in little endian format as described in the last paragraph. >> + >> + KVM_DEV_ARM_ITS_RESTORE_TABLES >> + restore the ITS tables from guest RAM to ITS internal structures. >> + >> + The GICV3 must be restored before the ITS and all ITS registers but >> + the GITS_CTLR must be restored before restoring the ITS tables. >> + >> + The GITS_IIDR read-only register must also be restored before >> + the table restore as the IIDR revision field encodes the ABI revision. > > the table restore? "before calling KVM_DEV_ARM_ITS_RESTORE_TABLES" ? > (there's a potential confusion between calling this ioctl and simply > restoring the memory containing the tables). OK > >> + >> + The expected ordering when restoring the GICv3/ITS is described in section >> + "ITS Restoration Sequence". >> + >> Errors: >> -ENXIO: ITS not properly configured as required prior to setting >> this attribute >> -ENOMEM: Memory shortage when allocating ITS internal data >> + -EINVAL: Inconsistent restored data >> + -EFAULT: Invalid guest ram access >> + -EBUSY: One or more VCPUS are running >> + >> + KVM_DEV_ARM_VGIC_GRP_ITS_REGS >> + Attributes: >> + The attr field of kvm_device_attr encodes the offset of the >> + ITS register, relative to the ITS control frame base address >> + (ITS_base). >> + >> + kvm_device_attr.addr points to a __u64 value whatever the width >> + of the addressed register (32/64 bits). 64 bit registers can only >> + be accessed with full length. >> + >> + Writes to read-only registers are ignored by the kernel except for: >> + - GITS_CREADR. It needs to be restored otherwise commands in the queue >> + will be re-executed after restoring CWRITER. GITS_CREADR must be >> + restored before restoring the GITS_CTLR which is likely to enable the >> + ITS. Also it needs to be restored after GITS_CBASER since a write to >> + GITS_CBASER resets GITS_CREADR. > > uber nit: s/needs to be/must be/g in all occurences above. OK > >> + - GITS_IIDR. Its Revision field encodes the table layout ABI revision. > > s/Its/The/ OK. Argh, I meant its (~his) ;-) > the revision of the table layout ABI. > >> + In the future we might implement direct injection of virtual LPIS. > > LPIs OK > >> + This will require an upgrade of the table layout and an evolution of >> + the ABI. GITS_IIDR must be restored before the table restoration. > > before calling KVM_DEV_ARM_ITS_RESTORE_TABLES. OK > >> + >> + For other registers, getting or setting a register has the same >> + effect as reading/writing the register on real hardware. >> + Errors: >> + -ENXIO: Offset does not correspond to any supported register >> + -EFAULT: Invalid user pointer for attr->addr >> + -EINVAL: Offset is not 64-bit aligned >> + -EBUSY: one or more VCPUS are running >> + >> + ITS Restoration Sequence: > > s/Restoration/restore/ OK > >> + ------------------------- >> + >> +The following ordering must be followed when restoring the GIC and the ITS: >> +a) restore all guest memory and create vcpus >> +b) initialize the GIC and ITS (KVM_DEV_ARM_VGIC_CTRL_INIT) > > are these really coupled or can you do everything related to the gic > first, and then everything related to the ITS afterwards? no they aren't. > >> +c) provide GIC and ITS base addresses (KVM_DEV_ARM_VGIC_GRP_ADDR) > > same no they aren't > >> +d) restore all redistributors >> +e) restore the ITS in the following order: >> + 1. Restore GITS_CBASER >> + 2. Restore all other GITS_ registers, except GITS_CTLR! >> + 3. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES) >> + 4. Restore GITS_CTLR >> +f) start vcpus > > I would omit (f) from here. In that case I think it would make sense to describe the restore sequence for GICV3 RDIST in GICV3 device doc and not here. This would alleviate this very description. restoring RDIST regs require a previous GICV3 KVM_DEV_ARM_VGIC_CTRL_INIT and KVM_DEV_ARM_VGIC_GRP_ADDR I think. So description here would become " a) restore all guest memory and create vcpus b) restore all redistributors c) initialize the ITS and then provide its base address (KVM_DEV_ARM_VGIC_CTRL_INIT, KVM_DEV_ARM_VGIC_GRP_ADDR) c) restore the ITS in the following order: 1. Restore GITS_CBASER 2. Restore all other GITS_ registers, except GITS_CTLR! 3. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES) 4. Restore GITS_CTLR Then vcpus can be started. " What do you think? > >> + >> + ITS Table ABI REV0: >> + ------------------- >> + >> + Revision 0 of the ABI only supports physical LPIs. >> + >> + The device table and ITT are indexed by the deviceid and eventid, >> + respectively. The collection table is not indexed by collectionid: >> + CTE are written in the table in the order of collection creation. All > > nit: CTEs OK > >> + entries are 8 bytes. >> + >> + Device Table Entry (DTE): >> + >> + bits: | 63| 62 ... 49 | 48 ... 5 | 4 ... 0 | >> + values: | V | next | ITT_addr | Size | >> + >> + where; >> + - V indicates whether the entry is valid. If not, other fields >> + are not meaningful. >> + - next: equals to 0 if this entry is the last one; otherwise it >> + corresponds to the deviceid offset to the next DTE, capped by >> + 2^14 -1. >> + - ITT_addr matches bits [51:8] of the ITT address (256 Byte aligned). >> + - Size specifies the supported number of bits for the eventid, >> + minus one >> + >> + Collection Table Entry (CTE): >> + >> + bits: | 63| 62 .. 52 | 51 ... 16 | 15 ... 0 | >> + values: | V | RES0 | RDBase | ICID | >> + >> + where: >> + - V indicates whether the entry is valid. If not, other fields are >> + not meaningful. >> + - RES0: reserved field with Should-Be-Zero-or-Preserved behavior. >> + - RDBase is the PE number (GICR_TYPER.Processor_Number semantic), >> + - ICID is the collection ID >> + >> + Interrupt Translation Entry (ITE): >> + >> + bits: | 63 ... 48 | 47 ... 16 | 15 ... 0 | >> + values: | next | pINTID | ICID | >> + >> + where: >> + - next: equals to 0 if this entry is the last one; otherwise it corresponds >> + to the eventid offset to the next ITE capped by 2^16 -1. >> + - pINTID is the physical LPI ID; if zero, it means the entry is not valid >> + and other fields are not meaningful. >> + - ICID is the collection ID >> + >> -- >> 2.5.5 >> > > The only outstanding issue here is the restore order part. I don't wont > to specify a stronger order than absolutely necessary here. see above Thanks Eric > > Otherwise this generally looks good. > > Thanks, > -Christoffer > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@xxxxxxxxxxxxxxxxxxx > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel >
