On Thu, May 04, 2017 at 04:50:23PM +0200, Auger Eric wrote: > Hi, > > On 04/05/2017 15:23, Christoffer Dall wrote: > > On Thu, May 04, 2017 at 01:44:21PM +0200, Eric Auger wrote: > >> Add description for how to access ITS registers and how to save/restore > >> ITS tables into/from memory. > >> > >> Signed-off-by: Eric Auger <eric.auger@xxxxxxxxxx> > >> > >> --- > >> v5 -> v6: > >> - add restoration ordering > >> - 256B -> 256 Byte aligned > >> - DTE Size is number of bits for the EVENTID > >> - s/GITS_READR/GITS_CREADR > >> > >> v4 -> v5: > >> - take into account Christoffer's comments > >> - pending table save on GICV3 side now > >> > >> v3 -> v4: > >> - take into account Peter's comments: > >> - typos > >> - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0 > >> - add a validity bit in DTE > >> - document all fields in CTE and ITE > >> - document ABI revision > >> - take into account Andre's comments: > >> - document restrictions about GITS_CREADR writing and GITS_IIDR > >> - document -EBUSY error if one or more VCPUS are runnning > >> - document 64b registers only can be accessed with 64b access > >> - itt_addr field matches bits [51:8] of the itt_addr > >> > >> v1 -> v2: > >> - DTE and ITE now are 8 bytes > >> - DTE and ITE now indexed by deviceid/eventid > >> - use ITE name instead of ITTE > >> - mentions ITT_addr matches bits [51:8] of the actual address > >> - mentions LE layout > >> --- > >> Documentation/virtual/kvm/devices/arm-vgic-its.txt | 117 +++++++++++++++++++++ > >> 1 file changed, 117 insertions(+) > >> > >> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt > >> index 6081a5b..863c359 100644 > >> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt > >> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt > >> @@ -32,7 +32,124 @@ Groups: > >> KVM_DEV_ARM_VGIC_CTRL_INIT > >> request the initialization of the ITS, no additional parameter in > >> kvm_device_attr.addr. > >> + > >> + KVM_DEV_ARM_ITS_SAVE_TABLES > >> + save the ITS table data into guest RAM, at the location provisioned > >> + by the guest in corresponding registers/table entries. > >> + > >> + The layout of the tables in guest memory defines an ABI. The entries > >> + are laid out in little endian format as described in the last paragraph. > >> + > >> + KVM_DEV_ARM_ITS_RESTORE_TABLES > >> + restore the ITS tables from guest RAM to ITS internal structures. > >> + > >> + The GICV3 must be restored before the ITS and all ITS registers but > >> + the GITS_CTLR must be restored before restoring the ITS tables. > >> + > >> + The GITS_IIDR read-only register must also be restored before > >> + the table restore as the IIDR revision field encodes the ABI revision. > > > > the table restore? "before calling KVM_DEV_ARM_ITS_RESTORE_TABLES" ? > > (there's a potential confusion between calling this ioctl and simply > > restoring the memory containing the tables). > OK > > > >> + > >> + The expected ordering when restoring the GICv3/ITS is described in section > >> + "ITS Restoration Sequence". > >> + > >> Errors: > >> -ENXIO: ITS not properly configured as required prior to setting > >> this attribute > >> -ENOMEM: Memory shortage when allocating ITS internal data > >> + -EINVAL: Inconsistent restored data > >> + -EFAULT: Invalid guest ram access > >> + -EBUSY: One or more VCPUS are running > >> + > >> + KVM_DEV_ARM_VGIC_GRP_ITS_REGS > >> + Attributes: > >> + The attr field of kvm_device_attr encodes the offset of the > >> + ITS register, relative to the ITS control frame base address > >> + (ITS_base). > >> + > >> + kvm_device_attr.addr points to a __u64 value whatever the width > >> + of the addressed register (32/64 bits). 64 bit registers can only > >> + be accessed with full length. > >> + > >> + Writes to read-only registers are ignored by the kernel except for: > >> + - GITS_CREADR. It needs to be restored otherwise commands in the queue > >> + will be re-executed after restoring CWRITER. GITS_CREADR must be > >> + restored before restoring the GITS_CTLR which is likely to enable the > >> + ITS. Also it needs to be restored after GITS_CBASER since a write to > >> + GITS_CBASER resets GITS_CREADR. > > > > uber nit: s/needs to be/must be/g in all occurences above. > OK > > > >> + - GITS_IIDR. Its Revision field encodes the table layout ABI revision. > > > > s/Its/The/ > OK. Argh, I meant its (~his) ;-) You wrote it correctly, it's just that it can be misleading, so I'm really in the business of beutifying the ABI text here :) > > the revision of the table layout ABI. > > > >> + In the future we might implement direct injection of virtual LPIS. > > > > LPIs > OK > > > >> + This will require an upgrade of the table layout and an evolution of > >> + the ABI. GITS_IIDR must be restored before the table restoration. > > > > before calling KVM_DEV_ARM_ITS_RESTORE_TABLES. > OK > > > >> + > >> + For other registers, getting or setting a register has the same > >> + effect as reading/writing the register on real hardware. > >> + Errors: > >> + -ENXIO: Offset does not correspond to any supported register > >> + -EFAULT: Invalid user pointer for attr->addr > >> + -EINVAL: Offset is not 64-bit aligned > >> + -EBUSY: one or more VCPUS are running > >> + > >> + ITS Restoration Sequence: > > > > s/Restoration/restore/ > OK > > > >> + ------------------------- > >> + > >> +The following ordering must be followed when restoring the GIC and the ITS: > >> +a) restore all guest memory and create vcpus > >> +b) initialize the GIC and ITS (KVM_DEV_ARM_VGIC_CTRL_INIT) > > > > are these really coupled or can you do everything related to the gic > > first, and then everything related to the ITS afterwards? > no they aren't. > > > > >> +c) provide GIC and ITS base addresses (KVM_DEV_ARM_VGIC_GRP_ADDR) > > > > same > no they aren't > > > > > >> +d) restore all redistributors > >> +e) restore the ITS in the following order: > >> + 1. Restore GITS_CBASER > >> + 2. Restore all other GITS_ registers, except GITS_CTLR! > >> + 3. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES) > >> + 4. Restore GITS_CTLR > >> +f) start vcpus > > > > I would omit (f) from here. > > In that case I think it would make sense to describe the restore > sequence for GICV3 RDIST in GICV3 device doc and not here. This would > alleviate this very description. > > restoring RDIST regs require a previous GICV3 KVM_DEV_ARM_VGIC_CTRL_INIT > and KVM_DEV_ARM_VGIC_GRP_ADDR I think. > > So description here would become > > " > a) restore all guest memory and create vcpus > b) restore all redistributors > c) initialize the ITS and then provide its base address > (KVM_DEV_ARM_VGIC_CTRL_INIT, KVM_DEV_ARM_VGIC_GRP_ADDR) > c) restore the ITS in the following order: > 1. Restore GITS_CBASER > 2. Restore all other GITS_ registers, except GITS_CTLR! > 3. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES) > 4. Restore GITS_CTLR > > Then vcpus can be started. > " > > What do you think? Yes, looks good. > > > > > >> + > >> + ITS Table ABI REV0: > >> + ------------------- > >> + > >> + Revision 0 of the ABI only supports physical LPIs. > >> + > >> + The device table and ITT are indexed by the deviceid and eventid, > >> + respectively. The collection table is not indexed by collectionid: > >> + CTE are written in the table in the order of collection creation. All > > > > nit: CTEs > OK > > > >> + entries are 8 bytes. > >> + > >> + Device Table Entry (DTE): > >> + > >> + bits: | 63| 62 ... 49 | 48 ... 5 | 4 ... 0 | > >> + values: | V | next | ITT_addr | Size | > >> + > >> + where; > >> + - V indicates whether the entry is valid. If not, other fields > >> + are not meaningful. > >> + - next: equals to 0 if this entry is the last one; otherwise it > >> + corresponds to the deviceid offset to the next DTE, capped by > >> + 2^14 -1. > >> + - ITT_addr matches bits [51:8] of the ITT address (256 Byte aligned). > >> + - Size specifies the supported number of bits for the eventid, > >> + minus one > >> + > >> + Collection Table Entry (CTE): > >> + > >> + bits: | 63| 62 .. 52 | 51 ... 16 | 15 ... 0 | > >> + values: | V | RES0 | RDBase | ICID | > >> + > >> + where: > >> + - V indicates whether the entry is valid. If not, other fields are > >> + not meaningful. > >> + - RES0: reserved field with Should-Be-Zero-or-Preserved behavior. > >> + - RDBase is the PE number (GICR_TYPER.Processor_Number semantic), > >> + - ICID is the collection ID > >> + > >> + Interrupt Translation Entry (ITE): > >> + > >> + bits: | 63 ... 48 | 47 ... 16 | 15 ... 0 | > >> + values: | next | pINTID | ICID | > >> + > >> + where: > >> + - next: equals to 0 if this entry is the last one; otherwise it corresponds > >> + to the eventid offset to the next ITE capped by 2^16 -1. > >> + - pINTID is the physical LPI ID; if zero, it means the entry is not valid > >> + and other fields are not meaningful. > >> + - ICID is the collection ID > >> + > >> -- > >> 2.5.5 > >> > > > > The only outstanding issue here is the restore order part. I don't wont > > to specify a stronger order than absolutely necessary here. > see above > Yep, I think we're there. Thanks, -Christoffer
