On 09/26/2018 06:42 PM, Alex Williamson wrote:
On Tue, 25 Sep 2018 19:16:41 -0400
Tony Krowiak <akrowiak@xxxxxxxxxxxxxxxxxx> wrote:
From: Tony Krowiak <akrowiak@xxxxxxxxxxxxx>
This patch provides documentation describing the AP architecture and
design concepts behind the virtualization of AP devices. It also
includes an example of how to configure AP devices for exclusive
use of KVM guests.
Signed-off-by: Tony Krowiak <akrowiak@xxxxxxxxxxxxx>
Reviewed-by: Halil Pasic <pasic@xxxxxxxxxxxxx>
---
Documentation/s390/vfio-ap.txt | 782 +++++++++++++++++++++++++++++++++
MAINTAINERS | 1 +
2 files changed, 783 insertions(+)
create mode 100644 Documentation/s390/vfio-ap.txt
...
+Example:
+=======
+Let's now provide an example to illustrate how KVM guests may be given
+access to AP facilities. For this example, we will show how to configure
+three guests such that executing the lszcrypt command on the guests would
+look like this:
+
+Guest1
+------
+CARD.DOMAIN TYPE MODE
+------------------------------
+05 CEX5C CCA-Coproc
+05.0004 CEX5C CCA-Coproc
+05.00ab CEX5C CCA-Coproc
+06 CEX5A Accelerator
+06.0004 CEX5A Accelerator
+06.00ab CEX5C CCA-Coproc
+
+Guest2
+------
+CARD.DOMAIN TYPE MODE
+------------------------------
+05 CEX5A Accelerator
+05.0047 CEX5A Accelerator
+05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Seems like an unfinished thought here.
+
+Guest2
+------
+CARD.DOMAIN TYPE MODE
+------------------------------
+06 CEX5A Accelerator
+06.0047 CEX5A Accelerator
+06.00ff CEX5A Accelerator
+
+These are the steps:
+
+1. Install the vfio_ap module on the linux host. The dependency chain for the
+ vfio_ap module is:
+ * iommu
+ * s390
+ * zcrypt
+ * vfio
+ * vfio_mdev
+ * vfio_mdev_device
+ * KVM
+
+ To build the vfio_ap module, the kernel build must be configured with the
+ following Kconfig elements selected:
+ * IOMMU_SUPPORT
+ * S390
+ * ZCRYPT
+ * S390_AP_IOMMU
+ * VFIO
+ * VFIO_MDEV
+ * VFIO_MDEV_DEVICE
+ * KVM
+
+ If using make menuconfig select the following to build the vfio_ap module:
+ -> Device Drivers
+ -> IOMMU Hardware Support
+ select S390 AP IOMMU Support
+ -> VFIO Non-Privileged userspace driver framework
+ -> Mediated device driver frramework
+ -> VFIO driver for Mediated devices
+ -> I/O subsystem
+ -> VFIO support for AP devices
+
+2. Secure the AP queues to be used by the three guests so that the host can not
+ access them. To secure them, there are two sysfs files that specify
+ bitmasks marking a subset of the APQN range as 'usable by the default AP
+ queue device drivers' or 'not usable by the default device drivers' and thus
+ available for use by the vfio_ap device driver'. The sysfs files containing
+ the sysfs locations of the masks are:
+
+ /sys/bus/ap/apmask
+ /sys/bus/ap/aqmask
+
+ The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
+ (APID). Each bit in the mask, from most significant to least significant bit,
+ corresponds to an APID from 0-255. If a bit is set, the APID is marked as
+ usable only by the default AP queue device drivers; otherwise, the APID is
+ usable by the vfio_ap device driver.
+
+ The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
+ (APQI). Each bit in the mask, from most significant to least significant bit,
+ corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as
+ usable only by the default AP queue device drivers; otherwise, the APQI is
+ usable by the vfio_ap device driver.
+
+ The APQN of each AP queue device assigned to the linux host is checked by the
+ AP bus against the set of APQNs derived from the cross product of APIDs
+ and APQIs marked as usable only by the default AP queue device drivers. If a
+ match is detected, only the default AP queue device drivers will be probed;
+ otherwise, the vfio_ap device driver will be probed.
+
+ By default, the two masks are set to reserve all APQNs for use by the default
+ AP queue device drivers. There are two ways the default masks can be changed:
+
+ 1. The masks can be changed at boot time with the kernel command line
+ like this:
+
+ ap.apmask=0xffff ap.aqmask=0x40
+
+ This would give these two pools:
+
+ default drivers pool: adapter 0-15, domain 1
+ alternate drivers pool: adapter 16-255, domains 2-255
What happened to domain 0? I'm also a little confused by the bit
ordering. If 0x40 is bit 1 and 0xffff is bits 0-15, then the least
significant bit is furthest left? Did I miss documentation of that?
+
+ 2. The sysfs mask files can also be edited by echoing a string into the
+ respective file in one of two formats:
+
+ * An absolute hex string starting with 0x - like "0x12345678" - sets
+ the mask. If the given string is shorter than the mask, it is padded
+ with 0s on the right. If the string is longer than the mask, the
+ operation is terminated with an error (EINVAL).
And this does say zero padding on the right, but then in the next
bullet our hex digits use normal least significant bit right notation,
ie. 0x41 is 65, not 82, correct?
+
+ * A plus ('+') or minus ('-') followed by a numerical value. Valid
+ examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0". Only
+ the corresponding bit in the mask is switched on ('+') or off ('-'). The
+ values may also be specified in a comma-separated list to switch more
+ than one bit on or off.
+
+ To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047,
+ 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding
+ APQNs must be removed from the masks as follows:
+
+ echo -5,-6 > /sys/bus/ap/apmask
+
+ echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask
Other than the bit ordering confusion, I like this +/- scheme.
The following fixup attempts to clarify the bit ordering confusion,
hopefully this is acceptable.
-----------------------------------8<-----------------------------------
From: Tony Krowiak <akrowiak@xxxxxxxxxxxxx>
Date: Thu, 27 Sep 2018 14:51:12 -0400
Subject: [FIXUP v10] fixup! s390: doc: detailed specifications for AP
virtualization
Better explains mask bit ordering.
Signed-off-by: Tony Krowiak <akrowiak@xxxxxxxxxxxxx>
---
Documentation/s390/vfio-ap.txt | 127 +++++++++++++++++++++++----------
1 file changed, 91 insertions(+), 36 deletions(-)
diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt
index bec67eb7141c..599eb0f75c07 100644
--- a/Documentation/s390/vfio-ap.txt
+++ b/Documentation/s390/vfio-ap.txt
@@ -123,21 +123,24 @@ to identify the adapters, usage domains and
control domains assigned to the KVM
guest:
* The AP Mask (APM) field is a bit mask that identifies the AP
adapters assigned
- to the KVM guest. Each bit in the mask, from most significant to least
- significant bit, corresponds to an APID from 0-255. If a bit is set, the
- corresponding adapter is valid for use by the KVM guest.
+ to the KVM guest. Each bit in the mask, from left to right (i.e. from
most
+ significant to least significant bit in big endian order), corresponds to
+ an APID from 0-255. If a bit is set, the corresponding adapter is
valid for
+ use by the KVM guest.
* The AP Queue Mask (AQM) field is a bit mask identifying the AP usage
domains
- assigned to the KVM guest. Each bit in the mask, from most significant to
- least significant bit, corresponds to an AP queue index (APQI) from
0-255. If
- a bit is set, the corresponding queue is valid for use by the KVM guest.
+ assigned to the KVM guest. Each bit in the mask, from left to right
(i.e. from
+ most significant to least significant bit in big endian order),
corresponds to
+ an AP queue index (APQI) from 0-255. If a bit is set, the
corresponding queue
+ is valid for use by the KVM guest.
* The AP Domain Mask field is a bit mask that identifies the AP
control domains
assigned to the KVM guest. The ADM bit mask controls which domains
can be
changed by an AP command-request message sent to a usage domain from the
- guest. Each bit in the mask, from least significant to most
significant bit,
- corresponds to a domain from 0-255. If a bit is set, the
corresponding domain
- can be modified by an AP command-request message sent to a usage domain.
+ guest. Each bit in the mask, from left to right (i.e. from most
significant to
+ least significant bit in big endian order), corresponds to a domain from
+ 0-255. If a bit is set, the corresponding domain can be modified by an AP
+ command-request message sent to a usage domain.
If you recall from the description of an AP Queue, AP instructions include
an APQN to identify the AP queue to which an AP command-request
message is to be
@@ -503,23 +506,34 @@ These are the steps:
access them. To secure them, there are two sysfs files that specify
bitmasks marking a subset of the APQN range as 'usable by the
default AP
queue device drivers' or 'not usable by the default device drivers'
and thus
- available for use by the vfio_ap device driver'. The sysfs files
containing
- the sysfs locations of the masks are:
+ available for use by the vfio_ap device driver'. The location of the
sysfs
+ files containing the masks are:
/sys/bus/ap/apmask
/sys/bus/ap/aqmask
The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
- (APID). Each bit in the mask, from most significant to least
significant bit,
- corresponds to an APID from 0-255. If a bit is set, the APID is
marked as
- usable only by the default AP queue device drivers; otherwise, the
APID is
- usable by the vfio_ap device driver.
+ (APID). Each bit in the mask, from left to right (i.e., from most
significant
+ to least significant bit in big endian order), corresponds to an
APID from
+ 0-255. If a bit is set, the APID is marked as usable only by the
default AP
+ queue device drivers; otherwise, the APID is usable by the vfio_ap
+ device driver.
The 'aqmask' is a 256-bit mask that identifies a set of AP queue
indexes
- (APQI). Each bit in the mask, from most significant to least
significant bit,
- corresponds to an APQI from 0-255. If a bit is set, the APQI is
marked as
- usable only by the default AP queue device drivers; otherwise, the
APQI is
- usable by the vfio_ap device driver.
+ (APQI). Each bit in the mask, from left to right (i.e., from most
significant
+ to least significant bit in big endian order), corresponds to an
APQI from
+ 0-255. If a bit is set, the APQI is marked as usable only by the
default AP
+ queue device drivers; otherwise, the APQI is usable by the vfio_ap
device
+ driver.
+
+ Take, for example, the following mask:
+
+ 0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
+
+ It indicates:
+
+ 1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and
0 and 6
+ belong to the vfio_ap device driver's pool.
The APQN of each AP queue device assigned to the linux host is
checked by the
AP bus against the set of APQNs derived from the cross product of APIDs
@@ -530,38 +544,79 @@ These are the steps:
By default, the two masks are set to reserve all APQNs for use by
the default
AP queue device drivers. There are two ways the default masks can
be changed:
- 1. The masks can be changed at boot time with the kernel command line
- like this:
+ 1. The sysfs mask files can be edited by echoing a string into the
+ respective sysfs mask file in one of two formats:
+
+ * An absolute hex string starting with 0x - like "0x12345678" - sets
+ the mask. If the given string is shorter than the mask, it is
padded
+ with 0s on the right; for example, specifying a mask value of
0x41 is
+ the same as specifying:
+
+
0x4100000000000000000000000000000000000000000000000000000000000000
+
+ Keep in mind that the mask reads from left to right (i.e., most
+ significant to least significant bit in big endian order), so
the mask
+ above identifies device numbers 1 and 7 (01000001).
+
+ If the string is longer than the mask, the operation is
terminated with
+ an error (EINVAL).
+
+ * Individual bits in the mask can be switched on and off by
specifying
+ each bit number to be switched in a comma separated list. Each bit
+ number string must be prepended with a ('+') or minus ('-') to
indicate
+ the corresponding bit is to be switched on ('+') or off ('-'). Some
+ valid values are:
+
+ "+0" switches bit 0 on
+ "-13" switches bit 13 off
+ "+0x41" switches bit 65 on
+ "-0xff" switches bit 255 off
+
+ The following example:
+ +0,-6,+0x47,-0xf0
+
+ Switches bits 0 and 71 (0x47) on
+ Switches bits 6 and 240 (0xf0) off
+
+ Note that the bits not specified in the list remain as they
were before
+ the operation.
+
+ 2. The masks can also be changed at boot time via parameters on the
kernel
+ command line like this:
ap.apmask=0xffff ap.aqmask=0x40
- This would give these two pools:
+ This would create the following masks:
- default drivers pool: adapter 0-15, domain 1
- alternate drivers pool: adapter 16-255, domains 2-255
+ apmask:
+
0xffff000000000000000000000000000000000000000000000000000000000000
- 2. The sysfs mask files can also be edited by echoing a string into the
- respective file in one of two formats:
+ aqmask:
+
0x4000000000000000000000000000000000000000000000000000000000000000
- * An absolute hex string starting with 0x - like "0x12345678" - sets
- the mask. If the given string is shorter than the mask, it is
padded
- with 0s on the right. If the string is longer than the mask, the
- operation is terminated with an error (EINVAL).
+ Resulting in these two pools:
- * A plus ('+') or minus ('-') followed by a numerical value. Valid
- examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and
"-0". Only
- the corresponding bit in the mask is switched on ('+') or off
('-'). The
- values may also be specified in a comma-separated list to
switch more
- than one bit on or off.
+ default drivers pool: adapter 0-15, domain 1
+ alternate drivers pool: adapter 16-255, domains 0, 2-255
+ Securing the APQNs for our example:
+ ----------------------------------
To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff,
06.0004, 06.0047,
06.00ab, and 06.00ff for use by the vfio_ap device driver, the
corresponding
- APQNs must be removed from the masks as follows:
+ APQNs can either be removed from the default masks:
echo -5,-6 > /sys/bus/ap/apmask
echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask
+ Or the masks can be set as follows:
+
+ echo
0xf9ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \
+ > apmask
+
+ echo
0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \
+ > aqmask
+
This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff,
06.0004,
06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device
driver. The
sysfs directory for the vfio_ap device driver will now contain
symbolic links
--
2.19.0.221.g150f307
