On 03.07.2018 16:56, Tony Krowiak wrote: > On 07/03/2018 03:46 AM, Harald Freudenberger wrote: >> On 02.07.2018 18:28, Halil Pasic wrote: >>> >>> On 06/29/2018 11:11 PM, Tony Krowiak wrote: >>>> This patch provides documentation describing the AP architecture and >>>> design concepts behind the virtualization of AP devices. It also >>>> includes an example of how to configure AP devices for exclusive >>>> use of KVM guests. >>>> >>>> Signed-off-by: Tony Krowiak <akrowiak@xxxxxxxxxxxxx> >>>> --- >>> [..] >>>> + >>>> +Reserve APQNs for exclusive use of KVM guests >>>> +--------------------------------------------- >>>> +The following block diagram illustrates the mechanism by which APQNs are >>>> +reserved: >>>> + >>>> + +------------------+ >>>> + remove | | unbind >>>> + +------------------->+ cex4queue driver +<-----------+ >>>> + | | | | >>>> + | +------------------+ | >>>> + | | >>>> + | | >>>> + | | >>>> ++--------+---------+ register +------------------+ +-----+------+ >>>> +| +<---------+ | bind | | >>>> +| ap_bus | | vfio_ap driver +<-----+ admin | >>>> +| +--------->+ | | | >>>> ++------------------+ probe +---+--------+-----+ +------------+ >>>> + | | >>>> + create | | store APQN >>>> + | | >>>> + v v >>>> + +---+--------+-----+ >>>> + | | >>>> + | matrix device | >>>> + | | >>>> + +------------------+ >>>> + >>>> +The process for reserving an AP queue for use by a KVM guest is: >>>> + >>>> +* The vfio-ap driver during its initialization will perform the following: >>>> + * Create the 'vfio_ap' root device - /sys/devices/vfio_ap >>>> + * Create the 'matrix' device in the 'vfio_ap' root >>>> + * Register the matrix device with the device core >>>> +* Register with the ap_bus for AP queue devices of type 10 devices (CEX4 and >>>> + newer) and to provide the vfio_ap driver's probe and remove callback >>>> + interfaces. The reason why older devices are not supported is because there >>>> + are no systems available on which to test. >>>> +* The admin unbinds queue cc.qqqq from the cex4queue device driver. This results >>>> + in the ap_bus calling the the device driver's remove interface which >>>> + unbinds the cc.qqqq queue device from the driver. >>> What if the queue cc.qqqq is already in use? AFAIU unbind is almost as radical as >>> pulling a cable. What is the proper procedure an admin should follow before doing >>> the unbind? >> What do you mean on this level with 'in use'? A unbind destroys the association >> between device and driver. There is no awareness of 'in use' or 'not in use' on this >> level. This is a hard unbind. > > According to my reading of the code, the remove callback for the AP queue drivers > flushes the queue before it is disconnected from the driver. Do you concur Harald? Yes, you are right. I checked this. The unbind triggers a zcrypt_cex4_queue_remove() which calls ap_queue_remove() which calls ap_flush_queue(). The ap_flush_queue() function does: - for all requests which are queued these are 'received' with -EAGAIN and thus the zcrypt api tries to re-schedule these requests on another apqn. - for all requests which have been sent to the ap but there is no answer yet these are 'reveived' with -EAGAIN and the zcrypt_api tries to re-schedule these requests. [Well, this may in the end lead to some requests sent double...] Looks like the unbind is handled in a smooth way :-) > >>>> +* The admin binds the cc.qqqq queue to the vfio_ap device driver. This results >>>> + in the ap_bus calling the device vfio_ap driver's probe interface to bind >>>> + queue cc.qqqq to the driver. The vfio_ap device driver will store the APQN for >>>> + the queue in the matrix device >>>> + >>> [..] >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-s390" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> > -- To unsubscribe from this list: send the line "unsubscribe linux-s390" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
