On Wed, 9 Sep 2020 14:40:33 +0200 Erik Skultety <eskultet@xxxxxxxxxx> wrote: > On Wed, Sep 09, 2020 at 08:23:39AM +0200, Bjoern Walk wrote: > > Erik Skultety <eskultet@xxxxxxxxxx> [2020-09-08, 06:01PM +0200]: > > > On Mon, Aug 24, 2020 at 01:59:15PM +0200, Bjoern Walk wrote: > > > > From: Boris Fiuczynski <fiuczy@xxxxxxxxxxxxx> > > > > + case VIR_NODE_DEV_CAP_SYSTEM: > > > > + case VIR_NODE_DEV_CAP_USB_DEV: > > > > + case VIR_NODE_DEV_CAP_USB_INTERFACE: > > > > + case VIR_NODE_DEV_CAP_NET: > > > > + case VIR_NODE_DEV_CAP_SCSI_HOST: > > > > + case VIR_NODE_DEV_CAP_SCSI_TARGET: > > > > + case VIR_NODE_DEV_CAP_SCSI: > > > > + case VIR_NODE_DEV_CAP_STORAGE: > > > > + case VIR_NODE_DEV_CAP_FC_HOST: > > > > + case VIR_NODE_DEV_CAP_VPORTS: > > > > + case VIR_NODE_DEV_CAP_SCSI_GENERIC: > > > > + case VIR_NODE_DEV_CAP_DRM: > > > > + case VIR_NODE_DEV_CAP_MDEV_TYPES: > > > > + case VIR_NODE_DEV_CAP_MDEV: > > > > + case VIR_NODE_DEV_CAP_CCW_DEV: > > > > > > What about ccw mdevs? I would think that it's CCW end point device that > > > we would want to assign concurrently, but we're only "slicing" the root channel > > > subsystem device. If you have and CSS mdev, are the CCW subchannels separate > > > for each VM or does it behave kind of like NPIV with its vHBAs? OR I'm > > > completely off here? > > > > You are correct. This is an unfortunate architectural decision on our > > platform. For vfio-ccw, we instatiate the mdevs on the CSS layer, which > > has a 1-to-1 relation to the underlying CCW device. That's why we need > > the information about the CSS devices in libvirt, to give the user a > > chance to easily find this relation when he wants to do passthrough with > > a DASD for example. It's a bit unintuitive and even silly for the > > end user, but it's architecturally correct and hence it was implemented > > like this in the vfio-ccw kernel driver. We don't even get the benefits > > of something like NPIV because for every subchannel there is only one > > CCW device (at least to my knowledge). > > Having read both Connie's blog on channels and the I/O chapter in the free > architectural publication you linked in the other reply, I now have a very > vague understating of the architecture. One thing that still > puzzles me though is that if there's a 1:1 subchannel mapping between CSS and an > CCW device which then communicates via a control unit (which there may be more > than 1) with CSS over up to 8 channel paths (this at least to me resembles > NPIV), what does the mdev really solve, can't you assign one of the channel > paths of one of the several control units the device is connected to to > plug into the VM? Well, clearly not, I'm just being curious and trying to > understand the use case here. I think it's easier to leave control units mostly out of the picture, as they are not really relevant for the modelling here. And yes, there is always exactly one subchannel for an I/O device. Subchannels (mostly a concept describing how to interact with a device) and channel paths (mostly corresponding to the way data flows to/from the device) are probably best understood as different ways to describe how devices are accessed. You don't really want to pass channel paths (as they are shared between many devices); the subchannel is a more natural unit to pass. The operating system also has way more control over subchannels than over individual channel paths. Regarding mdev, the idea here is less to be able to slice up a device as you do with GPUs, but more to be able to add a vendor driver that does the actual interesting work (channel program translations etc.) As to why we use the subchannel instead of the device, that's mostly following from how the low-level channel I/O layer in the Linux kernel is organized: The default I/O subchannel driver does a lot of things that a normal device driver should not need to deal with (path grouping, verification, status accumulation, etc.), but which would get into the way if you wanted to assign the device via vfio. Therefore, vfio-ccw implements an alternate I/O subchannel driver. HTH (at least a little bit).
