On 2021/6/24 12:03, David Gibson wrote:
On Fri, Jun 18, 2021 at 01:21:47PM +0800, Lu Baolu wrote:
Hi David,
On 6/17/21 1:22 PM, David Gibson wrote:
The iommu_group can guarantee the isolation among different physical
devices (represented by RIDs). But when it comes to sub-devices (ex. mdev or
vDPA devices represented by RID + SSID), we have to rely on the
device driver for isolation. The devices which are able to generate sub-
devices should either use their own on-device mechanisms or use the
platform features like Intel Scalable IOV to isolate the sub-devices.
This seems like a misunderstanding of groups. Groups are not tied to
any PCI meaning. Groups are the smallest unit of isolation, no matter
what is providing that isolation.
If mdevs are isolated from each other by clever software, even though
they're on the same PCI device they are in different groups from each
other*by definition*. They are also in a different group from their
parent device (however the mdevs only exist when mdev driver is
active, which implies that the parent device's group is owned by the
kernel).
You are right. This is also my understanding of an "isolation group".
But, as I understand it, iommu_group is only the isolation group visible
to IOMMU. When we talk about sub-devices (sw-mdev or mdev w/ pasid),
only the device and device driver knows the details of isolation, hence
iommu_group could not be extended to cover them. The device drivers
should define their own isolation groups.
So, "iommu group" isn't a perfect name. It came about because
originally the main mechanism for isolation was the IOMMU, so it was
typically the IOMMU's capabilities that determined if devices were
isolated. However it was always known that there could be other
reasons for failure of isolation. To simplify the model we decided
that we'd put things into the same group if they were non-isolated for
any reason.
Yes.
The kernel has no notion of "isolation group" as distinct from "iommu
group". What are called iommu groups in the kernel now*are*
"isolation groups" and that was always the intention - it's just not a
great name.
Fair enough.
Otherwise, the device driver has to fake an iommu_group and add hacky
code to link the related IOMMU elements (iommu device, domain, group
etc.) together. Actually this is part of the problem that this proposal
tries to solve.
Yeah, that's not ideal.
Under above conditions, different sub-device from a same RID device
could be able to use different IOASID. This seems to means that we can't
support mixed mode where, for example, two RIDs share an iommu_group and
one (or both) of them have sub-devices.
That doesn't necessarily follow. mdevs which can be successfully
isolated by their mdev driver are in a different group from their
parent device, and therefore need not be affected by whether the
parent device shares a group with some other physical device. They
*might* be, but that's up to the mdev driver to determine based on
what it can safely isolate.
If we understand it as multiple levels of isolation, can we classify the
devices into the following categories?
1) Legacy devices
- devices without device-level isolation
- multiple devices could sit in a single iommu_group
- only a single I/O address space could be bound to IOMMU
I'm not really clear on what that last statement means.
I mean a single iommu_domain should be used by all devices sharing a
single iommu_group.
2) Modern devices
- devices capable of device-level isolation
This will*typically* be true of modern devices, but I don't think we
can really make it a hard API distinction. Legacy or buggy bridges
can force modern devices into the same group as each other. Modern
devices are not immune from bugs which would force lack of isolation
(e.g. forgotten debug registers on function 0 which affect other
functions).
Yes.
I am thinking whether it's feasible to change "bind/attach a device to
an IOASID" to "bind/attach an isolated unit to an IOASID". An isolated
unit could be
1) an iommu_ group including single or multiple devices;
2) a physical device which have a 1-device iommu group + device ID
(PASID/subStreamID) which represents an isolated subdevice inside the
physical one.
3) anything that we might have in the future.
A handler which represents the connection between device and iommu is
returned on any successful binding. This handler could be used to
GET_INFO and attach/detach after binding.
Best regards,
baolu
