On 5/10/22 02:38, Tian, Kevin wrote:
>> From: Jason Gunthorpe <jgg@xxxxxxxxxx>
>> Sent: Friday, May 6, 2022 7:46 PM
>>
>> On Fri, May 06, 2022 at 03:51:40AM +0000, Tian, Kevin wrote:
>>>> From: Jason Gunthorpe <jgg@xxxxxxxxxx>
>>>> Sent: Thursday, May 5, 2022 10:08 PM
>>>>
>>>> On Thu, May 05, 2022 at 07:40:37AM +0000, Tian, Kevin wrote:
>>>>
>>>>> In concept this is an iommu property instead of a domain property.
>>>>
>>>> Not really, domains shouldn't be changing behaviors once they are
>>>> created. If a domain supports dirty tracking and I attach a new device
>>>> then it still must support dirty tracking.
>>>
>>> That sort of suggests that userspace should specify whether a domain
>>> supports dirty tracking when it's created. But how does userspace
>>> know that it should create the domain in this way in the first place?
>>> live migration is triggered on demand and it may not happen in the
>>> lifetime of a VM.
>>
>> The best you could do is to look at the devices being plugged in at VM
>> startup, and if they all support live migration then request dirty
>> tracking, otherwise don't.
>
> Yes, this is how a device capability can help.
>
>>
>> However, tt costs nothing to have dirty tracking as long as all iommus
>> support it in the system - which seems to be the normal case today.
>>
>> We should just always turn it on at this point.
>
> Then still need a way to report " all iommus support it in the system"
> to userspace since many old systems don't support it at all. If we all
> agree that a device capability flag would be helpful on this front (like
> you also said below), probably can start building the initial skeleton
> with that in mind?
>
This would capture device-specific and maybe iommu-instance features, but
there's some tiny bit odd semantic here. There's nothing that
depends on the device to support any of this, but rather the IOMMU instance that sits
below the device which is independent of device-own capabilities e.g. PRI on the other
hand would be a perfect fit for a device capability (?), but dirty tracking
conveying over a device capability would be a convenience rather than an exact
hw representation.
Thinking out loud if we are going as a device/iommu capability [to see if this matches
what people have or not in mind]: we would add dirty-tracking feature bit via the existent
kAPI for iommu device features (e.g. IOMMU_DEV_FEAT_AD) and on iommufd we would maybe add
an IOMMUFD_CMD_DEV_GET_IOMMU_FEATURES ioctl which would have an u64 dev_id as input (from
the returned vfio-pci BIND_IOMMUFD @out_dev_id) and u64 features as an output bitmap of
synthetic feature bits, having IOMMUFD_FEATURE_AD the only one we query (and
IOMMUFD_FEATURE_{SVA,IOPF} as potentially future candidates). Qemu would then at start of
day would check if /all devices/ support it and it would then still do the blind set
tracking, but bail out preemptively if any of device-iommu don't support dirty-tracking. I
don't think we have any case today for having to deal with different IOMMU instances that
have different features.
Either that or as discussed in the beginning perhaps add an iommufd (or iommufd hwpt one)
ioctl call (e.g.IOMMUFD_CMD_CAP) via a input value (e.g. subop IOMMU_FEATURES) which
would gives us a structure of things (e.g. for the IOMMU_FEATURES subop the common
featureset bitmap in all iommu instances). This would give the 'all iommus support it in
the system'. Albeit the device one might have more concrete longevity if there's further
plans aside from dirty tracking.
>>
>>> and if the user always creates domain to allow dirty tracking by default,
>>> how does it know a failed attach is due to missing dirty tracking support
>>> by the IOMMU and then creates another domain which disables dirty
>>> tracking and retry-attach again?
>>
>> The automatic logic is complicated for sure, if you had a device flag
>> it would have to figure it out that way
>>
>
> Yes. That is the model in my mind.
>
> Thanks
> Kevin
