> From: Joao Martins <joao.m.martins@xxxxxxxxxx>
> Sent: Tuesday, May 10, 2022 7:51 PM
>
> On 5/10/22 02:38, Tian, Kevin wrote:
> >> From: Jason Gunthorpe <jgg@xxxxxxxxxx>
> >> Sent: Friday, May 6, 2022 7:46 PM
> >>
> >> On Fri, May 06, 2022 at 03:51:40AM +0000, Tian, Kevin wrote:
> >>>> From: Jason Gunthorpe <jgg@xxxxxxxxxx>
> >>>> Sent: Thursday, May 5, 2022 10:08 PM
> >>>>
> >>>> On Thu, May 05, 2022 at 07:40:37AM +0000, Tian, Kevin wrote:
> >>>>
> >>>>> In concept this is an iommu property instead of a domain property.
> >>>>
> >>>> Not really, domains shouldn't be changing behaviors once they are
> >>>> created. If a domain supports dirty tracking and I attach a new device
> >>>> then it still must support dirty tracking.
> >>>
> >>> That sort of suggests that userspace should specify whether a domain
> >>> supports dirty tracking when it's created. But how does userspace
> >>> know that it should create the domain in this way in the first place?
> >>> live migration is triggered on demand and it may not happen in the
> >>> lifetime of a VM.
> >>
> >> The best you could do is to look at the devices being plugged in at VM
> >> startup, and if they all support live migration then request dirty
> >> tracking, otherwise don't.
> >
> > Yes, this is how a device capability can help.
> >
> >>
> >> However, tt costs nothing to have dirty tracking as long as all iommus
> >> support it in the system - which seems to be the normal case today.
> >>
> >> We should just always turn it on at this point.
> >
> > Then still need a way to report " all iommus support it in the system"
> > to userspace since many old systems don't support it at all. If we all
> > agree that a device capability flag would be helpful on this front (like
> > you also said below), probably can start building the initial skeleton
> > with that in mind?
> >
>
> This would capture device-specific and maybe iommu-instance features, but
> there's some tiny bit odd semantic here. There's nothing that
> depends on the device to support any of this, but rather the IOMMU instance
> that sits
> below the device which is independent of device-own capabilities e.g. PRI on
> the other
> hand would be a perfect fit for a device capability (?), but dirty tracking
> conveying over a device capability would be a convenience rather than an
> exact
> hw representation.
it is sort of getting certain iommu capability for a given device, as how
the iommu kAPI is moving toward.
>
> Thinking out loud if we are going as a device/iommu capability [to see if this
> matches
> what people have or not in mind]: we would add dirty-tracking feature bit via
> the existent
> kAPI for iommu device features (e.g. IOMMU_DEV_FEAT_AD) and on
> iommufd we would maybe add
> an IOMMUFD_CMD_DEV_GET_IOMMU_FEATURES ioctl which would have an
> u64 dev_id as input (from
> the returned vfio-pci BIND_IOMMUFD @out_dev_id) and u64 features as an
> output bitmap of
> synthetic feature bits, having IOMMUFD_FEATURE_AD the only one we query
> (and
> IOMMUFD_FEATURE_{SVA,IOPF} as potentially future candidates). Qemu
> would then at start of
> day would check if /all devices/ support it and it would then still do the blind
> set
> tracking, but bail out preemptively if any of device-iommu don't support
> dirty-tracking. I
> don't think we have any case today for having to deal with different IOMMU
> instances that
> have different features.
This heterogeneity already exists today. On Intel platform not all IOMMUs
support force snooping. I believe ARM has similar situation which is why
Robin is refactoring bus-oriented iommu_capable() etc. to device-oriented.
I'm not aware of such heterogeneity particularly for dirty tracking today. But
who knows it won't happen in the future? I just feel that aligning iommufd
uAPI to iommu kAPI for capability reporting might be more future proof here.
>
> Either that or as discussed in the beginning perhaps add an iommufd (or
> iommufd hwpt one)
> ioctl call (e.g.IOMMUFD_CMD_CAP) via a input value (e.g. subop
> IOMMU_FEATURES) which
> would gives us a structure of things (e.g. for the IOMMU_FEATURES subop
> the common
> featureset bitmap in all iommu instances). This would give the 'all iommus
> support it in
> the system'. Albeit the device one might have more concrete longevity if
> there's further
> plans aside from dirty tracking.
>
> >>
> >>> and if the user always creates domain to allow dirty tracking by default,
> >>> how does it know a failed attach is due to missing dirty tracking support
> >>> by the IOMMU and then creates another domain which disables dirty
> >>> tracking and retry-attach again?
> >>
> >> The automatic logic is complicated for sure, if you had a device flag
> >> it would have to figure it out that way
> >>
> >
> > Yes. That is the model in my mind.
> >
> > Thanks
> > Kevin
