Re: [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On 14.11.2023 14:22, Vivi, Rodrigo wrote:
> On Tue, 2023-11-14 at 12:37 +0000, Tvrtko Ursulin wrote:
>>
>> On 10/11/2023 18:22, Michal Wajdeczko wrote:
>>> The Single Root I/O Virtualization (SR-IOV) extension to the PCI
>>> Express (PCIe) specification suite is supported starting from 12th
>>> generation of Intel Graphics processors.
>>>
>>> This RFC aims to explain how do we want to add support for SR-IOV
>>> to the new Xe driver and to propose related additions to the sysfs.
>>>
>>> Signed-off-by: Michal Wajdeczko <michal.wajdeczko@xxxxxxxxx>
>>> Cc: Oded Gabbay <ogabbay@xxxxxxxxxx>
>>> Cc: Rodrigo Vivi <rodrigo.vivi@xxxxxxxxx>
>>> Cc: Joonas Lahtinen <joonas.lahtinen@xxxxxxxxxxxxxxx>
>>> Cc: Tvrtko Ursulin <tvrtko.ursulin@xxxxxxxxxxxxxxx>
>>> Cc: Daniel Vetter <daniel@xxxxxxxx>
>>> ---
>>>   Documentation/gpu/rfc/index.rst             |   5 +
>>>   Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501
>>> ++++++++++++++++++++
>>>   Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
>>>   3 files changed, 698 insertions(+)
>>>   create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>>   create mode 100644 Documentation/gpu/rfc/xe_sriov.rst
>>>
>>> diff --git a/Documentation/gpu/rfc/index.rst
>>> b/Documentation/gpu/rfc/index.rst
>>> index e4f7b005138d..fc5bc447f30d 100644
>>> --- a/Documentation/gpu/rfc/index.rst
>>> +++ b/Documentation/gpu/rfc/index.rst
>>> @@ -35,3 +35,8 @@ host such documentation:
>>>   .. toctree::
>>>   
>>>      xe.rst
>>> +
>>> +.. toctree::
>>> +   :maxdepth: 1
>>> +
>>> +   xe_sriov.rst
>>> diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>> b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>> new file mode 100644
>>> index 000000000000..77748204dd83
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>> @@ -0,0 +1,501 @@
>>> +.. Documentation/ABI/testing/sysfs-driver-xe-sriov
>>> +..
>>> +.. Intel Xe driver ABI (SR-IOV extensions)
>>> +..
>>> +    The Single Root I/O Virtualization (SR-IOV) extension to
>>> +    the PCI Express (PCIe) specification suite is supported
>>> +    starting from 12th generation of Intel Graphics processors.
>>> +
>>> +    This document describes Xe driver specific additions.
>>> +
>>> +    For description of generic SR-IOV sysfs attributes see
>>> +    "Documentation/ABI/testing/sysfs-bus-pci" document.
>>> +
>>> +    /sys/bus/pci/drivers/xe/BDF/
>>> +    ├── sriov_auto_provisioning
>>> +    │   ├── admin_mode
>>> +    │   ├── enabled
>>> +    │   ├── reset_defaults
>>> +    │   ├── resources
>>> +    │   │   ├── default_contexts_quota
>>> +    │   │   ├── default_doorbells_quota
>>> +    │   │   ├── default_ggtt_quota
>>> +    │   │   └── default_lmem_quota
>>> +    │   ├── scheduling
>>> +    │   │   ├── default_exec_quantum_ms
>>> +    │   │   └── default_preempt_timeout_us
>>> +    │   └── monitoring
>>> +    │       ├── default_cat_error_count
>>> +    │       ├── default_doorbell_time_us
>>> +    │       ├── default_engine_reset_count
>>> +    │       ├── default_h2g_time_us
>>> +    │       ├── default_irq_time_us
>>> +    │       └── default_page_fault_count
>>
>>  From the department of bike-shedding, one alternative could be to
>> have 
>> a directory called defaults which avoids having to have the default_ 
>> prefix on everything under it.
> 
> good idea. probably with a '.' prefix to make it hidden like we have
> for other stuff already.
> 
> '.defaults'

but then we will be inconsistent as in this 'other stuff' we use
".defaults" directory to hold RO attributes with min/default/max values,
while here we wanted to define RW attributes that will be applied to VFs

maybe ".template" instead ?

> 
>>
>>> +
>>> +    /sys/bus/pci/drivers/xe/BDF/
>>> +    ├── sriov_extensions
>>
>> Should this be xe_sriov_extensions or if not doesn't it need
>> agreement 
>> to reserve the keyword in Documentation/ABI/testing/sysfs-bus-pci? 
>> Sriov_auto_provisioning too I guess.
>>
>>> +    │   ├── monitoring_period_ms
>>> +    │   ├── strict_scheduling_enabled
>>> +    │   ├── pf
>>> +    │   │   ├── device -> ../../../BDF
>>> +    │   │   ├── priority
>>> +    │   │   ├── tile0
>>> +    │   │   │   ├── gt0
>>> +    │   │   │   │   ├── exec_quantum_ms
>>> +    │   │   │   │   ├── preempt_timeout_us
>>> +    │   │   │   │   └── thresholds
>>> +    │   │   │   │       ├── cat_error_count
>>> +    │   │   │   │       ├── doorbell_time_us
>>> +    │   │   │   │       ├── engine_reset_count
>>> +    │   │   │   │       ├── h2g_time_us
>>> +    │   │   │   │       ├── irq_time_us
>>> +    │   │   │   │       └── page_fault_count
>>> +    │   │   │   └── gtX
>>> +    │   │   └── tileT
>>> +    │   ├── vf1
>>> +    │   │   ├── device -> ../../../BDF+1
>>> +    │   │   ├── stop
>>> +    │   │   ├── tile0
>>> +    │   │   │   ├── ggtt_quota
>>> +    │   │   │   ├── lmem_quota
>>> +    │   │   │   ├── gt0
>>> +    │   │   │   │   ├── contexts_quota
>>> +    │   │   │   │   ├── doorbells_quota
>>> +    │   │   │   │   ├── exec_quantum_ms
>>> +    │   │   │   │   ├── preempt_timeout_us
>>> +    │   │   │   │   └── thresholds
>>> +    │   │   │   │       ├── cat_error_count
>>> +    │   │   │   │       ├── doorbell_time_us
>>> +    │   │   │   │       ├── engine_reset_count
>>> +    │   │   │   │       ├── h2g_time_us
>>> +    │   │   │   │       ├── irq_time_us
>>> +    │   │   │   │       └── page_fault_count
>>> +    │   │   │   └── gtX
>>> +    │   │   └── tileT
>>> +    │   └── vfN
>>> +..
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               This directory appears on the device when:
>>> +
>>> +                - device supports SR-IOV, and
>>> +                - device is a Physical Function (PF), and
>>> +                - xe driver supports SR-IOV PF on given device,
>>> and
>>> +                - xe driver supports automatic VFs provisioning.
>>> +
>>> +               This directory is used as a root for all attributes
>>> related to
>>> +               automatic provisioning of SR-IOV Physical Function
>>> (PF) and/or
>>> +               Virtual Functions (VFs).
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /enabled
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               (RW) bool (0, 1)
>>> +
>>> +               This file represents configuration flag for the
>>> automatic VFs
>>> +               (un)provisioning that could be performed by the PF.
>>> +
>>> +               The default value is 1 (true).
>>> +
>>> +               This flag can be set to false, unless manual
>>> provisioning is not
>>> +               applicable for given platform or it is not
>>> supported by current
>>> +               PF implementation. In such cases -EPERM will be
>>> returned.
>>> +
>>> +               This flag will be automatically set to false when
>>> there will be
>>> +               other attempts to change any of VF's resource
>>> provisioning.
>>> +               See "sriov_extensions" section for details.
>>> +
>>> +               This flag can be set back to true if and only if
>>> all VFs are
>>> +               fully unprovisioned, otherwise -EEXIST error will
>>> be returned.
>>> +
>>> +               false = "disabled"
>>> +                       When disabled, then PF will not attempt to
>>> do automatic
>>> +                       VFs provisioning when VFs are being enabled
>>> and will not
>>> +                       perform automatic unprovisioning of the VFs
>>> when VFs will
>>> +                       be disabled.
>>> +
>>> +               true = "enabled"
>>> +                       When enabled, then on VFs enabling PF will
>>> do automatic
>>> +                       VFs provisioning based on the default
>>> settings described
>>> +                       below.
>>> +
>>> +                       If automatic VFs provisioning fails due to
>>> some reasons,
>>> +                       then VFs will not be enabled.
>>> +
>>> +                       If enabled, all resources allocated during
>>> VFs enabling
>>> +                       will be released during VFs disabling
>>> (automatic unprovisioning).
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /admin_mode
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               (RW) bool (0, 1)
>>> +
>>> +               This file represents configuration flag for the
>>> automatic VFs
>>> +               provisioning that could be performed by the PF.
>>> +
>>> +               The default value depends on the platform type.
>>> +
>>> +               This flag can be changed any time, but will have no
>>> effect if
>>> +               VFs are already provisioned.
>>> +
>>> +               If enabled (default on discrete platforms) then the
>>> PF will
>>> +               retain only minimum hardcoded resources for its own
>>> use when
>>> +               doing VFs automatic provisioning and will not use
>>> any default
>>> +               values described below for its own configuration.
>>> +
>>> +               If disabled (default on integrated platforms) then
>>> the PF will
>>> +               treat itself like yet another additional VF in all
>>> fair resource
>>> +               allocations and will also try to apply default
>>> provisioning
>>> +               values described below for its own configuration.
>>> +
>>
>> One alternative could be to expose two sets of defaults, the PF and
>> VF 
>> ones. With the advantage of allowing the "admin mode" / "minimal PF"
>> to 
>> be explicitly configurable instead of hardcoded. Should be more
>> flexible.
>>
>> If the discrete vs integrated distinction is wanted it could simply
>> be 
>> made by initialy populating (driver init) the respective defaults
>> based 
>> on the platform type.
>>
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /reset_defaults
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               (WO) bool (1)
>>> +
>>> +               Writing to this file will reset all default
>>> provisioning parameters
>>> +               listed below to the default values.
>>
>> Maybe this isn't required if you can say it is the responsibility of 
>> whoever changes the defaults to either know what they are doing, or
>> to 
>> save and restore themselves if they. It is not a major concern but if
>> writing kernel code can be avoided perhaps it can be considered.
>>
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /resources/default_contexts_quota
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /resources/default_doorbells_quota
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /resources/default_ggtt_quota
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /resources/default_lmem_quota
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /scheduling/default_exec_quantum_ms
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /scheduling/default_preempt_timeout_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /monitoring/default_cat_error_count
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /monitoring/default_doorbell_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /monitoring/default_engine_reset_count
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /monitoring/default_h2g_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /monitoring/default_irq_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /monitoring/default_page_fault_count
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               These files represent default provisioning that
>>> should be used
>>> +               for VFs automatic provisioning.
>>> +
>>> +               These values can be changed any time, but will have
>>> no effect if
>>> +               VFs are already provisioned.
>>> +
>>> +               default_contexts_quota: (RW) integer 0..U32_MAX
>>> +                       The number of GuC context IDs to provide to
>>> the VF.
>>> +                       The default value is 0 (use fair
>>> allocations).
>>> +                       See
>>> "sriov_extensions/vfN/tileT/gtX/contexts_quota" for details.
>>> +
>>> +               default_doorbells_quota: (RW) integer 0..U32_MAX
>>> +                       The number of GuC doorbells to provide to
>>> the VF.
>>> +                       The default value is 0 (use fair
>>> allocations).
>>> +                       See
>>> "sriov_extensions/vfN/tileT/gtX/doorbells_quota" for details.
>>> +
>>> +               default_ggtt_quota: (RW) integer 0..U32_MAX
>>> +                       The size of the GGTT address space (in
>>> bytes) to provide to the VF.
>>> +                       The default value is 0 (use fair
>>> allocations).
>>> +                       See "sriov_extensions/vfN/tileT/ggtt_quota"
>>> for details.
>>> +
>>> +               default_lmem_quota: (RW) integer 0..U32_MAX
>>> +                       The size of the LMEM (in bytes) to provide
>>> to the VF.
>>> +                       The default value is 0 (use fair
>>> allocations).
>>> +                       See "sriov_extensions/vfN/tileT/lmem_quota"
>>> for details.
>>> +
>>> +               default_exec_quantum_ms: (RW) integer 0..U32_MAX
>>> +                       The GT execution quantum (in millisecs)
>>> assigned to the function.
>>> +                       The default value is 0 (infinify).
>>> +                       See
>>> "sriov_extensions/vfN/tileT/gtX/exec_quantum_ms" for details.
>>> +
>>> +               default_preempt_timeout_us: (RW) integer 0..U32_MAX
>>> +                       The GT preemption timeout (in microsecs)
>>> assigned to the function.
>>> +                       The default value is 0 (infinity).
>>> +                       See
>>> "sriov_extensions/vfN/tileT/gtX/preempt_timeout_us" for details.
>>
>> I have a slight concern here on the usability of GuC specific
>> tunables.
>>
>> Whereas one can imagine an external entity (some admin, somewhere) to
>> probably pretty much understand what it means to partition the local 
>> memory, address space, and set the scheduling timeouts (all are
>> intuitve 
>> and obvious concepts), how are they suppose to approach the GuC 
>> doorbells and contexts?
>>
>> It could be a matter of adding more documentation for those two, or
>> it 
>> even could make sense to shove them under a guc prefix (or 
>> subdirectory?) to signify the fact they are implementation details
>> and 
>> not a fundamental concept.
>>
>>> +
>>> +               default_cat_error_count: (RW) integer 0..U32_MAX
>>> +               default_doorbell_time_us: (RW) integer 0..U32_MAX
>>> +               default_engine_reset_count: (RW) integer 0..U32_MAX
>>> +               default_h2g_time_us: (RW) integer 0..U32_MAX
>>> +               default_irq_time_us: (RW) integer 0..U32_MAX
>>> +               default_page_fault_count: (RW) integer 0..U32_MAX
>>> +                       The monitoring threshold to be set for the
>>> function.
>>> +                       The default value is 0 (don't monitor).
>>> +                       See
>>> "sriov_extensions/vfN/tileT/gtX/thresholds" for details.
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               This directory appears on Xe device when:
>>> +
>>> +                - device supports SR-IOV, and
>>> +                - device is a Physical Function (PF), and
>>> +                - driver is enabled to support SR-IOV PF on given
>>> device.
>>> +
>>> +               This directory is used as a root for all attributes
>>> required to
>>> +               manage both Physical Function (PF) and Virtual
>>> Functions (VFs).
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/strict
>>> _scheduling_enabled
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               (RW) bool
>>> +
>>> +               This file represents a flag used to determine if
>>> scheduling
>>> +               parameters should be respected even if there is no
>>> active
>>> +               workloads submitted by the PF or VFs.
>>> +
>>> +               This flag is disabled by default, unless strict
>>> scheduling is
>>> +               not applicable on given platform. In such case this
>>> file will
>>> +               be read-only.
>>> +
>>> +               The change to this file may have no effect if VFs
>>> are not yet enabled.
>>> +               If strict scheduling can't be enabled in GuC then
>>> write will fail with -EIO.
>>
>> I think the semantics of this need to be documented ie. how it
>> interacts 
>> with exec_quantum_ms. If it does? I am guessing that it has to
>> otherwise 
>> I don't know what it would mean - presumably unused timeslices are
>> not 
>> given to other entities but time just goes wasted? But it is also a 
>> question on over what time interval. Or that too is purely defined by
>> the number of PF+VFs and their respective allocated quanta.
>>
>> Also, would there be benefit, assuming it is possible with GuC, to
>> allow 
>> configuring it per PF/VF?
>>
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/monito
>>> ring_period_ms
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               (RW) integer
>>> +
>>> +               This file represents the configuration knob used by
>>> adverse event
>>> +               monitoring. A value here is the period in millisecs
>>> during which
>>> +               events are counted and the total is checked against
>>> a threshold.
>>> +               See "sriov_extensions/vfN/tileT/gtX/thresholds" for
>>> more details.
>>> +
>>> +               Default is 0 (monitoring is disabled).
>>> +
>>> +               If monitoring capability is not available, then
>>> attempt to enable
>>> +               will fail with -EPERM error. If monitoring can't be
>>> enabled in
>>> +               GuC then write will fail with -EIO.
>>
>> Could the docs explain if there is a downside to enabling it, which
>> is 
>> probably why it isn't enabled by default? Because it does sound
>> natural 
>> that adverse events should be noticed.
>>
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               This directory holds all attributes related to the
>>> SR-IOV
>>> +               Physical Function (PF).
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               This directory holds all attributes related to the
>>> SR-IOV
>>> +               Virtual Function (VF).
>>> +
>>> +               Note that VF numbers (N) are 1-based as described
>>> in PCI SR-IOV specification.
>>> +               The Xe driver implementaton follows that naming
>>> schema.
>>> +
>>> +               There will be "vf1", "vf2" up to "vfN" directories,
>>> where N matches
>>> +               value of the PCI "sriov_totalvfs" attribute.
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               This directory holds all SR-IOV attributes related
>>> to the device tile.
>>> +               The tile numbers (T) start from 0.
>>> +
>>> +               There is at least one "tile0/" directory present.
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               This directory holds all SR-IOV attributes related
>>> to the device GT.
>>> +               The GT numbers (X) start from 0.
>>> +
>>> +               There is at least one "gt0/" directory present.
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/dev
>>> ice
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/de
>>> vice
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               (symbolic link)
>>> +
>>> +               Backlink to the PCI device entry representing given
>>> function.
>>> +               For PF this link is always present.
>>> +               For VF this link is present only for currently
>>> enabled VFs.
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/pri
>>> ority
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               (RW) string
>>> +
>>> +               This file represents a GuC Scheduler knob to
>>> override the default
>>> +               round-robin or FIFO scheduler policies implemented
>>> by the GuC.
>>> +
>>> +               The default value is "peer".
>>> +
>>> +               This flag can be changed, unless such change is not
>>> applicable
>>> +               for given platform or is not supported by current
>>> GuC firmware.
>>> +               In such case this file could be read-only or will
>>> return -EPERM
>>> +               on write attempt.
>>> +
>>> +               "immediate"
>>> +                       GuC will Schedule PF workloads immediately
>>> and PF
>>> +                       workloads only until the PF's work queues
>>> in GuC
>>> +                       are empty.
>>> +
>>> +               "lazy"
>>> +                       GuC will Schedule PF workloads at the next
>>> opportune
>>> +                       moment and PF workloads only until the PF
>>> work queues
>>> +                       in GuC are empty.
>>> +
>>> +               "peer"
>>> +                       GuC Scheduler will treat PF and VFs with
>>> equal priority.
>>
>> Hmmm this is too very GuC specific and I wonder what is the usecase
>> for 
>> lazy? Lazy = "don't care when it runs, but when it runs it will run 
>> everything queued so far", right? Feels a bit odd on first.
>>
>> "Immediate" may also not be depending on preemption granularity and 
>> workloads, right?
>>
>> Are there any ideas to express the knobs in a more generic fashion?
>>
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/st
>>> op
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               (WO) bool (1)
>>> +
>>> +               Write to this file will force GuC to stop handle
>>> any requests from
>>> +               this VF, but without triggering a FLR.
>>> +               To recover, the full FLR must be issued using
>>> generic "device/reset".
>>> +
>>> +               This file allows to implement custom policy
>>> mechanism when VF is
>>> +               misbehaving and triggering adverse events above
>>> defined thresholds.
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/exec_quantum_ms
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/preempt_timeout_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/exec_quantum_ms
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/preempt_timeout_us
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               These files represent scheduling parameters of the
>>> functions.
>>> +
>>> +               These scheduling parameters can be changed even if
>>> VFs are enabled
>>> +               and running, unless such change is not applicable
>>> on given platform
>>> +               due to fixed hardware or firmware assignment.
>>> +
>>> +               exec_quantum_ms: (RW) integer 0..U32_MAX
>>> +                       The GT execution quantum in [ms] assigned
>>> to the function.
>>> +                       Requested quantum might be aligned per
>>> HW/FW requirements.
>>> +
>>> +                       Default is 0 (unlimited).
>>> +
>>> +               preempt_timeout_us: (RW) integer 0..U32_MAX
>>> +                       The GT preemption timeout in [us] assigned
>>> to the function.
>>> +                       Requested timeout might be aligned per
>>> HW/FW requirements.
>>> +
>>> +                       Default is 0 (unlimited).
>>
>> Alignment for the above two will be visible after read-back?
>>
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/ggtt_quota
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/lmem_quota
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/contexts_quota
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/doorbells_quota
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               These files represent shared resource assigned to
>>> the functions.
>>> +
>>> +               These resource parameters can be changed, unless VF
>>> is already running,
>>> +               or such change is not applicable on given platform
>>> due to fixed hardware
>>> +               or firmware assignment.
>>> +
>>> +               Writes to these attributes may fail with:
>>> +                       -EPERM if change is not applicable on give
>>> HW/FW.
>>> +                       -E2BIG if value larger that HW/FW limit.
>>> +                       -EDQUOT if value is larger than maximum
>>> quota defined by the PF.
>>> +                       -ENOSPC if PF can't allocate required
>>> quota.
>>> +                       -EBUSY if the resource is currently in use
>>> by the VF.
>>> +                       -EIO if GuC refuses to change provisioning.
>>
>> Why it would refuse if input is valid? In other words, what is the 
>> user/admin supposed to do on -EIO?
>>
>>> +
>>> +               ggtt_quota: (RW) integer 0..U64_MAX
>>> +                       The size of the GGTT address space (in
>>> bytes) assigned to the VF.
>>> +                       The value might be aligned per HW/FW
>>> requirements.
>>> +
>>> +                       Default is 0 (unprovisioned).
>>> +
>>> +               lmem_quota: (RW) integer 0..U64_MAX
>>> +                       The size of the Local Memory (in bytes)
>>> assigned to the VF.
>>> +                       The value might be aligned per HW/FW
>>> requirements.
>>> +
>>> +                       This attribute is only available on
>>> discrete platforms.
>>> +
>>> +                       Default is 0 (unprovisioned).
>>> +
>>> +               contexts_quota: (RW) 0..U16_MAX
>>> +                       The number of GuC submission contexts
>>> assigned to the VF.
>>> +                       This value might be aligned per HW/FW
>>> requirements.
>>> +
>>> +                       Default is 0 (unprovisioned).
>>> +
>>> +               doorbells_quota: (RW) 0..U16_MAX
>>> +                       The number of GuC doorbells assigned to the
>>> VF.
>>> +                       This value might be aligned per HW/FW
>>> requirements.
>>> +
>>> +                       Default is 0 (unprovisioned).
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/thresholds/cat_error_count
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/thresholds/doorbell_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/thresholds/engine_reset_count
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/thresholds/h2g_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/thresholds/irq_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/thresholds/page_fault_count
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/thresholds/cat_error_count
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/thresholds/doorbell_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/thresholds/engine_reset_count
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/thresholds/h2g_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/thresholds/irq_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/thresholds/page_fault_count
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@xxxxxxxxxxxxxxxxxxxxx
>>> +Description:
>>> +               These files represent threshold values used by the
>>> GuC to trigger
>>> +               security events if adverse event monitoring is
>>> enabled.
>>
>> How are the security events delivered? There is mention of uevents in
>> a 
>> later paragraph - are they already defined or should be together with
>> this so the link can be place here?
>>
>>> +
>>> +               These thresholds are checked every
>>> "monitoring_period_ms".
>>> +               Refer to GuC ABI for details about each threshold
>>> category.
>>
>> Is it possible to have a link here to GuC ABI?
>>
>> Regards,
>>
>> Tvrtko
>>
>>> +
>>> +               Default value for all thresholds is 0 (disabled).
>>> +
>>> +               cat_error_count: (RW) integer
>>> +               doorbell_time_us: (RW) integer
>>> +               engine_reset_count: (RW) integer
>>> +               h2g_time_us: (RW) integer
>>> +               irq_time_us: (RW) integer
>>> +               page_fault_count: (RW) integer
>>> diff --git a/Documentation/gpu/rfc/xe_sriov.rst
>>> b/Documentation/gpu/rfc/xe_sriov.rst
>>> new file mode 100644
>>> index 000000000000..574f6414eabb
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/xe_sriov.rst
>>> @@ -0,0 +1,192 @@
>>> +.. SPDX-License-Identifier: MIT
>>> +
>>> +========================
>>> +Xe – SR-IOV Support Plan
>>> +========================
>>> +
>>> +The Single Root I/O Virtualization (SR-IOV) extension to the PCI
>>> Express (PCIe)
>>> +specification suite is supported starting from 12th generation of
>>> Intel Graphics
>>> +processors.
>>> +
>>> +This document describes planned ABI of the new Xe driver (see
>>> xe.rst) that will
>>> +provide flexible configuration and management options related to
>>> the SR-IOV.
>>> +It will also highlight few most important changes to the Xe driver
>>> +implementation to deal with Intel GPU SR-IOV specific
>>> requirements.
>>> +
>>> +
>>> +SR-IOV Capability
>>> +=================
>>> +
>>> +Due to SR-IOV complexity and required co-operation between
>>> hardware, firmware
>>> +and kernel drivers, not all Xe architecture platforms might have
>>> SR-IOV enabled
>>> +or fully functional.
>>> +
>>> +To control at the driver level which platform will provide support
>>> for SR-IOV,
>>> +as we can't just rely on the PCI configuration data exposed by the
>>> hardware,
>>> +we will introduce "has_sriov" flag to the struct xe_device_desc
>>> that describes
>>> +a device capabilities that driver checks during the probe.
>>> +
>>> +Initially this flag will be set to disabled even on platforms that
>>> we plan to
>>> +support. We will enable this flag only once we finish merging all
>>> required
>>> +changes to the driver and related validated firmwares are also
>>> made available.
>>> +
>>> +
>>> +SR-IOV Platforms
>>> +================
>>> +
>>> +Initially we plan to add SR-IOV functionality to the following SDV
>>> platforms
>>> +already supported by the Xe driver:
>>> +
>>> + - TGL (up to 7 VFs)
>>> + - ADL (up to 7 VFs)
>>> + - MTL (up to 7 VFs)
>>> + - ATSM (up to 31 VFs)
>>> + - PVC (up to 63 VFs)
>>> +
>>> +Newer platforms will be supported later, but we hope that enabling
>>> will be
>>> +much faster, as majority of the driver changes are either platform
>>> agnostic
>>> +or are similar between earlier platforms (hence we start with
>>> SDVs).
>>> +
>>> +
>>> +PF Mode
>>> +=======
>>> +
>>> +Support in the driver for acting in Physical Function (PF) mode,
>>> i.e. mode
>>> +that allows configuration of VFs, depends on the CONFIG_PCI_IOV
>>> and will be
>>> +enabled by default.
>>> +
>>> +However, due to potentially conflicting requirements for SR-IOV
>>> and other mega
>>> +features, we might want to have an option to disable SR-IOV PF
>>> mode support at
>>> +the driver load time.
>>> +
>>> +Thus, we plan to use additional modparam named "sriov_totalvfs"
>>> which if set to
>>> +0 will force the driver to operate in the native (non-virtualized)
>>> mode.
>>> +The same modparam could be used to limit number of supported
>>> Virtual Functions
>>> +(VFs) by the driver compared to the hardware limit exposed in PCI
>>> configuration.
>>> +
>>> +The name of this modparam corresponds to the existing PCI sysfs
>>> attribute, that
>>> +by default exposes hardware capability.
>>> +
>>> +The default value of this param will allow to support all possible
>>> VFs as
>>> +claimed by the hardware.
>>> +
>>> +This modparam will have no effect if driver is running on the VF
>>> device.
>>> +
>>> +
>>> +VFs Enabling
>>> +============
>>> +
>>> +To enable or disable VFs we plan to rely on existing sysfs
>>> attribute exposed by
>>> +the PCI subsystem named "sriov_numvfs". We will provide all
>>> necessary tweaks to
>>> +provision VFs in our custom implementation of the
>>> "sriov_configure" hook from
>>> +the struct pci_driver.
>>> +
>>> +If for some reason, including explicit request to disable SR-IOV
>>> PF mode using
>>> +modparam, we will not be able to correctly support any VFs, driver
>>> will change
>>> +number of supported VFs, exposed to the userspace by
>>> "sriov_totalvfs" attribute,
>>> +to 0, thus preventing configuration of the VFs.
>>> +
>>> +
>>> +VF Mode
>>> +=======
>>> +
>>> +When driver is running on the VF device, then due to hardware
>>> enforcements,
>>> +access to the privileged registers is not possible. To avoid
>>> relying on these
>>> +registers, we plan to perform early detection if we are running on
>>> the VF
>>> +device using dedicated VF_CAP(0x1901f8) register and then use
>>> global macro
>>> +IS_SRIOV_VF(xe) to control the driver logic.
>>> +
>>> +To speed up merging of the required changes, we might first
>>> introduce dummy
>>> +macro that is always set to false, to prepare driver to avoid some
>>> code paths
>>> +before we finalize our VF mode detection and other VFs enabling
>>> changes.
>>> +
>>> +
>>> +Resources
>>> +=========
>>> +
>>> +Most of the hardware (or firmware) resources available on the Xe
>>> architecture,
>>> +like GGTT, LMEM, GuC context IDs, GuC doorbells, will be shared
>>> between PF and
>>> +VFs and will require some provisioning steps to assign those
>>> resources for use
>>> +by the VF.
>>> +
>>> +Until VFs are provisioned with resources, the PF driver will be
>>> able to use all
>>> +resources, in the same way as it would be running in non-
>>> virtualized mode.
>>> +
>>> +If some resource (of part or region of it) is assigned to specific
>>> VF, then PF
>>> +is not allowed to use that part or region of the resource, but can
>>> continue to
>>> +use whatever is left available.
>>> +
>>> +Those resources are usually fully virtualized, so they will not
>>> require any
>>> +special handling when used by the VF driver, except that VF driver
>>> must know
>>> +the assigned quota.
>>> +
>>> +The most notable exception is the GGTT address space, as on some
>>> platforms,
>>> +the VF driver must additionally know the real range that it can
>>> access.
>>> +
>>> +Once the resources were assigned to the VF use and the VF driver
>>> has started,
>>> +then it is not allowed to change such provisioning, as that would
>>> break the
>>> +VF driver. To make changes the VF driver, which was using these
>>> resources,
>>> +must be unloaded (or the VM is terminated) and the VF device must
>>> be reset
>>> +using the FLR.
>>> +
>>> +
>>> +Scheduling
>>> +==========
>>> +
>>> +The workloads from PF driver and VF drivers must be submitted to
>>> the hardware
>>> +always by using the GuC submission mechanism. Unless VF has
>>> exclusive access
>>> +to the GT then submissions from different VFs are time-sliced and
>>> controlled
>>> +with additional "execution_quantum" and "preemption_timeout"
>>> parameters.
>>> +
>>> +In contrast to the resource provisioning, those scheduling
>>> parameters can be
>>> +changed even if VF drivers are already running and are active.
>>> +
>>> +
>>> +Automatic VFs Provisioning
>>> +==========================
>>> +
>>> +To provide out-of-the box experience when user will be enabling
>>> VFs using
>>> +generic "sriov_numvfs" attribute without requiring complex
>>> provisioning steps,
>>> +the SR-IOV PF driver will implement automatic VFs resource
>>> provisioning.
>>> +
>>> +By default, all VFs will be allocated with the fair amount of the
>>> mandatory
>>> +resources (like GGTT, GuC IDs) and with unrestricted scheduling
>>> parameters.
>>> +Such provisioning should be sufficient for most of the normal
>>> usages, when
>>> +no strict SLA is required.
>>> +
>>> +The PF driver will also expose some additional sysfs files to
>>> allow adjusting
>>> +this automatic VFs provisioning, like default values for most of
>>> the
>>> +provisioning parameters that PF will then apply for each enabled
>>> VF.
>>> +
>>> +    Details about those extension can be found in
>>> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
>>> +
>>> +
>>> +Manual VFs Provisioning
>>> +=======================
>>> +
>>> +If automatic VFs provisioning, which applies same configuration to
>>> every VF,
>>> +is not sufficient or there is a need for advanced customization of
>>> some VF,
>>> +the PF driver will also provide extended sysfs interface which
>>> will allow
>>> +control every provisioning attribute to the lowest feasible level.
>>> +
>>> +It is expected that these low-level attributes will be mostly used
>>> by the
>>> +advanced users or by the custom tools that will setup
>>> configurations that
>>> +meet predefined and validated SLA as required by the customers.
>>> +
>>> +    Details about those extension can be found in
>>> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
>>> +
>>> +
>>> +VFs Monitoring
>>> +==============
>>> +
>>> +In addition to the resource provisioning or changing scheduling
>>> parameters,
>>> +the PF driver might also allow configure some monitoring
>>> parameters, like
>>> +thresholds of adverse events or sample period, to track undesired
>>> behavior
>>> +of the VFs that could impact the whole system.
>>> +
>>> +Once those thresholds are setup and sampling period is defined,
>>> the GuC will
>>> +notify the PF driver about which VF is excessing the threshold and
>>> then PF is
>>> +able to trigger the uevent to notify the administrator (or VMM)
>>> that could
>>> +take some action against the VF.
> 



[Index of Archives]     [Linux DRI Users]     [Linux Intel Graphics]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [XFree86]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Linux Kernel]     [Linux SCSI]     [XFree86]
  Powered by Linux