On 23/09/2024 11.38, Danilo Krummrich wrote: > External email: Use caution opening links or attachments > > > On Sun, Sep 22, 2024 at 04:11:21PM +0300, Zhi Wang wrote: >> On Sun, 22 Sep 2024 05:49:22 -0700 >> Zhi Wang <zhiw@xxxxxxxxxx> wrote: >> >> +Ben. >> >> Forget to add you. My bad. > > Please also add the driver maintainers! > > I had to fetch the patchset from the KVM list, since they did not hit the > nouveau list (I'm trying to get @nvidia.com addresses whitelisted). > > - Danilo > My bad. Will do in the next iteration. Weird...never thought this cound happen since alll my previous emails landed in the mailing list. Did you see any email discussion in the thread in the nouveau list? Feel free to let me know if I should send them again to nouveau list. Maybe it is also easier that you can pull the patches from my tree. Note that I will be on vacation until Oct 11th. Email reply might be slow. But I will read the emails in the mailing list. Thanks, Zhi. >> >> >>> 1. Background >>> ============= >>> >>> NVIDIA vGPU[1] software enables powerful GPU performance for workloads >>> ranging from graphics-rich virtual workstations to data science and >>> AI, enabling IT to leverage the management and security benefits of >>> virtualization as well as the performance of NVIDIA GPUs required for >>> modern workloads. Installed on a physical GPU in a cloud or enterprise >>> data center server, NVIDIA vGPU software creates virtual GPUs that can >>> be shared across multiple virtual machines. >>> >>> The vGPU architecture[2] can be illustrated as follow: >>> >>> +--------------------+ +--------------------+ >>> +--------------------+ +--------------------+ | Hypervisor | >>> | Guest VM | | Guest VM | | Guest VM >>> | | | | +----------------+ | | >>> +----------------+ | | +----------------+ | | +----------------+ | >>> | |Applications... | | | |Applications... | | | |Applications... | | >>> | | NVIDIA | | | +----------------+ | | +----------------+ >>> | | +----------------+ | | | Virtual GPU | | | >>> +----------------+ | | +----------------+ | | +----------------+ | | >>> | Manager | | | | Guest Driver | | | | Guest Driver | | >>> | | Guest Driver | | | +------^---------+ | | +----------------+ >>> | | +----------------+ | | +----------------+ | | | >>> | +---------^----------+ +----------^---------+ >>> +----------^---------+ | | | | >>> | | | | >>> +--------------+-----------------------+----------------------+---------+ >>> | | | | >>> | | | | | >>> | | | >>> +--------+--------------------------+-----------------------+----------------------+---------+ >>> +---------v--------------------------+-----------------------+----------------------+----------+ >>> | NVIDIA +----------v---------+ >>> +-----------v--------+ +-----------v--------+ | | Physical GPU >>> | Virtual GPU | | Virtual GPU | | Virtual GPU >>> | | | +--------------------+ >>> +--------------------+ +--------------------+ | >>> +----------------------------------------------------------------------------------------------+ >>> >>> Each NVIDIA vGPU is analogous to a conventional GPU, having a fixed >>> amount of GPU framebuffer, and one or more virtual display outputs or >>> "heads". The vGPU’s framebuffer is allocated out of the physical >>> GPU’s framebuffer at the time the vGPU is created, and the vGPU >>> retains exclusive use of that framebuffer until it is destroyed. >>> >>> The number of physical GPUs that a board has depends on the board. >>> Each physical GPU can support several different types of virtual GPU >>> (vGPU). vGPU types have a fixed amount of frame buffer, number of >>> supported display heads, and maximum resolutions. They are grouped >>> into different series according to the different classes of workload >>> for which they are optimized. Each series is identified by the last >>> letter of the vGPU type name. >>> >>> NVIDIA vGPU supports Windows and Linux guest VM operating systems. The >>> supported vGPU types depend on the guest VM OS. >>> >>> 2. Proposal for upstream >>> ======================== >>> >>> 2.1 Architecture >>> ---------------- >>> >>> Moving to the upstream, the proposed architecture can be illustrated >>> as followings: >>> >>> +--------------------+ >>> +--------------------+ +--------------------+ | Linux VM | >>> | Windows VM | | Guest VM | | +----------------+ | >>> | +----------------+ | | +----------------+ | | |Applications... | | >>> | |Applications... | | | |Applications... | | | +----------------+ | >>> | +----------------+ | | +----------------+ | ... | >>> +----------------+ | | +----------------+ | | +----------------+ | | >>> | Guest Driver | | | | Guest Driver | | | | Guest Driver | | | >>> +----------------+ | | +----------------+ | | +----------------+ | >>> +---------^----------+ +----------^---------+ +----------^---------+ >>> | | | >>> +--------------------------------------------------------------------+ >>> |+--------------------+ +--------------------+ >>> +--------------------+| || QEMU | | QEMU >>> | | QEMU || || | | >>> | | || |+--------------------+ >>> +--------------------+ +--------------------+| >>> +--------------------------------------------------------------------+ >>> | | | >>> +-----------------------------------------------------------------------------------------------+ >>> | >>> +----------------------------------------------------------------+ | >>> | | VFIO >>> | | | | >>> | | | >>> +-----------------------+ | +------------------------+ >>> +---------------------------------+| | | | Core Driver vGPU | | >>> | | | || | | >>> | Support <--->| <----> >>> || | | +-----------------------+ | | NVIDIA vGPU >>> Manager | | NVIDIA vGPU VFIO Variant Driver || | | | NVIDIA >>> GPU Core | | | | | >>> || | | | Driver | | >>> +------------------------+ +---------------------------------+| | | >>> +--------^--------------+ >>> +----------------------------------------------------------------+ | >>> | | | | >>> | | >>> +-----------------------------------------------------------------------------------------------+ >>> | | | >>> | >>> +----------|--------------------------|-----------------------|----------------------|----------+ >>> | v +----------v---------+ >>> +-----------v--------+ +-----------v--------+ | | NVIDIA >>> | PCI VF | | PCI VF | | PCI VF >>> | | | Physical GPU | | | >>> | | | | | | >>> (Virtual GPU) | | (Virtual GPU) | | (Virtual GPU) | | | >>> +--------------------+ +--------------------+ >>> +--------------------+ | >>> +-----------------------------------------------------------------------------------------------+ >>> >>> The supported GPU generations will be Ada which come with the >>> supported GPU architecture. Each vGPU is backed by a PCI virtual >>> function. >>> >>> The NVIDIA vGPU VFIO module together with VFIO sits on VFs, provides >>> extended management and features, e.g. selecting the vGPU types, >>> support live migration and driver warm update. >>> >>> Like other devices that VFIO supports, VFIO provides the standard >>> userspace APIs for device lifecycle management and advance feature >>> support. >>> >>> The NVIDIA vGPU manager provides necessary support to the NVIDIA vGPU >>> VFIO variant driver to create/destroy vGPUs, query available vGPU >>> types, select the vGPU type, etc. >>> >>> On the other side, NVIDIA vGPU manager talks to the NVIDIA GPU core >>> driver, which provide necessary support to reach the HW functions. >>> >>> 2.2 Requirements to the NVIDIA GPU core driver >>> ---------------------------------------------- >>> >>> The primary use case of CSP and enterprise is a standalone minimal >>> drivers of vGPU manager and other necessary components. >>> >>> NVIDIA vGPU manager talks to the NVIDIA GPU core driver, which provide >>> necessary support to: >>> >>> - Load the GSP firmware, boot the GSP, provide commnication channel. >>> - Manage the shared/partitioned HW resources. E.g. reserving FB >>> memory, channels for the vGPU mananger to create vGPUs. >>> - Exception handling. E.g. delivering the GSP events to vGPU manager. >>> - Host event dispatch. E.g. suspend/resume. >>> - Enumerations of HW configuration. >>> >>> The NVIDIA GPU core driver, which sits on the PCI device interface of >>> NVIDIA GPU, provides support to both DRM driver and the vGPU manager. >>> >>> In this RFC, the split nouveau GPU driver[3] is used as an example to >>> demostrate the requirements of vGPU manager to the core driver. The >>> nouveau driver is split into nouveau (the DRM driver) and nvkm (the >>> core driver). >>> >>> 3 Try the RFC patches >>> ----------------------- >>> >>> The RFC supports to create one VM to test the simple GPU workload. >>> >>> - Host kernel: >>> https://github.com/zhiwang-nvidia/linux/tree/zhi/vgpu-mgr-rfc >>> - Guest driver package: NVIDIA-Linux-x86_64-535.154.05.run [4] >>> >>> Install guest driver: >>> # export GRID_BUILD=1 >>> # ./NVIDIA-Linux-x86_64-535.154.05.run >>> >>> - Tested platforms: L40. >>> - Tested guest OS: Ubutnu 24.04 LTS. >>> - Supported experience: Linux rich desktop experience with simple 3D >>> workload, e.g. glmark2 >>> >>> 4 Demo >>> ------ >>> >>> A demo video can be found at: https://youtu.be/YwgIvvk-V94 >>> >>> [1] https://www.nvidia.com/en-us/data-center/virtual-solutions/ >>> [2] >>> https://docs.nvidia.com/vgpu/17.0/grid-vgpu-user-guide/index.html#architecture-grid-vgpu >>> [3] >>> https://lore.kernel.org/dri-devel/20240613170211.88779-1-bskeggs@xxxxxxxxxx/T/ >>> [4] >>> https://us.download.nvidia.com/XFree86/Linux-x86_64/535.154.05/NVIDIA-Linux-x86_64-535.154.05.run >>> >>> Zhi Wang (29): >>> nvkm/vgpu: introduce NVIDIA vGPU support prelude >>> nvkm/vgpu: attach to nvkm as a nvkm client >>> nvkm/vgpu: reserve a larger GSP heap when NVIDIA vGPU is enabled >>> nvkm/vgpu: set the VF partition count when NVIDIA vGPU is enabled >>> nvkm/vgpu: populate GSP_VF_INFO when NVIDIA vGPU is enabled >>> nvkm/vgpu: set RMSetSriovMode when NVIDIA vGPU is enabled >>> nvkm/gsp: add a notify handler for GSP event >>> GPUACCT_PERFMON_UTIL_SAMPLES >>> nvkm/vgpu: get the size VMMU segment from GSP firmware >>> nvkm/vgpu: introduce the reserved channel allocator >>> nvkm/vgpu: introduce interfaces for NVIDIA vGPU VFIO module >>> nvkm/vgpu: introduce GSP RM client alloc and free for vGPU >>> nvkm/vgpu: introduce GSP RM control interface for vGPU >>> nvkm: move chid.h to nvkm/engine. >>> nvkm/vgpu: introduce channel allocation for vGPU >>> nvkm/vgpu: introduce FB memory allocation for vGPU >>> nvkm/vgpu: introduce BAR1 map routines for vGPUs >>> nvkm/vgpu: introduce engine bitmap for vGPU >>> nvkm/vgpu: introduce pci_driver.sriov_configure() in nvkm >>> vfio/vgpu_mgr: introdcue vGPU lifecycle management prelude >>> vfio/vgpu_mgr: allocate GSP RM client for NVIDIA vGPU manager >>> vfio/vgpu_mgr: introduce vGPU type uploading >>> vfio/vgpu_mgr: allocate vGPU FB memory when creating vGPUs >>> vfio/vgpu_mgr: allocate vGPU channels when creating vGPUs >>> vfio/vgpu_mgr: allocate mgmt heap when creating vGPUs >>> vfio/vgpu_mgr: map mgmt heap when creating a vGPU >>> vfio/vgpu_mgr: allocate GSP RM client when creating vGPUs >>> vfio/vgpu_mgr: bootload the new vGPU >>> vfio/vgpu_mgr: introduce vGPU host RPC channel >>> vfio/vgpu_mgr: introduce NVIDIA vGPU VFIO variant driver >>> >>> .../drm/nouveau/include/nvkm/core/device.h | 3 + >>> .../drm/nouveau/include/nvkm/engine/chid.h | 29 + >>> .../gpu/drm/nouveau/include/nvkm/subdev/gsp.h | 1 + >>> .../nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h | 45 ++ >>> .../nvidia/inc/ctrl/ctrl2080/ctrl2080gpu.h | 12 + >>> drivers/gpu/drm/nouveau/nvkm/Kbuild | 1 + >>> drivers/gpu/drm/nouveau/nvkm/device/pci.c | 33 +- >>> .../gpu/drm/nouveau/nvkm/engine/fifo/chid.c | 49 +- >>> .../gpu/drm/nouveau/nvkm/engine/fifo/chid.h | 26 +- >>> .../gpu/drm/nouveau/nvkm/engine/fifo/r535.c | 3 + >>> .../gpu/drm/nouveau/nvkm/subdev/gsp/r535.c | 14 +- >>> drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/Kbuild | 3 + >>> drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c | 302 +++++++++++ >>> .../gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c | 234 ++++++++ >>> drivers/vfio/pci/Kconfig | 2 + >>> drivers/vfio/pci/Makefile | 2 + >>> drivers/vfio/pci/nvidia-vgpu/Kconfig | 13 + >>> drivers/vfio/pci/nvidia-vgpu/Makefile | 8 + >>> drivers/vfio/pci/nvidia-vgpu/debug.h | 18 + >>> .../nvidia/inc/ctrl/ctrl0000/ctrl0000system.h | 30 + >>> .../nvidia/inc/ctrl/ctrl2080/ctrl2080gpu.h | 33 ++ >>> .../ctrl/ctrl2080/ctrl2080vgpumgrinternal.h | 152 ++++++ >>> .../common/sdk/nvidia/inc/ctrl/ctrla081.h | 109 ++++ >>> .../nvrm/common/sdk/nvidia/inc/dev_vgpu_gsp.h | 213 ++++++++ >>> .../common/sdk/nvidia/inc/nv_vgpu_types.h | 51 ++ >>> .../common/sdk/vmioplugin/inc/vmioplugin.h | 26 + >>> .../pci/nvidia-vgpu/include/nvrm/nvtypes.h | 24 + >>> drivers/vfio/pci/nvidia-vgpu/nvkm.h | 94 ++++ >>> drivers/vfio/pci/nvidia-vgpu/rpc.c | 242 +++++++++ >>> drivers/vfio/pci/nvidia-vgpu/vfio.h | 43 ++ >>> drivers/vfio/pci/nvidia-vgpu/vfio_access.c | 297 ++++++++++ >>> drivers/vfio/pci/nvidia-vgpu/vfio_main.c | 511 >>> ++++++++++++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu.c | >>> 352 ++++++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c | 144 >>> +++++ drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h | 89 +++ >>> drivers/vfio/pci/nvidia-vgpu/vgpu_types.c | 466 ++++++++++++++++ >>> include/drm/nvkm_vgpu_mgr_vfio.h | 61 +++ >>> 37 files changed, 3702 insertions(+), 33 deletions(-) >>> create mode 100644 drivers/gpu/drm/nouveau/include/nvkm/engine/chid.h >>> create mode 100644 >>> drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h create mode >>> 100644 drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/Kbuild create mode >>> 100644 drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c create mode >>> 100644 drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c create mode >>> 100644 drivers/vfio/pci/nvidia-vgpu/Kconfig create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/Makefile create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/debug.h create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl0000/ctrl0000system.h >>> create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080gpu.h >>> create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080vgpumgrinternal.h >>> create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrla081.h >>> create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/dev_vgpu_gsp.h >>> create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/nv_vgpu_types.h >>> create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/vmioplugin/inc/vmioplugin.h >>> create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/include/nvrm/nvtypes.h create mode >>> 100644 drivers/vfio/pci/nvidia-vgpu/nvkm.h create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/rpc.c create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/vfio.h create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/vfio_access.c create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/vfio_main.c create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/vgpu.c create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h create mode 100644 >>> drivers/vfio/pci/nvidia-vgpu/vgpu_types.c create mode 100644 >>> include/drm/nvkm_vgpu_mgr_vfio.h >>> >>
