On 5/27/22 02:50, Dmitry Osipenko wrote: > Hello, > > This patchset introduces memory shrinker for the VirtIO-GPU DRM driver > and adds memory purging and eviction support to VirtIO-GPU driver. > > The new dma-buf locking convention is introduced here as well. > > During OOM, the shrinker will release BOs that are marked as "not needed" > by userspace using the new madvise IOCTL, it will also evict idling BOs > to SWAP. The userspace in this case is the Mesa VirGL driver, it will mark > the cached BOs as "not needed", allowing kernel driver to release memory > of the cached shmem BOs on lowmem situations, preventing OOM kills. > > The Panfrost driver is switched to use generic memory shrinker. > > This patchset includes improvements and fixes for various things that > I found while was working on the shrinker. > > The Mesa and IGT patches will be kept on hold until this kernel series > will be approved and merged. > > This patchset was tested using Qemu and crosvm, including both cases of > IOMMU off/on. > > Mesa: https://gitlab.freedesktop.org/digetx/mesa/-/commits/virgl-madvise > IGT: https://gitlab.freedesktop.org/digetx/igt-gpu-tools/-/commits/virtio-madvise > https://gitlab.freedesktop.org/digetx/igt-gpu-tools/-/commits/panfrost-madvise > > Changelog: > > v6: - Added new VirtIO-related fix patch that previously was sent separately > and didn't get much attention: > > drm/gem: Properly annotate WW context on drm_gem_lock_reservations() error > > - Added new patch that fixes mapping of imported dma-bufs for > Tegra DRM and other affected drivers. It's also handy to have it > for switching to the new dma-buf locking convention scheme: > > drm/gem: Move mapping of imported dma-bufs to drm_gem_mmap_obj() > > - Added new patch that fixes shrinker list corruption for stable Panfrost > driver: > > drm/panfrost: Fix shrinker list corruption by madvise IOCTL > > - Added new minor patch-fix for drm-shmem: > > drm/shmem-helper: Add missing vunmap on error > > - Added fixes tag to the "Put mapping ..." patch like was suggested by > Steven Price. > > - Added new VirtIO-GPU driver improvement patch: > > drm/virtio: Return proper error codes instead of -1 > > - Reworked shrinker patches like was suggested by Daniel Vetter: > > - Introduced the new locking convention for dma-bufs. Tested on > VirtIO-GPU, Panfrost, Lima, Tegra and Intel selftests. > > - Dropped separate purge() callback. Now single evict() does > everything. > > - Dropped swap_in() callback from drm-shmem objects. DRM drivers > now could and should restore only the required mappings. > > - Dropped dynamic counting of evictable pages. This simplifies > code in exchange to *potentially* burning more CPU time on OOM. > > v5: - Added new for-stable patch "drm/panfrost: Put mapping instead of > shmem obj on panfrost_mmu_map_fault_addr() error" that corrects GEM's > refcounting in case of error. > > - The drm_gem_shmem_v[un]map() now takes a separate vmap_lock for > imported GEMs to avoid recursive locking of DMA reservations. > This addresses v4 comment from Thomas Zimmermann about the potential > deadlocking of vmapping. > > - Added ack from Thomas Zimmermann to "drm/shmem-helper: Correct > doc-comment of drm_gem_shmem_get_sg_table()" patch. > > - Dropped explicit shmem states from the generic shrinker patch as > was requested by Thomas Zimmermann. > > - Improved variable names and comments of the generic shrinker code. > > - Extended drm_gem_shmem_print_info() with the shrinker-state info in > the "drm/virtio: Support memory shrinking" patch. > > - Moved evict()/swap_in()/purge() callbacks from drm_gem_object_funcs > to drm_gem_shmem_object in the generic shrinker patch, for more > consistency. > > - Corrected bisectability of the patches that was broken in v4 > by accident. > > - The virtio_gpu_plane_prepare_fb() now uses drm_gem_shmem_pin() instead > of drm_gem_shmem_set_unpurgeable_and_unevictable() and does it only for > shmem BOs in the "drm/virtio: Support memory shrinking" patch. > > - Made more functions private to drm_gem_shmem_helper.c as was requested > by Thomas Zimmermann. This minimizes number of the public shmem helpers. > > v4: - Corrected minor W=1 warnings reported by kernel test robot for v3. > > - Renamed DRM_GEM_SHMEM_PAGES_STATE_ACTIVE/INACTIVE to PINNED/UNPINNED, > for more clarity. > > v3: - Hardened shrinker's count() with usage of READ_ONCE() since we don't > use atomic type for counting and technically compiler is free to > re-fetch counter's variable. > > - "Correct drm_gem_shmem_get_sg_table() error handling" now uses > PTR_ERR_OR_ZERO(), fixing typo that was made in v2. > > - Removed obsoleted shrinker from the Panfrost driver, which I missed to > do in v2 by accident and Alyssa Rosenzweig managed to notice it. > > - CCed stable kernels in all patches that make fixes, even the minor ones, > like was suggested by Emil Velikov and added his r-b to the patches. > > - Added t-b from Steven Price to the Panfrost's shrinker patch. > > - Corrected doc-comment of drm_gem_shmem_object.madv, like was suggested > by Steven Price. Comment now says that madv=1 means "object is purged" > instead of saying that value is unused. > > - Added more doc-comments to the new shmem shrinker API. > > - The "Improve DMA API usage for shmem BOs" patch got more improvements > by removing the obsoleted drm_dev_set_unique() quirk and its comment. > > - Added patch that makes Virtio-GPU driver to use common dev_is_pci() > helper, which was suggested by Robin Murphy. > > - Added new "drm/shmem-helper: Take GEM reservation lock instead of > drm_gem_shmem locks" patch, which was suggested by Daniel Vetter. > > - Added new "drm/virtio: Simplify error handling of > virtio_gpu_object_create()" patch. > > - Improved "Correct doc-comment of drm_gem_shmem_get_sg_table()" patch, > like was suggested by Daniel Vetter, by saying that function returns > ERR_PTR() and not errno. > > - virtio_gpu_purge_object() is fenced properly now, turned out > virtio_gpu_notify() doesn't do fencing as I was supposing before. > Stress testing of memory eviction revealed that. > > - Added new patch that corrects virtio_gpu_plane_cleanup_fb() to use > appropriate atomic plane state. > > - SHMEM shrinker got eviction support. > > - VirtIO-GPU driver now supports memory eviction. It's enabled for a > non-blob GEMs only, i.e. for VirGL. The blobs don't support dynamic > attaching/detaching of guest's memory, so it's not trivial to enable > them. > > - Added patch that removes obsoleted drm_gem_shmem_purge() > > - Added patch that makes drm_gem_shmem_get_pages() private. > > - Added patch that fixes lockup on dma_resv_reserve_fences() error. > > v2: - Improved shrinker by using a more fine-grained locking to reduce > contention during scan of objects and dropped locking from the > 'counting' callback by tracking count of shrinkable pages. This > was suggested by Rob Clark in the comment to v1. > > - Factored out common shrinker code into drm_gem_shmem_helper.c > and switched Panfrost driver to use the new common memory shrinker. > This was proposed by Thomas Zimmermann in his prototype series that > he shared with us in the comment to v1. Note that I only compile-tested > the Panfrost driver. > > - Shrinker now takes object_name_lock during scan to prevent racing > with dma-buf exporting. > > - Shrinker now takes vmap_lock during scan to prevent racing with shmem > vmap/unmap code. > > - Added "Correct doc-comment of drm_gem_shmem_get_sg_table()" patch, > which I sent out previously as a standalone change, since the > drm_gem_shmem_helper.c is now touched by this patchset anyways and > it doesn't hurt to group all the patches together. > > Dmitry Osipenko (22): > drm/gem: Properly annotate WW context on drm_gem_lock_reservations() > error > drm/gem: Move mapping of imported dma-bufs to drm_gem_mmap_obj() > drm/panfrost: Put mapping instead of shmem obj on > panfrost_mmu_map_fault_addr() error > drm/panfrost: Fix shrinker list corruption by madvise IOCTL > drm/virtio: Correct drm_gem_shmem_get_sg_table() error handling > drm/virtio: Check whether transferred 2D BO is shmem > drm/virtio: Unlock reservations on virtio_gpu_object_shmem_init() > error > drm/virtio: Unlock reservations on dma_resv_reserve_fences() error > drm/virtio: Use appropriate atomic state in > virtio_gpu_plane_cleanup_fb() > drm/shmem-helper: Add missing vunmap on error > drm/shmem-helper: Correct doc-comment of drm_gem_shmem_get_sg_table() ... Thomas, do you think it will be possible for you to take the fix patches 1-11 into the drm-fixes or would you prefer me to re-send them separately? The VirtIO patches 12-13 also are good to go into drm-next, IMO. I'm going to factor out the new dma-buf convention into a separate patchset, like was suggested by Christian. But it will take me some time to get the dma-buf patches ready and I also will be on a vacation soon. At minimum nothing should hold the fixes, so will be great if they could land sooner. Thank you! -- Best regards, Dmitry
