Re: drm/amdkfd: Relocate TBA/TMA to opposite side of VM hole (v2)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello Felix,
I think you are right. This problem has existed for years(just look at the
issue creation time in my link), and is thought caused by OpenGL-ROCM
interop(that's why I think this patch might help). It is very easy to
trigger this problem in blender(method is also mentioned in the link). Do
you have any idea about this?
Best regards,
Shengyu
在 2024/1/29 22:51, Felix Kuehling 写道:
On 2024-01-29 8:58, Shengyu Qu wrote:
Hi,
Seems rocm-opengl interop hang problem still exists[1]. Btw have you
discovered into this problem?
Best regards,
Shengyu
[1] https://projects.blender.org/blender/blender/issues/100353#issuecomment-1111599

Maybe you're having a different problem. Do you see this issue also without any version of the "Relocate TBA/TMA ..." patch?

Regards,
  Felix



在 2024/1/27 03:15, Shengyu Qu 写道:
Hello Felix,
This patch seems working on my system, also it seems fixes the ROCM/OpenGL
interop problem.
Is this intended to happen or not? Maybe we need more users to test it.
Besides,
Tested-by: Shengyu Qu <wiagn233@xxxxxxxxxxx>
Best Regards,
Shengyu

在 2024/1/26 06:27, Felix Kuehling 写道:
The TBA and TMA, along with an unused IB allocation, reside at low
addresses in the VM address space. A stray VM fault which hits these
pages must be serviced by making their page table entries invalid.
The scheduler depends upon these pages being resident and fails,
preventing a debugger from inspecting the failure state.

By relocating these pages above 47 bits in the VM address space they
can only be reached when bits [63:48] are set to 1. This makes it much
less likely for a misbehaving program to generate accesses to them.
The current placement at VA (PAGE_SIZE*2) is readily hit by a NULL
access with a small offset.

v2:
- Move it to the reserved space to avoid concflicts with Mesa
- Add macros to make reserved space management easier

Cc: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@xxxxxxx>
Cc: Christian Koenig <christian.koenig@xxxxxxx>
Signed-off-by: Jay Cornwall <jay.cornwall@xxxxxxx>
Signed-off-by: Felix Kuehling <felix.kuehling@xxxxxxx>
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c      |  4 +--
  drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c    |  7 ++---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h       | 12 ++++++--
  drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 30 +++++++++++---------
  4 files changed, 30 insertions(+), 23 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
index 823d31f4a2a3..53d0a458d78e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
@@ -28,9 +28,9 @@
    uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev)
  {
-    uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT;
+    uint64_t addr = AMDGPU_VA_RESERVED_CSA_START(
+        adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT);
  -    addr -= AMDGPU_VA_RESERVED_CSA_SIZE;
      addr = amdgpu_gmc_sign_extend(addr);
        return addr;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c
index 3d0d56087d41..9e769ef50f2e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c
@@ -45,11 +45,8 @@
   */
  static inline u64 amdgpu_seq64_get_va_base(struct amdgpu_device *adev)
  {
-    u64 addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT;
-
-    addr -= AMDGPU_VA_RESERVED_TOP;
-
-    return addr;
+    return AMDGPU_VA_RESERVED_SEQ64_START(
+        adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT);
  }
    /**
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index 666698a57192..f23b6153d310 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -135,11 +135,19 @@ struct amdgpu_mem_stats;
  #define AMDGPU_IS_MMHUB0(x) ((x) >= AMDGPU_MMHUB0_START && (x) < AMDGPU_MMHUB1_START)   #define AMDGPU_IS_MMHUB1(x) ((x) >= AMDGPU_MMHUB1_START && (x) < AMDGPU_MAX_VMHUBS)
  -/* Reserve 2MB at top/bottom of address space for kernel use */
+/* Reserve space at top/bottom of address space for kernel use */
  #define AMDGPU_VA_RESERVED_CSA_SIZE        (2ULL << 20)
+#define AMDGPU_VA_RESERVED_CSA_START(top)    ((top) \
+                         - AMDGPU_VA_RESERVED_CSA_SIZE)
  #define AMDGPU_VA_RESERVED_SEQ64_SIZE        (2ULL << 20)
+#define AMDGPU_VA_RESERVED_SEQ64_START(top) (AMDGPU_VA_RESERVED_CSA_START(top) \
+                         - AMDGPU_VA_RESERVED_SEQ64_SIZE)
+#define AMDGPU_VA_RESERVED_TRAP_SIZE        (2ULL << 12)
+#define AMDGPU_VA_RESERVED_TRAP_START(top) (AMDGPU_VA_RESERVED_SEQ64_START(top) \
+                         - AMDGPU_VA_RESERVED_TRAP_SIZE)
  #define AMDGPU_VA_RESERVED_BOTTOM        (2ULL << 20)
-#define AMDGPU_VA_RESERVED_TOP (AMDGPU_VA_RESERVED_SEQ64_SIZE + \
+#define AMDGPU_VA_RESERVED_TOP (AMDGPU_VA_RESERVED_TRAP_SIZE + \
+                         AMDGPU_VA_RESERVED_SEQ64_SIZE + \
                           AMDGPU_VA_RESERVED_CSA_SIZE)
    /* See vm_update_mode */
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
index 6604a3f99c5e..f899cce25b2a 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
@@ -36,6 +36,7 @@
  #include <linux/mm.h>
  #include <linux/mman.h>
  #include <linux/processor.h>
+#include "amdgpu_vm.h"
    /*
   * The primary memory I/O features being added for revisions of gfxip @@ -326,10 +327,16 @@ static void kfd_init_apertures_vi(struct kfd_process_device *pdd, uint8_t id)
       * with small reserved space for kernel.
       * Set them to CANONICAL addresses.
       */
-    pdd->gpuvm_base = SVM_USER_BASE;
+    pdd->gpuvm_base = max(SVM_USER_BASE, AMDGPU_VA_RESERVED_BOTTOM);
      pdd->gpuvm_limit =
pdd->dev->kfd->shared_resources.gpuvm_size - 1;
  +    /* dGPUs: the reserved space for kernel
+     * before SVM
+     */
+    pdd->qpd.cwsr_base = SVM_CWSR_BASE;
+    pdd->qpd.ib_base = SVM_IB_BASE;
+
      pdd->scratch_base = MAKE_SCRATCH_APP_BASE_VI();
      pdd->scratch_limit = MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base);
  }
@@ -339,18 +346,19 @@ static void kfd_init_apertures_v9(struct kfd_process_device *pdd, uint8_t id)
      pdd->lds_base = MAKE_LDS_APP_BASE_V9();
      pdd->lds_limit = MAKE_LDS_APP_LIMIT(pdd->lds_base);
  -        /* Raven needs SVM to support graphic handle, etc. Leave the small -         * reserved space before SVM on Raven as well, even though we don't
-         * have to.
-         * Set gpuvm_base and gpuvm_limit to CANONICAL addresses so that they
-         * are used in Thunk to reserve SVM.
-         */
-        pdd->gpuvm_base = SVM_USER_BASE;
+    pdd->gpuvm_base = AMDGPU_VA_RESERVED_BOTTOM;
      pdd->gpuvm_limit =
pdd->dev->kfd->shared_resources.gpuvm_size - 1;
        pdd->scratch_base = MAKE_SCRATCH_APP_BASE_V9();
      pdd->scratch_limit = MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base);
+
+    /*
+     * Place TBA/TMA on opposite side of VM hole to prevent
+     * stray faults from triggering SVM on these pages.
+     */
+    pdd->qpd.cwsr_base = AMDGPU_VA_RESERVED_TRAP_START(
+        pdd->dev->adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT);
  }
    int kfd_init_apertures(struct kfd_process *process)
@@ -407,12 +415,6 @@ int kfd_init_apertures(struct kfd_process *process)
                      return -EINVAL;
                  }
              }
-
-                        /* dGPUs: the reserved space for kernel
-                         * before SVM
-                         */
-                        pdd->qpd.cwsr_base = SVM_CWSR_BASE;
-                        pdd->qpd.ib_base = SVM_IB_BASE;
          }
            dev_dbg(kfd_device, "node id %u\n", id);

Attachment: OpenPGP_0xE3520CC91929C8E7.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature


[Index of Archives]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux