On 7/22/22 08:34, Jason Gunthorpe
wrote:
On Thu, Jul 21, 2022 at 07:00:23PM -0400, Felix Kuehling wrote:Hi all, We're noticing some unexpected behaviour when the amdgpu and Mellanox drivers are interacting on shared memory with hmm_range_fault. If the amdgpu driver migrated pages to DEVICE_PRIVATE memory, we would expect hmm_range_fault called by the Mellanox driver to fault them back to system memory. But that's not happening. Instead hmm_range_fault fails. For an experiment, Philip hacked hmm_vma_handle_pte to treat DEVICE_PRIVATE pages like device_exclusive pages, which gave us the expected behaviour. It would result in a dev_pagemap_ops.migrate_to_ram callback in our driver, and hmm_range_fault would return system memory pages to the Mellanox driver. So something is clearly wrong. It could be: * our expectations are wrong, * the implementation of hmm_range_fault is wrong, or * our driver is missing something when migrating to DEVICE_PRIVATE memory. Do you have any insights?I think it is a bug Jason
Yes, looks like a bug to me too. In hmm_vma_handle_pte(), it calls hmm_is_device_private_entry() which correctly handles the case where the device private entry is owned by the driver calling hmm_range_fault() but then does nothing to fault in the page if it is a device private entry not owned by the driver. I'll work with Alistair and one of us will post a fix. Thanks for finding this!
