On Tue, 2014-07-01 at 11:15 +0200, Joerg Roedel wrote: > On Mon, Jun 30, 2014 at 02:35:57PM -0400, Jerome Glisse wrote: > > We do intend to tear down all secondary mapping inside the relase > > callback but still we can not cleanup all the resources associated > > with it. > > > > And why can't you cleanup the other resources in the file close path? > Tearing down the mappings is all you need to do in the release function > anyway. > > > As said from the release call back you can not call > > mmu_notifier_unregister and thus you can not fully cleanup things. > > You don't need to call mmu_notifier_unregister when the release function > is already running from exit_mmap because this is equivalent to calling > mmu_notifier_unregister. > > > Only way to achieve so is to do it ouside mmu_notifier callback. > > The resources that can't be handled there can be cleaned up in the > file-close path. No need for a new notifier in mm code. > > In the end all you need to do in the release function is to tear down > the secondary mapping and make sure the device can no longer access the > address space when the release function returns. Everything else, like > freeing any resources can be done later when the file descriptors are > teared down. I will answer from the KFD perpective, as I'm AMD's maintainer of this driver. Little background: AMD's HSA Linux kernel driver (called radeon_kfd or KFD in short), has been developed for the past year by AMD, to support running Linux compute applications on AMD's HSA-enabled APUs, i.e Kaveri (A10-7850K/7700K). The driver will be up for kernel community review in about 2-3 weeks so we could push it during the 3.17 merge window. Prior discussions were made with gpu/drm subsystem maintainers about this driver. In the KFD, we need to maintain a notion of each compute process. Therefore, we have an object called "kfd_process" that is created for each process that uses the KFD. Naturally, we need to be able to track the process's shutdown in order to perform cleanup of the resources it uses (compute queues, virtual address space, gpu local memory allocations, etc.). To enable this tracking mechanism, we decided to associate the kfd_process with mm_struct to ensure that a kfd_process object has exactly the same lifespan as the process it represents. We preferred to use the mm_struct and not a file description because using a file descriptor to track “process” shutdown is wrong in two ways: * Technical: file descriptors can be passed to unrelated processes using AF_UNIX sockets. This means that a process can exit while the file stays open. Even if we implement this “correctly” i.e. holding the address space & page tables alive until the file is finally released, it’s really dodgy. * Philosophical: our ioctls are actually system calls in disguise. They operate on the process, not on a device. Moreover, because the GPU interacts with the process only through virtual memory (and not e.g. file descriptors), and because virtual address space is fundamental to an intuitive notion of what a process is, the decision to associate the kfd_process with mm_struct seems like a natural choice. Then arrived the issue of how the KFD is notified about an mm_struct destruction. Because the mmu_notifier release callback is called from an RCU read lock, it can't destory the mmu_notifier object, which is the kfd_process object itself. Therefore, I talked to Jerome and Andrew Morton on a way to implement this and after the discussion (which was in private emails), Jerome was kind enough to write a patch, which is the patch we are now discussing. You are more than welcomed to take a look at the entire driver, at http://cgit.freedesktop.org/~gabbayo/linux/?h=kfd-0.6.x although the driver will undergo some changes before sending the pull request to Dave Airle. I believe that converting amd_iommu_v2 driver to use this patch as well, will benefit all parties. AFAIK, KFD is the _only_ client of the amd_iommu_v2 driver, so it is imperative that we will work together on this. Oded > > If you know any other way to call mmu_notifier_unregister before the > > end of mmput function than i am all ear. I am not adding this call > > back just for the fun of it i spend serious time trying to find a > > way to do thing without it. I might have miss a way so if i did please > > show it to me. > > Why do you need to call mmu_notifier_unregister manually when it is done > implicitly in exit_mmap already? > > > Joerg > > ��.n������g��� �a��� �&ޖ)���)��h���&������梷���� �Ǟ�m������)������^�����������v���O��zf������