On Mon, Nov 19, 2018 at 11:48:54AM -0500, Jerome Glisse wrote: > Just to comment on this, any infiniband driver which use umem and do > not have ODP (here ODP for me means listening to mmu notifier so all > infiniband driver except mlx5) will be affected by same issue AFAICT. > > AFAICT there is no special thing happening after fork() inside any of > those driver. So if parent create a umem mr before fork() and program > hardware with it then after fork() the parent might start using new > page for the umem range while the old memory is use by the child. The > reverse is also true (parent using old memory and child new memory) > bottom line you can not predict which memory the child or the parent > will use for the range after fork(). > > So no matter what you consider the child or the parent, what the hw > will use for the mr is unlikely to match what the CPU use for the > same virtual address. In other word: > > Before fork: > CPU parent: virtual addr ptr1 -> physical address = 0xCAFE > HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE > > Case 1: > CPU parent: virtual addr ptr1 -> physical address = 0xCAFE > CPU child: virtual addr ptr1 -> physical address = 0xDEAD > HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE > > Case 2: > CPU parent: virtual addr ptr1 -> physical address = 0xBEEF > CPU child: virtual addr ptr1 -> physical address = 0xCAFE > HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE IIRC this is solved in IB by automatically calling madvise(MADV_DONTFORK) before creating the MR. MADV_DONTFORK .. This is useful to prevent copy-on-write semantics from changing the physical location of a page if the parent writes to it after a fork(2) .. Jason
