On Tue, Jun 04, 2013 at 02:14:45PM +0200, Frank Mehnert wrote: > On Tuesday 04 June 2013 13:58:07 Robin Holt wrote: > > This is probably more appropriate to be directed at the linux-mm > > mailing list. > > > > On Tue, Jun 04, 2013 at 09:22:10AM +0200, Frank Mehnert wrote: > > > Hi, > > > > > > our memory management on Linux hosts conflicts with NUMA page migration. > > > I assume this problem existed for a longer time but Linux 3.8 introduced > > > automatic NUMA page balancing which makes the problem visible on > > > multi-node hosts leading to kernel oopses. > > > > > > NUMA page migration means that the physical address of a page changes. > > > This is fatal if the application assumes that this never happens for > > > that page as it was supposed to be pinned. > > > > > > We have two kind of pinned memory: > > > > > > A) 1. allocate memory in userland with mmap() > > > > > > 2. madvise(MADV_DONTFORK) > > > 3. pin with get_user_pages(). > > > 4. flush dcache_page() > > > 5. vm_flags |= (VM_DONTCOPY | VM_LOCKED) > > > > > > (resulting flags are VM_MIXEDMAP | VM_DONTDUMP | VM_DONTEXPAND | > > > > > > VM_DONTCOPY | VM_LOCKED | 0xff) > > > > I don't think this type of allocation should be affected. The > > get_user_pages() call should elevate the pages reference count which > > should prevent migration from completing. I would, however, wait for > > a more definitive answer. > > Thanks Robin! Actually case B) is more important for us so I'm waiting > for more feedback :) If you have a good test case, you might want to try adding a get_page() in there to see if that mitigates the problem. It would at least be interesting to know if it has an effect. Robin > > Frank > > > > B) 1. allocate memory with alloc_pages() > > > > > > 2. SetPageReserved() > > > 3. vm_mmap() to allocate a userspace mapping > > > 4. vm_insert_page() > > > 5. vm_flags |= (VM_DONTEXPAND | VM_DONTDUMP) > > > > > > (resulting flags are VM_MIXEDMAP | VM_DONTDUMP | VM_DONTEXPAND | > > > 0xff) > > > > > > At least the memory allocated like B) is affected by automatic NUMA page > > > migration. I'm not sure about A). > > > > > > 1. How can I prevent automatic NUMA page migration on this memory? > > > 2. Can NUMA page migration also be handled on such kind of memory without > > > > > > preventing migration? > > > > > > Thanks, > > > > > > Frank > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > > the body of a message to majordomo@xxxxxxxxxxxxxxx > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Please read the FAQ at http://www.tux.org/lkml/ > > -- > Dr.-Ing. Frank Mehnert | Software Development Director, VirtualBox > ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt, Germany > > Hauptverwaltung: Riesstr. 25, D-80992 München > Registergericht: Amtsgericht München, HRA 95603 > Geschäftsführer: Jürgen Kunz > > Komplementärin: ORACLE Deutschland Verwaltung B.V. > Hertogswetering 163/167, 3543 AS Utrecht, Niederlande > Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 > Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
Attachment:
signature.asc
Description: Digital signature