On Tuesday 04 June 2013 16:02:30 Michal Hocko wrote: > On Tue 04-06-13 14:14:45, Frank Mehnert wrote: > > On Tuesday 04 June 2013 13:58:07 Robin Holt wrote: > > > This is probably more appropriate to be directed at the linux-mm > > > mailing list. > > > > > > On Tue, Jun 04, 2013 at 09:22:10AM +0200, Frank Mehnert wrote: > > > > Hi, > > > > > > > > our memory management on Linux hosts conflicts with NUMA page > > > > migration. I assume this problem existed for a longer time but Linux > > > > 3.8 introduced automatic NUMA page balancing which makes the problem > > > > visible on multi-node hosts leading to kernel oopses. > > > > > > > > NUMA page migration means that the physical address of a page > > > > changes. This is fatal if the application assumes that this never > > > > happens for that page as it was supposed to be pinned. > > > > > > > > We have two kind of pinned memory: > > > > > > > > A) 1. allocate memory in userland with mmap() > > > > > > > > 2. madvise(MADV_DONTFORK) > > > > 3. pin with get_user_pages(). > > > > 4. flush dcache_page() > > > > 5. vm_flags |= (VM_DONTCOPY | VM_LOCKED) > > > > > > > > (resulting flags are VM_MIXEDMAP | VM_DONTDUMP | VM_DONTEXPAND > > > > | > > > > > > > > VM_DONTCOPY | VM_LOCKED | 0xff) > > > > > > I don't think this type of allocation should be affected. The > > > get_user_pages() call should elevate the pages reference count which > > > should prevent migration from completing. I would, however, wait for > > > a more definitive answer. > > > > Thanks Robin! Actually case B) is more important for us so I'm waiting > > for more feedback :) > > The manual node migration code seems to be OK in case B as well because > Reserved are skipped (check check_pte_range called from down the > do_migrate_pages path). > > Maybe auto-numa code is missing this check assuming that it cannot > encounter reserved pages. > > migrate_misplaced_page relies on numamigrate_isolate_page which relies > on isolate_lru_page and that one expects a LRU page. Is your Reserved > page on the LRU list? That would be a bit unexpected. I will check this. In the meantime I verified that my testcase does not fail if I pass 'numa_balancing=false' to the kernel, so it's definitely a NUMA balancing problem. I also did 'get_page()' on all pages of method B but the testcase so this didn't help. Frank -- Dr.-Ing. Frank Mehnert | Software Development Director, VirtualBox ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt, Germany Hauptverwaltung: Riesstr. 25, D-80992 München Registergericht: Amtsgericht München, HRA 95603 Geschäftsführer: Jürgen Kunz Komplementärin: ORACLE Deutschland Verwaltung B.V. Hertogswetering 163/167, 3543 AS Utrecht, Niederlande Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
Attachment:
signature.asc
Description: This is a digitally signed message part.