On Wednesday 05 June 2013 12:10:19 Mel Gorman wrote: > On Tue, Jun 04, 2013 at 06:58:07AM -0500, Robin Holt wrote: > > > B) 1. allocate memory with alloc_pages() > > > > > > 2. SetPageReserved() > > > 3. vm_mmap() to allocate a userspace mapping > > > 4. vm_insert_page() > > > 5. vm_flags |= (VM_DONTEXPAND | VM_DONTDUMP) > > > > > > (resulting flags are VM_MIXEDMAP | VM_DONTDUMP | VM_DONTEXPAND | > > > 0xff) > > > > > > At least the memory allocated like B) is affected by automatic NUMA > > > page migration. I'm not sure about A). > > > > > > 1. How can I prevent automatic NUMA page migration on this memory? > > > 2. Can NUMA page migration also be handled on such kind of memory > > > without > > > > > > preventing migration? > > Page migration does not expect a PageReserved && PageLRU page. The only > reserved check that is made by migration is for the zero page and that > happens in the syscall path for move_pages() which is not used by either > compaction or automatic balancing. > > At some point you must have a driver that is setting PageReserved on > anonymous pages that is later encountered by automatic numa balancing > during a NUMA hinting fault. I expect this is an out-of-tree driver or > a custom kernel of some sort. Memory should be pinned by elevating the > reference count of the page, not setting PageReserved. Yes, this is ring 0 code from VirtualBox. The VBox ring 0 driver does the steps which are shown above. Setting PageReserved is not only for pinning but also for fork() protection. I've tried to do get_page() as well but it did not help preventing the migration during NUMA balancing. As I wrote, the code for allocating + mapping the memory assumes that the memory is finally pinned and will be never unmapped. That assumption might be wrong or wrong under certain/rare conditions. I would like to know these conditions and how we can prevent them from happening or how we can handle them correctly. > It's not particularly clear how you avoid hitting the same bug due to THP > and memory compaction to be honest but maybe your setup hits a steady > state that simply never hit the problem or it happens rarely and it was > not identified. I'm currently using the stock Ubuntu 13.04 generic kernel (3.8.0-23), patched with some additional logging code. It is true that this problem could also be triggered by other kernel mechanisms as you described. Thanks, Frank -- Dr.-Ing. Frank Mehnert | Software Development Director, VirtualBox ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt, Germany Hauptverwaltung: Riesstr. 25, D-80992 München Registergericht: Amtsgericht München, HRA 95603 Geschäftsführer: Jürgen Kunz Komplementärin: ORACLE Deutschland Verwaltung B.V. Hertogswetering 163/167, 3543 AS Utrecht, Niederlande Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
Attachment:
signature.asc
Description: This is a digitally signed message part.