Re: [Part3 PATCH v2 0/4] Support hot-remove local pagetable pages.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

On Thu, Jun 13, 2013 at 09:03:52PM +0800, Tang Chen wrote:
> The following patch-set from Yinghai allocates pagetables to local nodes.
> v1: https://lkml.org/lkml/2013/3/7/642
> v2: https://lkml.org/lkml/2013/3/10/47
> v3: https://lkml.org/lkml/2013/4/4/639
> v4: https://lkml.org/lkml/2013/4/11/829
> 
> Since pagetable pages are used by the kernel, they cannot be offlined.
> As a result, they cannot be hot-remove.
> 
> This patch fix this problem with the following solution:
> 
>      1.   Introduce a new bootmem type LOCAL_NODE_DATAL, and register local
>           pagetable pages as LOCAL_NODE_DATAL by setting page->lru.next to
>           LOCAL_NODE_DATAL, just like we register SECTION_INFO pages.
> 
>      2.   Skip LOCAL_NODE_DATAL pages in offline/online procedures. When the
>           whole memory block they reside in is offlined, the kernel can
>           still access the pagetables.
>           (This changes the semantics of offline/online a little bit.)

This could be a design problem of part3: if we allow local pagetable memory
to not be offlined but allow the offlining to return successfully, then
hot-remove is going to succeed. But the direct mapped pagetable pages are still
mapped in the kernel. The hot-removed memblocks will suddenly disappear (think
physical DIMMs getting disabled in real hardware, or in a VM case the
corresponding guest memory getting freed from the emulator e.g. qemu/kvm). The
system can crash as a result.

I think these local pagetables do need to be unmapped from kernel, offlined and
removed somehow - otherwise hot-remove should fail. Could they be migrated
alternatively e.g. to node 0 memory?  But Iiuc direct mapped pages cannot be
migrated, correct?

What is the original reason for local node pagetable allocation with regards
to memory hotplug? I assume we want to have hotplugged nodes use only their local
memory, so that there are no inter-node memory dependencies for hot-add/remove.
Are there other reasons that I am missing?


I get a crash with the following:
- boot 2-node VM with 4G on node0 and 0G on node1
- guest kernel linux_next20130607 + part1 + part2 + part3 patchsets
- hotplugged 2 dimms on node1 (previosuly node 1 had no memory)
- reboot with movablecore=acpi, ZONE_MOVABLE is created for node1
- I verified that in this case LOCAL_NODE_DATA pages are allocated on node1
- remove a dimm on node1

[  260.615677] Offlined Pages 32768
[  260.619650] Offlined Pages 32768
[  260.630514] Offlined Pages 32768
[  260.634342] Offlined Pages 32768
[  260.637989] Offlined Pages 32768
[  260.642021] Offlined Pages 32768
[  260.643852] Offlined Pages 32768
[  260.645555] Offlined Pages 32768
[  260.817960] general protection fault: 0000 [#1] SMP 
[  260.818384] Modules linked in: netconsole pci_hotplug lp loop kvm_amd kvm ppdev psmouse parport_pc i2c_piix4 parport i2c_core serio_raw evdev processor microcode button thermal_sys ext3 jbd mbcache sr_mod cdrom ata_generic virtio_blk virtio_net ata_piix libata scsi_mod virtio_pci virtio_ring virtio
[  260.820449] CPU: 0 PID: 177 Comm: kworker/0:2 Not tainted 3.10.0-rc4-next-20130607-guest #1
[  260.820449] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[  260.820449] Workqueue: kacpi_hotplug acpi_os_execute_deferred
[  260.820449] task: ffff880115c80080 ti: ffff880115564000 task.ti: ffff880115564000
[  260.820449] RIP: 0010:[<ffffffff812b92b9>]  [<ffffffff812b92b9>] memchr_inv+0x89/0x110
[  260.820449] RSP: 0018:ffff880115565b90  EFLAGS: 00010206
[  260.820449] RAX: 0000000008000000 RBX: ffff8801e0000000 RCX: 0000000000000000
[  260.820449] RDX: 0000000040000000 RSI: 00000000000000fd RDI: 7fff8801c0000000
[  260.820449] RBP: ffff8801e0000000 R08: 00000000000000fd R09: 0101010101010101
[  260.820449] R10: 0000000000000000 R11: ffff880116e38e50 R12: ffff880220000000
[  260.820449] R13: ffff8801c0000000 R14: 00003ffffffff000 R15: ffff880116e38e01
[  260.820449] FS:  00007fa6d8de37c0(0000) GS:ffff88011f000000(0000) knlGS:0000000000000000
[  260.820449] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  260.820449] CR2: 00007fa6d8dea000 CR3: 000000000180b000 CR4: 00000000000006f0
[  260.820449] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  260.820449] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  260.820449] Stack:
[  260.820449]  ffffffff8151e3bc 0000160000000000 ffffc00000000fff ffff880001a5d038
[  260.820449]  ffffffff8150e39e ffff880116e38e50 0000000000000100 0000000000000000
[  260.820449]  ffff8801e0000000 ffff8801181fb060 ffffffff8150e424 ffff88021ffff000
[  260.820449] Call Trace:
[  260.820449]  [<ffffffff8151e3bc>] ? remove_pagetable+0x20c/0x7f7
[  260.820449]  [<ffffffff8150e39e>] ? klist_put+0x4e/0xc0
[  260.820449]  [<ffffffff8150e424>] ? klist_iter_exit+0x14/0x20
[  260.820449]  [<ffffffff8150f385>] ? arch_remove_memory+0x75/0xd0
[  260.820449]  [<ffffffff81510e18>] ? remove_memory+0x68/0x80
[  260.820449]  [<ffffffff8132baa5>] ? acpi_memory_device_remove+0x90/0xea
[  260.820449]  [<ffffffff81308157>] ? acpi_bus_device_detach+0x37/0x5c
[  260.820449]  [<ffffffff813081b5>] ? acpi_bus_trim+0x39/0x6e
[  260.820449]  [<ffffffff81308c74>] ? acpi_scan_hot_remove+0x136/0x264
[  260.820449]  [<ffffffff81304b15>] ? acpi_evaluate_hotplug_ost+0x86/0x8b
[  260.820449]  [<ffffffff81308e3a>] ? acpi_bus_device_eject+0x98/0xcc
[  260.820449]  [<ffffffff813042ea>] ? acpi_os_execute_deferred+0x1f/0x2b
[  260.820449]  [<ffffffff8105a183>] ? process_one_work+0x153/0x480
[  260.820449]  [<ffffffff8105b2b6>] ? worker_thread+0x116/0x3d0
[  260.820449]  [<ffffffff8105b1a0>] ? manage_workers+0x2b0/0x2b0
[  260.820449]  [<ffffffff8105b1a0>] ? manage_workers+0x2b0/0x2b0
[  260.820449]  [<ffffffff81060b56>] ? kthread+0xc6/0xd0
[  260.820449]  [<ffffffff81060a90>] ? kthread_freezable_should_stop+0x60/0x60
[  260.820449]  [<ffffffff8152c6ec>] ? ret_from_fork+0x7c/0xb0
[  260.820449]  [<ffffffff81060a90>] ? kthread_freezable_should_stop+0x60/0x60
[  260.820449] Code: 75 f0 45 89 c0 4c 01 c7 4c 29 c2 0f 1f 80 00 00 00 00 48 89 d0 48 c1 e8 03 85 c0 74 2a 44 0f b6 c6 49 b9 01 01 01 01 01 01 01 01 <48> 8b 0f 4d 0f af c1 4c 39 c1 74 08 eb 41 90 48 3b 0f 75 3b 48 
[  260.820449] RIP  [<ffffffff812b92b9>] memchr_inv+0x89/0x110
[  260.820449]  RSP <ffff880115565b90>
[  260.852908] ---[ end trace 5e9b131e0294cfb6 ]---

I haven't analyzed more yet.

Other times instead of a crash on hot-remove I get a "fork: cannot allocate memory"
always after hot-remove.

thanks,

- Vasilis

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>




[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]