On Mon, May 01, 2017 at 02:56:34PM -0700, John Hubbard wrote:
On 05/01/2017 02:04 PM, Reza Arbab wrote:
On Mon, May 01, 2017 at 01:41:55PM -0700, John Hubbard wrote:
1. A way to move pages between NUMA nodes, both virtual address
and physical address-based, from kernel mode.
Jérôme's migrate_vma() and migrate_dma() should have this covered,
including DMA-accelerated copy.
Yes, that's good. I wasn't sure from this discussion here if either or
both of those would be used, but now I see.
Are those APIs ready for moving pages between NUMA nodes? As there is
no NUMA node id in the API, are we relying on the pages' membership
(using each page and updating which node it is on)?
Yes. Those APIs work by callback. The alloc_and_copy() function you
provide will be called at the appropriate point in the migration. Yours
would allocate from a specific destination node, and copy using DMA.
5. Something to handle the story of bringing NUMA nodes online and
putting them back offline, given that they require a device driver
that may not yet have been loaded. There are a few minor missing bits
there.
This has been prototyped with the driver doing memory
hotplug/hotremove. Could you elaborate a little on what you feel is
missing?
We just worked through how to deal with this in our driver, and I
remember feeling worried about the way NUMA nodes can only be put
online via a user space action (through sysfs). It seemed like you'd
want to do that from kernel as well, when a device driver gets loaded.
That's true. I don't think we have a way to online/offline from a
driver. To online, the alternatives are memhp_auto_online (incapable of
doing online_movable), or udev rules (not ideal in this driver
controlled memory use case). To offline, nothing that I know of.
I was also uneasy about user space trying to bring a node online before
the associated device driver was loaded, and I think it would be nice
to be sure that that whole story is looked at.
The theme here is that driver load/unload is, today, independent from
the NUMA node online/offline, and that's a problem. Not a huge one,
though, just worth enumerating here.
--
Reza Arbab
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>
