On Thu, 2017-04-20 at 10:29 -0500, Christoph Lameter wrote: > On Thu, 20 Apr 2017, Balbir Singh wrote: > > Couple of things are needed > > > > 1. Isolation of allocation > > cgroups, memory policy and cpuset provide that > Yes and we are building on top of mempolicies. The problem becomes a little worse when the coherent device memory node is seen as CPUless node. I was trying to solve 1 and 2 with the same approach. > > 2. Isolation of certain algorithms like kswapd/auto-numa balancing > > Ok that may mean adding some generic functionality to limit those As in per-algorithm tunables? I think it would be definitely good to have that. I do not know how well that would scale? > > > > The approach sounds pretty invasive to me. > > > > Could you please elaborate, you mean the user space programming bits? > > No I mean the modification of the memory policies in particular. We are > adding more exceptions to an already complex and fragile system. > > Can we do this in a generic way just using hotplug nodes and some of the > existing isolation mechanisms? > Yes, that was the first approach we tried and we are reusing whatever we can -- HMM for driver driven migration, mempolicies for allocation control and N_COHERENT_MEMORY for isolation because of 1 and 2 above combined. > > > Ideally we need the following: > > > > 1. Transparency about being able to allocate memory anywhere and the ability > > to migrate memory between coherent device memory and normal system memory > > If it is a memory node then you have that already. > > > 2. The ability to explictly allocate memory from coherent device memory > > Ditto > > > 3. Isolation of normal allocations from coherent device memory unless > > explictly stated, same as (2) above > > memory policies etc do that. > > > 4. The ability to hotplug in and out the memory at run-time > > hotplug code does that. > > > > 5. Exchange pointers between coherent device memory and normal memory > > for the compute on the coherent device memory to use > > I dont see anything preventing that from occurring right now. Thats a > device issue with doing proper virtual to physical mapping right? > Some of these requirements come from whether we use NUMA or HMM-CDM. We prefer NUMA and it meets the above requirements quite well. > > I could list further things, but largely coherent device memory is like > > system memory except that we believe that things like auto-numa balancing > > and kswapd will not work well due to lack of information about references > > and faults. > > Ok so far I do not see that we need coherent nodes at all. > I presume you are suggesting this based on the fact that we add additional infrastructure for auto-numa/kswapd/etc isolation? > > Some of the mm-summit notes are at https://lwn.net/Articles/717601/ > > The goals align with HMM, except that the device memory is coherent. HMM > > has a CDM variation as well. > > I was at the presentation but at that point you were interested in a > different approach it seems. I do remember you were present, I don't think things have changed since then. > > > We've been using the term coherent device memory (CDM). I could rephrase the > > text and documentation for consistency. Would you prefer a different term? > > Hotplug memory node? > Normal memory is hotpluggable too.. but I'd be fine as long as everyone agrees Balbir -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>
