On Thu, 2019-02-21 at 19:27 -0500, Jerome Glisse wrote: > On Thu, Feb 21, 2019 at 11:11:51PM +0000, Adam Manzanares wrote: > > Hello, > > > > I would like to attend the LSF/MM Summit 2019. I'm interested in > > several MM topics that are mentioned below as well as Zoned Block > > Devices and any io determinism topics that come up in the storage > > track. > > > > I have been working on a caching layer, hmmap (heterogeneous memory > > map) [1], for emerging NVM and it is in spirit close to the page > > cache. The key difference being that the backend device and caching > > layer of hmmap is pluggable. In addition, hmmap supports DAX and > > write > > protection, which I believe are key features for emerging NVMs that > > may > > have write/read asymmetry as well as write endurance constraints. > > Lastly we can leverage hardware, such as a DMA engine, when moving > > pages between the cache while also allowing direct access if the > > device > > is capable. > > > > I am proposing that as an alternative to using NVMs as a NUMA node > > we expose the NVM through the page cache or a viable alternative > > and > > have userspace applications mmap the NVM and hand out memory with > > their favorite userspace memory allocator. > > > > This would isolate the NVMs to only applications that are well > > aware > > of the performance implications of accessing NVM. I believe that > > all > > of this work could be solved with the NUMA node approach, but the > > two > > approaches are seeming to blur together. > > > > The main points I would like to discuss are: > > > > * Is the page cache model a viable alternative to NVM as a NUMA > > NODE? > > * Can we add more flexibility to the page cache? > > * Should we force separation of NVM through an explicit mmap? > > > > I believe this discussion could be merged with NUMA, memory > > hierarchy > > and device memory, Use NVDIMM as NUMA node and NUMA API, or memory > > reclaim with NUMA balancing. > > What about cache coherency and atomic ? If device block are expose > through PCIE then there is no cache coherency or atomic and thus > direct mmap will not have the expected memory model which would > break program expectation of a mmap. For the PCIE cache coherency case I would envision that you would map the memory as read only into the process address space. Once a write occurs I would then remap the PCIE memory to a page in the proposed caching mechanism. I have to think more about what this means for atomic operations. > > This is also one of the reasons i do not see a way forward with NUMA > and device memory. It can depart from the usual memory too much to > be drop in like that to unaware application. I have similar concerns and am trying to segregate the device memory to aware applications. > > In any case yes this kind of memory falls into the device memory i > wish to discuss during LSF/MM. > > Cheers, > Jérôme >
