This is something that Dan Williams and I discussed off the mailing list sometime ago, but I want to have a broader discussion about this problem so I could send out a fix that would be acceptable. We have a 2G pmem device that is carved out of regular memory that we use to pass data across reboots. After the machine is rebooted we hotplug that memory back, so we do not lose 2G of system memory (machine is small, only 8G of RAM total). In order to hotplug pmem memory it first must be converted to devdax. Devdax has a label 2M in size that is placed at the beginning of the pmem device memory which brings the problem. The section size is a hotplugging unit on Linux. Whatever gets hot-plugged or hot-removed must be section size aligned. On x86 section size is 128M on arm64 it is 1G (because arm64 supports 64K pages, and 128M does not work with 64K pages). Because the first 2M are subtracted from the pmem device to create devdax, that actual hot-pluggable memory is not 1G/128M aligned, and instead we lose 126M on x86 or 1022M on arm64 of memory that is getting hot-plugged, the whole first section is skipped when memory gets hot plugged because of 2M label. As a workaround, so we do not lose 1022M out of 8G of memory on arm64 we have section size reduced to 128M. We are using this patch [1]. This way we are losing 126M (which I still hate!) I would like to get rid of this workaround. First, because I would like us to switch to 64K pages to gain performance, and second so we do not depend on an unofficial patch which already has given us some headache with kdump support. Here are some solutions that I think we can do: 1. Instead of carving the memory at 1G aligned address, do it at 1G - 2M address, this way when devdax is created it is perfectly 1G aligned. On ARM64 it causes a panic because there is a 2M hole in memory. Even if panic is fixed, I do not think this is a proper fix. This is simply a workaround to the underlying problem. 2. Dan Williams introduced subsections [2]. They, however do not work with devdax, and hot-plugging in general. Those patches take care of __add_pages() side of things, and not add_memory(). Also, it is unclear what kind of user interface changes need to be made in order to enable subsection features to online/offline pages. 3. Allow to hot plug daxdev together with the label, but teach the kernel not to touch label (i.e. allocate its memory). IMO, kind of ugly solution, because when devdax is hot-plugged it is not even aware of label size. But, perhaps that can be changed. 4. Other ideas? (move dax label to the end? a special case without a label? label outside of data?) Thank you, Pasha [1] https://lore.kernel.org/lkml/20190423203843.2898-1-pasha.tatashin@xxxxxxxxxx [2] https://lore.kernel.org/lkml/156092349300.979959.17603710711957735135.stgit@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
