"Aneesh Kumar K.V" <aneesh.kumar@xxxxxxxxxxxxx> writes:
> Vishal Verma <vishal.l.verma@xxxxxxxxx> writes:
>
>> The MHP_MEMMAP_ON_MEMORY flag for hotplugged memory is currently
>> restricted to 'memblock_size' chunks of memory being added. Adding a
>> larger span of memory precludes memmap_on_memory semantics.
>>
>> For users of hotplug such as kmem, large amounts of memory might get
>> added from the CXL subsystem. In some cases, this amount may exceed the
>> available 'main memory' to store the memmap for the memory being added.
>> In this case, it is useful to have a way to place the memmap on the
>> memory being added, even if it means splitting the addition into
>> memblock-sized chunks.
>>
>> Change add_memory_resource() to loop over memblock-sized chunks of
>> memory if caller requested memmap_on_memory, and if other conditions for
>> it are met,. Teach try_remove_memory() to also expect that a memory
>> range being removed might have been split up into memblock sized chunks,
>> and to loop through those as needed.
>>
>> Cc: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
>> Cc: David Hildenbrand <david@xxxxxxxxxx>
>> Cc: Oscar Salvador <osalvador@xxxxxxx>
>> Cc: Dan Williams <dan.j.williams@xxxxxxxxx>
>> Cc: Dave Jiang <dave.jiang@xxxxxxxxx>
>> Cc: Dave Hansen <dave.hansen@xxxxxxxxxxxxxxx>
>> Cc: Huang Ying <ying.huang@xxxxxxxxx>
>> Suggested-by: David Hildenbrand <david@xxxxxxxxxx>
>> Signed-off-by: Vishal Verma <vishal.l.verma@xxxxxxxxx>
>> ---
>> mm/memory_hotplug.c | 154 +++++++++++++++++++++++++++++++---------------------
>> 1 file changed, 91 insertions(+), 63 deletions(-)
>>
>> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
>> index e9bcacbcbae2..20456f0d28e6 100644
>> --- a/mm/memory_hotplug.c
>> +++ b/mm/memory_hotplug.c
>> @@ -1286,6 +1286,35 @@ bool mhp_supports_memmap_on_memory(unsigned long size)
>> }
>> EXPORT_SYMBOL_GPL(mhp_supports_memmap_on_memory);
>>
>> +static int add_memory_create_devices(int nid, struct memory_group *group,
>> + u64 start, u64 size, mhp_t mhp_flags)
>> +{
>> + struct mhp_params params = { .pgprot = pgprot_mhp(PAGE_KERNEL) };
>> + struct vmem_altmap mhp_altmap = {};
>> + int ret;
>> +
>> + if ((mhp_flags & MHP_MEMMAP_ON_MEMORY)) {
>> + mhp_altmap.free = PHYS_PFN(size);
>> + mhp_altmap.base_pfn = PHYS_PFN(start);
>> + params.altmap = &mhp_altmap;
>> + }
>> +
>> + /* call arch's memory hotadd */
>> + ret = arch_add_memory(nid, start, size, ¶ms);
>> + if (ret < 0)
>> + return ret;
>> +
>> + /* create memory block devices after memory was added */
>> + ret = create_memory_block_devices(start, size, mhp_altmap.alloc,
>> + group);
>> + if (ret) {
>> + arch_remove_memory(start, size, NULL);
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> /*
>> * NOTE: The caller must call lock_device_hotplug() to serialize hotplug
>> * and online/offline operations (triggered e.g. by sysfs).
>> @@ -1294,11 +1323,10 @@ EXPORT_SYMBOL_GPL(mhp_supports_memmap_on_memory);
>> */
>> int __ref add_memory_resource(int nid, struct resource *res, mhp_t mhp_flags)
>> {
>> - struct mhp_params params = { .pgprot = pgprot_mhp(PAGE_KERNEL) };
>> + unsigned long memblock_size = memory_block_size_bytes();
>> enum memblock_flags memblock_flags = MEMBLOCK_NONE;
>> - struct vmem_altmap mhp_altmap = {};
>> struct memory_group *group = NULL;
>> - u64 start, size;
>> + u64 start, size, cur_start;
>> bool new_node = false;
>> int ret;
>>
>> @@ -1339,27 +1367,20 @@ int __ref add_memory_resource(int nid, struct resource *res, mhp_t mhp_flags)
>> /*
>> * Self hosted memmap array
>> */
>> - if (mhp_flags & MHP_MEMMAP_ON_MEMORY) {
>> - if (!mhp_supports_memmap_on_memory(size)) {
>> - ret = -EINVAL;
>> + if ((mhp_flags & MHP_MEMMAP_ON_MEMORY) &&
>> + mhp_supports_memmap_on_memory(memblock_size)) {
>> + for (cur_start = start; cur_start < start + size;
>> + cur_start += memblock_size) {
>> + ret = add_memory_create_devices(nid, group, cur_start,
>> + memblock_size,
>> + mhp_flags);
>> + if (ret)
>> + goto error;
>> + }
>
> We should handle the below error details here.
>
> 1) If we hit an error after some blocks got added, should we iterate over rest of the dev_dax->nr_range.
> 2) With some blocks added if we return a failure here, we remove the
> resource in dax_kmem. Is that ok?
>
> IMHO error handling with partial creation of memory blocks in a resource range should be
> documented with this change.
Or, should we remove all added memory blocks upon error?
--
Best Regards,
Huang, Ying
