Deferred Memory Init: How to bring rest of memory online after limiting it with `mem=XG`?

[Paul Menzel <pmenzel@xxxxxxxxxxxxx>]

Dear Feng,


I am trying to reduce the startup time of Debian’s Linux 5.9.9 on a 
Intel Kaby Lake system with 32 GB of memory (TUXEDO Book BU1406 (Clevo 
N240BU)). On your Linux Plumbers Conference 2019 slides of your talk 
*Linux Kernel Fastboot On the Way* [1], you mention *Deferred Memory Init*:

> Deferred Memory Init
>
> •   8GB RAM’s initialization costs 100+ ms
> •   In early boot phase, we don’t need that much memory
> •   Utilize the memory hotplug feature
>     •   “mem=4096m” in cmdline to only init 2 GB
>     •   Use systemd service to add rest memory in parallel

Starting Linux with `mem=2G` indeed reduces the startup time, but I am 
unable to get the rest of the memory online. Comparing it with a boot 
without `mem=2G` the `memoryX`  devices under 
`/sys/devices/system/memory/` are missing.

With `mem=2G`:

$ lsmem --output-all
RANGE                                  SIZE  STATE REMOVABLE BLOCK NODE 
ZONES
0x0000000000000000-0x0000000007ffffff  128M online       yes     0    0 
None
0x0000000008000000-0x000000007fffffff  1,9G online       yes  1-15    0 
DMA32

Memory block size:       128M
Total online memory:       2G
Total offline memory:      0B
$ ls -d /sys/devices/system/memory/memory*
/sys/devices/system/memory/memory0   /sys/devices/system/memory/memory2
/sys/devices/system/memory/memory1   /sys/devices/system/memory/memory3
/sys/devices/system/memory/memory10  /sys/devices/system/memory/memory4
/sys/devices/system/memory/memory11  /sys/devices/system/memory/memory5
/sys/devices/system/memory/memory12  /sys/devices/system/memory/memory6
/sys/devices/system/memory/memory13  /sys/devices/system/memory/memory7
/sys/devices/system/memory/memory14  /sys/devices/system/memory/memory8
/sys/devices/system/memory/memory15  /sys/devices/system/memory/memory9
```

Without `mem=2G`:

```
$ lsmem --output-all
RANGE                                  SIZE  STATE REMOVABLE  BLOCK NODE 
 ZONES
0x0000000000000000-0x0000000007ffffff  128M online       yes      0    0 
  None
0x0000000008000000-0x0000000087ffffff    2G online       yes   1-16    0 
 DMA32
0x0000000088000000-0x000000008fffffff  128M online       yes     17    0 
  None
0x0000000100000000-0x0000000867ffffff 29,6G online       yes 32-268    0 
Normal
0x0000000868000000-0x000000086fffffff  128M online       yes    269    0 
  None

Memory block size:       128M
Total online memory:      32G
Total offline memory:      0B
```

Can the deferred memory initialization be done with the upstream Linux 
kernel, or were you using patches on top?


Kind regards,

Paul


[1]: 
https://www.linuxplumbersconf.org/event/4/contributions/281/attachments/216/617/LPC_2019_kernel_fastboot_on_the_way.pdf