On Monday, April 13, 2015 at 6:20:05 PM UTC+8, Mel Gorman wrote:
> Memory initialisation had been identified as one of the reasons why large
> machines take a long time to boot. Patches were posted a long time ago
> that attempted to move deferred initialisation into the page allocator
> paths. This was rejected on the grounds it should not be necessary to hurt
> the fast paths to parallelise initialisation. This series reuses much of
> the work from that time but defers the initialisation of memory to kswapd
> so that one thread per node initialises memory local to that node. The
> issue is that on the machines I tested with, memory initialisation was not
> a major contributor to boot times. I'm posting the RFC to both review the
> series and see if it actually helps users of very large machines.
>
> After applying the series and setting the appropriate Kconfig variable I
> see this in the boot log on a 64G machine
>
> [ 7.383764] kswapd 0 initialised deferred memory in 188ms
> [ 7.404253] kswapd 1 initialised deferred memory in 208ms
> [ 7.411044] kswapd 3 initialised deferred memory in 216ms
> [ 7.411551] kswapd 2 initialised deferred memory in 216ms
>
> On a 1TB machine, I see
>
> [ 11.913324] kswapd 0 initialised deferred memory in 1168ms
> [ 12.220011] kswapd 2 initialised deferred memory in 1476ms
> [ 12.245369] kswapd 3 initialised deferred memory in 1500ms
> [ 12.271680] kswapd 1 initialised deferred memory in 1528ms
>
> Once booted the machine appears to work as normal. Boot times were measured
> from the time shutdown was called until ssh was available again. In the
> 64G case, the boot time savings are negligible. On the 1TB machine, the
> savings were 10 seconds (about 8% improvement on kernel times but 1-2%
> overall as POST takes so long).
>
> It would be nice if the people that have access to really large machines
> would test this series and report back if the complexity is justified.
Nice work!
On an older Numascale system with 1TB memory and 256 cores/32 NUMA nodes, platform init takes 52s (cold boot), firmware takes 84s (includes one warm reboot), stock linux 4.0 then takes 732s to boot [1] (due to the 700ns roundtrip, RMW cache-coherent cycles due to the temporal writes for pagetable init and per-core store queue limits), so there is huge potential.
Alas I ran into crashing during list manipulation [2] which list debugging detects [3]; I had started adding some debug [4], but need to look a bit deeper into it. I annotated the time of the output from cold power on.
Thanks,
Daniel
-- [4]
static void free_pcppages_bulk(struct zone *zone, int count,
struct per_cpu_pages *pcp)
...
pr_err("migrate_type=%d\n", migratetype);
/* This is the only non-empty list. Free them all. */
if (batch_free == MIGRATE_PCPTYPES)
batch_free = to_free;
do {
int mt; /* migratetype of the to-be-freed page */
pr_err("list_empty=%d\n", list_empty(list));