Re: [PATCH] mm: meminit: Finish initialisation of struct pages before basic setup

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I am just noticed a hang on my largest box.
I can only reproduce with large core counts, if I turn down the number of cpus it doesn't have an issue.

Also as time goes on the amount of time required to initialize pages goes up.


log_uv48_05121052:[  177.250385] node 0 initialised, 14950072 pages in 544ms
log_uv48_05121052:[  177.269629] node 1 initialised, 15990505 pages in 564ms
log_uv48_05121052:[ 177.436047] node 215 initialised, 3600110 pages in 724ms log_uv48_05121052:[ 177.464056] node 102 initialised, 3604205 pages in 756ms log_uv48_05121052:[ 178.073822] node 30 initialised, 7732972 pages in 1368ms log_uv48_05121052:[ 178.082888] node 31 initialised, 7728877 pages in 1372ms log_uv48_05121052:[ 178.080060] node 29 initialised, 7728877 pages in 1376ms
....
log_uv48_05121052:[ 178.217980] node 197 initialised, 7728877 pages in 1504ms log_uv48_05121052:[ 178.217851] node 196 initialised, 7732972 pages in 1504ms log_uv48_05121052:[ 178.219992] node 247 initialised, 7726418 pages in 1504ms log_uv48_05121052:[ 178.325299] node 3 initialised, 15986409 pages in 1624ms log_uv48_05121052:[ 178.328455] node 2 initialised, 15990505 pages in 1624ms log_uv48_05121052:[ 178.383371] node 4 initialised, 15990505 pages in 1680ms
...
log_uv48_05121052:[ 178.438401] node 19 initialised, 15986409 pages in 1728ms

I apologize for the tardiness of this report but I have not been able to get to the largest boxes reliably.
Hopefully I will have more access this week.


On 05/07/2015 05:09 PM, Andrew Morton wrote:
On Thu, 7 May 2015 08:25:18 +0100 Mel Gorman <mgorman@xxxxxxx> wrote:

Waiman Long reported that 24TB machines hit OOM during basic setup when
struct page initialisation was deferred. One approach is to initialise memory
on demand but it interferes with page allocator paths. This patch creates
dedicated threads to initialise memory before basic setup. It then blocks
on a rw_semaphore until completion as a wait_queue and counter is overkill.
This may be slower to boot but it's simplier overall and also gets rid of a
section mangling which existed so kswapd could do the initialisation.
Seems a reasonable compromise.  It makes a bit of a mess of the patch
sequencing.

Have some tweaklets:



From: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
Subject: mm-meminit-finish-initialisation-of-struct-pages-before-basic-setup-fix

include rwsem.h, use DECLARE_RWSEM, fix comment, remove unneeded cast

Cc: Daniel J Blueman <daniel@xxxxxxxxxxxxx>
Cc: Dave Hansen <dave.hansen@xxxxxxxxx>
Cc: Mel Gorman <mgorman@xxxxxxx>
Cc: Nathan Zimmer <nzimmer@xxxxxxx>
Cc: Scott Norton <scott.norton@xxxxxx>
Cc: Waiman Long <waiman.long@xxxxxx
Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
---

  mm/page_alloc.c |    8 ++++----
  1 file changed, 4 insertions(+), 4 deletions(-)

diff -puN mm/page_alloc.c~mm-meminit-finish-initialisation-of-struct-pages-before-basic-setup-fix mm/page_alloc.c
--- a/mm/page_alloc.c~mm-meminit-finish-initialisation-of-struct-pages-before-basic-setup-fix
+++ a/mm/page_alloc.c
@@ -18,6 +18,7 @@
  #include <linux/mm.h>
  #include <linux/swap.h>
  #include <linux/interrupt.h>
+#include <linux/rwsem.h>
  #include <linux/pagemap.h>
  #include <linux/jiffies.h>
  #include <linux/bootmem.h>
@@ -1075,12 +1076,12 @@ static void __init deferred_free_range(s
  		__free_pages_boot_core(page, pfn, 0);
  }
-static struct rw_semaphore __initdata pgdat_init_rwsem;
+static __initdata DECLARE_RWSEM(pgdat_init_rwsem);
/* Initialise remaining memory on a node */
  static int __init deferred_init_memmap(void *data)
  {
-	pg_data_t *pgdat = (pg_data_t *)data;
+	pg_data_t *pgdat = data;
  	int nid = pgdat->node_id;
  	struct mminit_pfnnid_cache nid_init_state = { };
  	unsigned long start = jiffies;
@@ -1096,7 +1097,7 @@ static int __init deferred_init_memmap(v
  		return 0;
  	}
- /* Bound memory initialisation to a local node if possible */
+	/* Bind memory initialisation thread to a local node if possible */
  	if (!cpumask_empty(cpumask))
  		set_cpus_allowed_ptr(current, cpumask);
@@ -1200,7 +1201,6 @@ void __init page_alloc_init_late(void)
  {
  	int nid;
- init_rwsem(&pgdat_init_rwsem);
  	for_each_node_state(nid, N_MEMORY) {
  		down_read(&pgdat_init_rwsem);
  		kthread_run(deferred_init_memmap, NODE_DATA(nid), "pgdatinit%d", nid);
_


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>




[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]