At 09/20/2012 01:16 PM, Minchan Kim Wrote: > On Thu, Sep 20, 2012 at 11:12:40AM +0800, Wen Congyang wrote: >> At 09/20/2012 10:30 AM, Minchan Kim Wrote: >>> On Thu, Sep 20, 2012 at 09:17:43AM +0800, Wen Congyang wrote: >>>> At 09/20/2012 04:17 AM, Minchan Kim Wrote: >>>>> Hi KOSAKI, >>>>> >>>>> On Wed, Sep 19, 2012 at 02:05:20PM -0400, KOSAKI Motohiro wrote: >>>>>> On Wed, Sep 19, 2012 at 3:29 AM, Minchan Kim <minchan@xxxxxxxxxx> wrote: >>>>>>> During memory-hotplug stress test, I found NR_ISOLATED_[ANON|FILE] >>>>>>> are increasing so that kernel are hang out. >>>>>>> >>>>>>> The cause is that when we do memory-hotadd after memory-remove, >>>>>>> __zone_pcp_update clear out zone's ZONE_STAT_ITEMS in setup_pageset >>>>>>> without draining vm_stat_diff of all CPU. >>>>>>> >>>>>>> This patch fixes it. >>>>>> >>>>>> zone_pcp_update() is called from online pages path. but IMHO, >>>>>> the statistics should be drained offline path. isn't it? >>>>> >>>>> It isn't necessary because statistics is right until we reset it to zero >>>>> in online path. >>>>> Do you have something on your mind that we have to drain it in offline path? >>>> >>>> When a node is offlined and onlined again. We create node_data[i] in the >>> >>> I would like to clarify your word. >>> Create or recreate? >>> Why I have a question is as I look over the source code, hotadd_new_pgdat >>> seem to be called if we do hotadd *new* memory. It's not the case for >>> offline and online again you mentioned. If you're right, I should find >>> arch_free_nodedata to free pgdat when node is disappear but I can't find it. >>> Do I miss something? >> >> Hmm, when a memory is removed, we don't do cleanup now. We(Fujitsu) posted >> a patchset to do this: >> https://lkml.org/lkml/2012/9/5/201 >> >> We don't free pgdat in this patchset now. We have two choice: >> 1. free pgdat >> 2. don't free it, and reuse it when it is onlined again >> >> I'm not sure which choice is better. > > I have no idea because I don't know how you guys uses. > If there is use case that sometime you ues many node burstly but > ues a few node in most time, 1) would be good POV memory efficiency > although it makes code rather complicated. > > Anyway, it's another story with this patch because it's not merged yet. > >> >>> >>> >>>> function hotadd_new_pgdat(), and we will lost the statistics stored in >>>> zone->pageset. So we should drain it in offline path. >>> >>> Even we drain in offline patch, it still has a problem. >>> >>> 1. offline >>> 2. drain -> OKAY >>> 3. schedule >>> 4. Process A increase zone stat >>> 5. Process B increase zone stat >>> 6. online >>> 7. reset it -> we ends up lost zone stat counter which is modified between 2-6 >>> >> >> I understand why you drain it in online path now. But it still should drain it >> in offline path because if all pages in this zone are offlined, we will call >> zone_pcp_reset() to reset zone's pcp. We should also drop it in the function >> zone_pcp_reset(). > > Good point. > How about this? > >>From e92bf3e96720c89cb18ec32c5db095a27ad4133c Mon Sep 17 00:00:00 2001 > From: Minchan Kim <minchan@xxxxxxxxxx> > Date: Thu, 20 Sep 2012 14:11:49 +0900 > Subject: [PATCH v2] memory-hotplug: fix zone stat mismatch > > During memory-hotplug, I found NR_ISOLATED_[ANON|FILE] > are increasing so that kernel are hang out. > > The cause is that when we do memory-hotadd after memory-remove, > __zone_pcp_update clear out zone's ZONE_STAT_ITEMS in setup_pageset > although vm_stat_diff of all CPU still have value. > > In addtion, when we offline all pages of the zone, we reset them > in zone_pcp_reset without drain so that we lost zone stat item. > > This patch fixes it. > > * from v1 > * drain offline patch - KOSAKI, Wen > > Cc: Kamezawa Hiroyuki <kamezawa.hiroyu@xxxxxxxxxxxxxx> > Cc: Yasuaki Ishimatsu <isimatu.yasuaki@xxxxxxxxxxxxxx> > Cc: Wen Congyang <wency@xxxxxxxxxxxxxx> > Cc: KOSAKI Motohiro <kosaki.motohiro@xxxxxxxxxxxxxx> > Signed-off-by: Minchan Kim <minchan@xxxxxxxxxx> > --- > include/linux/vmstat.h | 4 ++++ > mm/page_alloc.c | 7 +++++++ > mm/vmstat.c | 12 ++++++++++++ > 3 files changed, 23 insertions(+) > > diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h > index ad2cfd5..5d31876 100644 > --- a/include/linux/vmstat.h > +++ b/include/linux/vmstat.h > @@ -198,6 +198,8 @@ extern void __dec_zone_state(struct zone *, enum zone_stat_item); > void refresh_cpu_vm_stats(int); > void refresh_zone_stat_thresholds(void); > > +void drain_zonestat(struct zone *zone, struct per_cpu_pageset *); > + > int calculate_pressure_threshold(struct zone *zone); > int calculate_normal_threshold(struct zone *zone); > void set_pgdat_percpu_threshold(pg_data_t *pgdat, > @@ -251,6 +253,8 @@ static inline void __dec_zone_page_state(struct page *page, > static inline void refresh_cpu_vm_stats(int cpu) { } > static inline void refresh_zone_stat_thresholds(void) { } > > +static inline void drain_zonestat(struct zone *zone, > + struct per_cpu_pageset *pset) { } > #endif /* CONFIG_SMP */ > > extern const char * const vmstat_text[]; > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index ab58346..980f2e7 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -5904,6 +5904,7 @@ static int __meminit __zone_pcp_update(void *data) > local_irq_save(flags); > if (pcp->count > 0) > free_pcppages_bulk(zone, pcp->count, pcp); > + drain_zonestat(zone, pset); > setup_pageset(pset, batch); > local_irq_restore(flags); > } > @@ -5920,10 +5921,16 @@ void __meminit zone_pcp_update(struct zone *zone) > void zone_pcp_reset(struct zone *zone) > { > unsigned long flags; > + int cpu; > + struct per_cpu_pageset *pset; > > /* avoid races with drain_pages() */ > local_irq_save(flags); > if (zone->pageset != &boot_pageset) { > + for_each_online_cpu(cpu) { A cpu can be offlined before the pages in the zone are offlined. So I think you should drain it on all possible cpu, not online cpu. Thanks Wen Congyang > + pset = per_cpu_ptr(zone->pageset, cpu); > + drain_zonestat(zone, pset); > + } > free_percpu(zone->pageset); > zone->pageset = &boot_pageset; > } > diff --git a/mm/vmstat.c b/mm/vmstat.c > index b3e3b9d..d4cc1c2 100644 > --- a/mm/vmstat.c > +++ b/mm/vmstat.c > @@ -495,6 +495,18 @@ void refresh_cpu_vm_stats(int cpu) > atomic_long_add(global_diff[i], &vm_stat[i]); > } > > +void drain_zonestat(struct zone *zone, struct per_cpu_pageset *pset) > +{ > + int i; > + > + for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) > + if (pset->vm_stat_diff[i]) { > + int v = pset->vm_stat_diff[i]; > + pset->vm_stat_diff[i] = 0; > + atomic_long_add(v, &zone->vm_stat[i]); > + atomic_long_add(v, &vm_stat[i]); > + } > +} > #endif > > #ifdef CONFIG_NUMA -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>