From: Atsushi Kumagai <kumagai-atsushi@xxxxxxxxxxxxxxxxx> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption. Date: Fri, 13 Jul 2012 17:10:39 +0900 > Hello HATAYAMA-san, > > On Fri, 13 Jul 2012 14:18:01 +0900 (JST) > HATAYAMA Daisuke <d.hatayama at jp.fujitsu.com> wrote: > >> From: Atsushi Kumagai <kumagai-atsushi at mxc.nes.nec.co.jp> >> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption. >> Date: Fri, 13 Jul 2012 09:36:11 +0900 >> >> > Hello HATAYAMA-san, >> > >> > On Wed, 11 Jul 2012 14:23:59 +0900 (JST) >> > HATAYAMA Daisuke <d.hatayama at jp.fujitsu.com> wrote: >> > >> > [...] >> >> >>> Hi Atushi san, >> >> >>> >> >> >>> Good to see this work making progress. I have few queries. >> >> >>> >> >> >>> - Do you have some numbers for bigger machines like 1TB or higher memory. >> >> >>> I am curious to know how bad is the time penalty. >> >> >> >> >> >> I'm afraid that I don't have such a large machine, so I need someone who can >> >> >> measure execution time in large machine. >> >> >> >> >> > >> >> > I can prepare such machine but I cannot say I can use the machine on >> >> > this day precisely for example. But I think it must be until the end >> >> > of this month at most. So, I would like to fix content of benchmark >> >> > first. >> >> > >> >> >> >> Hello Kumagai-san, >> >> >> >> I'm now trying to reserve the machine with big memory, but I'm not >> >> accurately sure when I can use the machine; expecting within this >> >> month? In advance, I want to make sure what I measure in this >> >> benchmark in more detail. >> > >> > OK, I'm glad for your help. >> > >> >> I'm going to collect at least what you showed in your benchmark: RSS >> >> size for no option, -cd31 and -Ed31. Is there another to collect? >> > >> > Would you collect RSS also for --split ? >> > While I expect the RSS will be increase based on number of child processes, >> > I want to see measured data. >> > >> >> Yes, I'll collect RSS information. But due to COW, child processes >> share parent process's memory until they modify their memory, so >> simply looking value of RSS is not enough. We need to measure size of >> shared part e.g. by looking at /proc/<PID>/smaps. > > You're right. Would you choose the way which can correct effective > memory consumption ? > OK. PSS in smaps seems proper here, and though /proc/<PID>/pagemap would also allow us to evaluate exact RSS, it's very slow. This is not RSS, valgrind seems useful for evaluating usage of heap memory. I'll consider how to evalute.. >> And, what version of makedumpfile should I use for this bench? If you >> have additonal changes locally, it might be better to use the version. > > Please use v1.4.4 + v2 patch of cyclic + performance improvement patch which > I sent today, it's the latest version internally. > I'll use this version. Thanks. HATAYAMA, Daisuke
