From: Atsushi Kumagai <kumagai-atsushi@xxxxxxxxxxxxxxxxx>
Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
Date: Wed, 15 Aug 2012 15:27:10 +0900
> Hello HATAYAMA-san,
>
> On Tue, 14 Aug 2012 20:55:32 +0900 (JST)
> HATAYAMA Daisuke <d.hatayama at jp.fujitsu.com> wrote:
>
>> From: Vivek Goyal <vgoyal at redhat.com>
>> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
>> Date: Fri, 10 Aug 2012 10:36:55 -0400
>>
>> > On Fri, Aug 10, 2012 at 05:39:38PM +0900, HATAYAMA Daisuke wrote:
>> >
>> > [..]
>> >>
>> >> I finished benchmarking filtering time and demonstrate the result.
>> >> But I failed to collect amount of memory consumption by my mistake. If
>> >> they are necessary, I'll again try to collect them. But we have 9 days
>> >> vacation starting tommorow, so I'll do that after the vacation.
>> >>
>
> Thank you for your help.
> Could you continue to measure amount of memory consumption ?
>
Hello Kumagai-san, here is the benchmark result for the amount of
actual memory consumption when there are multiple child processes
using --split option.
I measured the data in the following way:
* for buffer size, 1MB, 2MB, 4MB, 8MB and 16MB are chosen.
* for the number of splitted dumpfiles, 2, 4, 8 and 16 are chosen.
* I collected RSS, SHA(RED) and PRI(VATE) memory for each process
from /proc/self/smaps; RSS is the value of the column "Rss:", SHA
is the value of the column "Shared_Clean" plus the value of the
column "Shared_Dirty" and PRI is the value of the column
"Private_Clean" plus the value of the clumn "Private_Dirty".
Note that RSS is the amount of memory actually mapped on physical
memory, and RSS = SHA + PRI always holds.
Note also that the reason of collecting SHA is the fact that
makedumpfile --split is implemented using fork and child processes
shares the same parent's memory by COW.
Notice that I write only one child process data below because each
result is exactly the same and also purely due to space issue;
there's no room to put all childs' data here.
* In the following table, I'm using [BEGIN] => [END] notation, such
as 3.16 => 0.57. This indicates at what point the data is
measured.
For the parent process, the left-hand side, [BEGIN] part, is
before waitpid() in writeout_multiple_dumpfiles(), while the
right-hand side, [END] part, after waitpid().
For child process, the left-hand side, [BEGIN] part, is the
beginning of write_kdump_pages_and_bitmap_cyclic(), while the
right-hand side, [END] part, the end of
write_kdump_pages_and_bitmap_cyclic().
buffer size 1MB
split parent child
------------------------------------------
RSS 3.42 => 3.42 2.98 => 3.27
SHA 2 3.16 => 0.57 2.85 => 0.89
PRI 0.25 => 2.84 0.13 => 2.38
------------------------------------------
RSS 3.42 => 3.42 2.98 => 3.27
SHA 4 3.18 => 0.57 2.85 => 0.89
PRI 0.24 => 2.84 0.13 => 2.38
------------------------------------------
RSS 3.42 => 3.42 2.98 => 3.27
SHA 8 3.18 => 0.57 2.85 => 0.89
PRI 0.23 => 2.84 0.13 => 2.37
------------------------------------------
RSS 3.41 => 3.41 2.98 => 3.26
SHA 16 3.18 => 0.57 2.84 => 0.89
PRI 0.23 => 2.84 0.13 => 2.37
buffer size 2MB
split parent child
------------------------------------------
RSS 5.42 => 5.42 4.98 => 5.27
SHA 2 5.17 => 0.57 4.85 => 0.89
PRI 0.25 => 4.84 0.13 => 4.37
------------------------------------------
RSS 5.42 => 5.42 4.98 => 5.27
SHA 4 5.17 => 0.57 4.85 => 0.89
PRI 0.25 => 4.84 0.13 => 4.37
------------------------------------------
RSS 5.41 => 5.41 4.98 => 5.26
SHA 8 5.18 => 0.57 4.84 => 0.89
PRI 0.23 => 4.84 0.13 => 4.37
------------------------------------------
RSS 5.42 => 5.42 4.98 => 5.27
SHA 16 5.19 => 0.57 4.85 => 0.89
PRI 0.23 => 4.84 0.13 => 4.37
buffer size 4MB
split parent child
------------------------------------------
RSS 9.42 => 9.42 8.98 => 9.27
SHA 2 9.18 => 0.57 8.85 => 0.89
PRI 0.24 => 8.84 0.13 => 8.38
------------------------------------------
RSS 9.41 => 9.41 8.98 => 9.26
SHA 4 9.18 => 0.57 8.84 => 0.89
PRI 0.24 => 8.84 0.13 => 8.37
------------------------------------------
RSS 9.41 => 9.41 8.98 => 9.26
SHA 8 9.18 => 0.57 8.84 => 0.89
PRI 0.24 => 8.84 0.13 => 8.37
------------------------------------------
RSS 9.42 => 9.42 8.98 => 9.27
SHA 16 9.18 => 0.57 8.85 => 0.89
PRI 0.23 => 8.84 0.13 => 8.38
buffer size 8MB
split parent child
---------------------------------------------
RSS 17.41 => 17.41 16.98 => 17.26
SHA 2 17.17 => 0.57 16.84 => 0.89
PRI 0.25 => 16.84 0.13 => 16.37
---------------------------------------------
RSS 17.41 => 17.41 16.98 => 17.27
SHA 4 17.18 => 0.57 16.84 => 0.89
PRI 0.24 => 16.84 0.13 => 16.38
---------------------------------------------
RSS 17.42 => 17.42 16.98 => 17.27
SHA 8 17.17 => 0.57 16.85 => 0.89
PRI 0.25 => 16.84 0.13 => 16.37
---------------------------------------------
RSS 17.42 => 17.42 16.98 => 17.27
SHA 16 17.18 => 0.57 16.85 => 0.89
PRI 0.23 => 16.84 0.13 => 16.38
buffer size 16MB
split parent child
---------------------------------------------
RSS 33.41 => 33.41 32.98 => 33.27
SHA 2 33.16 => 0.57 32.84 => 0.89
PRI 0.25 => 32.84 0.13 => 32.38
---------------------------------------------
RSS 33.42 => 33.42 32.98 => 33.27
SHA 4 33.18 => 0.57 32.85 => 0.89
PRI 0.24 => 32.84 0.13 => 32.38
---------------------------------------------
RSS 33.42 => 33.42 32.98 => 33.27
SHA 8 33.18 => 0.57 32.85 => 0.89
PRI 0.23 => 32.84 0.13 => 32.37
---------------------------------------------
RSS 33.41 => 33.41 32.98 => 33.26
SHA 16 33.18 => 0.57 32.84 => 0.89
PRI 0.24 => 32.84 0.13 => 32.37
Thanks.
HATAYAMA, Daisuke
