[RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



From: Atsushi Kumagai <kumagai-atsushi@xxxxxxxxxxxxxxxxx>
Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
Date: Wed, 15 Aug 2012 15:27:10 +0900

> Hello HATAYAMA-san,
> 
> On Tue, 14 Aug 2012 20:55:32 +0900 (JST)
> HATAYAMA Daisuke <d.hatayama at jp.fujitsu.com> wrote:
> 
>> From: Vivek Goyal <vgoyal at redhat.com>
>> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
>> Date: Fri, 10 Aug 2012 10:36:55 -0400
>> 
>> > On Fri, Aug 10, 2012 at 05:39:38PM +0900, HATAYAMA Daisuke wrote:
>> > 
>> > [..]
>> >> 
>> >> I finished benchmarking filtering time and demonstrate the result.
>> >> But I failed to collect amount of memory consumption by my mistake. If
>> >> they are necessary, I'll again try to collect them. But we have 9 days
>> >> vacation starting tommorow, so I'll do that after the vacation.
>> >> 
> 
> Thank you for your help.
> Could you continue to measure amount of memory consumption ?
> 

Hello Kumagai-san, here is the benchmark result for the amount of
actual memory consumption when there are multiple child processes
using --split option.

I measured the data in the following way:

  * for buffer size, 1MB, 2MB, 4MB, 8MB and 16MB are chosen.

  * for the number of splitted dumpfiles, 2, 4, 8 and 16 are chosen.

  * I collected RSS, SHA(RED) and PRI(VATE) memory for each process
    from /proc/self/smaps; RSS is the value of the column "Rss:", SHA
    is the value of the column "Shared_Clean" plus the value of the
    column "Shared_Dirty" and PRI is the value of the column
    "Private_Clean" plus the value of the clumn "Private_Dirty".

    Note that RSS is the amount of memory actually mapped on physical
    memory, and RSS = SHA + PRI always holds.

    Note also that the reason of collecting SHA is the fact that
    makedumpfile --split is implemented using fork and child processes
    shares the same parent's memory by COW.

    Notice that I write only one child process data below because each
    result is exactly the same and also purely due to space issue;
    there's no room to put all childs' data here.

  * In the following table, I'm using [BEGIN] => [END] notation, such
    as 3.16 => 0.57. This indicates at what point the data is
    measured.

    For the parent process, the left-hand side, [BEGIN] part, is
    before waitpid() in writeout_multiple_dumpfiles(), while the
    right-hand side, [END] part, after waitpid().

    For child process, the left-hand side, [BEGIN] part, is the
    beginning of write_kdump_pages_and_bitmap_cyclic(), while the
    right-hand side, [END] part, the end of
    write_kdump_pages_and_bitmap_cyclic().


buffer size 1MB

      split     parent         child
------------------------------------------
RSS          3.42 => 3.42   2.98 => 3.27
SHA     2    3.16 => 0.57   2.85 => 0.89
PRI          0.25 => 2.84   0.13 => 2.38
------------------------------------------
RSS          3.42 => 3.42   2.98 => 3.27
SHA     4    3.18 => 0.57   2.85 => 0.89
PRI          0.24 => 2.84   0.13 => 2.38
------------------------------------------
RSS          3.42 => 3.42   2.98 => 3.27
SHA     8    3.18 => 0.57   2.85 => 0.89
PRI          0.23 => 2.84   0.13 => 2.37
------------------------------------------
RSS          3.41 => 3.41   2.98 => 3.26
SHA    16    3.18 => 0.57   2.84 => 0.89
PRI          0.23 => 2.84   0.13 => 2.37


buffer size 2MB

      split     parent         child
------------------------------------------
RSS          5.42 => 5.42   4.98 => 5.27
SHA     2    5.17 => 0.57   4.85 => 0.89
PRI          0.25 => 4.84   0.13 => 4.37
------------------------------------------
RSS          5.42 => 5.42   4.98 => 5.27
SHA     4    5.17 => 0.57   4.85 => 0.89
PRI          0.25 => 4.84   0.13 => 4.37
------------------------------------------
RSS          5.41 => 5.41   4.98 => 5.26
SHA     8    5.18 => 0.57   4.84 => 0.89
PRI          0.23 => 4.84   0.13 => 4.37
------------------------------------------
RSS          5.42 => 5.42   4.98 => 5.27
SHA    16    5.19 => 0.57   4.85 => 0.89
PRI          0.23 => 4.84   0.13 => 4.37


buffer size 4MB

      split     parent         child
------------------------------------------
RSS          9.42 => 9.42   8.98 => 9.27
SHA     2    9.18 => 0.57   8.85 => 0.89
PRI          0.24 => 8.84   0.13 => 8.38
------------------------------------------
RSS          9.41 => 9.41   8.98 => 9.26
SHA     4    9.18 => 0.57   8.84 => 0.89
PRI          0.24 => 8.84   0.13 => 8.37
------------------------------------------
RSS          9.41 => 9.41   8.98 => 9.26
SHA     8    9.18 => 0.57   8.84 => 0.89
PRI          0.24 => 8.84   0.13 => 8.37
------------------------------------------
RSS          9.42 => 9.42   8.98 => 9.27
SHA    16    9.18 => 0.57   8.85 => 0.89
PRI          0.23 => 8.84   0.13 => 8.38


buffer size 8MB

      split      parent          child
---------------------------------------------
RSS          17.41 => 17.41  16.98 => 17.26
SHA     2    17.17 =>  0.57  16.84 =>  0.89
PRI           0.25 => 16.84   0.13 => 16.37
---------------------------------------------
RSS          17.41 => 17.41   16.98 => 17.27
SHA     4    17.18 =>  0.57   16.84 =>  0.89
PRI           0.24 => 16.84    0.13 => 16.38
---------------------------------------------
RSS          17.42 => 17.42   16.98 => 17.27
SHA     8    17.17 =>  0.57   16.85 =>  0.89
PRI           0.25 => 16.84    0.13 => 16.37
---------------------------------------------
RSS          17.42 => 17.42  16.98 => 17.27
SHA    16    17.18 =>  0.57  16.85 =>  0.89
PRI           0.23 => 16.84   0.13 => 16.38


buffer size 16MB

      split      parent           child
---------------------------------------------
RSS          33.41 => 33.41  32.98 => 33.27
SHA     2    33.16 =>  0.57  32.84 =>  0.89
PRI           0.25 => 32.84   0.13 => 32.38
---------------------------------------------
RSS          33.42 => 33.42   32.98 => 33.27
SHA     4    33.18 =>  0.57   32.85 =>  0.89
PRI           0.24 => 32.84    0.13 => 32.38
---------------------------------------------
RSS          33.42 => 33.42   32.98 => 33.27
SHA     8    33.18 =>  0.57   32.85 =>  0.89
PRI           0.23 => 32.84    0.13 => 32.37
---------------------------------------------
RSS          33.41 => 33.41  32.98 => 33.26
SHA    16    33.18 =>  0.57  32.84 =>  0.89
PRI           0.24 => 32.84   0.13 => 32.37


Thanks.
HATAYAMA, Daisuke




[Index of Archives]     [LM Sensors]     [Linux Sound]     [ALSA Users]     [ALSA Devel]     [Linux Audio Users]     [Linux Media]     [Kernel]     [Gimp]     [Yosemite News]     [Linux Media]

  Powered by Linux