From: HATAYAMA Daisuke <d.hatayama@xxxxxxxxxxxxxx>
Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
Date: Tue, 7 Aug 2012 16:31:20 +0900
> From: Vivek Goyal <vgoyal at redhat.com>
> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
> Date: Mon, 6 Aug 2012 16:47:31 -0400
>
>> On Fri, Jun 29, 2012 at 11:13:20AM +0900, Atsushi Kumagai wrote:
>>> Hello,
>>>
>>> I improved prototype of cyclic processing as version 2.
>>> If there is no objection to basic idea, I want to consider the things
>>> related to performance as next step. (Concretely, buffer size and the patch set
>>> HATAYAMA-san sent a short time ago.)
>>
>> Hi Atsushi San,
>>
>> Just checking that what's the state of these patches now. Are they ready
>> to be included in makedumpfile?
>>
>> I would love to see new makedumpfile where memory usage does not grow
>> by physical memory present in the system. (Assuming computig overhead
>> of cycles is bearable).
>>
>> Thanks
>> Vivek
>>
>
> Hello Vivek,
>
> I'm just now benchmarking cycle processing on our machine. Please wait
> for a while.
>
> Thanks.
> HATAYAMA, Daisuke
>
I finished benchmarking filtering time and demonstrate the result.
But I failed to collect amount of memory consumption by my mistake. If
they are necessary, I'll again try to collect them. But we have 9 days
vacation starting tommorow, so I'll do that after the vacation.
The machine spec I used is as follows:
Memory: 2TB
CPU: Intel(R) Xeon(R) CPU E7- 8870 @ 2.40GHz
(8 sockets, 10 cores, 2 threads)
In the first step, I chosed buffer size 10KB and it took about 3h 45m
57s. So, next I changed the buffer size to 512KB and measured up to
8MB.
The result is as follows:
| buffer size | time |
|-------------+------------|
| 8 MB | 48.32 sec |
| 4 MB | 55.76 sec |
| 2 MB | 69.91 sec |
| 1 MB | 98.25 sec |
| 512 KB | 154.42 sec |
BTW, the existing free_list logic took about 48 sec for the same
vmcore as below.
STEP [Excluding free pages ] : 49.846321 seconds
STEP [Excluding unnecessary pages] : 6.339228 seconds
STEP [Excluding free pages ] : 48.595884 seconds
STEP [Excluding unnecessary pages] : 6.530479 seconds
STEP [Excluding free pages ] : 48.598879 seconds
STEP [Excluding unnecessary pages] : 6.527133 seconds
STEP [Excluding free pages ] : 48.602401 seconds
STEP [Excluding unnecessary pages] : 6.502681 seconds
STEP [Excluding free pages ] : 48.602010 seconds
STEP [Excluding unnecessary pages] : 6.469853 seconds
STEP [Excluding free pages ] : 48.601637 seconds
STEP [Excluding unnecessary pages] : 6.431381 seconds
STEP [Excluding free pages ] : 48.601195 seconds
STEP [Excluding unnecessary pages] : 6.416676 seconds
STEP [Excluding free pages ] : 48.602221 seconds
STEP [Excluding unnecessary pages] : 6.387611 seconds
STEP [Excluding free pages ] : 48.589972 seconds
STEP [Excluding unnecessary pages] : 0.816955 seconds
Original pages : 0x0000000040049690
Excluded pages : 0x000000001f3c1564
Pages filled with zero : 0x0000000000000000
Cache pages : 0x000000000000467d
Cache pages + private : 0x000000000000103c
User process data pages : 0x00000000000015d6
Free pages : 0x000000001f3ba8d5
Remaining pages : 0x0000000020c8812c
(The number of pages is reduced to 51%.)
Memory Hole : 0xffffffffe0036970
--------------------------------------------------
Total pages : 0x0000000020080000
There are other log files. I can directly email them if necessary; the
reason why I didn't attach the log files in this mail is that sending
the mail with attachment to this ML requires authentication and it
would take some time.
Thanks.
HATAYAMA, Daisuke
