[PATCH] makedumpfile: keep dumpfile pages in a cache

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello Petr,

On Thu, 6 Sep 2012 17:50:52 +0200
Petr Tesarik <ptesarik at suse.cz> wrote:

> Dne Po 3. z??? 2012 09:04:03 Petr Tesarik napsal(a):
> > Dne Po 3. z??? 2012 05:42:33 Atsushi Kumagai napsal(a):
> > > Hello Petr,
> > > 
> > > On Tue, 28 Aug 2012 19:49:49 +0200
> > > 
> > > Petr Tesarik <ptesarik at suse.cz> wrote:
> > > > Add a simple cache for pages read from the dumpfile.
> > > > 
> > > > This is a big win if we read consecutive data from one page, e.g.
> > > > page descriptors, or even page table entries.
> > > > 
> > > > Note that makedumpfile now always reads a complete page. This was
> > > > already the case with kdump-compressed and sadump formats, but
> > > > makedumpfile was throwing most of the data away. For the
> > > > kdump-compressed case, we may actually save a lot of decompression,
> > > > too.
> > > > 
> > > > I tried to keep the cache small to minimize memory footprint, but it
> > > > should be big enough to hold all pages to do 4-level paging plus some
> > > > data. This is needed e.g. for vmalloc areas or Xen page frame table
> > > > data, which are not contiguous in physical memory.
> > > > 
> > > > Signed-off-by: Petr Tesarik <ptesarik at suse.cz>

Sorry for the late reply.
According to your measurement, it looks good on performance.

However, I found the issue below in v1.5.1-beta and made sure that this patch
causes it by git bisect (but I don't find the true cause yet).

  result on kernel 3.4:
    $ makedumpfile --non-cyclic vmcore dumpfile
    Copying data                       : [ 62 %] 
    readpage_elf: Can't convert a physical address(a0000) to offset.
    readmem: type_addr: 1, addr:1000a0000, size:4096
    read_pfn: Can't get the page data.

    makedumpfile Failed.
    $

It seems critical issue for all users, so I will postpone merging this patch
until this issue is solved.


Thanks
Atsushi Kumagai

> > > 
> > > It's interesting to me. I want to know how performance will be improved
> > > with this patch, so do you have speed measurements ?
> > 
> > Not really. I only measured the hit/miss ratio, and with filtering Xen domU
> > and dump level 0, I got the following on a small system (2G RAM):
> > 
> > cache hit: 1818880  cache miss: 1873
> > 
> > The improvement isn't much for non-Xen case, because the hits are mostly
> > due to virtual-to-physical translations, and most of Linux data is stored
> > at virtual addresses that can be resolved by adding/subtracting a fixed
> > offset.
> > 
> > Of course, you will also win only the syscall overhead, because Linux keeps
> > the data in the kernel pagecache anyway. I'll measure the times for you on
> > a reasonably large system (~256G) and send the results here.
> 
> I couldn't get a medium-sized system for testing, so I performed some 
> measurements on a 64G system. I ran makedumpfile repeatedly from the kdump 
> environment. First run was used to cache target filesystem metadata, and the 
> cache was not dropped between runs to minimize effects of the target 
> filesystem. I ran it against /proc/vmcore, i.e. the input file was always 
> resident, nothing to skew the results.
> 
> I tried with a kdump file with no compression (to get gzip/LZO out of the 
> picture) and an ELF file. For the Xen case I only did the ELF file, because 
> kdump is not available.
> 
> First I ran it on bare metal. There was a slight improvement for -d31:
> 
> kdump no cache:
> 6.32user 55.20system 1:15.60elapsed 81%CPU (0avgtext+0avgdata 
> 4800maxresident)k
> 2080inputs+5714296outputs (2major+342minor)pagefaults 0swaps
> 
> kdump with cache:
> 6.02user 24.58system 0:46.51elapsed 65%CPU (0avgtext+0avgdata 
> 4912maxresident)k
> 1864inputs+5714288outputs (2major+350minor)pagefaults 0swaps
> 
> ELF no cache:
> 7.58user 74.25system 1:59.52elapsed 68%CPU (0avgtext+0avgdata 
> 4800maxresident)k
> 728inputs+9288824outputs (1major+342minor)pagefaults 0swaps
> 
> ELF with cache:
> 7.43user 44.21system 1:17.41elapsed 66%CPU (0avgtext+0avgdata 
> 4896maxresident)k
> 728inputs+9288792outputs (1major+349minor)pagefaults 0swaps
> 
> To sum it up, I can see an improvement of approx. 50% in system time. The 
> increase in memory consumption is a bit more than I would expect (why do I see 
> ~100k for a cache of 12k?), but acceptable nevertheless. I can see a slight 
> increase in user time (approx. 25%) for the kdump case, which could be 
> attributed to the cache overhead. I don't have any explanation for the 
> decreased user time for the ELF case, but it's consistent.
> 
> I also tried running makedumpfile with -d1. This results in long sequential 
> reads, so it's the worst case for a simple LRU-policy cache. The results are 
> too unstable to make a reliable measurement, but there seems to be a slight 
> performance hit. It is certainly less than 5% total time.
> 
> I think there are two reasons for that:
> 
> 1. We're copying file data twice for each page (once from the kernel page 
> cache to the process space, and once from the internal cache to the 
> destination).
> 2. Instead of reusing the same data location, we're rotating 8 different pages 
> (or even up to twice as much if the allocated space is neither continuous  nor 
> page-aligned). This stresses both for the CPU's L1 d-cache and the TLB a tiny 
> bit more. Note that in the /proc/vmcore case, the kernel sequentially maps all 
> physical memory of the crashed system, so every cache page may be evicted 
> before we get to using it again. This could explain why I observe an increase 
> in system time despite making less system calls.
> 
> There's a lot of things I could do to regain the old performance, if anybody 
> is concerned about the slight performance regression for this worst case. Just 
> let me know.
> 
> Second, I ran with the Xen hypervisor. Since dump levels greater than 1 don't 
> work, I ran with '-E -X -d1'. Even though this includes the inefficient page 
> walk described above, the improvement was immense.
> 
> no cache:
> 95.33user 657.18system 13:08.40elapsed 95%CPU (0avgtext+0avgdata 
> 5440maxresident)k
> 704inputs+6563856outputs (1major+388minor)pagefaults 0swaps
> 
> with cache:
> 61.14user 110.15system 3:24.24elapsed 83%CPU (0avgtext+0avgdata 
> 5584maxresident)k
> 2360inputs+6563872outputs (2major+396minor)pagefaults 0swaps
> 
> In short, almost 80% shorter total time.
> 
> Petr Tesarik
> SUSE Linux
> 
> _______________________________________________
> kexec mailing list
> kexec at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec



[Index of Archives]     [LM Sensors]     [Linux Sound]     [ALSA Users]     [ALSA Devel]     [Linux Audio Users]     [Linux Media]     [Kernel]     [Gimp]     [Yosemite News]     [Linux Media]

  Powered by Linux