From: Vivek Goyal <vgoyal@xxxxxxxxxx> Subject: Re: [PATCH v1 0/2] x86, apic: Disable BSP if boot cpu is AP Date: Wed, 17 Oct 2012 10:12:34 -0400 > On Tue, Oct 16, 2012 at 01:35:17PM +0900, HATAYAMA Daisuke wrote: >> Multiple CPUs are useful for CPU-bound processing like compression and >> I do want to use compression to generate crash dump quickly. But now >> we cannot wakeup the 2nd and later cpus in the kdump 2nd kernel if >> crash happens on AP. If crash happens on AP, kexec enters the 2nd >> kernel with the AP, and there BSP in the 1st kernel is expected to be >> haling in the 1st kernel or possibly in any fatal system error state. > > Hatayama san, > Hello Vivek, > Do you have any rough numbers on what kind of speed up we are looking > at. IOW, what % of time is gone compressing a filetered dump. On large > memory machines, saving huge dump files is anyway not an option due to > time it takes. So we need to filter it to bare minimum and after that > vmcore size should be reasonable and compression time might not be a > big factor. Hence I am curious what kind of gains we are looking at. > I did two kinds of benchmark 1) to evaluate how well compression and writing dump into multiple disks performs on crash dump and 2) to compare three kinds of compression algorhythm --- zlib, lzo and snappy --- for use of crash dump. >From 1), 4 disks with 4 cpus performs 300 MB/s on compression with snappy. 1 hour for 1 TB. But on this benchmark, sample data is intentionally randomized enough so data size is not reduced during compression, it must be quicker on most of actual dumps. See also bench_comp_multi_IO.tar.xz for image of graph. In the future, I'm going to do this benchmark again using quicker SSD disks if I get them. >From 2), zlib, used when doing makedumpfile -c, turns out to be too slow to use it for crash dump. lzo and snappy is quick and relatively as good compression ratio as zlib. In particular, snappy speed is stable on any ratio of randomized part. See also bench_compare_zlib_lzo_snappy.tar.xz for image of graph. BTW, makedumpfile has already supported lzo since v1.4.4 and is going to support snappy on v1.5.1. OTOH, we have some requirements where we cannot use filtering. Examples are: - high-availability cluster system where application triggers crash dump to switch the active node to inactive node quickly. We retrieve the application image as process core dump later and analize it. We cannot filter user-space memory. - On qemu/kvm environment, we sometimes face a complicated bug caused by interaction between guest and host. For example, previously, there was a bug causing guest machine to hang, where IO scheduler handled guest's request as wrongly lower request than the actual one and guest was waiting for IO completion endlessly, repeating VMenter-VMexit forever. To address such kind of bug, we first reproduce the bug, get host's crash dump to capture the situation, and then analyze the bug by investigating the situation from both host's and guest's views. On the bug above, we saw guest machine was waiting for IO, and we could resolve the issue relatively quickly. For this kind of complicated bug relevant to qemu/kvm, both host and guest views are very helpful. guest image is in user-space memory, qemu process, and again we cannot filter user-space memory. - Filesystem people say page cache is often necessary for analysis of crash dump. Of course, we use filtering positively on the system where no such requreirement is given. >> >> To wake up AP, we use the method called INIT-INIT-SIPI. INIT causes >> BSP to jump into BIOS init code. A typical visible behaviour is hang >> or immediate reset, depending on the BIOS init code. >> >> AP can be initiated by INIT even in a fatal state: MP spec explains >> that processor-specific INIT can be used to recover AP from a fatal >> system error. On the other hand, there's no method for BSP to recover; >> it might be possible to do so by NMI plus any hand-coded reset code >> that is carefully designed, but at least I have no idea in this >> direction now. >> >> Therefore, the idea I do in this patch set is simply to disable BSP if >> vboot cpu is AP. > > So in regular boot BSP still works as we boot on BSP. So this will take > effect only in kdump kernel? > Yes, this patch is effective only for the case where boot cpu is not BSP, AP, and this happens in kexec case only. > How well does it work with nr_cpus kernel parameter. Currently we boot > with nr_cpus=1 to save upon amount of memory to be reserved. I guess > you might want to boot with nr_cpus=2 or nr_cpus=4 in your case to > speed up compression? Exactly, it seems reasonable to specify at most nr_cpus=4 on usual machines becaue reserved memory is severely limited, and many disks are difficult to connect only for crash dump use without special requrement. But there might be the system where crash dump is definitely done quickly and for it, more reserved memory and more disks are no problem. On such system, I think it's necessary to be able to set up more reserved memory and more cpus. > > > [..] >> Note: recent upstream kernel fails reserving memory for kdump 2nd >> kernel. To run kdump, please apply the patch below on top of this >> patch set: >> https://lkml.org/lkml/2012/8/31/238 > > Above is a big issue. 3.6 kernel is broken and I can't take dump on F18 > either. (works only on one machine). I have not looked enough into it > the issue to figure out what's the issue at hand, but we really need > atleast a stop gap fix (assuming others are working on longer term > fix). > To be honest, I didn't follow this in some monthes, but I expect the topic below to fix this issue. What's the status of bzImage limitation? http://lists.infradead.org/pipermail/kexec/2012-September/006855.html Thanks. HATAYAMA, Daisuke -------------- next part -------------- A non-text attachment was scrubbed... Name: bench_comp_multi_IO.tar.xz Type: application/octet-stream Size: 62112 bytes Desc: not available URL: <http://lists.infradead.org/pipermail/kexec/attachments/20121018/d2733ed0/attachment-0002.obj> -------------- next part -------------- A non-text attachment was scrubbed... Name: bench_compare_zlib_lzo_snappy.tar.xz Type: application/octet-stream Size: 147120 bytes Desc: not available URL: <http://lists.infradead.org/pipermail/kexec/attachments/20121018/d2733ed0/attachment-0003.obj>
