Hello Lisa, On Tue, 04 Dec 2012 06:31:39 -0700 Lisa Mitchell <lisa.mitchell at hp.com> wrote: > On Tue, 2012-11-20 at 05:14 -0700, Lisa Mitchell wrote: > > > > > > > I tested this makedumpfile v1.5.1-rc on a 4 TB DL980, on 2.6.32 based > > kernel, and got good results. With crashkernel=256M, and default > > settings (i.e. no cyclic buffer option selected), the dump successfully > > completed in about 2 hours, 40 minutes, and then I specified a cyclic > > buffer size of 48 M, and the dump completed in the same time, no > > measurable differences within the accuracy of our measurements. > > > > We are still evaluating perfomance data, and don't have very precise > > measurements here for comparisons, but the results look promising so > > far. > > > > Lisa Mitchell > > Update: > > I did another test over the last few days that was a better apples-to- > apples comparison, contrasting the performance of makedumpfile 1.4 with > makedumpfile v1.5.1-rc on a RHEL 6.3 system with 4 TB of memory. Thanks for your measurement. It's very helpful because I can't get a chance to use such a large memory machine. > Earlier I had not taken good comparable measurements of the dump times, > from the exact same machine configuration comparisons of the timing > differences between the two makedumpfiles. I had noted that > makedumpfile 1.5.1-rc seemed a performance improvemnt over makedumpfile > v1.5.0 results seen earlier. > > Unfortunately this weekend, the results showed a significant performance > regression still with makedumpfile v1.5.1-rc compared to makedumpfile > 1.4 > > This time my performance measurements were based on comparing the file > system timestamps in the /var/crash directory, showing the difference > from when the crash directory was created by makedumpfile, to the > timestamp on the vmcore file, to show when the copy of the memory to > this file was complete. > > 1. Baseline: On the 4 TB DL980, with the RHEL 6.3 installation,(2.6.32 > based kernel) with a crashkernel size of 512M or 384M (both big enough > to contain the 256M bit map required, plus the kernel).The makedumpfile > command line was the same for both tests: " -c --message-level 1 -d 31" > The timestamps shown for the dump copy were: > > # cd /var/crash > # ls > 127.0.0.1-2012-11-30-15:28:22 > # cd 127.0.0.1-2012-11-30-15:28:22 > #ls -al 127.0.0.1-2012-11-30-15:28:22 > total 10739980^M > drwxr-xr-x. 2 root root 4096 Nov 30 17:07 .^M > drwxr-xr-x. 3 root root 4096 Nov 30 15:28 ..^M > -rw-------. 1 root root 10997727069 Nov 30 17:07 vmcore > > >From the time stamps above the dump started at 15:28, completed at > 17:07, the dump time was 1 hour, 41 minutes. > > 2. Makedumpfile-v1.5.1-rc on the same system configuration as (1.) > above, but with crashkernel size set to 256 M to insure the use of the > cyclic buffer feature to fit in smaller crashkernel space. The same > makedumpfile command line of "-c --message-level 1 -d 31" was used. > > #cd /var/crash > # ls -al > total 12 > drwxr-xr-x. 3 root root 4096 Nov 30 23:25 . > drwxr-xr-x. 22 root root 4096 Nov 30 08:41 .. > drwxr-xr-x. 2 root root 4096 Dec 1 02:05 127.0.0.1-2012-11-30-23:25:20 > > #ls -al * > total 10734932 > drwxr-xr-x. 2 root root 4096 Dec 1 02:05 . > drwxr-xr-x. 3 root root 4096 Nov 30 23:25 .. > -rw-------. 1 root root 10992554141 Dec 1 02:05 vmcore > > >From the timestamps above, the dump started at 23:25 and completed at > 2:05 after midnight, so the total dump time was 2 hours and 40 minutes. > > So for this 4 TB system, in this test, the dump write phase took 1 hour > longer for makedumpfile-v1.5.1-rc, versus makedumpfile v1.4. This time > seems dominated by the dump filtering activity, assuming the copy to > disk times should have been the same, though I don't have a good > breakdown. As you may understand, the number of cycle is two (or larger) in your test (2.). And it seems that you used free_list logic because you specified neither -x vmlinux option nor -i vmcoreinfo_text_file option. (Please see release note for how to use mem_map array logic.) http://lists.infradead.org/pipermail/kexec/2012-December/007460.html This combination means that redundant scans was done in your test, I think makedumpfile-v1.5.1-rc couldn't show the best performance we expected. So, could you do the same test with v1.5.1-GA (but, the logic isn't different from rc) and -i vmcoreinfo_text_file option ? We should see its result and discuss it. In addition, you need to include vmcoreinfo_text_file in initramfs in order to use -i option. If you have RedHat OS, you can refer to /sbin/mkdumprd to know how to do it. Thanks Atushi Kumagai
