[makedumpfile PATCH] Allow PFN_EXCLUDED to be tunable via command line option --exclude-threshold

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



>The PFN_EXCLUDED value is used to control at which point a run of
>zeros in the bitmap (zeros denote excluded pages) is large enough
>to warrant truncating the current output segment and to create a
>new output segment (containing non-excluded pages), in an ELF dump.
>
>If the run is smaller than PFN_EXCLUDED, then those excluded pages
>are still output in the ELF dump, for the current output segment.
>
>By using smaller values of PFN_EXCLUDED, the resulting dump file
>size can be made smaller by actually removing more excluded pages
>from the resulting dump file.
>
>This patch adds the command line option --exclude-threshold=<value>
>to indicate the threshold. The default is 256, the legacy value
>of PFN_EXCLUDED. The smallest value permitted is 1.
>
>Using an existing vmcore, this was tested by the following:
>
>% makedumpfile -E -d31 --exclude-threshold=256 -x vmlinux vmcore newvmcore256
>% makedumpfile -E -d31 --exclude-threshold=4 -x vmlinux vmcore newvmcore4
>
>I utilize -d31 in order to exclude as many page types as possible,
>resulting in a [significantly] smaller file sizes than the original
>vmcore.
>
>-rwxrwx--- 1 edevolde edevolde 4034564096 Jun 27 10:24 vmcore
>-rw------- 1 edevolde edevolde 119808156 Jul  6 13:01 newvmcore256
>-rw------- 1 edevolde edevolde 100811276 Jul  6 13:08 newvmcore4
>
>The use of smaller value of PFN_EXCLUDED increases the number of
>output segments (the 'Number of program headers' in the readelf
>output) in the ELF dump file.

How will you tune the value ? I'm not sure what is the benefit of the
tunable PFN_EXCLUDED. If there is no regression caused by too many PT_LOAD
entries, I think we can decide a concrete PFN_EXCLUDED. 

The penalty for splitting PT_LOAD is the size of a PT_LOAD header,
so the best PFN_EXCLUDED is the minimum number which meets the condition
below:

      (size of PT_LOAD header)  <  (PFN_EXCLUDED <<  PAGE_SIZE)
     
>% readelf -h vmcore
>ELF Header:
>  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
>  Class:                             ELF64
>  Data:                              2's complement, little endian
>  Version:                           1 (current)
>  OS/ABI:                            UNIX - System V
>  ABI Version:                       0
>  Type:                              CORE (Core file)
>  Machine:                           Advanced Micro Devices X86-64
>  Version:                           0x1
>  Entry point address:               0x0
>  Start of program headers:          64 (bytes into file)
>  Start of section headers:          0 (bytes into file)
>  Flags:                             0x0
>  Size of this header:               64 (bytes)
>  Size of program headers:           56 (bytes)
>  Number of program headers:         6
>                                     ^^^
>  Size of section headers:           0 (bytes)
>  Number of section headers:         0
>  Section header string table index: 0
>
>% readelf -h newvmcore256
>ELF Header:
>  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
>  Class:                             ELF64
>  Data:                              2's complement, little endian
>  Version:                           1 (current)
>  OS/ABI:                            UNIX - System V
>  ABI Version:                       0
>  Type:                              CORE (Core file)
>  Machine:                           Advanced Micro Devices X86-64
>  Version:                           0x1
>  Entry point address:               0x0
>  Start of program headers:          64 (bytes into file)
>  Start of section headers:          0 (bytes into file)
>  Flags:                             0x0
>  Size of this header:               64 (bytes)
>  Size of program headers:           56 (bytes)
>  Number of program headers:         18
>                                     ^^^
>  Size of section headers:           0 (bytes)
>  Number of section headers:         0
>  Section header string table index: 0
>
>% readelf -h newvmcore4
>ELF Header:
>  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
>  Class:                             ELF64
>  Data:                              2's complement, little endian
>  Version:                           1 (current)
>  OS/ABI:                            UNIX - System V
>  ABI Version:                       0
>  Type:                              CORE (Core file)
>  Machine:                           Advanced Micro Devices X86-64
>  Version:                           0x1
>  Entry point address:               0x0
>  Start of program headers:          64 (bytes into file)
>  Start of section headers:          0 (bytes into file)
>  Flags:                             0x0
>  Size of this header:               64 (bytes)
>  Size of program headers:           56 (bytes)
>  Number of program headers:         244
>                                     ^^^
>  Size of section headers:           0 (bytes)
>  Number of section headers:         0
>  Section header string table index: 0
>
>The newvmcore4 has an even smaller file size than newvmcore256, with
>the small price being that there are now 244 rather than 18 segments
>in the dump file.
>
>And with a larger number of segments, loading both vmcore and newvmcore4
>into 'crash' resulted in identical outputs when run with the dmesg, ps,
>files, mount, and net sub-commands.

What about the processing speed of crash, is there no slow down ?


Thanks,
Atsushi Kumagai

>Signed-off-by: Eric DeVolder <eric.devolder at oracle.com>
>---
>v1: Posted 06jul2017 to kexec-tools mailing list
> - original
>---
> makedumpfile.c | 20 +++++++++++++++++---
> makedumpfile.h |  4 +++-
> 2 files changed, 20 insertions(+), 4 deletions(-)
>
>diff --git a/makedumpfile.c b/makedumpfile.c
>index e69b6df..940f64c 100644
>--- a/makedumpfile.c
>+++ b/makedumpfile.c
>@@ -7236,7 +7236,7 @@ get_loads_dumpfile_cyclic(void)
>
> 				/*
> 				 * If the number of the contiguous pages to be excluded
>-				 * is 256 or more, those pages are excluded really.
>+				 * is PFN_EXCLUDED or more, those pages are excluded really.
> 				 * And a new PT_LOAD segment is created.
> 				 */
> 				if (num_excluded >= PFN_EXCLUDED) {
>@@ -7352,7 +7352,7 @@ write_elf_pages_cyclic(struct cache_data *cd_header, struct cache_data *cd_page)
> 					continue;
> 					/*
> 					 * If the number of the contiguous pages to be excluded
>-					 * is 255 or less, those pages are not excluded.
>+					 * is less than PFN_EXCLUDED, those pages are not excluded.
> 					 */
> 				} else if (num_excluded < PFN_EXCLUDED) {
> 					if ((pfn == pfn_end - 1) && frac_tail) {
>@@ -7370,7 +7370,7 @@ write_elf_pages_cyclic(struct cache_data *cd_header, struct cache_data *cd_page)
>
> 				/*
> 				 * If the number of the contiguous pages to be excluded
>-				 * is 256 or more, those pages are excluded really.
>+				 * is PFN_EXCLUDED or more, those pages are excluded really.
> 				 * And a new PT_LOAD segment is created.
> 				 */
> 				load.p_memsz = memsz;
>@@ -11007,6 +11007,7 @@ static struct option longopts[] = {
> 	{"splitblock-size", required_argument, NULL, OPT_SPLITBLOCK_SIZE},
> 	{"work-dir", required_argument, NULL, OPT_WORKING_DIR},
> 	{"num-threads", required_argument, NULL, OPT_NUM_THREADS},
>+	{"exclude-threshold", required_argument, NULL, OPT_PFN_EXCLUDE_THRESHOLD},
> 	{0, 0, 0, 0}
> };
>
>@@ -11044,6 +11045,14 @@ main(int argc, char *argv[])
> 	 */
> 	info->flag_usemmap = MMAP_TRY;
>
>+	/*
>+	 * A run of zeros in the bitmap (excluded pages) of less than
>+	 * pfn_excluded_threshold in length will still be dumped. Runs greater
>+	 * than or equal to pfn_excluded_threshold will result in the creation
>+	 * of a new output segment, for ELF dumps.
>+	 */
>+	info->pfn_exclude_threshold = 256;
>+
> 	info->block_order = DEFAULT_ORDER;
> 	message_level = DEFAULT_MSG_LEVEL;
> 	while ((opt = getopt_long(argc, argv, "b:cDd:eEFfg:hi:lpRvXx:", longopts,
>@@ -11163,6 +11172,11 @@ main(int argc, char *argv[])
> 		case OPT_NUM_THREADS:
> 			info->num_threads = MAX(atoi(optarg), 0);
> 			break;
>+		case OPT_PFN_EXCLUDE_THRESHOLD:
>+			info->pfn_exclude_threshold = strtoul(optarg, NULL, 0);
>+			if (0 == info->pfn_exclude_threshold)
>+				info->pfn_exclude_threshold = 1;
>+			break;
> 		case '?':
> 			MSG("Commandline parameter is invalid.\n");
> 			MSG("Try `makedumpfile --help' for more information.\n");
>diff --git a/makedumpfile.h b/makedumpfile.h
>index e32e567..33d3eb0 100644
>--- a/makedumpfile.h
>+++ b/makedumpfile.h
>@@ -216,7 +216,7 @@ isAnon(unsigned long mapping)
>
> #define BITPERBYTE		(8)
> #define PGMM_CACHED		(512)
>-#define PFN_EXCLUDED		(256)
>+#define PFN_EXCLUDED		(info->pfn_exclude_threshold)
> #define BUFSIZE			(1024)
> #define BUFSIZE_FGETS		(1500)
> #define BUFSIZE_BITMAP		(4096)
>@@ -1139,6 +1139,7 @@ struct DumpInfo {
> 	long		page_size;           /* size of page */
> 	long		page_shift;
> 	mdf_pfn_t	max_mapnr;   /* number of page descriptor */
>+	unsigned long	pfn_exclude_threshold;
> 	unsigned long   page_offset;
> 	unsigned long   section_size_bits;
> 	unsigned long   max_physmem_bits;
>@@ -2143,6 +2144,7 @@ struct elf_prstatus {
> #define OPT_SPLITBLOCK_SIZE	OPT_START+14
> #define OPT_WORKING_DIR         OPT_START+15
> #define OPT_NUM_THREADS	OPT_START+16
>+#define OPT_PFN_EXCLUDE_THRESHOLD	OPT_START+17
>
> /*
>  * Function Prototype.
>--
>2.7.4





[Index of Archives]     [LM Sensors]     [Linux Sound]     [ALSA Users]     [ALSA Devel]     [Linux Audio Users]     [Linux Media]     [Kernel]     [Gimp]     [Yosemite News]     [Linux Media]

  Powered by Linux