Hello Pingfan,
Thank you for the v2 update.
On 8/2/2018 11:16 PM, Pingfan Liu wrote:
> When refiltering on kdump format file, there is no info about pt_load[] for
> exclude_nodata_pages(), and also we can not expect more data than the kdump
> file can provide, hence this patch suggests to initialize the refiltered
> bitmap2 from the kdump file's bitmap2. As for the statics original_pfn, it
> should also be calculated based on kdump file's bitmap2.
As for statistics, since we now count up "Excluded pages" even if
the page is already excluded (the bit is already off in bitmap2), so
"Remaining pages" (= pfn_original - pfn_excluded) becomes incorrect
with the patch.
[first filtering]
# makedumpfile --message-level 22 -l -d 31 /proc/vmcore dump.ld31
...
Original pages : 0x000000000007311b
Excluded pages : 0x00000000000694ed
Pages filled with zero : 0x0000000000001442
Non-private cache pages : 0x00000000000058ea
Private cache pages : 0x0000000000000000
User process data pages : 0x000000000000031f
Free pages : 0x00000000000624a2
Hwpoison pages : 0x0000000000000000
Remaining pages : 0x0000000000009c2e
(The number of pages is reduced to 8%.)
Memory Hole : 0x000000000000cee5
--------------------------------------------------
Total pages : 0x0000000000080000
[re-filtering]
# makedumpfile --message-level 22 -l -d 31 dump.ld31 dump.ld31.ld31
...
Original pages : 0x000000000000f9e3
Excluded pages : 0x00000000000694ed <<-- same with the above
Pages filled with zero : 0x0000000000001442
Non-private cache pages : 0x00000000000058ea
Private cache pages : 0x0000000000000000
User process data pages : 0x000000000000031f
Free pages : 0x00000000000624a2
Hwpoison pages : 0x0000000000000000
Remaining pages : 0xfffffffffffa64f6 <<-- incorrect
(The number of pages is reduced to 288361039747273%.) <<--
Memory Hole : 0x0000000000080000
--------------------------------------------------
Total pages : 0x0000000000080000
So how about counting memory hole with the kdump file's bitmap1
instead of the bitmap2? i.e. the report shows statistics between
the original vmcore and the re-filtered kdump file.
My comments on this suggestion:
>
> Note about the bug reported by the following ops:
> makedumpfile -l --message-level 1 -d 31 /proc/vmcore /path/to/vmcore
> makedumpfile --split -d 31 ./vmcore dumpfile_{1,2,3} 2>&1
> And get the following error:
> Excluding unnecessary pages : [100.0 %] \
> readpage_kdump_compressed: pfn(9b) is excluded from /var/crash/127.0.0.1-2018-07-02-22:10:38/vmcore.
> readmem: type_addr: 1, addr:9b000, size:4096
> read_pfn: Can't get the page data.
> writeout_multiple_dumpfiles: Child process(2277) finished incompletely.(256)
> Copying data : [ 24.6 %] - eta: 2s
> makedumpfile Failed.
>
> Cc: Kazuhito Hagio <k-hagio@xxxxxxxxxxxxx>
> Signed-off-by: Pingfan Liu <piliu@xxxxxxxxxx>
> ---
> v1 -> v2
> initialized refiltered bitmap2 from file's bitmap2
> improve statics of original_pfn
>
> makedumpfile.c | 54 +++++++++++++++++++++++++++++++++++++++++++++---------
> 1 file changed, 45 insertions(+), 9 deletions(-)
>
> diff --git a/makedumpfile.c b/makedumpfile.c
> index 915cbf4..bf2f806 100644
> --- a/makedumpfile.c
> +++ b/makedumpfile.c
> @@ -6090,26 +6090,59 @@ copy_bitmap_buffer(void)
> return TRUE;
> }
>
> +static mdf_pfn_t count_bits(unsigned char *buf, int sz)
> +{
> + unsigned char *p = buf;
> + int i, j;
> + mdf_pfn_t cnt = 0;
> + for (i = 0; i < sz; i++, p++) {
> + if (*p == 0)
> + continue;
> + else if (*p == 0xff)
> + cnt += 8;
should have a continue?
> + for (j = 0; j < 8; j++) {
> + if (*p & 1<<j)
> + cnt++;
> + }
> + }
> + return cnt;
> +}
> +
> +static mdf_pfn_t orig_pfn = 0;
If counting memory hole with bitmap1, orig_pfn is not needed.
> +
> int
> copy_bitmap_file(void)
> {
> - off_t offset;
> + off_t base, offset = 0;
> unsigned char buf[info->page_size];
> const off_t failed = (off_t)-1;
> + int fd;
> + struct disk_dump_header *dh = info->dh_memory;
>
> - offset = 0;
> + if (info->flag_refiltering) {
> + fd = info->fd_memory;
> + base = info->len_bitmap / 2;
> + base += (DISKDUMP_HEADER_BLOCKS + dh->sub_hdr_size) * dh->block_size;
> + orig_pfn = 0;
> + } else {
> + fd = info->bitmap1->fd;
> + base = info->bitmap1->offset;
> + }
> while (offset < (info->len_bitmap / 2)) {
> - if (lseek(info->bitmap1->fd, info->bitmap1->offset + offset,
> - SEEK_SET) == failed) {
> + if (lseek(fd, base + offset, SEEK_SET) == failed) {
> ERRMSG("Can't seek the bitmap(%s). %s\n",
> info->name_bitmap, strerror(errno));
> return FALSE;
> }
> - if (read(info->bitmap1->fd, buf, sizeof(buf)) != sizeof(buf)) {
> + if (read(fd, buf, sizeof(buf)) != sizeof(buf)) {
> ERRMSG("Can't read the dump memory(%s). %s\n",
> info->name_memory, strerror(errno));
> return FALSE;
> }
> + /* counting the original pfn */
> + if (info->flag_refiltering) {
> + orig_pfn += count_bits(buf, sizeof(buf));
> + }
Shall we move this to copy_1st_bitmap_from_memory() and
do the following?
pfn_memhole -= count_bits(buf, sizeof(buf));
> if (lseek(info->bitmap2->fd, info->bitmap2->offset + offset,
> SEEK_SET) == failed) {
> ERRMSG("Can't seek the bitmap(%s). %s\n",
> @@ -9713,10 +9746,13 @@ print_report(void)
> {
> mdf_pfn_t pfn_original, pfn_excluded, shrinking;
>
> - /*
> - * /proc/vmcore doesn't contain the memory hole area.
> - */
> - pfn_original = info->max_mapnr - pfn_memhole;
> + if (info->flag_refiltering)
> + pfn_original = orig_pfn;
> + else
> + /*
> + * /proc/vmcore doesn't contain the memory hole area.
> + */
> + pfn_original = info->max_mapnr - pfn_memhole;
>
> pfn_excluded = pfn_zero + pfn_cache + pfn_cache_private
> + pfn_user + pfn_free + pfn_hwpoison;
>
Thanks,
Kazu
_______________________________________________
kexec mailing list
kexec@xxxxxxxxxxxxxxxxxxx
http://lists.infradead.org/mailman/listinfo/kexec
