----- "Toshikazu Nakayama" <nakayama.ts@xxxxxxxxxxxxxx> wrote:
> Hi,
>
> The crash utility could not get module percpu symbols.
> Since percpu symbols are not included in kallsyms nor module_core area,
> they were not found out in module_init().
>
> However, symbols are existing in module object file
> .data..percpu (legacy kernel revision .data.percpu) section,
> load_module_symbols() can find and resolve them.
> (percpu symbols exist in mod_load_symtable, not in mod_ext_symtable)
>
> 'p' command is also updated to be made redundant legacy per_cpu__
> prefix.
>
> Note:
> I have not tested these paches enough over any arch or kernel revisions,
> also any distros.
> Just have been tested x86 (64bit) with linux-2.6.10 and linux-2.6.35.
>
> This is my first post to crash utility.
> I don't understand well the manner of this mailing list.
> (I'm sorry if there's anything wrong)
>
> Thanks,
> Toshi.
You've come to the right place... ;-)
And your patch submission is fine, although I do prefer that you send the
the patches as email attachments instead of inline. (Although that's my
problem because of the Zimbra email client that I use)
I tested your patch with two kernel types, one of them the "legacy" version
with "pcpu_num_used" and "pcpu_size" symbols (RHEL5), and the second one with
current kernels with the "pcpu_reserved_chunk_limit" symbol (RHEL6 and 2.6.37-rc1).
Here's what I see, first without, and then with your patch applied.
(1) Testing with a 2.6.18-based RHEL5 kernel:
Without your patch, the stored __per_cpu_start and __per_cpu_end
values are as shown here in the symbol_table_data structure:
crash> help -s
...
__per_cpu_start: ffffffff80419000
__per_cpu_end: ffffffff8041f508
...
which reflect the beginning and end of the __per_cpu kernel
symbol range:
crash> sym -l
...
ffffffff80419000 (A) __per_cpu_start
ffffffff80419000 (D) per_cpu__init_tss
ffffffff8041b080 (D) per_cpu__orig_ist
ffffffff8041b100 (d) per_cpu__idle_state
...
ffffffff8041f480 (d) per_cpu__flow_flush_tasklets
ffffffff8041f4c0 (d) per_cpu__rt_cache_stat
ffffffff8041f500 (d) per_cpu____icmp_socket
ffffffff8041f508 (A) __per_cpu_end
With your patch applied, the __per_cpu_end gets incremented based
upon the "pcpu_num_used" and "pcpu_size" contents:
crash> help -s
...
__per_cpu_start: ffffffff80419000
__per_cpu_end: ffffffff80427a10
...
But that doesn't seem right, because the resultant value
pushes it into the bss symbol range:
crash> sym ffffffff80427a10
ffffffff80427a10 (B) boot_exception_stacks+6672
crash>
crash> sym -l
...
ffffffff8041f4c0 (d) per_cpu__rt_cache_stat
ffffffff8041f500 (d) per_cpu____icmp_socket
ffffffff8041f508 (A) __per_cpu_end <-- original value
ffffffff80420000 (d) .data_nosave
ffffffff80420000 (A) __nosave_begin
ffffffff80420000 (D) in_suspend
ffffffff80421000 (b) .bss
ffffffff80421000 (A) __bss_start
ffffffff80421000 (A) __nosave_end
ffffffff80421000 (B) empty_zero_page
ffffffff80422000 (B) boot_cpu_stack
ffffffff80426000 (B) boot_exception_stacks <-- bumped to ffffffff80427a10
ffffffff8042c000 (B) idt_table
ffffffff8042d000 (B) boot_delay
ffffffff8042d008 (B) printk_delay_msec
...
(2) Testing with a 2.6.32-based RHEL6 kernel:
Without your patch, the stored __per_cpu_start and __per_cpu_end
values are as shown here in the symbol_table_data structure:
crash> help -s
...
__per_cpu_start: 0
__per_cpu_end: 17358
...
which reflect the beginning and end of the kernel's __per_cpu
symbol range:
crash> sym -l
0 (D) __per_cpu_start
0 (V) per_cpu__irq_stack_union
4000 (V) per_cpu__gdt_page
5000 (V) per_cpu__exception_stacks
...
16940 (V) per_cpu__cpu_tlbstate
16980 (V) per_cpu__runqueues
17340 (V) per_cpu__sched_clock_data
17358 (D) __per_cpu_end
ffffffff81000000 (T) _text
...
With your patch applied, the st->__per_cpu_end value gets changed
like so:
crash> help -s
...
__per_cpu_start: 0
__per_cpu_end: ffffffff81bf36b8
...
But that doesn't seem right, where ffffffff81bf36b8 is the
"pcpu_reserved_chunk_limit" symbol value:
crash> sym ffffffff81bf36b8
ffffffff81bf36b8 (b) pcpu_reserved_chunk_limit
crash>
I'm guessing that you want it st->__per_cpu_end to be bumped from
0x17358 to 0x19358:
crash> px pcpu_reserved_chunk_limit
pcpu_reserved_chunk_limit = $4 = 0x19358
crash>
(3) Testing with a generic upstream 2.6.37-rc1 kernel, I see the same
behaviour as with the RHEL6 kernel.
In any case, there's no way I can change the st->__per_cpu_end values
as you propose -- even if it somehow works with your testing.
Comments?
Dave
> -----------------------------------------------------------------------------
> CHANGES: module percpu symbols are resolved while module object loading.
> cmd_p() can read percpu syombols without prefix in legacy environs.
>
> SRPM: crash-5.0.9-0.src.rpm
>
> TEST#1: Result on linux-2.6.10 x86(64) SMP(4 CPUs) * with test module
>
> [before]
>
> crash> sym -m test | grep abc
> crash> p abc
> p: gdb request failed: p abc
>
> [after]
>
> crash> sym -m test | grep abc
> ffffffff80871928 (d) per_cpu__abc
> crash> sym abc
> symbol not found: abc
> possible alternatives:
> ffffffff80871928 (d) per_cpu__abc
> crash> p abc
> PER-CPU DATA TYPE:
> int per_cpu__abc;
> PER-CPU ADDRESSES:
> [0]: 10008af0888
> [1]: 10008b00888
> [2]: 10008b10888
> [3]: 10008b20888
>
> TEST#2: Result on linux-2.6.35 x86(64) SMP (8 CPUs) * with KVM
> modules
>
> crash> mod | grep kvm
> ffffffffa0035c40 kvm_intel 39368
> /lib/modules/2.6.35/kernel/arch/x86/kvm/kvm-intel.ko
> ffffffffa008dba0 kvm 175160
> /lib/modules/2.6.35/kernel/arch/x86/kvm/kvm.ko
>
> [before]
>
> crash> sym -m kvm_intel | grep vcpus_on_cpu
> crash> p vcpus_on_cpu
> p: gdb request failed: p vcpus_on_cpu
>
> [after]
>
> crash> sym -m kvm_intel | grep vcpus_on_cpu
> 12a40 (d) vcpus_on_cpu
> crash> p vcpus_on_cpu
> PER-CPU DATA TYPE:
> struct list_head vcpus_on_cpu;
> PER-CPU ADDRESSES:
> [0]: ffff880001812a40
> [1]: ffff880001852a40
> [2]: ffff880001892a40
> [3]: ffff8800018d2a40
> [4]: ffff880001912a40
> [5]: ffff880001952a40
> [6]: ffff880001992a40
> [7]: ffff8800019d2a40
>
> Toshikazu Nakayama (4):
> Add percpu member.
> module's percpu basic procedure.
> expand percpu area.
> update cmd_p().
>
> defs.h | 2 ++
> kernel.c | 1 +
> symbols.c | 50 +++++++++++++++++++++++++++++++++++++++++++++-----
> 3 files changed, 48 insertions(+), 5 deletions(-)
>
> --
> 1.7.3.2.161.g3089c
--
Crash-utility mailing list
Crash-utility@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/crash-utility
