Hello everybody,
After complete re-installation of the machine, I'm back on stage.
Dave, to answer to your question, on gdb, the "bt" shows
following:
(gdb) bt
#0 0xb776a416 in __kernel_vsyscall ()
#1 0xb7698b43 in __write_nocancel () from /lib/libc.so.6
#2 0xb76343b4 in _IO_new_file_write () from /lib/libc.so.6
#3 0xb7635c90 in _IO_new_do_write () from /lib/libc.so.6
#4 0xb7634e80 in _IO_new_file_overflow () from
/lib/libc.so.6
#5 0xb7637d2a in __overflow () from /lib/libc.so.6
#6 0xb76312b5 in putc () from /lib/libc.so.6
#7 0x0809b64b in echo_builtin ()
So your guess was correct, most of the functions are from libc.
Also stated with the 'vm' command.
Performing again the whole process, I think that I now know where
I was wrong.
The error message at the origin of this thread was due to the fact
that I do not use the correct "bash" binary when launching 'gdb'.
I made a mismatch between the "bash" on the crashed system and the
"bash" on the machine where I analyze the crash.
Using the correct "bash" allows to dump the memory and disassemble
the code without problem.
BTW, using the 'vm' and 'vtop' command (Thanks Dave for the tip)
offers interesting information that I will explore later.
For now, I wrote a user-space program that will trigger the page
fault, so that I can more easily make the relationship between
assembly and C language.
To be followed.
Patrick Agrain
Le 06/01/2014 22:08, Dave Anderson a écrit :
as evidenced by the fact that you were able to at least read stack address
bfd1b6d8 from gdb. But it's not clear -- after you walked through the
user stack -- that the dumpfile contains the text page that you're trying
to disassemble. I don't know why it wouldn't be available given that
it's a "hot" code path, and shouldn't be swapped out or otherwise not
mapped.
I'm also presuming that your user-stack walk-through is correct.
When you enter gdb, what does the "bt" command show?
Dave
----- Original Message -----
Hello all,
First of all, I wish a Happy New Year (with less crash, but still enhanced
tools...)
Thanks for the links, they were very useful.
I dig further in the way of analyzing the User Space, but it seems that I'm
linked to a dead-end way.
Below is a snapshot of kernel / userland stack dump.
What I've done :
- Crash is triggered by a page fault inside a kernel module (write 0 in
0xFFFFFFFF, classic).
- Using gcore to create the 'core.<pid>.bash (which is the user task running
at time of crash).
I'm curious as to how the bash task was related to the module crash?
Did the bash task write to a procfs interface that the module created
to then generate the "write 0 to 0xFFFFFFFF"? Does the crash utility
indicate that the bash task is the panic task? And if so, what does
its "bt" show? (i.e., the kernel-mode backtrace)
- Evaluating an EBP (between { }) chaining value (hypothesis), EIP value
(between [ ]) is then just pushed beside
The purpose of this study is to find a method to analyze futur crashes from
kernel space down to user space applications.
Do you have an idea about the cause of this non-dumping of the memory in
user-space ?
Should I use other extension as 'gcore' ?
Thank in advance.
Best regards,
Patrick Agrain
-------
===============================================================================
--------------------- Go down into User Space Territory
-----------------------
Last pt_regs of kernel stack is:
| pt_regs
00000001 094a5408 00000003 ..~......TJ..... | bx cx dx
c2699fc0: 00000003 094a5408 bfd1b704 00000004 .....TJ......... | si di bp ax
c2699fd0: 0000007b ffff007b c07e0000 00000033 {...{.....~.3... | ds es fs gs
c2699fe0: 00000004 b776a416 00000073 00000246 ......v.s...F... | orig_eax ip
cs flags
c2699ff0: bfd1b6d8 0000007b | sp ss
v cccccccc cccccccc ....{........... | padding
|
|----------------------------------------------------------------|
|
(gdb) x/32xw 0xbfd1b680 |
0xbfd1b680: 0xbfd1b6d0 0x0000000f 0x094b4568 0x080c90b9 |
0xbfd1b690: 0x094b4568 0x080cd160 0x00001936 0x00000001 |
0xbfd1b6a0: 0x094ab9c8 0x00000000 0x094b4b48 0xbfd1b7c8 |
0xbfd1b6b0: 0x080ce9e8 0x094b4b48 0x094b4b48 0xbfd1b728 |
0xbfd1b6c0: 0x094aed28 0x00000020 0x00000000 0x00000070 |
0xbfd1b6d0: 0x094b4588 0x080cc080 |
0xb7698b43 <--|
0xb7757ff4
0xbfd1b6e0: 0xb76343b4 0x00000001 0x094a5408 0x00000003
0xbfd1b6f0: 0xb77584e0 0x080cc080 0xbfd1b728 0xb77584e0
|------------------------------------------ Hypothesis : this is an EBP
|value...
v
0xbfd1b700: 0x00000003 {0xbfd1b72c} [0xb7635c90] 0xb77584e0
0xbfd1b710: 0x094a5408 0x00000003 0x094b4b48 0xbfd1b7c8
0xbfd1b720: 0xb7757ff4 0xb77584e0 0x0000000a {0xbfd1b750}
0xbfd1b730: [0xb7634e80] 0xb77584e0 0x094a5408 0x00000003
0xbfd1b740: 0x0000000a 0xb7757ff4 0xb77584e0 0x0000000a
0xbfd1b750: {0xbfd1b768} [0xb7637d2a] 0xb77584e0 0x0000000a
0xbfd1b760: 0xb7757ff4 0xb77584e0 {0xbfd1b788} [0xb76312b5] >-|
0xbfd1b770: 0xb77584e0 0x0000000a 0xb75c9940 0x094a3e48 |
0xbfd1b780: 0x00000001 0x00000000 0x00000000 0x0809b64b |
|
Disassemble Try: EIP@0xb76312b5
<---------------------------------------------|
(gdb) disassemble 0xb7631200, 0xb7631300
Dump of assembler code from 0xb7631200 to 0xb7631300:
0xb7631200: Cannot access memory at address 0xb7631200
(gdb)
----------
Anyway, I'm guessing that the 0xb76312b5 IP address is in some
library, probably libc? If you do a "vm" on the active bash task
from within the crash utility, you will see where it comes from.
Try reading the user-space address from the crash utility to see
if it was available to copy to the core.<pid>.bash file, i.e.,
try this command:
crash> rd -u 0xb76312b5
The command above presumes that you are in the context of the
"bash" task while running crash. (i.e., if you enter "set" alone,
it shows that particular task)
Dave
Le 17/12/2013 19:12, Buland Kumar Singh a écrit :
Hi Patrick,
The following links may also be helpful to understand gdb and
it's usage for application core analysis.
http://web.eecs.umich.edu/~sugih/pointers/gdb_core.html
https://sourceware.org/gdb/onlinedocs/gdb/
-- BKS
On 17 December 2013 21:36, Patrick Agrain < patrick.agrain@xxxxxxxxxxxxxxxxxx
wrote:
Hello all,
Now that we have dumped the kernel stack, I'm intesresting in the user
process from which we came just before the 'panic'.
Googling around, I found mention of the 'gcore' extension.
I compiled version 1.22 and installed it.
Using it on crash 6.1.0-1.el6, I get a file core.845.bash on process 'bash'
(in which I trigger a kernel panic) :
crash> gcore -v 1 845
gcore: Opening file core.845.bash ...
gcore: done.
gcore: Writing ELF header ...
gcore: done.
gcore: Retrieving and writing note information ...
gcore: done.
gcore: Writing PT_NOTE program header ...
gcore: done.
gcore: Writing PT_LOAD program headers ...
gcore: done.
gcore: Writing PT_LOAD segment ...
gcore: PT_LOAD[0]: 8048000 - 8048000
gcore: PT_LOAD[1]: 80e2000 - 80e9000
gcore: PT_LOAD[2]: 80e9000 - 80ed000
gcore: PT_LOAD[3]: 94a2000 - 94d1000
gcore: PT_LOAD[4]: b7374000 - b7374000
gcore: PT_LOAD[5]: b7375000 - b7376000
gcore: PT_LOAD[6]: b7376000 - b7377000
gcore: PT_LOAD[7]: b7377000 - b7377000
gcore: PT_LOAD[8]: b737e000 - b737e000
gcore: PT_LOAD[9]: b737f000 - b737f000
gcore: PT_LOAD[10]: b73bb000 - b73bb000
gcore: PT_LOAD[11]: b75bb000 - b75bb000
gcore: PT_LOAD[12]: b75c7000 - b75c8000
gcore: PT_LOAD[13]: b75c8000 - b75c9000
gcore: PT_LOAD[14]: b75c9000 - b75ca000
gcore: PT_LOAD[15]: b75ca000 - b75ca000
gcore: PT_LOAD[16]: b7756000 - b7758000
gcore: PT_LOAD[17]: b7758000 - b7759000
gcore: PT_LOAD[18]: b7759000 - b775c000
gcore: PT_LOAD[19]: b775c000 - b775c000
gcore: PT_LOAD[20]: b775f000 - b7760000
gcore: PT_LOAD[21]: b7760000 - b7761000
gcore: PT_LOAD[22]: b7761000 - b7761000
gcore: PT_LOAD[23]: b7764000 - b7765000
gcore: PT_LOAD[24]: b7769000 - b776a000
gcore: PT_LOAD[25]: b776a000 - b776b000
gcore: PT_LOAD[26]: b776b000 - b776b000
gcore: PT_LOAD[27]: b7789000 - b778a000
gcore: PT_LOAD[28]: b778a000 - b778b000
gcore: PT_LOAD[29]: bfd07000 - bfd1d000
gcore: done.
Saved core.845.bash
crash>
So far, so good... But
Question: Are there anywhere some hints about how to use this core.<pid> file
?
Thanks in advance.
Regards,
Patrick Agrain
--
Crash-utility mailing list
Crash-utility@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/crash-utility
--
BKS
--
Crash-utility mailing list Crash-utility@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/crash-utility
