----- Original Message -----
> Hi Dave,
>
> Thanks again for the response.
>
> When I check the address of tty->read_buf using kmem command in crash,
> below is the output. Which shows that the memory is still not freed. If
> the tty_close handler had been invoked, before accessing tty->read_buf
> we should not get this output. Please correct me if I am erring.
>
> kmem 0xffff8802cbfe6000
> CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE
> ffff88031f8039c0 size-4096 4096 6322 6393 6393 4k
> SLAB MEMORY TOTAL ALLOCATED FREE
> ffff8802ac4811c0 ffff8802cbfe6000 1 1 0
> FREE / [ALLOCATED]
> [ffff8802cbfe6000]
>
> PAGE PHYSICAL MAPPING INDEX CNT FLAGS
> ffffea0009c9fa50 2cbfe6000 0 0 1 200000000000080
>
>
> I would like to thank every one for your time and effort in analyzing
> this issue.
>
> Thanks and Regards
> Shashidhara
That particular 4k buffer was definitely allocated at the point
in time when the crash occurred.
However, it doesn't make logical sense because the code path saw
a non-zero read_cnt and a NULL read_buf pointer. So if by some
chance there were an illegal intervening n_tty_close() call, it
would have had to have happened:
(1) after tty->read_cnt was read, and
(2) before the "c = " assignment (because read_buf was NULL), and
(3) then would have to have been re-opened, reallocated the buffer
that's being seen in the dump.
while (nr && tty->read_cnt) {
int eol;
eol = test_and_clear_bit(tty->read_tail,
tty->read_flags);
c = tty->read_buf[tty->read_tail];
spin_lock_irqsave(&tty->read_lock, flags);
So it just seems impossible to recreate even with an intervening
n_tty_close().
Dave
Dave
>
> -----Original Message-----
> From: crash-utility-bounces@xxxxxxxxxx
> [mailto:crash-utility-bounces@xxxxxxxxxx] On Behalf Of Dave Anderson
> Sent: Friday, June 24, 2011 7:10 PM
> To: Discussion list for crash utility usage,maintenance and
> development
> Subject: Re: Unable to switch stack frames while using
> crash
>
>
> ---- Original Message -----
> >>
> >> BTW, are you sure about that?
> >>
> >> Presuming that the "tty" pointer is ffff8802cbd54800 as you've
> >> shown
> below,
> >> and therefore tty->read_buf is 0xffff8802cbfe6000 and
> >> tty->read_tail
> is 0,
> >> then the statement above would be simply be reading
> >> tty->read_buf[0],
> or
> >> virtual address 0xffff8802cbfe6000. But the oops shows it faulting
> on a
> >> virtual address of "5":
>
> Well, as it turns out, you have every reason to be sure about that...
>
> Anyway, I don't understand why line numbers are not available with
> your kernel:
>
> > crash> dis -rl n_tty_read+0x58c
> > dis: line numbers are not available
> ... [ cut ] ...
> > 0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
> > 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
> > 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
> > 0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
> > 0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx
>
> But nonetheless, there is only on movsbl instruction in n_tty_read(),
> and looking at a RHEL6 kernel, you were correct in your original
> determination of the faulting instruction:
>
> crash> dis n_tty_read | grep movsbl
> 0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx
> crash> dis -rl 0xffffffff812f88c9 | tail
> ... [ cut ] ...
> /usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/drivers/char/n_tty.c:
> 1821
> 0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx
> crash>
>
> which is here:
>
> 1814 if (tty->icanon) {
> 1815 /* N.B. avoid overrun if nr == 0 */
> 1816 while (nr && tty->read_cnt) {
> 1817 int eol;
> 1818
> 1819 eol =
> test_and_clear_bit(tty->read_tail,
> 1820
> tty->read_flags);
> 1821 c =
> tty->read_buf[tty->read_tail];
>
> The tty_struct offsets are these:
>
> crash> tty_struct -o
> struct tty_struct {
> ... [ cut ]...
> [0x250] char *read_buf;
> [0x258] int read_head;
> [0x25c] int read_tail;
> ...
>
> And you can see in the previous instructions the tty->read_buf (0x250)
> and tty->read_tail (0x25c) offsets being added to the tty_struct
> base address that is located in R13:
>
> > 0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
> > 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
> > 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
> > 0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
> > 0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx
>
> And as you originally reported, the tty_struct address in R13
> is ffff8802cbd54800:
>
> PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
> #0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
> #1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
> #2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
> #3 [ffff88031ce75b50] no_context at ffffffff8102d801
> #4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
> #5 [ffff88031ce75c70] bad_area at ffffffff8102da41
> #6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
> #7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
> [exception RIP: n_tty_read+1420]
> RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
> RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
> RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
> RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
> R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
> R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> #8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
> #9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
> #10 [ffff88031ce75f40] sys_read at ffffffff810ebe48
> #11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
> RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
> RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
> RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
> RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
> R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
> R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
> ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
>
> But for whatever reason -- and I cannot explain it -- after these
> instructions were done:
>
> > 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
> > 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
>
> the RDX register ended up with 0000000000000005, and the RAX register
> with
> a 0000000000000000, leading to the:
>
> BUG: unable to handle kernel NULL pointer dereference at
> 0000000000000005
>
> But when you display the tty_struct at ffff8802cbd54800, you see the
> read_buf and read_tail with seemingly legitimate values:
>
> crash> tty_struct ffff8802cbd54800
> struct tty_struct { ...
> magic = 21505,
> driver = 0xffff88031b54ea00,
> ops = 0xffffffff8130f650,
> name = "pts9\000\...",
> driver_data = 0xffff88029c8a9668,
> icanon = 1 '\001',
> read_buf = 0xffff8802cbfe6000 "",
> read_head = 0,
> read_tail = 0,
> read_cnt = 0,
> ...
>
> So everything in your analysis was correct, but how it is possible
> that the RDX and RAX registers to have ended up with 0 and 5 is hard
> to explain. And for that matter, since tty->read_cnt is 0 above,
> your original question as to how that code path was taken to
> begin with is also valid.
>
> So I don't know -- anybody on the list ever seen anything like this?
>
> Stumped,
> Dave
>
>
> ----- Original Message -----
> > Hi Dave,
> >
> > Thank you so much for your help.
> >
> > Below is the output of dis -rl n_tty_read+0x58c
> >
> > crash> dis -rl n_tty_read+0x58c
> > dis: line numbers are not available
> > 0xffffffff811efe27 <n_tty_read>: push %rbp
> > 0xffffffff811efe28 <n_tty_read+1>: mov %gs:0xb500,%rax
> > 0xffffffff811efe31 <n_tty_read+10>: mov %rsp,%rbp
> > 0xffffffff811efe34 <n_tty_read+13>: push %r15
> > 0xffffffff811efe36 <n_tty_read+15>: push %r14
> > 0xffffffff811efe38 <n_tty_read+17>: push %r13
> > 0xffffffff811efe3a <n_tty_read+19>: mov %rdi,%r13
> > 0xffffffff811efe3d <n_tty_read+22>: lea -0x70(%rbp),%rdi
> > 0xffffffff811efe41 <n_tty_read+26>: push %r12
> > 0xffffffff811efe43 <n_tty_read+28>: push %rbx
> > 0xffffffff811efe44 <n_tty_read+29>: lea 0x490(%r13),%rbx
> > 0xffffffff811efe4b <n_tty_read+36>: sub $0xe8,%rsp
> > 0xffffffff811efe52 <n_tty_read+43>: mov %rax,-0x98(%rbp)
> > 0xffffffff811efe59 <n_tty_read+50>: mov %rcx,-0x78(%rbp)
> > 0xffffffff811efe5d <n_tty_read+54>: xor %eax,%eax
> > 0xffffffff811efe5f <n_tty_read+56>: mov $0xa,%ecx
> > 0xffffffff811efe64 <n_tty_read+61>: mov %rdx,-0xd8(%rbp)
> > 0xffffffff811efe6b <n_tty_read+68>: mov %rsi,-0xd0(%rbp)
> > 0xffffffff811efe72 <n_tty_read+75>: mov %rdx,-0x40(%rbp)
> > 0xffffffff811efe76 <n_tty_read+79>: rep stos %eax,%es:(%rdi)
> > 0xffffffff811efe78 <n_tty_read+81>: lea 0x1c0(%r13),%rax
> > 0xffffffff811efe7f <n_tty_read+88>: lea 0x1c8(%r13),%rcx
> > 0xffffffff811efe86 <n_tty_read+95>: mov %rbx,-0xc0(%rbp)
> > 0xffffffff811efe8d <n_tty_read+102>: lea 0xd8(%r13),%rbx
> > 0xffffffff811efe94 <n_tty_read+109>: movq
> > $0xffffffff81045f84,-0x60(%rbp)
> > 0xffffffff811efe9c <n_tty_read+117>: movq $0x0,-0xa8(%rbp)
> > 0xffffffff811efea7 <n_tty_read+128>: mov -0x98(%rbp),%rdx
> > 0xffffffff811efeae <n_tty_read+135>: mov %rax,-0xc8(%rbp)
> > 0xffffffff811efeb5 <n_tty_read+142>: mov -0x98(%rbp),%rax
> > 0xffffffff811efebc <n_tty_read+149>: mov %rcx,-0x90(%rbp)
> > 0xffffffff811efec3 <n_tty_read+156>: lea 0x51c(%r13),%rcx
> > 0xffffffff811efeca <n_tty_read+163>: mov %rbx,-0x80(%rbp)
> > 0xffffffff811efece <n_tty_read+167>: mov %rdx,-0x68(%rbp)
> > 0xffffffff811efed2 <n_tty_read+171>: lea 0x268(%r13),%rdx
> > 0xffffffff811efed9 <n_tty_read+178>: mov %rcx,-0xb8(%rbp)
> > 0xffffffff811efee0 <n_tty_read+185>: mov %rax,-0xf8(%rbp)
> > 0xffffffff811efee7 <n_tty_read+192>: mov %rax,-0x100(%rbp)
> > 0xffffffff811efeee <n_tty_read+199>: mov %rdx,-0x88(%rbp)
> > 0xffffffff811efef5 <n_tty_read+206>: mov %rax,-0x108(%rbp)
> > 0xffffffff811efefc <n_tty_read+213>: mov %rax,-0x110(%rbp)
> > 0xffffffff811eff03 <n_tty_read+220>: cmpq $0x0,0x250(%r13)
> > 0xffffffff811eff0b <n_tty_read+228>: jne 0xffffffff811eff11
> > <n_tty_read+234>
> > 0xffffffff811eff0d <n_tty_read+230>: ud2a
> > 0xffffffff811eff0f <n_tty_read+232>: jmp 0xffffffff811eff0f
> > <n_tty_read+232>
> > 0xffffffff811eff11 <n_tty_read+234>: mov -0xd0(%rbp),%rdx
> > 0xffffffff811eff18 <n_tty_read+241>: mov 0x20(%rdx),%rax
> > 0xffffffff811eff1c <n_tty_read+245>: cmpq
> > $0xffffffff811ed61f,0x18(%rax)
> > 0xffffffff811eff24 <n_tty_read+253>: je 0xffffffff811effef
> > <n_tty_read+456>
> > 0xffffffff811eff2a <n_tty_read+259>: mov -0xf8(%rbp),%rcx
> > 0xffffffff811eff31 <n_tty_read+266>: mov 0x478(%rcx),%rax
> > 0xffffffff811eff38 <n_tty_read+273>: cmp %r13,0x180(%rax)
> > 0xffffffff811eff3f <n_tty_read+280>: jne 0xffffffff811effef
> > <n_tty_read+456>
> > 0xffffffff811eff45 <n_tty_read+286>: mov 0xc8(%r13),%rdx
> > 0xffffffff811eff4c <n_tty_read+293>: test %rdx,%rdx
> > 0xffffffff811eff4f <n_tty_read+296>: jne 0xffffffff811eff64
> > <n_tty_read+317>
> > 0xffffffff811eff51 <n_tty_read+298>: mov $0xffffffff8139c972,%rdi
> > 0xffffffff811eff58 <n_tty_read+305>: xor %eax,%eax
> > 0xffffffff811eff5a <n_tty_read+307>: callq 0xffffffff812d4abf
> > <printk>
> > 0xffffffff811eff5f <n_tty_read+312>: jmpq 0xffffffff811effef
> > <n_tty_read+456>
> > 0xffffffff811eff64 <n_tty_read+317>: mov -0xf8(%rbp),%rbx
> > 0xffffffff811eff6b <n_tty_read+324>: mov 0x1e0(%rbx),%rax
> > 0xffffffff811eff72 <n_tty_read+331>: cmp %rdx,0x238(%rax)
> > 0xffffffff811eff79 <n_tty_read+338>: je 0xffffffff811effef
> > <n_tty_read+456>
> > 0xffffffff811eff7b <n_tty_read+340>: mov -0x98(%rbp),%rax
> > 0xffffffff811eff82 <n_tty_read+347>: testb $0x10,0x48a(%rax)
> > 0xffffffff811eff89 <n_tty_read+354>: jne 0xffffffff811f0611
> > <n_tty_read+2026>
> > 0xffffffff811eff8f <n_tty_read+360>: mov 0x480(%rax),%rax
> > 0xffffffff811eff96 <n_tty_read+367>: cmpq $0x1,0x288(%rax)
> > 0xffffffff811eff9e <n_tty_read+375>: jne 0xffffffff811f0604
> > <n_tty_read+2013>
> > 0xffffffff811effa4 <n_tty_read+381>: jmpq 0xffffffff811f0611
> > <n_tty_read+2026>
> > 0xffffffff811effa9 <n_tty_read+386>: mov -0x98(%rbp),%rcx
> > 0xffffffff811effb0 <n_tty_read+393>: mov $0x1,%edx
> > 0xffffffff811effb5 <n_tty_read+398>: mov $0x15,%esi
> > 0xffffffff811effba <n_tty_read+403>: mov 0x1e0(%rcx),%rax
> > 0xffffffff811effc1 <n_tty_read+410>: mov 0x238(%rax),%rdi
> > 0xffffffff811effc8 <n_tty_read+417>: callq 0xffffffff8105953a
> > <kill_pgrp>
> > 0xffffffff811effcd <n_tty_read+422>: mov %gs:0xb508,%rdx
> > 0xffffffff811effd6 <n_tty_read+431>: lea -0x1fc8(%rdx),%rax
> > 0xffffffff811effdd <n_tty_read+438>: lock orb $0x4,-0x1fc8(%rdx)
> > 0xffffffff811effe5 <n_tty_read+446>: mov $0xfffffe00,%eax
> > 0xffffffff811effea <n_tty_read+451>: jmpq 0xffffffff811f0616
> > <n_tty_read+2031>
> > 0xffffffff811effef <n_tty_read+456>: testb $0x10,0x21c(%r13)
> > 0xffffffff811efff7 <n_tty_read+464>: je 0xffffffff811f000f
> > <n_tty_read+488>
> > 0xffffffff811efff9 <n_tty_read+466>: movl $0x0,-0xb0(%rbp)
> > 0xffffffff811f0003 <n_tty_read+476>: movl $0x0,-0xac(%rbp)
> > 0xffffffff811f000d <n_tty_read+486>: jmp 0xffffffff811f0076
> > <n_tty_read+591>
> > 0xffffffff811f000f <n_tty_read+488>: mov 0x70(%r13),%rdx
> > 0xffffffff811f0013 <n_tty_read+492>: movzbl 0x16(%rdx),%eax
> > 0xffffffff811f0017 <n_tty_read+496>: imul $0x19,%eax,%eax
> > 0xffffffff811f001a <n_tty_read+499>: mov %eax,-0xac(%rbp)
> > 0xffffffff811f0020 <n_tty_read+505>: movzbl 0x17(%rdx),%edx
> > 0xffffffff811f0024 <n_tty_read+509>: test %edx,%edx
> > 0xffffffff811f0026 <n_tty_read+511>: mov %edx,-0xb0(%rbp)
> > 0xffffffff811f002c <n_tty_read+517>: je 0xffffffff811f0082
> > <n_tty_read+603>
> > 0xffffffff811f002e <n_tty_read+519>: test %eax,%eax
> > 0xffffffff811f0030 <n_tty_read+521>: je 0xffffffff811f003e
> > <n_tty_read+535>
> > 0xffffffff811f0032 <n_tty_read+523>: movw $0x1,0x21e(%r13)
> > 0xffffffff811f003c <n_tty_read+533>: jmp 0xffffffff811f0076
> > <n_tty_read+591>
> > 0xffffffff811f003e <n_tty_read+535>: mov -0x90(%rbp),%rbx
> > 0xffffffff811f0045 <n_tty_read+542>: cmp %rbx,0x1c8(%r13)
> > 0xffffffff811f004c <n_tty_read+549>: je 0xffffffff811f0068
> > <n_tty_read+577>
> > 0xffffffff811f004e <n_tty_read+551>: movzwl 0x21e(%r13),%eax
> > 0xffffffff811f0056 <n_tty_read+559>: mov $0x7fffffffffffffff,%r15
> > 0xffffffff811f0060 <n_tty_read+569>: cmp -0xb0(%rbp),%eax
> > 0xffffffff811f0066 <n_tty_read+575>: jle 0xffffffff811f00b7
> > <n_tty_read+656>
> > 0xffffffff811f0068 <n_tty_read+577>: mov -0xb0(%rbp),%eax
> > 0xffffffff811f006e <n_tty_read+583>: mov %ax,0x21e(%r13)
> > 0xffffffff811f0076 <n_tty_read+591>: mov $0x7fffffffffffffff,%r15
> > 0xffffffff811f0080 <n_tty_read+601>: jmp 0xffffffff811f00b7
> > <n_tty_read+656>
> > 0xffffffff811f0082 <n_tty_read+603>: movslq -0xac(%rbp),%r15
> > 0xffffffff811f0089 <n_tty_read+610>: cmpl $0x0,-0xac(%rbp)
> > 0xffffffff811f0090 <n_tty_read+617>: mov $0x0,%eax
> > 0xffffffff811f0095 <n_tty_read+622>: movw $0x1,0x21e(%r13)
> > 0xffffffff811f009f <n_tty_read+632>: movl $0x1,-0xb0(%rbp)
> > 0xffffffff811f00a9 <n_tty_read+642>: movl $0x0,-0xac(%rbp)
> > 0xffffffff811f00b3 <n_tty_read+652>: cmove %rax,%r15
> > 0xffffffff811f00b7 <n_tty_read+656>: mov -0xd0(%rbp),%rdx
> > 0xffffffff811f00be <n_tty_read+663>: testb $0x8,0x39(%rdx)
> > 0xffffffff811f00c2 <n_tty_read+667>: je 0xffffffff811f00e4
> > <n_tty_read+701>
> > 0xffffffff811f00c4 <n_tty_read+669>: mov -0xc0(%rbp),%rdi
> > 0xffffffff811f00cb <n_tty_read+676>: callq 0xffffffff812d5ec7
> > <mutex_trylock>
> > 0xffffffff811f00d0 <n_tty_read+681>: test %eax,%eax
> > 0xffffffff811f00d2 <n_tty_read+683>: jne 0xffffffff811f0104
> > <n_tty_read+733>
> > 0xffffffff811f00d4 <n_tty_read+685>: movq
> > $0xfffffffffffffff5,-0xa8(%rbp)
> > 0xffffffff811f00df <n_tty_read+696>: jmpq 0xffffffff811f05eb
> > <n_tty_read+1988>
> > 0xffffffff811f00e4 <n_tty_read+701>: mov -0xc0(%rbp),%rdi
> > 0xffffffff811f00eb <n_tty_read+708>: callq 0xffffffff812d6358
> > <mutex_lock_interruptible>
> > 0xffffffff811f00f0 <n_tty_read+713>: test %eax,%eax
> > 0xffffffff811f00f2 <n_tty_read+715>: je 0xffffffff811f0104
> > <n_tty_read+733>
> > 0xffffffff811f00f4 <n_tty_read+717>: movq
> > $0xfffffffffffffe00,-0xa8(%rbp)
> > 0xffffffff811f00ff <n_tty_read+728>: jmpq 0xffffffff811f05eb
> > <n_tty_read+1988>
> > 0xffffffff811f0104 <n_tty_read+733>: mov 0xec(%r13),%al
> > 0xffffffff811f010b <n_tty_read+740>: mov -0xc8(%rbp),%rdi
> > 0xffffffff811f0112 <n_tty_read+747>: lea -0x70(%rbp),%rsi
> > 0xffffffff811f0116 <n_tty_read+751>: shr $0x3,%al
> > 0xffffffff811f0119 <n_tty_read+754>: mov %eax,%ecx
> > 0xffffffff811f011b <n_tty_read+756>: and $0x1,%ecx
> > 0xffffffff811f011e <n_tty_read+759>: mov %ecx,-0x9c(%rbp)
> > 0xffffffff811f0124 <n_tty_read+765>: callq 0xffffffff8106201b
> > <add_wait_queue>
> > 0xffffffff811f0129 <n_tty_read+770>: movslq -0xb0(%rbp),%rbx
> > 0xffffffff811f0130 <n_tty_read+777>: movslq -0xac(%rbp),%rax
> > 0xffffffff811f0137 <n_tty_read+784>: mov -0xd8(%rbp),%rdx
> > 0xffffffff811f013e <n_tty_read+791>: inc %rdx
> > 0xffffffff811f0141 <n_tty_read+794>: mov %rbx,-0xe0(%rbp)
> > 0xffffffff811f0148 <n_tty_read+801>: mov %rax,-0xe8(%rbp)
> > 0xffffffff811f014f <n_tty_read+808>: mov %rdx,-0xf0(%rbp)
> > 0xffffffff811f0156 <n_tty_read+815>: jmpq 0xffffffff811f0522
> > <n_tty_read+1787>
> > 0xffffffff811f015b <n_tty_read+820>: cmpl $0x0,-0x9c(%rbp)
> > 0xffffffff811f0162 <n_tty_read+827>: je 0xffffffff811f01ef
> > <n_tty_read+968>
> > 0xffffffff811f0168 <n_tty_read+833>: mov 0xf8(%r13),%rax
> > 0xffffffff811f016f <n_tty_read+840>: cmpb $0x0,0xed(%rax)
> > 0xffffffff811f0176 <n_tty_read+847>: je 0xffffffff811f01ef
> > <n_tty_read+968>
> > 0xffffffff811f0178 <n_tty_read+849>: mov -0xd8(%rbp),%rcx
> > 0xffffffff811f017f <n_tty_read+856>: cmp %rcx,-0x40(%rbp)
> > 0xffffffff811f0183 <n_tty_read+860>: jne 0xffffffff811f052d
> > <n_tty_read+1798>
> > 0xffffffff811f0189 <n_tty_read+866>: lea 0x68(%rax),%rdi
> > 0xffffffff811f018d <n_tty_read+870>: callq 0xffffffff812d6fb8
> > <_spin_lock_irqsave>
> > 0xffffffff811f0192 <n_tty_read+875>: mov 0xf8(%r13),%rdi
> > 0xffffffff811f0199 <n_tty_read+882>: mov %rax,%rsi
> > 0xffffffff811f019c <n_tty_read+885>: mov 0xed(%rdi),%bl
> > 0xffffffff811f01a2 <n_tty_read+891>: movb $0x0,0xed(%rdi)
> > 0xffffffff811f01a9 <n_tty_read+898>: add $0x68,%rdi
> > 0xffffffff811f01ad <n_tty_read+902>: callq 0xffffffff812d70c1
> > <_spin_unlock_irqrestore>
> > 0xffffffff811f01b2 <n_tty_read+907>: mov -0x40(%rbp),%r12
> > 0xffffffff811f01b6 <n_tty_read+911>: lea -0x31(%rbp),%rsi
> > 0xffffffff811f01ba <n_tty_read+915>: mov $0x1,%edx
> > 0xffffffff811f01bf <n_tty_read+920>: mov %r13,%rdi
> > 0xffffffff811f01c2 <n_tty_read+923>: mov %bl,-0x31(%rbp)
> > 0xffffffff811f01c5 <n_tty_read+926>: lea 0x1(%r12),%rax
> > 0xffffffff811f01ca <n_tty_read+931>: mov %rax,-0x40(%rbp)
> > 0xffffffff811f01ce <n_tty_read+935>: callq 0xffffffff812008ac
> > <tty_audit_add_data>
> > 0xffffffff811f01d3 <n_tty_read+940>: mov -0x31(%rbp),%al
> > 0xffffffff811f01d6 <n_tty_read+943>: mov %r12,%rcx
> > 0xffffffff811f01d9 <n_tty_read+946>: callq 0xffffffff811949a0
> > <__put_user_1>
> > 0xffffffff811f01de <n_tty_read+951>: test %eax,%eax
> > 0xffffffff811f01e0 <n_tty_read+953>: jne 0xffffffff811f043d
> > <n_tty_read+1558>
> > 0xffffffff811f01e6 <n_tty_read+959>: decq -0x78(%rbp)
> > 0xffffffff811f01ea <n_tty_read+963>: jmpq 0xffffffff811f052d
> > <n_tty_read+1798>
> > 0xffffffff811f01ef <n_tty_read+968>: mov $0x1,%eax
> > 0xffffffff811f01f4 <n_tty_read+973>: mov -0x100(%rbp),%rbx
> > 0xffffffff811f01fb <n_tty_read+980>: xchg %rax,(%rbx)
> > 0xffffffff811f01fe <n_tty_read+983>: mov -0x40(%rbp),%rcx
> > 0xffffffff811f0202 <n_tty_read+987>: mov -0xd8(%rbp),%rax
> > 0xffffffff811f0209 <n_tty_read+994>: mov -0xe0(%rbp),%rbx
> > 0xffffffff811f0210 <n_tty_read+1001>: sub %rcx,%rax
> > 0xffffffff811f0213 <n_tty_read+1004>: lea (%rax,%rbx,1),%rdx
> > 0xffffffff811f0217 <n_tty_read+1008>: movzwl 0x21e(%r13),%eax
> > 0xffffffff811f021f <n_tty_read+1016>: cmp %rax,%rdx
> > 0xffffffff811f0222 <n_tty_read+1019>: jge 0xffffffff811f0240
> > <n_tty_read+1049>
> > 0xffffffff811f0224 <n_tty_read+1021>: test %rdx,%rdx
> > 0xffffffff811f0227 <n_tty_read+1024>: jle 0xffffffff811f0240
> > <n_tty_read+1049>
> > 0xffffffff811f0229 <n_tty_read+1026>: mov -0xd8(%rbp),%eax
> > 0xffffffff811f022f <n_tty_read+1032>: sub %cx,%ax
> > 0xffffffff811f0232 <n_tty_read+1035>: add -0xb0(%rbp),%eax
> > 0xffffffff811f0238 <n_tty_read+1041>: mov %ax,0x21e(%r13)
> > 0xffffffff811f0240 <n_tty_read+1049>: mov %r13,%rdi
> > 0xffffffff811f0243 <n_tty_read+1052>: callq 0xffffffff811f37f3
> > <tty_flush_to_ldisc>
> > 0xffffffff811f0248 <n_tty_read+1057>: testb $0x10,0x21c(%r13)
> > 0xffffffff811f0250 <n_tty_read+1065>: je 0xffffffff811f0261
> > <n_tty_read+1082>
> > 0xffffffff811f0252 <n_tty_read+1067>: cmpl $0x0,0x478(%r13)
> > 0xffffffff811f025a <n_tty_read+1075>: jne 0xffffffff811f026f
> > <n_tty_read+1096>
> > 0xffffffff811f025c <n_tty_read+1077>: jmpq 0xffffffff811f0621
> > <n_tty_read+2042>
> > 0xffffffff811f0261 <n_tty_read+1082>: cmpl $0x0,0x260(%r13)
> > 0xffffffff811f0269 <n_tty_read+1090>: jle 0xffffffff811f0621
> > <n_tty_read+2042>
> > 0xffffffff811f026f <n_tty_read+1096>: mov -0x110(%rbp),%rax
> > 0xffffffff811f0276 <n_tty_read+1103>: movq $0x0,(%rax)
> > 0xffffffff811f027d <n_tty_read+1110>: cmpl $0x0,-0x9c(%rbp)
> > 0xffffffff811f0284 <n_tty_read+1117>: mov -0x40(%rbp),%rax
> > 0xffffffff811f0288 <n_tty_read+1121>: je 0xffffffff811f0376
> > <n_tty_read+1359>
> > 0xffffffff811f028e <n_tty_read+1127>: cmp -0xd8(%rbp),%rax
> > 0xffffffff811f0295 <n_tty_read+1134>: jne 0xffffffff811f0376
> > <n_tty_read+1359>
> > 0xffffffff811f029b <n_tty_read+1140>: jmpq 0xffffffff811f033b
> > <n_tty_read+1300>
> > 0xffffffff811f02a0 <n_tty_read+1145>: mov -0xd0(%rbp),%rdi
> > 0xffffffff811f02a7 <n_tty_read+1152>: callq 0xffffffff811eb980
> > <tty_hung_up_p>
> > 0xffffffff811f02ac <n_tty_read+1157>: test %eax,%eax
> > 0xffffffff811f02ae <n_tty_read+1159>: jne 0xffffffff811f052d
> > <n_tty_read+1798>
> > 0xffffffff811f02b4 <n_tty_read+1165>: test %r15,%r15
> > 0xffffffff811f02b7 <n_tty_read+1168>: je 0xffffffff811f052d
> > <n_tty_read+1798>
> > 0xffffffff811f02bd <n_tty_read+1174>: mov -0xd0(%rbp),%rdx
> > 0xffffffff811f02c4 <n_tty_read+1181>: testb $0x8,0x39(%rdx)
> > 0xffffffff811f02c8 <n_tty_read+1185>: je 0xffffffff811f02da
> > <n_tty_read+1203>
> > 0xffffffff811f02ca <n_tty_read+1187>: movq
> > $0xfffffffffffffff5,-0xa8(%rbp)
> > 0xffffffff811f02d5 <n_tty_read+1198>: jmpq 0xffffffff811f052d
> > <n_tty_read+1798>
> > 0xffffffff811f02da <n_tty_read+1203>: mov -0x108(%rbp),%rcx
> > 0xffffffff811f02e1 <n_tty_read+1210>: mov 0x8(%rcx),%rax
> > 0xffffffff811f02e5 <n_tty_read+1214>: testb $0x4,0x10(%rax)
> > 0xffffffff811f02e9 <n_tty_read+1218>: je 0xffffffff811f02fb
> > <n_tty_read+1236>
> > 0xffffffff811f02eb <n_tty_read+1220>: movq
> > $0xfffffffffffffe00,-0xa8(%rbp)
> > 0xffffffff811f02f6 <n_tty_read+1231>: jmpq 0xffffffff811f052d
> > <n_tty_read+1798>
> > 0xffffffff811f02fb <n_tty_read+1236>: mov $0xfff,%eax
> > 0xffffffff811f0300 <n_tty_read+1241>: sub 0x260(%r13),%eax
> > 0xffffffff811f0307 <n_tty_read+1248>: test %eax,%eax
> > 0xffffffff811f0309 <n_tty_read+1250>: jg 0xffffffff811f0324
> > <n_tty_read+1277>
> > 0xffffffff811f030b <n_tty_read+1252>: xor %eax,%eax
> > 0xffffffff811f030d <n_tty_read+1254>: testb $0x10,0x21c(%r13)
> > 0xffffffff811f0315 <n_tty_read+1262>: je 0xffffffff811f0324
> > <n_tty_read+1277>
> > 0xffffffff811f0317 <n_tty_read+1264>: xor %eax,%eax
> > 0xffffffff811f0319 <n_tty_read+1266>: cmpl $0x0,0x478(%r13)
> > 0xffffffff811f0321 <n_tty_read+1274>: sete %al
> > 0xffffffff811f0324 <n_tty_read+1277>: mov %r15,%rdi
> > 0xffffffff811f0327 <n_tty_read+1280>: mov %eax,0xf0(%r13)
> > 0xffffffff811f032e <n_tty_read+1287>: callq 0xffffffff812d5a02
> > <schedule_timeout>
> > 0xffffffff811f0333 <n_tty_read+1292>: mov %rax,%r15
> > 0xffffffff811f0336 <n_tty_read+1295>: jmpq 0xffffffff811f0522
> > <n_tty_read+1787>
> > 0xffffffff811f033b <n_tty_read+1300>: mov -0xf0(%rbp),%rbx
> > 0xffffffff811f0342 <n_tty_read+1307>: lea -0x31(%rbp),%rsi
> > 0xffffffff811f0346 <n_tty_read+1311>: mov $0x1,%edx
> > 0xffffffff811f034b <n_tty_read+1316>: mov %r13,%rdi
> > 0xffffffff811f034e <n_tty_read+1319>: movb $0x0,-0x31(%rbp)
> > 0xffffffff811f0352 <n_tty_read+1323>: mov %rbx,-0x40(%rbp)
> > 0xffffffff811f0356 <n_tty_read+1327>: callq 0xffffffff812008ac
> > <tty_audit_add_data>
> > 0xffffffff811f035b <n_tty_read+1332>: mov -0x31(%rbp),%al
> > 0xffffffff811f035e <n_tty_read+1335>: mov -0xd8(%rbp),%rcx
> > 0xffffffff811f0365 <n_tty_read+1342>: callq 0xffffffff811949a0
> > <__put_user_1>
> > 0xffffffff811f036a <n_tty_read+1347>: test %eax,%eax
> > 0xffffffff811f036c <n_tty_read+1349>: jne 0xffffffff811f043d
> > <n_tty_read+1558>
> > 0xffffffff811f0372 <n_tty_read+1355>: decq -0x78(%rbp)
> > 0xffffffff811f0376 <n_tty_read+1359>: testb $0x10,0x21c(%r13)
> > 0xffffffff811f037e <n_tty_read+1367>: jne 0xffffffff811f0456
> > <n_tty_read+1583>
> > 0xffffffff811f0384 <n_tty_read+1373>: jmpq 0xffffffff811f047a
> > <n_tty_read+1619>
> > 0xffffffff811f0389 <n_tty_read+1378>: mov 0x25c(%r13),%eax
> > 0xffffffff811f0390 <n_tty_read+1385>: mov -0x88(%rbp),%rbx
> > 0xffffffff811f0397 <n_tty_read+1392>: lock btr %eax,(%rbx)
> > 0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
> > 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
> > 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
> > 0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
> > 0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx
> >
> > Below is the output of bt -a command in crash
> >
> > bt -a
> > PID: 0 TASK: ffffffff814204b0 CPU: 0 COMMAND: "swapper"
> > #0 [ffff880033007e80] crash_nmi_callback at ffffffff8101fbc9
> > #1 [ffff880033007e90] notifier_call_chain at ffffffff81065893
> > #2 [ffff880033007ed0] atomic_notifier_call_chain at ffffffff810658dd
> > #3 [ffff880033007ee0] notify_die at ffffffff8106597f
> > #4 [ffff880033007f10] do_nmi at ffffffff8100dc5d
> > #5 [ffff880033007f50] nmi at ffffffff812d76b0
> > [exception RIP: mwait_idle+163]
> > RIP: ffffffff81013029 RSP: ffffffff813e3eb8 RFLAGS: 00000246
> > RAX: 0000000000000000 RBX: ffffffff813e3fd8 RCX: 0000000000000000
> > RDX: 0000000000000000 RSI: ffffffff813e3fd8 RDI: ffffffff81522308
> > RBP: ffffffff813e3ec8 R8: 0000000000000000 R9: ffff88003306e290
> > R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368
> > R13: ffffffff814ccb30 R14: ffffffff814cdfa0 R15: ffffffff813e3fa8
> > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> > --- <NMI exception stack> ---
> > #6 [ffffffff813e3eb8] mwait_idle at ffffffff81013029
> > #7 [ffffffff813e3ed0] cpu_idle at ffffffff8100af21
> >
> > PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
> > #0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
> > #1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
> > #2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
> > #3 [ffff88031ce75b50] no_context at ffffffff8102d801
> > #4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
> > #5 [ffff88031ce75c70] bad_area at ffffffff8102da41
> > #6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
> > #7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
> > [exception RIP: n_tty_read+1420]
> > RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
> > RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
> > RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
> > RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
> > R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
> > R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
> > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> > #8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
> > #9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
> > #10 [ffff88031ce75f40] sys_read at ffffffff810ebe48
> > #11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
> > RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
> > RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
> > RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
> > RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
> > R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
> > R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
> > ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
> >
> > PID: 0 TASK: ffff88031e0e3540 CPU: 2 COMMAND: "swapper"
> > #0 [ffff880033047e80] crash_nmi_callback at ffffffff8101fbc9
> > #1 [ffff880033047e90] notifier_call_chain at ffffffff81065893
> > #2 [ffff880033047ed0] atomic_notifier_call_chain at ffffffff810658dd
> > #3 [ffff880033047ee0] notify_die at ffffffff8106597f
> > #4 [ffff880033047f10] do_nmi at ffffffff8100dc5d
> > #5 [ffff880033047f50] nmi at ffffffff812d76b0
> > [exception RIP: mwait_idle+163]
> > RIP: ffffffff81013029 RSP: ffff88031e0e5ef8 RFLAGS: 00000246
> > RAX: 0000000000000000 RBX: ffff88031e0e5fd8 RCX: 0000000000000000
> > RDX: 0000000000000000 RSI: ffff88031e0e5fd8 RDI: ffffffff81522308
> > RBP: ffff88031e0e5f08 R8: 0000000000000000 R9: ffff88003302e290
> > R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368
> > R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> > --- <NMI exception stack> ---
> > #6 [ffff88031e0e5ef8] mwait_idle at ffffffff81013029
> > #7 [ffff88031e0e5f10] cpu_idle at ffffffff8100af21
> >
> > PID: 0 TASK: ffff88031e113580 CPU: 3 COMMAND: "swapper"
> > #0 [ffff880033067e80] crash_nmi_callback at ffffffff8101fbc9
> > #1 [ffff880033067e90] notifier_call_chain at ffffffff81065893
> > #2 [ffff880033067ed0] atomic_notifier_call_chain at ffffffff810658dd
> > #3 [ffff880033067ee0] notify_die at ffffffff8106597f
> > #4 [ffff880033067f10] do_nmi at ffffffff8100dc5d
> > #5 [ffff880033067f50] nmi at ffffffff812d76b0
> > [exception RIP: mwait_idle+163]
> > RIP: ffffffff81013029 RSP: ffff88031e115ef8 RFLAGS: 00000246
> > RAX: 0000000000000000 RBX: ffff88031e115fd8 RCX: 0000000000000000
> > RDX: 0000000000000000 RSI: ffff88031e115fd8 RDI: ffffffff81522308
> > RBP: ffff88031e115f08 R8: 0000000000000000 R9: 0000000000000000
> > R10: 0000000000000800 R11: 0000000000000000 R12: ffffffff8147e368
> > R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> > --- <NMI exception stack> ---
> > #6 [ffff88031e115ef8] mwait_idle at ffffffff81013029
> > #7 [ffff88031e115f10] cpu_idle at ffffffff8100af21
> >
> > Please let me know if you need any other details.
> >
> > Thanks and Regards
> > Shashidhara
> >
> >
> > -----Original Message-----
> > From: crash-utility-bounces@xxxxxxxxxx
> > [mailto:crash-utility-bounces@xxxxxxxxxx] On Behalf Of Dave Anderson
> > Sent: Thursday, June 23, 2011 9:35 PM
> > To: Discussion list for crash utility usage,maintenance and
> > development
> > Subject: Re: Unable to switch stack frames while
> > using
> > crash
> >
> >
> >
> > ----- Original Message -----
> >
> > > BTW, are you sure about that?
> > >
> > > Presuming that the "tty" pointer is ffff8802cbd54800 as you've
> > > shown
> > below,
> > > and therefore tty->read_buf is 0xffff8802cbfe6000 and
> > > tty->read_tail
> > is 0,
> > > then the statement above would be simply be reading
> > > tty->read_buf[0],
> > or
> > > virtual address 0xffff8802cbfe6000. But the oops shows it faulting
> > > on
> > a
> > > virtual address of "5":
> > >
> > > BUG: unable to handle kernel NULL pointer dereference at
> > 0000000000000005
> >
> > Just for my own sanity, can you either attach the
> > "drivers/char/n_tty.c"
> >
> > from *your* specific kernel, or get the source-code/line-number data
> > from
> > the embedded gdb module?
> >
> > If you don't have the n_tty.c file readily available, you can get
> > the
> > source-code/line-number data of a particular function by doing
> > something
> > like this:
> >
> > Get the line number of the beginning of n_tty_read(), which in my
> > kernel
> > is at 1698 -- your's will probably be different:
> >
> > crash> gdb list n_tty_read
> > 1695 * This code must be sure never to sleep through a hangup.
> > 1696 */
> > 1697
> > 1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file
> > *file,
> > 1699 unsigned char __user *buf, size_t nr)
> > 1700 {
> > 1701 unsigned char __user *b = buf;
> > 1702 DECLARE_WAITQUEUE(wait, current);
> > 1703 int c;
> > 1704 int minimum, time;
> > crash>
> >
> > Then get the line number of the next function in the file, which is
> > n_tty_write():
> >
> > crash> gdb list n_tty_write
> > 1918 * lock themselves)
> > 1919 */
> > 1920
> > 1921 static ssize_t n_tty_write(struct tty_struct *tty, struct file
> > *file,
> > 1922 const unsigned char *buf, size_t nr)
> > 1923 {
> > 1924 const unsigned char *b = buf;
> > 1925 DECLARE_WAITQUEUE(wait, current);
> > 1926 int c;
> > 1927 ssize_t retval = 0;
> >
> > And then dump the whole n_tty_read() function (plus some extra
> > stuff):
> >
> > crash> gdb list 1698,1920
> > 1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file
> > *file,
> > 1699 unsigned char __user *buf, size_t nr)
> > 1700 {
> > 1701 unsigned char __user *b = buf;
> > 1702 DECLARE_WAITQUEUE(wait, current);
> > 1703 int c;
> > 1704 int minimum, time;
> > 1705 ssize_t retval = 0;
> > 1706 ssize_t size;
> > 1707 long timeout;
> > 1708 unsigned long flags;
> > 1709 int packet;
> > 1710
> > 1711 do_it_again:
> > 1712
> > 1713 BUG_ON(!tty->read_buf);
> > 1714
> > 1715 c = job_control(tty, file);
> > 1716 if (c < 0)
> > 1717 return c;
> > 1718
> > 1719 minimum = time = 0;
> > 1720 timeout = MAX_SCHEDULE_TIMEOUT;
> > 1721 if (!tty->icanon) {
> > 1722 time = (HZ / 10) * TIME_CHAR(tty);
> > 1723 minimum = MIN_CHAR(tty);
> > ...
> >
> > And lastly, since the crash occurred at
> >
> > IP: [<ffffffff811f03b3>] n_tty_read+0x58c/0x818
> >
> > Do this:
> >
> > crash> dis -rl n_tty_read+0x58c
> > ...
> >
> > And then post all of that data.
> >
> > Dave
> >
> >
> > --
> > Crash-utility mailing list
> > Crash-utility@xxxxxxxxxx
> > https://www.redhat.com/mailman/listinfo/crash-utility
> >
> > Information transmitted by this e-mail is proprietary to MphasiS,
> > its
> > associated companies and/ or its customers and is intended
> > for use only by the individual or entity to which it is addressed,
> > and
> > may contain information that is privileged, confidential or
> > exempt from disclosure under applicable law. If you are not the
> > intended recipient or it appears that this mail has been forwarded
> > to you without proper authority, you are notified that any use or
> > dissemination of this information in any manner is strictly
> > prohibited. In such cases, please notify us immediately at
> > mailmaster@xxxxxxxxxxx and delete this mail from your records.
> >
> >
> > --
> > Crash-utility mailing list
> > Crash-utility@xxxxxxxxxx
> > https://www.redhat.com/mailman/listinfo/crash-utility
>
> --
> Crash-utility mailing list
> Crash-utility@xxxxxxxxxx
> https://www.redhat.com/mailman/listinfo/crash-utility
>
> Information transmitted by this e-mail is proprietary to MphasiS, its
> associated companies and/ or its customers and is intended
> for use only by the individual or entity to which it is addressed, and
> may contain information that is privileged, confidential or
> exempt from disclosure under applicable law. If you are not the
> intended recipient or it appears that this mail has been forwarded
> to you without proper authority, you are notified that any use or
> dissemination of this information in any manner is strictly
> prohibited. In such cases, please notify us immediately at
> mailmaster@xxxxxxxxxxx and delete this mail from your records.
>
>
> --
> Crash-utility mailing list
> Crash-utility@xxxxxxxxxx
> https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/crash-utility
