----- Original Message ----- > Hello Dave, > > Changed according to your suggestions. But need to add some explanation > here. > > change 1: > > Perhaps there could be a way to pre-verify the addresses with > > accessible(), and if the address is bogus, display an error message, > > but allow the command to continue on with the other cpus? > > Error comes from rb_last() which used to calculate the space for > printing expires. > > I fixed rb_last(), adding pre-verify the addresses. But I didn't use > rb_last() any more, for change 2 helps to find the last hrtimer. > > change 2: > > > > Maybe you can use hq_open()/hq_enter()/hq_close() on the hrtimer addresses > > to prevent this from happening, warn the user when it does, and continue on > > with the next cpu? > > As you suggested. > > change3: > When doing some snap test, I found rb_node's data may be destroyed, so > I also fixed other rbtree manipulation functions to avoid abort of > command. > Hi Qiao, Thanks -- this second version makes it through my set of test kernels successfully. The only vmcore I have that needs examination is this one, where a softlockup timeout occurred: crash> bt PID: 7589 TASK: ffff8802292fa040 CPU: 0 COMMAND: "crash" #0 [ffff880028203ce0] machine_kexec at ffffffff810329db #1 [ffff880028203d40] crash_kexec at ffffffff810bdbd2 #2 [ffff880028203e10] panic at ffffffff815097a9 #3 [ffff880028203e90] watchdog_timer_fn at ffffffff810de51d #4 [ffff880028203ec0] __run_hrtimer at ffffffff810986ee #5 [ffff880028203f10] hrtimer_interrupt at ffffffff81098a56 #6 [ffff880028203f90] smp_apic_timer_interrupt at ffffffff8151350b #7 [ffff880028203fb0] apic_timer_interrupt at ffffffff8100bc13 --- <IRQ stack> --- #8 [ffff8802292a1df8] apic_timer_interrupt at ffffffff8100bc13 [exception RIP: copy_user_generic_string+45] RIP: ffffffff81283ddd RSP: ffff8802292a1ea0 RFLAGS: 00010246 RAX: ffff8802292a0000 RBX: ffff8802292a1ee8 RCX: 0000000000000200 RDX: 0000000000000000 RSI: ffff880020000000 RDI: 0000000000d91dc0 RBP: ffffffff8100bc0e R8: ffffffff81346e70 R9: 0000000000000001 R10: 0000000000105c52 R11: 0000000000000246 R12: 0000000000d91dc0 R13: 0000000000020000 R14: 0000000000000000 R15: 0000000020000000 ORIG_RAX: ffffffffffffff10 CS: 0010 SS: 0018 #9 [ffff8802292a1ea0] crash_read at ffffffff81346fbd #10 [ffff8802292a1ef0] vfs_read at ffffffff8117fd65 #11 [ffff8802292a1f30] sys_read at ffffffff8117fea1 #12 [ffff8802292a1f80] system_call_fastpath at ffffffff8100b0f2 RIP: 00007f3a6cb7e5f0 RSP: 00007fff740b8070 RFLAGS: 00000246 RAX: 0000000000000000 RBX: ffffffff8100b0f2 RCX: 00007f3a6cb7e5f0 RDX: 0000000000001000 RSI: 0000000000d91dc0 RDI: 0000000000000004 RBP: 0000000000000004 R8: 0000000020000000 R9: 0000000020000000 R10: 0000000000105c52 R11: 0000000000000246 R12: 0000000000001000 R13: 0000000000d91dc0 R14: 0000000000001000 R15: 0000000000d91dc0 ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b crash> And the timer command fails like so, again, without being able to complete the rest of the cpus: crash> timer -r UPTIME: 300039(1000HZ) cpu: 0 clock: 0 .base: ffff8800282115a8 .offset: 1354043557752205725 .get_time: ktime_get_real EXPIRES HRTIMER FUNCTION (empty) clock: 1 .base: ffff8800282115e8 .offset: 0 .get_time: ktime_get EXPIRES HRTIMER FUNCTION 300040000000-300040000000 ffff8800282116a0 ffffffff810a4b70 <tick_sched_timer> 3660368239466-3660368239466 ffff880224f07c68 ffffffff81071c00 <it_real_fn> 3660700491472-3660700491472 ffff880224f07068 ffffffff81071c00 <it_real_fn> 14461470907794-14461570907794 ffff88022750fa68 ffffffff81098160 <hrtimer_wakeup> clock: 2 .base: ffff880028211628 .offset: 0 .get_time: timer: invalid kernel virtual address: 2c type: "ktime_t tv64" crash> Presumably the kernel was in the midst of tinkering with the hrtimer-related structures, and that was what causes the bad readmem() above. But I'm thinking that since softlockups are a fairly common cause of kernel crashes, that this option should be able to handle it better than above? I've put copies of the vmlinux/vmcore pair above here for you to take a look: http://people.redhat.com/anderson/qiao The next thing that needs to be reworked is the display itself. I really don't like the ".base", ".offset" and ".get_time" lines, primarily because none of the other crash commands do anything like that. Also, the UPTIME display is kind of strange. It only applies to the "ktime_get" type clocks, right? I understand why you selected those particular fields -- although I can't find a ".base" member in either the hr_clock_base or hrtimer_cpu_base structures? But more importantly, what if a user wants to see the other fields in either of those two data structures? There's no obvious way to find their addresses. Anyway, for consistency with other crash commands, I think that it's more useful to display the hrtimer_cpu_base and hr_clock_base structure addresses, and perhaps the most important fields within them, in tabular form, grouped together something like: crash> timer -r CPU: 0 CLOCK: 0 HRTIMER_CPU_BASE HR_CLOCK_BASE OFFSET FUNCTION <address> <address> <value> ktime_get EXPIRES HRTIMER FUNCTION 396384243603-396384293603 ffff88003c905eb8 ffffffff8107da00 <hrtimer_wakeup> 397000645784-397004641782 ffff88003d2b1a58 ffffffff8107da00 <hrtimer_wakeup> 407501746988-407531746986 ffff88003c2b5a58 ffffffff8107da00 <hrtimer_wakeup> 446387171280-446446754509 ffff88003bb43908 ffffffff8107da00 <hrtimer_wakeup> 900000001418-900000001418 ffff8800368e53c0 ffffffff811c57d0 <timerfd_tmrproc> CLOCK: 1 HRTIMER_CPU_BASE HR_CLOCK_BASE OFFSET FUNCTION <address> <address> <value> ktime_get_real EXPIRES HRTIMER FUNCTION (empty) CLOCK: 2 HRTIMER_CPU_BASE HR_CLOCK_BASE OFFSET FUNCTION <address> <address> <value> ktime_get EXPIRES HRTIMER FUNCTION 396366000000-396366000000 ffff88003fcce6a0 ffffffff810ac4a0 <tick_sched_timer> 394748203985-394749203930 ffff880039f4f908 ffffffff8107da00 <hrtimer_wakeup> 400056995196-400056995196 ffff88003fcce820 ffffffff810e2800 <watchdog_timer_fn> 3616558364176-3616558414176 ffff88003cf6beb8 ffffffff8107da00 <hrtimer_wakeup> CPU: 1 ... I think a display like the above shows everything that you are doing now -- plus a bit more. What do you think? Dave -- Crash-utility mailing list Crash-utility@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/crash-utility