Hi Kevin,
On Mon, 2011-01-03 at 08:14 -0800, Kevin D. Kissell wrote:
> The very first SMTC implementations didn't support full kernel-mode
> preemption, which anyway wasn't a priority, given the hardware event
> response support in MIPS MT. I believe it was later made compatible,
> but it was never extensively exercised. Since SMTC has fingers in some
> pretty low-level atomicity mechanisms, if a new, parallel set was
> implemented for RCU, I can easily imagine that nobody has yet
> implemented SMTC-ified variants of that set.
>
> Your last statement isn't very clear, though. Are you saying that if
> you configure for no forced preemption and with TREE_CPU, the 2.6.37
> kernel boots all the way up, or that it simply hangs later? What's the
> last rev kernel that actually boots all the way up?
I have debugged this a bit more. It seems that kernel getting stalled
while executing on TC's of second VPE .
INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
by 1, t=2504 jiffies)
INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
by 1, t=10036 jiffies)
INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
by 1, t=17568 jiffies)
INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
by 1, t=25100 jiffies)
INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
by 1, t=32632 jiffies)
INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
by 1, t=40164 jiffies)
With CONFIG_TREE_CPU we were not hitting this scenario very often.
However with CONFIG_PREEMPT_TREE_CPU stall happens most of the time.
I presume some issue in my timer setup . I am not seeing timer interrupt
(or IPI interrupt) getting incremented for VPE1 tcs on a completely
booted 2.6.32-stable kernel.
/ # cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
CPU6
1: 148 15023 15140 15093 3779 8
2 MIPS SMTC_IPI
6: 0 0 0 0 0 0
0 MIPS MSP CIC cascade
8: 0 0 0 0 0 0
0 MSP_CIC Softreset button
9: 0 0 0 0 0 0
0 MSP_CIC Standby switch
21: 0 0 0 0 0 0
0 MSP_CIC MSP PER cascade
25: 15113 341 4 7 0 0
0 MSP_CIC timer
27: 260 9 0 1 0 0
0 MSP_CIC serial
34: 0 0 0 0 0 0
0 MSP_CIC timer
Can't we use separate timer interrupts for VPE1 and VPE0 in SMTC ?.
I have tried setting up VPE1 timer from get_co_compare_int as follows
unsigned int __cpuinit get_c0_compare_int(void)
{
if ((1==get_current_vpe()) && !vpe1_timr_installed){
memcpy(&timer_vpe1,&c0_compare_irqaction,sizeof(timer_vpe1));
setup_irq(MSP_INT_VPE1_TIMER, &timer_vpe1);
vpe1_timr_installed++;
}
return (get_current_vpe() ? MSP_INT_VPE1_TIMER :
MSP_INT_VPE0_TIMER);
}
Thanks
Anoop
>
> Regards,
>
> Kevin K.
>
> On 1/3/2011 7:12 AM, Anoop P A wrote:
> > Hi ,
> >
> > Following patch restricts TREE_CPU RCU implementation only for !PREEMPT
> > SMP kernel.
> > http://git.linux-mips.org/?p=linux.git;a=commit;h=687d7a960aea46e016182c7ce346d62c4dbd0366
> >
> > CONFIG_TREE_PREEMPT_RCU option seems to be not working for SMTC kernel
> > ( which will be only available RCU implementation for SMTC kernel from
> > 2.6.37 onwards) .
> >
> > With no forced preemption and selecting TREE_CPU I am able to boot
> > further to the hang that I have reported.
> >
> > Thanks
> > Anoop
> >
> > On Sat, 2011-01-01 at 00:42 -0800, Kevin D. Kissell wrote:
> >> At this point the logical thing to do would seem to look at your kernel
> >> image and disassemble smtc_ipi_replay(), which is where the EPC of VPE 0
> >> shows the last exception to have been taken. That's a critical SMTC
> >> routine that gets called whenever an xxx_irq_restore() enables
> >> interrupts, so that virtual per-TC IPI interrupts that were posted while
> >> the TC had interrupts disabled can be handled deterministically. As I
> >> mentioned in an earlier message, there was some cleanup work from David
> >> Howell that changed a number of irq management-related function names
> >> and prototypes across all architectures, which went into linux-mips.org
> >> at very roughly the time of the breakage. The SMTC overlay over the irq
> >> implementation has been pretty robust, but it's written in a perhaps
> >> doomed attempt to be both efficient and using a maximum amount of common
> >> code with the general case. A mechanical or semi-mechanical change
> >> could conceivably have broken things.
> >>
> >> Regards,
> >>
> >> Kevin K.
> >>
> >>
> >> On 12/31/2010 4:27 AM, Anoop P A wrote:
> >>> Hi ,
> >>>
> >>> Kernel hangs on stop_machine call. Please find mt reg dump below.
> >>> Another important observation is even though 2.6.33 kernel + stackframe
> >>> patch well passes calibration hang , I am still unable boot in to a
> >>> initramfs root ( verified ramfs working with VSMP). So it looks like
> >>> still some issue to fix between 2.6.32 and 2.6.33 .
> >>> ######################## Log ###########################
> >>>
> >>> === MIPS MT State Dump ===
> >>> -- Global State --
> >>> MVPControl Passed: 00000005
> >>> MVPControl Read: 00000004
> >>> MVPConf0 : a8008406
> >>> -- per-VPE State --
> >>> VPE 0
> >>> VPEControl : 00008000
> >>> VPEConf0 : 800f0003
> >>> VPE0.Status : 11004201
> >>> VPE0.EPC : 8010dc54 smtc_ipi_replay+0xcc/0x108
> >>> VPE0.Cause : 50804000
> >>> VPE0.Config7 : 00010000
> >>> VPE 1
> >>> VPEControl : 00068006
> >>> VPEConf0 : 80cf0003
> >>> VPE1.Status : 11008301
> >>> VPE1.EPC : 801022a0 r4k_wait+0x20/0x40
> >>> VPE1.Cause : 50800000
> >>> VPE1.Config7 : 00010000
> >>> -- per-TC State --
> >>> TC 0 (current TC with VPE EPC above)
> >>> TCStatus : 18102000
> >>> TCBind : 00000000
> >>> TCRestart : 803fa19c printk+0xc/0x30
> >>> TCHalt : 00000000
> >>> TCContext : 00000000
> >>> TC 1
> >>> TCStatus : 18902000
> >>> TCBind : 00200000
> >>> TCRestart : 801022a0 r4k_wait+0x20/0x40
> >>> TCHalt : 00000000
> >>> TCContext : 00140000
> >>> TC 2
> >>> TCStatus : 18902000
> >>> TCBind : 00400000
> >>> TCRestart : 801022a0 r4k_wait+0x20/0x40
> >>> TCHalt : 00000000
> >>> TCContext : 00280000
> >>> TC 3
> >>> TCStatus : 18902000
> >>> TCBind : 00600000
> >>> TCRestart : 801022a0 r4k_wait+0x20/0x40
> >>> TCHalt : 00000000
> >>> TCContext : 003c0000
> >>> TC 4
> >>> TCStatus : 18902000
> >>> TCBind : 00800001
> >>> TCRestart : 8010229c r4k_wait+0x1c/0x40
> >>> TCHalt : 00000000
> >>> TCContext : 00500000
> >>> TC 5
> >>> TCStatus : 18902000
> >>> TCBind : 00a00001
> >>> TCRestart : 8010229c r4k_wait+0x1c/0x40
> >>> TCHalt : 00000000
> >>> TCContext : 00640000
> >>> TC 6
> >>> TCStatus : 18902000
> >>> TCBind : 00c00001
> >>> TCRestart : 8010229c r4k_wait+0x1c/0x40
> >>> TCHalt : 00000000
> >>> TCContext : 00780000
> >>> Counter Interrupts taken per CPU (TC)
> >>> 0: 0
> >>> 1: 0
> >>> 2: 0
> >>> 3: 0
> >>> 4: 0
> >>> 5: 0
> >>> 6: 0
> >>> 7: 0
> >>> Self-IPI invocations:
> >>> 0: 12
> >>> 1: 0
> >>> 2: 0
> >>> 3: 0
> >>> 4: 0
> >>> 5: 5
> >>> 6: 4
> >>> 7: 0
> >>> IPIQ[0]: head = 0x0, tail = 0x0, depth = 0
> >>> IPIQ[1]: head = 0x0, tail = 0x0, depth = 0
> >>> IPIQ[2]: head = 0x0, tail = 0x0, depth = 0
> >>> IPIQ[3]: head = 0x0, tail = 0x0, depth = 0
> >>> IPIQ[4]: head = 0x0, tail = 0x0, depth = 0
> >>> IPIQ[5]: head = 0x0, tail = 0x0, depth = 0
> >>> IPIQ[6]: head = 0x0, tail = 0x0, depth = 0
> >>> IPIQ[7]: head = 0x0, tail = 0x0, depth = 0
> >>> 0 Recoveries of "stolen" FPU
> >>> ===========================
> >>>
> >>> ################################################################
> >>>
> >>> Thanks
> >>> Anoop
> >>>
> >>> On Tue, 2010-12-28 at 00:43 -0800, Kevin D. Kissell wrote:
> >>>> I took a quick look last night, and the only thing that looked vaguely
> >>>> dangerous in changes since the timer changes I alluded to earlier was
> >>>> the global naming cleanup of irq-related function names that David
> >>>> Howell submitted. The diff didn't look dangerous in itself, but some of
> >>>> the definitions are nested subtly for SMTC to maximize the amount of
> >>>> common code, and I could imagine something getting lost in translation
> >>>> there. If that were really the problem, it would of course affect much
> >>>> more than just the timer subsystem, but early in the boot process,
> >>>> timers are pretty much the only interrupts that have to be handled
> >>>> correctly.
> >>>>
> >>>> I'm travelling today, but will take a look at timekeeping_notify()
> >>>> tomorrow or the next day...
> >>>>
> >>>> /K.
> >>>>
> >>>> On 12/28/10 12:19 AM, Anoop P A wrote:
> >>>>> Hi,
> >>>>>
> >>>>> I had a glance into the code diff without notice of any suspect-able
> >>>>> code .
> >>>>> Tracing the hang showed that it is getting hanged in timekeeping_notify
> >>>>> function.
> >>>>>
> >>>>> Thanks,
> >>>>> Anoop
> >>>>>
> >>>>> PS: I may not be available until Thursday
> >>>>>
> >>>>> On Mon, 2010-12-27 at 22:49 +0530, Anoop P A wrote:
> >>>>>> Hi Kevin,
> >>>>>>
> >>>>>> It is very unlikely that the patch you pointed has any impact on the the
> >>>>>> hang I am seeing. The patch you have mentioned got into kernel around
> >>>>>> 2.6.32 timeframe. I am able to boot both 2.6.32 and 2.6.33 kernel ( +
> >>>>>> stackframe patch) .
> >>>>>>
> >>>>>> Hi Stuart,
> >>>>>>
> >>>>>> I haven't got much time to spend on this today.
> >>>>>>
> >>>>>> I had got 2.6.36-stable(+ stack frame patch) booting last day and I have
> >>>>>> observed hang issue with 2.6.37-rc1 ( Same as rc6 and current git head)
> >>>>>>
> >>>>>> So probably some patches in 2.6.37 branch introduced this hang.
> >>>>>>
> >>>>>> Hopefully I will get some free slot tomorrow so that I can look into
> >>>>>> code diff .
> >>>>>>
> >>>>>> Thanks
> >>>>>> Anoop
> >>>>>>
> >>>>>> On Mon, 2010-12-27 at 09:49 -0600, STUART VENTERS wrote:
> >>>>>>> Kevin,
> >>>>>>>
> >>>>>>> Outstanding, sometimes it's better to be lucky than good.
> >>>>>>>
> >>>>>>>
> >>>>>>> Anoop,
> >>>>>>>
> >>>>>>> Maybe we can get lucky again.
> >>>>>>>
> >>>>>>> If you can isolate the .33 works/.37 works_not bug to a specific pair of versions,
> >>>>>>> I'll be happy to do another diff.
> >>>>>>>
> >>>>>>>
> >>>>>>> Hope you'll have had a good Christmas as well.
> >>>>>>> We've had snow in Alabama since Christmas eve!
> >>>>>>>
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>> Stuart
> >>>>>>>
> >>>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Kevin D. Kissell [mailto:kevink@xxxxxxxxxxxxx]
> >>>>>>> Sent: Friday, December 24, 2010 5:34 PM
> >>>>>>> To: Anoop P A
> >>>>>>> Cc: STUART VENTERS; Anoop P.A.; linux-mips@xxxxxxxxxxxxxx
> >>>>>>> Subject: Re: SMTC support status in latest git head.
> >>>>>>>
> >>>>>>>
> >>>>>>> Ah, well, at least we have a stackframe.h fix that preserves David's
> >>>>>>> performance tweak for the deeper pipelined processors. In looking for
> >>>>>>> this, I did notice that someone did some modification to the SMTC clock
> >>>>>>> tick logic that I was skeptical had ever been tested. If you've still
> >>>>>>> got that kernel binary handy, you might check to see if it boots with
> >>>>>>> maxtcs=1 maxvpes=1, maxtcs=2 maxvpes=1, and/or maxtcs=2 maxvpes=2.
> >>>>>>>
> >>>>>>> Oh, yes, and Merry Christmas one and all!
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>> Kevin K.
> >>>>>>>
> >>>>>>> On 12/24/10 8:02 AM, Anoop P A wrote:
> >>>>>>>> On Fri, 2010-12-24 at 06:53 -0800, Kevin D. Kissell wrote:
> >>>>>>>>> Excellent! Now, does the attached patch (relative to 2.6.37.11) also
> >>>>>>>>> fix things, while preserving the other fixes and performance enhancements?
> >>>>>>>>>
> >>>>>>>> I have tested that patch with 2.6.37 branch it well passes calibration
> >>>>>>>> loop but hangs after switching to mips closource
> >>>>>>>>
> >>>>>>>> TC 6 going on-line as CPU 6
> >>>>>>>> Brought up 7 CPUs
> >>>>>>>> bio: create slab<bio-0> at 0
> >>>>>>>> SCSI subsystem initialized
> >>>>>>>> Switching to clocksource MIPS
> >>>>>>>>
> >>>>>>>> I Presume this is a different issue as restoring older file didn't help
> >>>>>>>> much to get rid of this hang.
> >>>>>>>>
> >>>>>>>> diff --git a/arch/mips/include/asm/stackframe.h
> >>>>>>>> b/arch/mips/include/asm/stackframe.h
> >>>>>>>> index 58730c5..7fc9f10 100644
> >>>>>>>> --- a/arch/mips/include/asm/stackframe.h
> >>>>>>>> +++ b/arch/mips/include/asm/stackframe.h
> >>>>>>>> @@ -195,9 +195,9 @@
> >>>>>>>> * to cover the pipeline delay.
> >>>>>>>> */
> >>>>>>>> .set mips32
> >>>>>>>> - mfc0 v1, CP0_TCSTATUS
> >>>>>>>> + mfc0 v0, CP0_TCSTATUS
> >>>>>>>> .set mips0
> >>>>>>>> - LONG_S v1, PT_TCSTATUS(sp)
> >>>>>>>> + LONG_S v0, PT_TCSTATUS(sp)
> >>>>>>>> #endif /* CONFIG_MIPS_MT_SMTC */
> >>>>>>>> LONG_S $4, PT_R4(sp)
> >>>>>>>> LONG_S $5, PT_R5(sp)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> /K.
> >>>>>>>>>
> >>>>>>>>> On 12/24/10 6:39 AM, Anoop P A wrote:
> >>>>>>>>>> Hi Kevin, Stuart ,
> >>>>>>>>>>
> >>>>>>>>>> Woohooo You guys spotted !.
> >>>>>>>>>>
> >>>>>>>>>> http://git.linux-mips.org/?p=linux.git;a=commit;h=d5ec6e3c seems to be
> >>>>>>>>>> the culprit
> >>>>>>>>>>
> >>>>>>>>>> Once I restored previous version of stackframe.h 2.6.33-stable started
> >>>>>>>>>> booting !.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Anoop
> >>>>>>>>>>
> >>>>>>>>>> On Fri, 2010-12-24 at 04:32 -0800, Kevin D. Kissell wrote:
> >>>>>>>>>>> Thank you, Stuart! I've spotted some definite breakage to SMTC between
> >>>>>>>>>>> those versions. In arch/mips/include/asm/stackframe.h, someone moved
> >>>>>>>>>>> the store of the Status register value in SAVE_SOME (line 169 or 204,
> >>>>>>>>>>> depending on the version) from two instructions after the mfc0 to a
> >>>>>>>>>>> point after the #ifdef for SMTC, presumably to get better pipelining of
> >>>>>>>>>>> the register access. Unfortunately, the v1 register is also used in the
> >>>>>>>>>>> SMTC-specific fragment to save TCStatus, so the Status value gets
> >>>>>>>>>>> clobbered before it gets stored. This will eventually result in the
> >>>>>>>>>>> Status register getting a TCStatus value, which has some bits on common,
> >>>>>>>>>>> but isn't identical and sooner or later Bad Things will happen.
> >>>>>>>>>>>
> >>>>>>>>>>> I'm a little surprised this wasn't caught by visual inspection of the patch.
> >>>>>>>>>>>
> >>>>>>>>>>> Possible solutions would include reverting the store of the CP0_STATUS
> >>>>>>>>>>> value to the block above the #ifdef, or, to retain whatever performance
> >>>>>>>>>>> advantage was obtained by moving the store downward, to use v0/$2
> >>>>>>>>>>> instead of v1/$3, as the staging register for the TCStatus value. I'd
> >>>>>>>>>>> lean toward the second option, but I'm not in a position to test and
> >>>>>>>>>>> submit a patch just now.
> >>>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>>
> >>>>>>>>>>> Kevin K.
> >>>>>>>>>>>
> >>>>>>>>>>> On 12/23/10 1:09 PM, STUART VENTERS wrote:
> >>>>>>>>>>>> Kevin,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm not sure if it's useful,
> >>>>>>>>>>>> but finally I got the time to look at the two kernel versions Anoop pointed out.
> >>>>>>>>>>>> works 2.6.32-stable with patch 804
> >>>>>>>>>>>> works_not 2.6.33-stable
> >>>>>>>>>>>>
> >>>>>>>>>>>> greping for files with CONFIG_MIPS_MT_SMTC
> >>>>>>>>>>>> and looking for timer interrupt related stuff found the following differences:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> arch/mips/include/asm/irq.h
> >>>>>>>>>>>> arch/mips/kernel/irq.c
> >>>>>>>>>>>> do_IRQ
> >>>>>>>>>>>>
> >>>>>>>>>>>> arch/mips/include/asm/stackframe.h
> >>>>>>>>>>>> SAVE_SOME SAVE_TEMP get/set_saved_sp
> >>>>>>>>>>>>
> >>>>>>>>>>>> arch/mips/include/asm/time.h
> >>>>>>>>>>>> clocksource_set_clock
> >>>>>>>>>>>>
> >>>>>>>>>>>> arch/mips/kernel/process.c
> >>>>>>>>>>>> cpu_idle
> >>>>>>>>>>>>
> >>>>>>>>>>>> arch/mips/kernel/smtc.c
> >>>>>>>>>>>> __irq_entry
> >>>>>>>>>>>> ipi_decode
> >>>>>>>>>>>> SMTC_CLOCK_TICK
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Enclosed are the two subsets of files for a more expert look.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'll try to look in more detail after Christmas.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Stuart
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >
> >
>
