On 01/04/11 05:02, Anoop P A wrote:
On Tue, 2011-01-04 at 00:17 -0800, Kevin D. Kissell wrote:
Those interrupt counters show that IPIs are being taken everywhere,
though very few by CPUs 5 and 6. If I understand the configuration
correctly, CPU 4 is a TC in VPE 1, and it's getting a reasonable IPI
Yes CPU4 is in second VPE
rate, *if* we're looking at a tickless kernel under low load. But there
No it was not the tickless kernel.I had selected 250 MHz timer. can't we
expect IPI / timer interrupt for all the threads in this case ?.
In that case, you should expect a distribution of timer interrupts that
favors the low-numbered TCs within the VPE, as you do in VPE0, and a
distribution of IPIs that is sort-of the inverse, as you do in VPE0.
But the low counts on VPE1 are indeed suspicious, as you note.
may be a clue there to part of your problem. I have no idea why the
behavior would have changed from 2.6.36 to 2.6.37, but it looks as if
you're getting your clock interrupts through the MSP CIC interrupt
controller on VPE 0. There's nothing symmetric for VPE1. The Malta
example code is perhaps deceptively simple, in that both VPEs have their
count/compare indication wired directly to the 2 clock interrupt inputs,
so that having both of them running with only a single set of irq state
just works. I don't know whether the MSP CIC timer interrupt is a
In my case it is separate irq. MSP_INT_VPE1_TIMER (34) and
MSP_INT_VPE0_TIMER (25) are wired to CIC . CIC interrupt has been
connected to cpu irq 6.
I can reproduce cpu stall in VSMP mode If I don't setup VPE1 timer
interrupt . Don't we have support for separate irq in SMTC
implementation ?..
There are hooks for platform-specific SMTC support, which is implemented
for the Malta in arch/mips/mti-malta/malta-smtc.c. See
msmtc_init_secondary(), for example, where the clock/compare, profile,
and IPI interrupts are armed for VPE 1, while I/O peripheral interrupts
are inhibited.
gating of the VPE0 count/compare output, or whether it's it's own
interval timer, but I suspect that you may need to do some further
low-level initialization in the platform-specific code to set up an
interrupt on the VPE1 side. I don't think the snippet you've got below
would work as written.
The routine which I copied works fine for VSMP mode .
/ # cat /proc/interrupts
CPU0 CPU1
0: 187 254 MIPS IPI_resched
1: 77 174 MIPS IPI_call
6: 0 0 MIPS MSP CIC cascade
8: 0 0 MSP_CIC Softreset button
9: 0 0 MSP_CIC Standby switch
21: 0 0 MSP_CIC MSP PER cascade
25: 37077 0 MSP_CIC timer
27: 188 0 MSP_CIC serial
34: 0 36986 MSP_CIC timer
Do I want to change anything specific for SMTC ? .
If it works (which I doubt), then we can critique stylistic points like
using
if ((1==get_current_vpe())
Instead of the more readable and general
if (get_current_vpe()> 0)
But I think you're generally looking in the wrong place. Look at the
Malta code and see what's done where. The initial SMTC code had a lot
of Malta assumptions in the main line that I pushed out to platform code
in later patches. I can see how things could be made even more modular,
but for the moment I think it's just that there's some stuff that ought
to be done in a "msp_smtc.c" file that doesn't exist in 2.6.37.
Regards,
Kevin K.
If it's purely an issue with clock distribution on VPE1, then a boot
with maxvpes=1 maxtcs=4 should be stable.
Yes the kernel seems to be stable if I boot with maxvpes=1 maxtcs=4 .
/K.
On 1/3/2011 11:20 AM, Anoop P A wrote:
Hi Kevin,
On Mon, 2011-01-03 at 08:14 -0800, Kevin D. Kissell wrote:
The very first SMTC implementations didn't support full kernel-mode
preemption, which anyway wasn't a priority, given the hardware event
response support in MIPS MT. I believe it was later made compatible,
but it was never extensively exercised. Since SMTC has fingers in some
pretty low-level atomicity mechanisms, if a new, parallel set was
implemented for RCU, I can easily imagine that nobody has yet
implemented SMTC-ified variants of that set.
Your last statement isn't very clear, though. Are you saying that if
you configure for no forced preemption and with TREE_CPU, the 2.6.37
kernel boots all the way up, or that it simply hangs later? What's the
last rev kernel that actually boots all the way up?
I have debugged this a bit more. It seems that kernel getting stalled
while executing on TC's of second VPE .
INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
by 1, t=2504 jiffies)
INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
by 1, t=10036 jiffies)
INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
by 1, t=17568 jiffies)
INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
by 1, t=25100 jiffies)
INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
by 1, t=32632 jiffies)
INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
by 1, t=40164 jiffies)
With CONFIG_TREE_CPU we were not hitting this scenario very often.
However with CONFIG_PREEMPT_TREE_CPU stall happens most of the time.
I presume some issue in my timer setup . I am not seeing timer interrupt
(or IPI interrupt) getting incremented for VPE1 tcs on a completely
booted 2.6.32-stable kernel.
/ # cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
CPU6
1: 148 15023 15140 15093 3779 8
2 MIPS SMTC_IPI
6: 0 0 0 0 0 0
0 MIPS MSP CIC cascade
8: 0 0 0 0 0 0
0 MSP_CIC Softreset button
9: 0 0 0 0 0 0
0 MSP_CIC Standby switch
21: 0 0 0 0 0 0
0 MSP_CIC MSP PER cascade
25: 15113 341 4 7 0 0
0 MSP_CIC timer
27: 260 9 0 1 0 0
0 MSP_CIC serial
34: 0 0 0 0 0 0
0 MSP_CIC timer
Can't we use separate timer interrupts for VPE1 and VPE0 in SMTC ?.
I have tried setting up VPE1 timer from get_co_compare_int as follows
unsigned int __cpuinit get_c0_compare_int(void)
{
if ((1==get_current_vpe())&& !vpe1_timr_installed){
memcpy(&timer_vpe1,&c0_compare_irqaction,sizeof(timer_vpe1));
setup_irq(MSP_INT_VPE1_TIMER,&timer_vpe1);
vpe1_timr_installed++;
}
return (get_current_vpe() ? MSP_INT_VPE1_TIMER :
MSP_INT_VPE0_TIMER);
}
Thanks
Anoop
