On 06.11.19 23:36, Leonard Crestez wrote: > On 06.11.2019 13:59, Martin Kepplinger wrote: >> On 04.11.19 11:35, Abel Vesa wrote: >>> On 19-11-04 09:49:18, Martin Kepplinger wrote: >>>> On 30.10.19 09:08, Abel Vesa wrote: >>>>> On 19-10-30 07:11:37, Martin Kepplinger wrote: >>>>>> On 23.06.19 13:47, Martin Kepplinger wrote: >>>>>>> On 10.06.19 14:13, Abel Vesa wrote: >>>>>>>> This is another alternative for the RFC: >>>>>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml.org%2Flkml%2F2019%2F3%2F27%2F545&data=02%7C01%7Cleonard.crestez%40nxp.com%7C6ca438b3b9e44d70ac7608d762b0c030%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C637086383589318475&sdata=NyFLkQ8PUfC7PGejDK7NBJoQu36ZfaYvg9yuJvHedzo%3D&reserved=0 >>>>>>>> >>>>>>>> This new workaround proposal is a little bit more hacky but more contained >>>>>>>> since everything is done within the irq-imx-gpcv2 driver. >>>>>>>> >>>>>>>> Basically, it 'hijacks' the registered gic_raise_softirq __smp_cross_call >>>>>>>> handler and registers instead a wrapper which calls in the 'hijacked' >>>>>>>> handler, after that calling into EL3 which will take care of the actual >>>>>>>> wake up. This time, instead of expanding the PSCI ABI, we use a new vendor SIP. >>>>>>>> >>>>>>>> I also have the patches ready for TF-A but I'll hold on to them until I see if >>>>>>>> this has a chance of getting in. >>>>>>> >>>>>> >>>>>> Hi Abel, >>>>>> >>>>>> Running this workaround doesn't seem to work anymore on 5.4-rcX. Linux >>>>>> doesn't boot, with ATF unchanged (includes your workaround changes). I >>>>>> can try to add more details to this... >>>>>> >>>>> >>>>> This is happening because the system counter is now enabled on 8mq. >>>>> And since the irq-imx-gpcv2 is using as irq_set_affinity the >>>>> irq_chip_set_affinity_parent. This is because the actual implementation >>>>> of the driver relies on GIC to set the right affinity. On a SoC >>>>> that has the wake_request signales linked to the power controller this >>>>> works fine. Since the system counter is actually the tick broadcast >>>>> device and the set affinity relies only on GIC, the cores can't be >>>>> woken up by the broadcast interrupt. >>>>> >>>>>> Have you tested this for 5.4? Could you update this workaround? Please >>>>>> let me know if I missed any earlier update on this (having a cpu-sleep >>>>>> idle state). >>>>>> >>>>> >>>>> The solution is to implement the set affinity in the irq-imx-gpcv2 driver >>>>> which would allow the gpc to wake up the target core when the broadcast >>>>> irq arrives. >>>>> >>>>> I have a patch for this. I just need to clean it up a little bit. >>>>> Unfortunately, it won't go upstream since everuone thinks the gic >>>>> should be the one to control the affinity. This obviously doesn't work >>>>> on 8mq. >>>>> >>>>> Currently, I'm at ELCE in Lyon. Will get back at the office tomorrow >>>>> and sned you what I have. >>>>> >>>> >>>> Hi Abel, >>>> >>>> Do you have any news on said patch for testing? That'd be great for my >>>> plannings. >>>> >>> >>> Sorry for the late answer. >>> >>> I'm dropping here the diff. >>> >>> Please keep in mind that this is _not_ an official solution. >>> >>> --- >>> drivers/irqchip/irq-imx-gpcv2.c | 42 ++++++++++++++++++++++++++++++++++++++++- >>> 1 file changed, 41 insertions(+), 1 deletion(-) >>> >>> diff --git a/drivers/irqchip/irq-imx-gpcv2.c b/drivers/irqchip/irq-imx-gpcv2.c >>> index 01ce6f4..3150588 100644 >>> --- a/drivers/irqchip/irq-imx-gpcv2.c >>> +++ b/drivers/irqchip/irq-imx-gpcv2.c >>> @@ -41,6 +41,24 @@ static void __iomem *gpcv2_idx_to_reg(struct gpcv2_irqchip_data *cd, int i) >>> return cd->gpc_base + cd->cpu2wakeup + i * 4; >>> } >>> >>> +static void __iomem *gpcv2_idx_to_reg_cpu(struct gpcv2_irqchip_data *cd, >>> + int i, int cpu) >>> +{ >>> + u32 offset = GPC_IMR1_CORE0; >>> + switch(cpu) { >>> + case 1: >>> + offset = GPC_IMR1_CORE1; >>> + break; >>> + case 2: >>> + offset = GPC_IMR1_CORE2; >>> + break; >>> + case 3: >>> + offset = GPC_IMR1_CORE3; >>> + break; >>> + } >>> + return cd->gpc_base + offset + i * 4; >>> +} >>> + >>> static int gpcv2_wakeup_source_save(void) >>> { >>> struct gpcv2_irqchip_data *cd; >>> @@ -163,6 +181,28 @@ static void imx_gpcv2_irq_mask(struct irq_data *d) >>> irq_chip_mask_parent(d); >>> } >>> >>> +static int imx_gpcv2_irq_set_affinity(struct irq_data *d, >>> + const struct cpumask *dest, bool force) >>> +{ >>> + struct gpcv2_irqchip_data *cd = d->chip_data; >>> + void __iomem *reg; >>> + u32 val; >>> + int cpu; >>> + >>> + for_each_possible_cpu(cpu) { >>> + raw_spin_lock(&cd->rlock); >>> + reg = gpcv2_idx_to_reg_cpu(cd, d->hwirq / 32, cpu); >>> + val = readl_relaxed(reg); >>> + val |= BIT(d->hwirq % 32); >>> + if (cpumask_test_cpu(cpu, dest)) >>> + val &= ~BIT(d->hwirq % 32); >>> + writel_relaxed(val, reg); >>> + raw_spin_unlock(&cd->rlock); >>> + } >>> + >>> + return irq_chip_set_affinity_parent(d, dest, force); >>> +} >>> + >>> static struct irq_chip gpcv2_irqchip_data_chip = { >>> .name = "GPCv2", >>> .irq_eoi = irq_chip_eoi_parent, >>> @@ -172,7 +212,7 @@ static struct irq_chip gpcv2_irqchip_data_chip = { >>> .irq_retrigger = irq_chip_retrigger_hierarchy, >>> .irq_set_type = irq_chip_set_type_parent, >>> #ifdef CONFIG_SMP >>> - .irq_set_affinity = irq_chip_set_affinity_parent, >>> + .irq_set_affinity = imx_gpcv2_irq_set_affinity, >>> #endif >>> }; > > This is prone to race conditions. > > In NXP tree there is different gpcv2 irqchip driver which does all GPC > IMR register manipulation in TF-A through SMC calls. The cpuidle > workaround also manipulates the same registers and does so safely under > a lock.> > If OS also writes to same IMR register then set_affinity for SPIs 1-31 > can potentially race with one those cores being woken up. This is very > unlikely (set_affinity calls are rare) but in the worst case the system > could still hang on lost IPI. > >> I guess this diff does not apply when using this reworked change: >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsource.puri.sm%2FLibrem5%2Flinux-next%2Fcommit%2Fe59807ae0e236512761b751abc84a9b129d7fcda&data=02%7C01%7Cleonard.crestez%40nxp.com%7C6ca438b3b9e44d70ac7608d762b0c030%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C637086383589318475&sdata=Mf%2BFtqFSG4xHL3IGPrD%2FOweR8qoJHV0IKuziPIUK%2Bsw%3D&reserved=0 >> which has worked for me when running 5.3. >> >> At least on 5.4-rc5, using your change, I still get >> >> cat /sys/devices/system/cpu/cpuidle/current_driver >> none > > This reads "psci_idle" for me in linux-next on imx8mm. Your problem > seems to be related to probing the cpuidle driver, not related to any > hardware workarounds. > >> But also when trying to rewrite your patch against irq-gic-v3.c at least >> nothing changes for me (I might have done that wrong as well though). >> >> What needs to change (in order to have the cpu-sleep state / idle >> driver) based on the above "reworked" workaround? >> >> Could the config have changed? CONFIG_ARM_CPUIDLE should be the only >> needed path, or did things change there in 5.4? > > It seems there were some recent cleanups in the cpuidle psci core code, > maybe you need config updates? > > https://patchwork.kernel.org/cover/11052723/ > >> I know all this is no real solution, but currently the only way to have >> said sleep state on top of mainline. so be it for now. > Can you use the gpcv2 driver from NXP tree? hm. what driver do you mean? at least https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/irqchip/irq-imx-gpcv2.c?h=imx_4.19.35_1.0.0 seems not support imx8mq yet. > > -- > Regards, > Leonard > Hi Leonard, hi Abel, Thanks for having a look! To sum up this problem and not to get confused: We have the workaround that changes irq-imx-gpcv2 from this very email thread, to be used with mainline ATF. when applying Abel's recent diff, Linux 5.4 boots but I still don't have a cpuidle driver. When I enable CONFIG_ARM_PSCI_CPUIDLE, the kernel hangs during boot (after probing mmc, but that doesn't tell much) What do I miss? Then (in parallel) we have NXP's ATF: https://source.codeaurora.org/external/imx/imx-atf/log/?h=imx_4.19.35_1.0.0 that I test in parallel (and will actually want to have cpuidle right now too). The workaround in Linux in that case looks like so: https://source.codeaurora.org/external/imx/linux-imx/commit/?h=imx_4.19.35_1.0.0&id=26a59057f88997dfe48ab7f81898ddd6b6d3903e which changes irq-gic-v3 only. Since 5.4, also no cpu-sleep state anymore. What would need to change in that "NXP" case, for 5.4 to have cpuidle again? When I enable ARM_PSCI_CPUIDLE here, right now Linux hangs during boot (during probing sdhci but again that seems random). I'm still happy for hints :) Thanks, martin
