On Tue, Sep 19, 2023 at 11:54:10AM -0700, Jan Bottorff wrote:
> On 9/19/2023 7:51 AM, Catalin Marinas wrote:
> > While smp_* is ok, it really depends on what the regmap_write() does. Is
> > it a write to a shared peripheral (if not, you may need a DSB)? Does the
> > regmap_write() caller know this? That's why I think having the barrier
> > in dw_reg_write() is better.
> >
> > If you do want to stick to a fix in i2c_dw_xfer_init(), you could go for
> > dma_wmb(). While this is not strictly DMA, it's sharing data with
> > another coherent agent (a different CPU in this instance). The smp_wmb()
> > is more about communication via memory not involving I/O. But this still
> > assumes that the caller knows regmap_write() ends up with an I/O
> > write*() (potentially relaxed).
>
> If we wanted maximum correctness wouldn't we need something like
> writel_triggers_interrupt/regmap_write_triggers_interrupt or maybe
> preinterrupt_wmb?
Well, if you want to have an API for all things that can be triggered
(interrupts, device DMA), you can try but I think it would make things
more confusing and driver writers won't bother (if, say, they only test
on x86 and never see a problem). The other way around - barriers by
default and only relax if you see a performance issue - seems more
sensible. But I don't maintain these drivers, so it's up to you guys.
> The ARM docs do have a specific example case where the device write triggers
> an interrupt, and that example specifically says a DSB barrier is needed.
Yeah, the Arm ARM is not very precise here on what the mailbox is,
whether it's a local or shared peripheral and they went for the
stronger DMB. Will added a good explanation on why a DMB is sufficient
in commit 22ec71615d82 ("arm64: io: Relax implicit barriers in default
I/O accessors"). It talks about DMA but it applies equally to another
CPU accessing the memory. It's pretty subtle though.
> If I look at the ARM GIC IPI send function gic_ipi_send_mask in
> https://elixir.bootlin.com/linux/v6.6-rc2/source/drivers/irqchip/irq-gic-v3.c#L1354
> is says:
>
> /*
> * Ensure that stores to Normal memory are visible to the
> * other CPUs before issuing the IPI.
> */
> dsb(ishst);
>
> I would think the IPI send code is very carefully tuned for performance, and
> would not use a barrier any stronger than required.
That's why I mentioned in my previous reply that it really depends on
what the regmap_write() does, where the I/O go shared peripheral or some
local CPU interface). In the GIC example above, there's not even an I/O
access but a system register write (MSR, see gic_write_sgi1r()), hence
the DSB. If you look at gic_ipi_send_mask() in irq-gic.c (GICv2), there
is a dmb(ishst) since the interrupt is sent with an I/O write to the GIC
distributor (shared peripheral).
> I believe dma_wmb maps to DMB on ARM64.
Yes, it does.
--
Catalin
