On Wed, Apr 20, 2022 at 7:38 PM Hao Lee <haolee.swjtu@xxxxxxxxx> wrote:
>
> On Tue, Apr 19, 2022 at 10:36:16AM -0700, Paul E. McKenney wrote:
> > On Mon, Apr 18, 2022 at 06:23:59AM +0000, Hao Lee wrote:
> > > On Sun, Apr 17, 2022 at 09:52:39AM -0700, Paul E. McKenney wrote:
> > > > On Sun, Apr 17, 2022 at 11:09:47AM +0000, Hao Lee wrote:
> > > > > CPU caches need to be flushed before DMA operations to avoid data
> > > > > overwriting.
> > > > >
> > > > > Signed-off-by: Hao Lee <haolee.swjtu@xxxxxxxxx>
> > > > > ---
> > > > > appendix/whymb/whymemorybarriers.tex | 2 +-
> > > > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/appendix/whymb/whymemorybarriers.tex b/appendix/whymb/whymemorybarriers.tex
> > > > > index b49d5fe8..4ec27962 100644
> > > > > --- a/appendix/whymb/whymemorybarriers.tex
> > > > > +++ b/appendix/whymb/whymemorybarriers.tex
> > > > > @@ -1640,7 +1640,7 @@ future such problems:
> > > > > in any DMA buffer before presenting that buffer to the
> > > > > I/O device.
> > > >
> > > > The paragraph above is before the I/O starts. Any changes to the buffer
> > > > need to be flushed to memory so that the I/O device's DMA actually
> > > > sees them.
> > > >
> > > > The next paragraph is the other end of I/O request, after it completes.
> > > > So we cannot have "before". But maybe we need "invalidate" instead of
> > > > "flush".
> > >
> > > Got it!
> > >
> > > > With a note that itcan be more efficient to both flush and
> > > > invalidate prior to presenting the DMA buffer to the device.
> > > >
> > > > The point is that any pre-DMA data in the CPU caches must be gotten
> > > > rid of in order for the CPUs to see the data that was actually DMA'ed
> > > > into memory.
> > >
> > > Great explanation! Now I know we need something like WBINVD instruction
> > > for the former case and INVD instruction for the latter.
> >
> > Glad that it helped!
> >
> > > > > Similarly, you need to flush the CPU caches of any location
> > >
> > > Should we change the word "flush" to "invalidate" to differentiate these
> > > two opposite DMA operations?
> >
> > If after the DMA, we need to care about the difference, hasn't the CPU
> > incorrectly written to part of the DMA buffer during the DMA operation?
>
> Sorry, I can't understand this passage... Maybe I didn't make myself
> clear. I find a great example in arch/powerpc/include/asm/cacheflush.h
>
> Before DMA provides output buffer to devices, we should call
> flush_dcache_range or clean_dcache_range to let the devices see the
> latest data generated by the CPU. After DMA writes data into the input
> buffer, we should use invalidate_dcache_range to invalidate the CPU
> cache so the CPU can read the latest data written by devices.
>
> Should we change the word "flush" in the following sentence to "invalidate"?
>
> Similarly, you need to flush the CPU caches of any location in
> any DMA buffer after DMA to that buffer completes.
>
Hello,
My above reply may be missed :)
Thanks,
Hao Lee
>
> > If you are thinking of some other scenario, could you please list the
> > events leading up to that scenrio?
> >
> > > > > - in any DMA buffer after DMA to that buffer completes.
> > > > > + in any DMA buffer before DMA to that buffer completes.
> > > > > And even then, you need to be \emph{very} careful to avoid
> > > > > pointer bugs, as even a misplaced read to an input buffer
> > > > > can result in corrupting the data input!
> > > >
> > > > And this last clause could just wordsmithed a bit.
> > >
> > > Maybe it's not quite rigorous?
> > >
> > > If I understand correctly, a misplaced read will load the data from
> > > input buffer area to cache line. If the cpu doesn't write to this cache
> > > line, I think this case can only result in that the cpu reads stale data
> > > in subsequent reads but won't corrupt the input buffer given that this
> > > cache line is clean and won't writeback to memory.
> >
> > If the user code that accesses the buffer runs on this same CPU, it will
> > look to that CPU as if the DMA buffer was corrupted, correct?
>
> Ah, this indeed makes sense!
>
> > After
> > all, that user code will see the value loaded by the CPU, not the value
> > actually DMAed into memory.
> >
> > Hmmm... It might be necessary to spell out these DMA-corruption
> > scenarios. Thoughts?
>
> Yes. I think spelling out these scenarios should make this passage easier to
> understand.
>
> Thanks,
> Hao Lee
>
> >
> > Thanx, Paul
