2010/5/17 luca ellero <lroluk@xxxxxxxxx>:
> Hi list,
> I have some (maybe stupid) questions which I can't answer even after reading
> lots of documentation.
> Suppose I have a PCI device which has some I/O registers mapped to memory
> (here I mean access are made through memory, not I/O space).
> As far as I know the right way to access them is through functions such as
> iowrite8 and friends:
>
> spin_lock(Q)
> iowrite8(some_address, ADDR)
> iowrite8(some_data, DATA);
> spin_unlock(Q);
>
> My questions are:
>
> 1) Do I need a write memory barrier (wmb) between the two iowrite8?
> I think I need it because I've read the implementation of iowrite8 and (in
> kernel 2.6.30.6) this expands to:
>
> void iowrite8(u8 val, void *addr)
> {
> do {
> unsigned long port = (unsigned long )addr;
> if (port >= 0x40000UL) {
> writeb(val, addr);
> } else if (port > 0x10000UL) {
> port &= 0x0ffffUL;
> outb(val,port);
> } else bad_io_access(port, "outb(val,port)" );
> } while (0);
> }
>
> where writeb is:
>
> static inline void writeb(unsigned char val, volatile void *addr) {
> asm volatile("movb %0,%1":
> :"q" (val), "m" (*(volatile unsigned char *)addr)
> :"memory");
> }
>
> which contains only a compiler barrier (the :"memory" in the asm statement)
> but no CPU barrier. So, without wmb(), CPU can reorder the iowrite8 with
> disastrous effect. Am I right?
>
>
> 2) do I need mmiowb() before spin_unlock()?
> Documentation about mmiowb() is really confusing me, so any explanation
> about his use is really welcome.
See the documentation which explains it clearly.
http://lxr.linux.no/linux+v2.6.27.46/Documentation/memory-barriers.txt
1295LOCKS VS I/O ACCESSES
1296---------------------
1297
1298Under certain circumstances (especially involving NUMA), I/O accesses within
1299two spinlocked sections on two different CPUs may be seen as
interleaved by the
1300PCI bridge, because the PCI bridge does not necessarily participate in the
1301cache-coherence protocol, and is therefore incapable of issuing the required
1302read memory barriers.
1303
1304For example:
1305
1306 CPU 1 CPU 2
1307 =============================== ===============================
1308 spin_lock(Q)
1309 writel(0, ADDR)
1310 writel(1, DATA);
1311 spin_unlock(Q);
1312 spin_lock(Q);
1313 writel(4, ADDR);
1314 writel(5, DATA);
1315 spin_unlock(Q);
1316
1317may be seen by the PCI bridge as follows:
1318
1319 STORE *ADDR = 0, STORE *ADDR = 4, STORE *DATA = 1, STORE *DATA = 5
1320
1321which would probably cause the hardware to malfunction.
1322
1323
1324What is necessary here is to intervene with an mmiowb() before dropping the
1325spinlock, for example:
1326
1327 CPU 1 CPU 2
1328 =============================== ===============================
1329 spin_lock(Q)
1330 writel(0, ADDR)
1331 writel(1, DATA);
1332 mmiowb();
1333 spin_unlock(Q);
1334 spin_lock(Q);
1335 writel(4, ADDR);
1336 writel(5, DATA);
1337 mmiowb();
1338 spin_unlock(Q);
1339
1340this will ensure that the two stores issued on CPU 1 appear at the
PCI bridge
1341before either of the stores issued on CPU 2.
1342
1343
1344Furthermore, following a store by a load from the same device
obviates the need
1345for the mmiowb(), because the load forces the store to complete
before the load
1346is performed:
1347
1348 CPU 1 CPU 2
1349 =============================== ===============================
1350 spin_lock(Q)
1351 writel(0, ADDR)
1352 a = readl(DATA);
1353 spin_unlock(Q);
1354 spin_lock(Q);
1355 writel(4, ADDR);
1356 b = readl(DATA);
1357 spin_unlock(Q);
1358
1359
1360See Documentation/DocBook/deviceiobook.tmpl for more information.
>
> Thanks in advance
> regards
> Luca
>
> --
> To unsubscribe from this list: send an email with
> "unsubscribe kernelnewbies" to ecartis@xxxxxxxxxxxx
> Please read the FAQ at http://kernelnewbies.org/FAQ
>
>
--
Best Regards
Lin
--
To unsubscribe from this list: send an email with
"unsubscribe kernelnewbies" to ecartis@xxxxxxxxxxxx
Please read the FAQ at http://kernelnewbies.org/FAQ
