2010/5/17 luca ellero <lroluk@xxxxxxxxx>: > Hi list, > I have some (maybe stupid) questions which I can't answer even after reading > lots of documentation. > Suppose I have a PCI device which has some I/O registers mapped to memory > (here I mean access are made through memory, not I/O space). > As far as I know the right way to access them is through functions such as > iowrite8 and friends: > > spin_lock(Q) > iowrite8(some_address, ADDR) > iowrite8(some_data, DATA); > spin_unlock(Q); > > My questions are: > > 1) Do I need a write memory barrier (wmb) between the two iowrite8? > I think I need it because I've read the implementation of iowrite8 and (in > kernel 2.6.30.6) this expands to: > > void iowrite8(u8 val, void *addr) > { > do { > unsigned long port = (unsigned long )addr; > if (port >= 0x40000UL) { > writeb(val, addr); > } else if (port > 0x10000UL) { > port &= 0x0ffffUL; > outb(val,port); > } else bad_io_access(port, "outb(val,port)" ); > } while (0); > } > > where writeb is: > > static inline void writeb(unsigned char val, volatile void *addr) { > asm volatile("movb %0,%1": > :"q" (val), "m" (*(volatile unsigned char *)addr) > :"memory"); > } > > which contains only a compiler barrier (the :"memory" in the asm statement) > but no CPU barrier. So, without wmb(), CPU can reorder the iowrite8 with > disastrous effect. Am I right? > > > 2) do I need mmiowb() before spin_unlock()? > Documentation about mmiowb() is really confusing me, so any explanation > about his use is really welcome. See the documentation which explains it clearly. http://lxr.linux.no/linux+v2.6.27.46/Documentation/memory-barriers.txt 1295LOCKS VS I/O ACCESSES 1296--------------------- 1297 1298Under certain circumstances (especially involving NUMA), I/O accesses within 1299two spinlocked sections on two different CPUs may be seen as interleaved by the 1300PCI bridge, because the PCI bridge does not necessarily participate in the 1301cache-coherence protocol, and is therefore incapable of issuing the required 1302read memory barriers. 1303 1304For example: 1305 1306 CPU 1 CPU 2 1307 =============================== =============================== 1308 spin_lock(Q) 1309 writel(0, ADDR) 1310 writel(1, DATA); 1311 spin_unlock(Q); 1312 spin_lock(Q); 1313 writel(4, ADDR); 1314 writel(5, DATA); 1315 spin_unlock(Q); 1316 1317may be seen by the PCI bridge as follows: 1318 1319 STORE *ADDR = 0, STORE *ADDR = 4, STORE *DATA = 1, STORE *DATA = 5 1320 1321which would probably cause the hardware to malfunction. 1322 1323 1324What is necessary here is to intervene with an mmiowb() before dropping the 1325spinlock, for example: 1326 1327 CPU 1 CPU 2 1328 =============================== =============================== 1329 spin_lock(Q) 1330 writel(0, ADDR) 1331 writel(1, DATA); 1332 mmiowb(); 1333 spin_unlock(Q); 1334 spin_lock(Q); 1335 writel(4, ADDR); 1336 writel(5, DATA); 1337 mmiowb(); 1338 spin_unlock(Q); 1339 1340this will ensure that the two stores issued on CPU 1 appear at the PCI bridge 1341before either of the stores issued on CPU 2. 1342 1343 1344Furthermore, following a store by a load from the same device obviates the need 1345for the mmiowb(), because the load forces the store to complete before the load 1346is performed: 1347 1348 CPU 1 CPU 2 1349 =============================== =============================== 1350 spin_lock(Q) 1351 writel(0, ADDR) 1352 a = readl(DATA); 1353 spin_unlock(Q); 1354 spin_lock(Q); 1355 writel(4, ADDR); 1356 b = readl(DATA); 1357 spin_unlock(Q); 1358 1359 1360See Documentation/DocBook/deviceiobook.tmpl for more information. > > Thanks in advance > regards > Luca > > -- > To unsubscribe from this list: send an email with > "unsubscribe kernelnewbies" to ecartis@xxxxxxxxxxxx > Please read the FAQ at http://kernelnewbies.org/FAQ > > -- Best Regards Lin -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecartis@xxxxxxxxxxxx Please read the FAQ at http://kernelnewbies.org/FAQ