Hi all. On the powerpc arch, there is no ioremap_cache() version. Could someone to explain why? Thanks -----Original Message----- From: kernelnewbies-bounce@xxxxxxxxxxxx [mailto:kernelnewbies-bounce@xxxxxxxxxxxx] On Behalf Of Jason Nymble Sent: Friday, September 25, 2009 6:24 PM To: kernelnewbies@xxxxxxxxxxxx Subject: Re: interrupt/tasklet issue in custom driver on recent kernels For posterity, I've finally solved this issue. It ended up having nothing to do with the interrupts/tasklets themselves. The driver uses ioremap() to get hold of some reserved memory, and it seems from about 2.6.25 onwards or so this defaults to ioremap_nocache(), so our driver was doing memory operations in the tasklet on uncacheable pages. Calling ioremap_cache() explicitly in the driver solved the issue (nice when you can fix a perf regression of 50-100x with a single line fix!). Oprofile was of tremendous help in solving this issue. On 09 Sep 2009, at 10:39 AM, Jason Nymble wrote: > Hi, > > Background: We use a custom kernel driver module for our PCIe device > which processes bulk data between the host and the card. The card > issues MSI interrupts at up to 20kHz to the host, and the driver > interrupt routine essentially just calls tasklet_schedule() and > returns IRQ_HANDLED, and the work is performed inside the tasklet > routine. This has worked very well for us for the past several > years, with acceptably low overhead on the processor servicing the > interrupts and running the tasklet, using Linux kernel versions from > about 2.6.13 to 2.6.24. > > Recent tests on kernels from 2.6.25 to 2.6.30 indicate some serious > regression however. The CPU core servicing the interrupts/tasklets > shows 100% si usage in top for ksoftirqd, and the driver can > consequently only handle a very small fraction of what it was able > to handle using kernel <=2.6.24 (slowdown of around 50-100x)... Even > when we scale back our interrupt rate to 1kHz, we still see this > poor behavior, and from what we can tell the time isn't actually > spent in our tasklet code itself (not 100% sure of this). > > The question is, does anybody know of something that has changed in > kernels >= 2.6.25 that might cause this behavior? I've pored over > changelogs and lwn.net articles and lwn.net kernel API change lists > and kernelnewbies kernel change webpage etc., and cannot find > anything which could explain my phenomenon. > > Any suggestions for ways to track down where the problem lies? I've > tried running kernels with all the debugging+sanity checks enabled, > and they don't report any badness in the driver. My next step is to > get oprofile going and try to determine exactly where that time is > spent. I would _maybe_ have believed it could perhaps be a Linux > kernel bug (e.g. the softirq that handles tasklets somehow not > ending its loop or something) if it only happened on one kernel > version, but it seems to happen on all kernels from 2.6.25 onwards ... > > Thanks in advance -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecartis@xxxxxxxxxxxx Please read the FAQ at http://kernelnewbies.org/FAQ The information in this e-mail transmission contains proprietary and business sensitive information. Unauthorized interception of this e-mail may constitute a violation of law. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. You are also asked to contact the sender by reply email and immediately destroy all copies of the original message. -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecartis@xxxxxxxxxxxx Please read the FAQ at http://kernelnewbies.org/FAQ
