On Tue, Jan 27, 2015 at 11:08:46PM -0500, Steven Rostedt wrote: > On Fri, 23 Jan 2015 20:03:41 -0600 > Josh Cartwright <joshc@xxxxxx> wrote: > > > Hey folks- > > > > We've recently undertaken an upgrade of our kernel from 3.2-rt to > > 3.14-rt, and have run into a performance regression on our ARM boards. > > We're still in the process of trying to isolate what we can, but > > hopefully someone's already run into this and has a solution or might > > have some useful debugging ideas. > > > > The first test we did was to run cyclictest[1] for comparison: > > > > 3.2.35-rt52 > > # Total: 312028761 312028761 624057522 > > # Min Latencies: 00010 00011 > > # Avg Latencies: 00018 00020 > > # Max Latencies: 00062 00066 00066 > > # Histogram Overflows: 00000 00000 00000 > > > > 3.14.25-rt22 > > # Total: 304735655 304735657 609471312 > > # Min Latencies: 00013 00013 > > # Avg Latencies: 00023 00024 > > # Max Latencies: 00086 00083 00086 > > # Histogram Overflows: 00000 00000 00000 > > > > I'm curious if the vanilla kernels (non-rt) show the same regression. > > Max latencies for vanilla kernels will probably go through the roof, > but the min and average should give you some hint. Yes, it's likely a non-rt related problem. I'll be running a test overnight comparing min/avg latencies on 3.2-rt vs 3.14-rt, both built without PREEMPT_RT_FULL. We'll get a test going on stable (non-rt) going over the next couple days. In parallel we're working to do a bisection, but this is a difficult task given a dependence on a ARM vendor tree. We'll see where this gets us. Thanks! Josh -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
