On Fri, 20 May 2011, Matt Garman wrote:
> We have several latency-sensitive "pipeline"-style programs that have
> a measurable performance degredation when run on CentOS 5.x versus
> CentOS 4.x.
>
> By "pipeline" program, I mean one that has multiple threads. The
> mutiple threads work on shared data. Between each thread, there is a
> queue. So thread A gets data, pushes into Qab, thread B pulls from
> Qab, does some processing, then pushes into Qbc, thread C pulls from
> Qbc, etc. The initial data is from the network (generated by a 3rd
> party).
>
> We basically measure the time from when the data is received to when
> the last thread performs its task. In our application, we see an
> increase of anywhere from 20 to 50 microseconds when moving from
> CentOS 4 to CentOS 5.
> Anyone have any experience with this? Perhaps some more areas to investigate?
We do procesing similar to this with financials markets
datastreams. You do not say, but I assume you are blocking on
a select, rather than polling [polling is bad here]. Also you
do not say if all threds are under a common process'
ownership. If not, mod complexity of debugging threading, you
may want to do so
I say this, because in our testing (both with all housed in a
single process, and when using co-processes fed through an
anaoymous pipe), we will occasionally get hit with a context
or process switch, which messes up the latencies something
fierce. An 'at' or 'cron' job firing off can ruin the day as
well
Also, system calls are to be avoided, as the timing on when
(and if, and in what order) one gets returned to, is not
something controllable in userspace
Average latencies are not so meaningful here ... collecton of
all dispatch and return data and explaining the outliers is
probably a good place to continue with afer addresing the
foregoing. graphviz, and gnuplot are lovely for doing this
kind of visualization
-- Russ herrold
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
