On Dec 3, 2014, at 10:56 AM, Tejun Heo <tj@xxxxxxxxxx> wrote: > Hello, Neil, Jeff. > > On Tue, Dec 02, 2014 at 08:29:46PM -0500, Jeff Layton wrote: >> That's a good point. I had originally thought that max_active on an >> unbound workqueue would be the number of concurrent jobs that could run >> across all the CPUs, but now that I look I'm not sure that's really >> the case. > > @max_active is a per-pool number. By default, unbound wqs use > per-node pools, so @max_active would be per-node. Currently, > @max_active is mostly meant as a protection against run-away > workqueues creating crazy number of workers, which has been enough for > the existing wq users. *Maybe* it makes sense to make it actually > mean maximum concurrency which would prolly involve aggregated per-cpu > distribution mechanism so that we don't end up inc'ing and dec'ing the > same counter from all CPUs on each work item execution. > > However, I do agree with Neil that making it user configurable is > almost always painful. It's usually a question without a good answer > and the same value may behave differently depending on a lot of > implementation details and a better approach, probably, is to use > @max_active as the last resort protection mechanism while providing > automatic throttling of in-flight work items which is meaningful for > the specific use cases. > >> I've heard random grumblings from various people in the past that >> workqueues have significant latency, but this is the first time I've >> really hit it in practice. If we can get this fixed, then that may be a >> significant perf win for all workqueue users. For instance, rpciod in >> the NFS client is all workqueue-based. Getting that latency down could >> really help things. >> >> I'm currently trying to roll up a kernel module for benchmarking the >> workqueue dispatching code in the hopes that we can use that to help >> nail it down. > > Definitely, there were some reportings but nothing really got tracked > down properly. It'd be awesome to actually find out where the latency > is coming from. I’ve measured some long (>10usec) latencies for queue_workqueue() and wake_up_bit() calls with CPU-intensive workloads. Often these APIs are invoked while one or more spinlocks are held. That makes it easy to back up a lot of work if one of these calls is slow for any reason. For wake_up_bit(), this doesn’t include how long it takes before the awoken process is run. With CPU-intensive workloads, it’s often the case that hundreds of usecs elapse, and the awoken process is migrated to another CPU (observed via ftrace). -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
