On Tue, Aug 25, 2015 at 1:41 PM, Junio C Hamano <gitster@xxxxxxxxx> wrote: > Jeff King <peff@xxxxxxxx> writes: > >> On Tue, Aug 25, 2015 at 10:28:25AM -0700, Stefan Beller wrote: >> >>> By treating each object as its own task the workflow is easier to follow >>> as the function used in the worker threads doesn't need any control logic >>> any more. >> >> Have you tried running t/perf/p5302 on this? >> >> I seem to get a pretty consistent 2%-ish slowdown, both against git.git >> and linux.git. That's not a lot, but I'm wondering if there is some >> low-hanging fruit in the locking, or in the pattern of work being >> dispatched. Or it may just be noise, but it seems fairly consistent. > > The pattern of work dispatch hopefully is the same, no? add_task() > does the "append at the end" thing and next_task() picks from the > front of the queue. The original is "we have globally N things, > so far M things have been handled, and we want a new one, so we pick > the M+1th one and do it". > > The amount of memory that is used to represent a single task may be > much larger than the original, with overhead coming from job_list > structure and the doubly-linked list. We may not be able to spin up > 30 threads and throw a million tasks at them using this, because of > the overhead. I thought about making the add_task block after a certain size of job_list such that there is always just enough tasks available to process. Then we would not need to used a huge amount of memory for having a long line of waiting tasks; the shorter the line is the less we can tolerate burst behavior in the threads. > It would be more suited to handle a pattern in which > an overlord actively creates new tasks while worker threads chew > them, using the add_task/dispatch as the medium for communication > between them. Not sure I follow there. Original implementation: We have M threads sitting around the table, all of them trying to obtain food from the one bowl on the table and then eating it. Once the bowl is all eaten, we can stop. New pattern: One cook puts all the food items on the sushi-go-round-belt with a fancy plate and the threads grab them one by one still using locks (but they are internal to the belt). Are you saying we're content with just a bowl and everyone helps themselves for getting food? -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html