A multi-threaded version of chunkd has been checked into the
repository[1], on a branch aptly named "thread."
The design:
- uses GLib's thread pools[2]
- for each poll(2)
for each active fd
push fd to worker thread via thread pool
- worker thread
handle one unit of activity (tcp_cli_event func call)
wake up main thread, to requeue fd for further polling
return to thread pool, to seek more work
Concurrent performance (multiple TCP streams) is definitely improved,
but single stream suffers a bit, probably due to excessive wakeups.
Both single-thread and MT performance can definitely be improved with
epoll, but I wanted to get the working-across-all-platforms poll(2) code
solid first.
One possible alternative model is to leave chunkd single-threaded, yet
use aio_read() and aio_write() for file I/O. That would eliminate the
current bottleneck common to single-threaded designs: the serialization
of file input (READs) requested by multiple parallel TCP clients.
If you have multiple TCP clients issuing multiple reads at the same time
-- a common case -- multiple threads are the only way to submit I/Os to
the kernel in parallel, thereby permitting the kernel elevator and
hardware (SCSI/ATA) elevator opportunity to optimize for seek performance.
In most applications, reads are synchronous, which in the current
single-threaded chunkd becomes a visible performance wart.
Regardless... I do not currently plan on pushing the "thread" branch
onto git upstream until I explore which epoll-based design is best (ST
or MT), and perhaps tune this poll(2) design a bit more.
Feedback and contributions always welcome, of course...
Jeff
[1] git://git.kernel.org/pub/scm/daemon/distsrv/chunkd.git
[2] http://library.gnome.org/devel/glib/stable/glib-Thread-Pools.html
--
To unsubscribe from this list: send the line "unsubscribe hail-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
