I've been digging into a seemingly difficult performance issues over the last few days. We're running glusterfs mainline 2.5 patch 628, fuse-2.7.2 with Marian's glfs patch, kernel 2.6.23, currently one server and two clients (soon to be two and four, respectively). The server is a dual-core Opteron with a SATA2 disk (one, we're planning on AFR redundancy), the clients are dual-core Intel machines. The network transport is gigabit ethernet. The server is 32-bit and the clients are 64-bit (I can rebuild the server no problem if that is the issue). Throughput is good, and activity by one process seems to work fine. Our issue is with a PHP script running on the client via the glusterfs share. The script has a number of includes, and those files have a few more includes. This means a lot of stats as the webserver checks to make sure none of the files have changed. If we make one call to the script, everything is fine - the code completes in 300ms. Similarly, if you run "ls -l" on a large directory (1700 files) everything appears to work fine (from local disk the code completes in 100ms). However, if we make two concurrent calls to the PHP script, or run two copies of ls -l on the large directory, everything slows down by an order of magnitude. The output of the ls commands appears to stutter on each copy - usually one will stop and the other will start, but sometimes both will stop for a second or two. Adding a third process makes it worse. The PHP script takes 2.5 or 3 seconds to complete, instead of 300ms, and again more requests makes it worse - if you request four operations concurrently, the finish time jumps to 7 seconds. This issue occurs whether you are on a single client with two processes, or if you are on two clients with one process each. Inserting the trace translator doesn't turn up anything unusual that I can see, with the exception that it makes the processes run even slower (which is expected, of course). A tcpdump of the filesystem traffic shows inexplicable gaps of 100ms or more with no traffic. The single process "ls -l" test does not show these gaps. I stripped the server and client to the bare minimum with unify. This didn't seem to make a difference. I'm currently running this server/client stack, also without success: ns brick (x2) posix-locks io-threads(16, 64MB) server (ns, brick1, brick2) brick1 brick2 unify(alu) io-threads(16, 64MB) io-cache(256MB) At various times I've tried read-ahead with no discernable difference. An strace of the client process doesn't return anything interesting except a lot of these: futex(0x12345678, FUTEX_WAIT, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable) These also appear during a single process test, but they are much more prevalent when two processes are running. What am I doing wrong? :)
