bigendian+gfs@xxxxxxxxx wrote:
I've just set up a new two-node GFS cluster on a CORAID sr1520
ATA-over-Ethernet. My nodes are each quad dual-core Opteron CPU
systems with 32GB RAM each. The CORAID unit exports a 1.6TB block
device that I have a GFS file system on.
I seem to be having performance issues where certain read system calls
take up to three seconds to complete. My test app is bonnie++, and
the slow-downs appear to be happen in the "Rewriting" portion of the
test, though I'm not sure if this is exclusive. If I watch top and
iostat for the device in question, I see activity on the device, then
long (up to three second) periods of no apparent I/O. During the
periods of no I/O the bonnie++ process is blocked on disk I/O, so it
seems that the system it trying to do something. Network traces seem
to show that the host machine is not waiting on the RAID array, and
the packet following the dead-period seems to always be sent from the
host to the coraid device. Unfortunately, I don't know how to dig in
any deeper to figure out what the problem is.
I think we know about this issue. Note that bonnie++ doesn't keep the
file size within the benchmark's local memory, it always invokes a
"stat" system call to poll for the file size before it can do read and
rewrite. GFS1 has a known performance issue with stat system call (that
we hope it can be addressed by GFS2) and since file size in bonnie++
tend to be small, the stat() call overhead becomes very obvious. It will
be worse in your case due to the filesystem size.
The issue here is whether bonnie++ faithfully represents your
application ? More specifically, does your application need to do a
"stat" system call before each and every IO ? On the other hand, there
are few tunables that could help out (e.g., increasing statfs_slots via
"gfs_tool settune" command). Will work with support group to document
the detailed tuning steps into Red Hat Knowledgebase
(http://kbase.redhat.com/faq/). When they're ready, we'll post here.
-- Wendy
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
