On 01/15/13 07:06, glusterzhxue wrote: > About the following issue, anyone could help us? Why using VM mounted by > NFS V3(which is also provided by gluster) could approach full > bandwidth(100MB/s),while VM mounted by gluster only reach half of > bandwidth (50MB/s). > The following is the mount command: > 1)gluster native client: > mount -t glusterfs server1:/test-volume /var/lib/nova/instance/ > 2) NFS V3: > mount -t nfs -o vers=3 server1:/test-volume /var/lib/nova/instance/ > /var/lib/nova/instance/ is the file directory storing VMs > Thanks. > ------------------------------------------------------------------------ > zhxue What is your volume layout? The output of "gluster volume info test-volume" What type of network hardware do you have? What type of ethernet ports on the host running the VMs? What type of ethernet ports on the hosts containing the storage (the gluster servers)? How are they configured? My configuration has 4 1Gb ports LACP bonded on the VM host and single 10Gb ports on the the storage nodes (gluster servers.) Our volume is: (4 bricks per gluster server) Volume Name: vol01 Type: Distributed-Replicate Volume ID: 5ebcaed8-dd9b-4c48-be75-46457fe2e978 Status: Started Number of Bricks: 24 x 2 = 48 Transport-type: tcp With the iozone testing on a FUSE mount, I am currently doing, I get: ~100MB/s writes if the replicated bricks chosen hash out in the LACP bonding to the different 1Gb ethernet ports. ~50MB/s writes if the replicated bricks chosen hash out in the LACP bonding to the same 1Gb ethernet ports. ~20MB/s writes for the test using the O_SYNC option for iozone, no matter what 1Gb ethernet ports end up being used. Reads are always ~100MB/s as it comes from one brick. If you have a replica volume, then mount using FUSE, the writes are duplicated on the client. If you mount via NFS, the writes are duplicated on the server. Also of note, with the iozone going full speed, the FUSE client takes 50% of a CPU core. -- Mr. Flibble King of the Potato People
