Hi,
I hope I can explain this properly.
1. I have a two brick system replicating each other. (10.1.4.181 and 10.1.40.2)
2. I have a third system that mounts the gluster fileshares (192.168.30.111)
3. I export the share on 192.168.30.111 as NFS to a XenServer.
What I'm hoping for, is the aggregate speeds of gluster to the 2 servers shows up when exported as NFS (roughly 2 GigE).
When I use the linux kernel nfs server, my speeds are atrocious.
When I use the gluster NFS server, I get complete failure.
The error I get when I try to use the gluster nfs server on a gluster mounted disk is:
[2011-09-13 10:38:45.265011] E [socket.c:1685:socket_connect_finish] 0-datastore-client-0: connection to 192.168.30.111:24007 failed (Connection refused)
[2011-09-13 10:39:06.742844] I [nfs.c:704:init] 0-nfs: NFS service started
[2011-09-13 10:39:06.742980] W [write-behind.c:3030:init] 0-datastore-write-behind: disabling write-behind for first 0 bytes
[2011-09-13 10:39:06.745879] I [client.c:1935:notify] 0-datastore-client-0: parent translators are ready, attempting connect on transport
Given volfile:
+------------------------------------------------------------------------------+
1: volume datastore-client-0
2: type protocol/client
3: option remote-host 192.168.30.111
4: option remote-subvolume /filer1
5: option transport-type tcp
6: end-volume
7:
8: volume datastore-write-behind
9: type performance/write-behind
10: subvolumes datastore-client-0
11: end-volume
12:
13: volume datastore-read-ahead
14: type performance/read-ahead
15: subvolumes datastore-write-behind
16: end-volume
17:
18: volume datastore-io-cache
19: type performance/io-cache
20: subvolumes datastore-read-ahead
21: end-volume
22:
23: volume datastore-quick-read
24: type performance/quick-read
25: subvolumes datastore-io-cache
26: end-volume
27:
28: volume datastore
29: type debug/io-stats
30: option latency-measurement off
31: option count-fop-hits off
32: subvolumes datastore-quick-read
33: end-volume
34:
35: volume nfs-server
36: type nfs/server
37: option nfs.dynamic-volumes on
38: option rpc-auth.addr.datastore.allow *
39: option nfs3.datastore.volume-id 1ce79bc5-0e1a-4ab9-98ba-be38166101fa
40: option nfs.port 2049
41: subvolumes datastore
42: end-volume
+------------------------------------------------------------------------------+
[2011-09-13 10:39:06.747340] I [rpc-clnt.c:1531:rpc_clnt_reconfig] 0-datastore-client-0: changing port to 24009 (from 0)
[2011-09-13 10:39:09.748308] I [client-handshake.c:1082:select_server_supported_programs] 0-datastore-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2011-09-13 10:39:09.751963] I [client-handshake.c:913:client_setvolume_cbk] 0-datastore-client-0: Connected to 192.168.30.111:24009, attached to remote volume '/filer1'.
[2011-09-13 10:39:09.758352] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-datastore-client-0: remote operation failed: No data available
[2011-09-13 10:39:09.758383] C [nfs.c:240:nfs_start_subvol_lookup_cbk] 0-nfs: Failed to lookup root: No data available
This is the 'gluster volume info' for each server:
The two backend gluster servers:
Volume Name: datastore
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 10.1.40.2:/nfs/disk3
Brick2: 10.1.4.181:/glusetr
Options Reconfigured:
nfs.port: 2049
nfs.trusted-sync: on
The server that mounts the two gluster servers and re-exports it as NFS
fstab:
192.168.30.1:/datastore /filer1 glusterfs noatime 0 0
Volume Name: datastore
Type: Distribute
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 192.168.30.111:/filer1
Options Reconfigured:
nfs.port: 2049
A rough diagram of the network:
brick1 brick2 (physical machines) (machine 1 and 2)
| |
| (a) | (b)
| |
-----||-----
||
||(c)
||
gluster client (XenServer VM) (machine 3 - virtual)
|
|(d)
|
NFS (Same XenServer VM as above) (machine 3 - virtual)
|
|(e)
|
XenServer (mounts NFS export from VM)
Where:
(a) 1 GigE TCP/IP (physical NIC) (10.1.4.181)
(b) 1 GigE TCP/IP (physical NIC) (10.1.40.2)
(c) aggregate ~ 2GiGE (2 physical NICs)
(d) aggregate ~ 2GigE (XenServer virtual NIC) (192.168.30.111)
(e) aggregate ~ 2GigE (XenServer virtual NIC)
I know I'll get a performance hitting going through multiple layers/machines, but I'm hoping the aggregate throughput offsets that.
Does anyone know if gluster can export a gluster mounted fileshare?
Thanks,
Gerald
