Hi, I have three machines in a NUFA arrangment. When the local disk is full no writes can be made to the gluster disk, even though there is space on the other disks?! Thanks for any help. Perhaps I have done something wrong, this is very odd. Here is what df says: tur-awc1:~> df -hl Filesystem Size Used Avail Use% Mounted on /dev/mapper/sysvg-root 7.7G 3.1G 4.3G 42% / /dev/sda1 92M 25M 62M 29% /boot tmpfs 7.9G 0 7.9G 0% /dev/shm /dev/sdb2 6.3T 3.4T 2.9T 55% /data /dev/sdb1 5.4T 2.8T 2.7T 51% /export/brick0 glusterfs 26T 12T 15T 46% /awc/data /dev/mapper/sysvg-newgluster 886G 886G 63M 100% /export/brick-newgluster glusterfs#/usr/local/glusterfs-2.0.0rc9/etc/glusterfs/glusterfsd.vol 2.8T 887G 1.9T 32% /ng Looks like plenty of room on the gluster, but my local brick is full, and I can't write to /ng. Here is the log: ================================================================================ Version : glusterfs 2.0.0git built on Apr 29 2009 11:08:01 TLA Revision : git://git.sv.gnu.org/gluster.git Starting Time: 2009-04-29 11:35:46 Command line : /usr/local/glusterfs-2.0.0rc9/sbin/glusterfsd -f /usr/local/glusterfs-2.0.0rc9/etc/glusterfs/glusterfsd.vol /ng PID : 22711 System name : Linux Nodename : tur-awc1 Kernel Release : 2.6.18-92.1.13.el5 Hardware Identifier: x86_64 Given volfile: +------------------------------------------------------------------------------+ 1: volume posix0 2: type storage/posix 3: option directory /export/brick-newgluster 4: end-volume 5: 6: volume locks0 7: type features/locks 8: subvolumes posix0 9: end-volume 10: 11: volume brick0 12: type performance/io-threads 13: subvolumes locks0 14: end-volume 15: 16: volume server 17: type protocol/server 18: option transport-type tcp 19: option listen-port 16996 20: option auth.addr.brick0.allow * 21: subvolumes brick0 22: end-volume 23: 24: volume tur-awc1-0 25: type protocol/client 26: option transport-type tcp 27: option remote-port 16996 28: option remote-host tur-awc1 29: option remote-subvolume brick0 30: option ping-timeout 120 31: end-volume 32: 33: volume tur-awc2-0 34: type protocol/client 35: option transport-type tcp 36: option remote-port 16996 37: option remote-host tur-awc2 38: option remote-subvolume brick0 39: option ping-timeout 120 40: end-volume 41: 42: volume tur-awc3-0 43: type protocol/client 44: option transport-type tcp 45: option remote-port 16996 46: option remote-host tur-awc3 47: option remote-subvolume brick0 48: option ping-timeout 120 49: end-volume 50: 51: volume nufa 52: type cluster/nufa 53: option local-volume-name tur-awc1-0 54: subvolumes tur-awc1-0 tur-awc2-0 tur-awc3-0 55: end-volume 56: 57: volume writebehind 58: type performance/write-behind 59: option cache-size 1MB 60: subvolumes nufa 61: end-volume 62: 63: # before or after writebehind? 64: volume ra 65: type performance/read-ahead 66: subvolumes writebehind 67: end-volume 68: 69: volume cache 70: type performance/io-cache 71: option cache-size 512MB 72: subvolumes ra 73: end-volume 74: +------------------------------------------------------------------------------+ 2009-04-29 11:35:46 W [xlator.c:555:validate_xlator_volume_options] server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction 2009-04-29 11:35:46 N [glusterfsd.c:1152:main] glusterfs: Successfully started 2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc2-0: connection failed (Connection refused) 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected 2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc2-0: connection failed (Connection refused) 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected 2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc1-0: connection failed (Connection refused) 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected 2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc1-0: connection failed (Connection refused) 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected 2009-04-29 11:35:46 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc3-0: connection and handshake succeeded 2009-04-29 11:35:46 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc3-0: connection and handshake succeeded 2009-04-29 11:35:51 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.35:1020 2009-04-29 11:35:51 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.35:1019 2009-04-29 11:35:52 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.28:1017 2009-04-29 11:35:52 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.28:1016 2009-04-29 11:35:57 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.129.121:1022 2009-04-29 11:35:57 W [client-protocol.c:6320:client_setvolume_cbk] tur-awc1-0: attaching to the local volume 'brick0' 2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc1-0: connection and handshake succeeded 2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc2-0: connection and handshake succeeded 2009-04-29 11:35:57 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.129.121:1016 2009-04-29 11:35:57 W [client-protocol.c:6320:client_setvolume_cbk] tur-awc1-0: attaching to the local volume 'brick0' 2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc1-0: connection and handshake succeeded 2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc2-0: connection and handshake succeeded 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device <snip> 2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device 2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device 2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device 2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device 2009-04-29 15:18:23 E [fuse-bridge.c:923:fuse_err_cbk] glusterfs-fuse: 4939521: FLUSH() ERR => -1 (No space left on device)