OK to answer my own question, the problem was due to me writing a really big file. there was space on the local brick at the start of the write, but I filled that brick up during the write. No problem making new files. Matt On Wed, Apr 29, 2009 at 03:34:06PM +1200, m.c.wilkins at massey.ac.nz wrote: > > > Hi, > > I have three machines in a NUFA arrangment. When the local disk is > full no writes can be made to the gluster disk, even though there is > space on the other disks?! Thanks for any help. > > Perhaps I have done something wrong, this is very odd. Here is what > df says: > > tur-awc1:~> df -hl > Filesystem Size Used Avail Use% Mounted on > /dev/mapper/sysvg-root > 7.7G 3.1G 4.3G 42% / > /dev/sda1 92M 25M 62M 29% /boot > tmpfs 7.9G 0 7.9G 0% /dev/shm > /dev/sdb2 6.3T 3.4T 2.9T 55% /data > /dev/sdb1 5.4T 2.8T 2.7T 51% /export/brick0 > glusterfs 26T 12T 15T 46% /awc/data > /dev/mapper/sysvg-newgluster > 886G 886G 63M 100% /export/brick-newgluster > glusterfs#/usr/local/glusterfs-2.0.0rc9/etc/glusterfs/glusterfsd.vol > 2.8T 887G 1.9T 32% /ng > > Looks like plenty of room on the gluster, but my local brick is full, > and I can't write to /ng. Here is the log: > > > ================================================================================ > Version : glusterfs 2.0.0git built on Apr 29 2009 11:08:01 > TLA Revision : git://git.sv.gnu.org/gluster.git > Starting Time: 2009-04-29 11:35:46 > Command line : /usr/local/glusterfs-2.0.0rc9/sbin/glusterfsd -f /usr/local/glusterfs-2.0.0rc9/etc/glusterfs/glusterfsd.vol /ng > PID : 22711 > System name : Linux > Nodename : tur-awc1 > Kernel Release : 2.6.18-92.1.13.el5 > Hardware Identifier: x86_64 > > Given volfile: > +------------------------------------------------------------------------------+ > 1: volume posix0 > 2: type storage/posix > 3: option directory /export/brick-newgluster > 4: end-volume > 5: > 6: volume locks0 > 7: type features/locks > 8: subvolumes posix0 > 9: end-volume > 10: > 11: volume brick0 > 12: type performance/io-threads > 13: subvolumes locks0 > 14: end-volume > 15: > 16: volume server > 17: type protocol/server > 18: option transport-type tcp > 19: option listen-port 16996 > 20: option auth.addr.brick0.allow * > 21: subvolumes brick0 > 22: end-volume > 23: > 24: volume tur-awc1-0 > 25: type protocol/client > 26: option transport-type tcp > 27: option remote-port 16996 > 28: option remote-host tur-awc1 > 29: option remote-subvolume brick0 > 30: option ping-timeout 120 > 31: end-volume > 32: > 33: volume tur-awc2-0 > 34: type protocol/client > 35: option transport-type tcp > 36: option remote-port 16996 > 37: option remote-host tur-awc2 > 38: option remote-subvolume brick0 > 39: option ping-timeout 120 > 40: end-volume > 41: > 42: volume tur-awc3-0 > 43: type protocol/client > 44: option transport-type tcp > 45: option remote-port 16996 > 46: option remote-host tur-awc3 > 47: option remote-subvolume brick0 > 48: option ping-timeout 120 > 49: end-volume > 50: > 51: volume nufa > 52: type cluster/nufa > 53: option local-volume-name tur-awc1-0 > 54: subvolumes tur-awc1-0 tur-awc2-0 tur-awc3-0 > 55: end-volume > 56: > 57: volume writebehind > 58: type performance/write-behind > 59: option cache-size 1MB > 60: subvolumes nufa > 61: end-volume > 62: > 63: # before or after writebehind? > 64: volume ra > 65: type performance/read-ahead > 66: subvolumes writebehind > 67: end-volume > 68: > 69: volume cache > 70: type performance/io-cache > 71: option cache-size 512MB > 72: subvolumes ra > 73: end-volume > 74: > > +------------------------------------------------------------------------------+ > 2009-04-29 11:35:46 W [xlator.c:555:validate_xlator_volume_options] server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction > 2009-04-29 11:35:46 N [glusterfsd.c:1152:main] glusterfs: Successfully started > 2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc2-0: connection failed (Connection refused) > 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected > 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected > 2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc2-0: connection failed (Connection refused) > 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected > 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected > 2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc1-0: connection failed (Connection refused) > 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected > 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected > 2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc1-0: connection failed (Connection refused) > 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected > 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected > 2009-04-29 11:35:46 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc3-0: connection and handshake succeeded > 2009-04-29 11:35:46 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc3-0: connection and handshake succeeded > 2009-04-29 11:35:51 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.35:1020 > 2009-04-29 11:35:51 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.35:1019 > 2009-04-29 11:35:52 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.28:1017 > 2009-04-29 11:35:52 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.28:1016 > 2009-04-29 11:35:57 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.129.121:1022 > 2009-04-29 11:35:57 W [client-protocol.c:6320:client_setvolume_cbk] tur-awc1-0: attaching to the local volume 'brick0' > 2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc1-0: connection and handshake succeeded > 2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc2-0: connection and handshake succeeded > 2009-04-29 11:35:57 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.129.121:1016 > 2009-04-29 11:35:57 W [client-protocol.c:6320:client_setvolume_cbk] tur-awc1-0: attaching to the local volume 'brick0' > 2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc1-0: connection and handshake succeeded > 2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc2-0: connection and handshake succeeded > 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device > 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device > 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device > 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device > 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device > 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device > 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device > 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device > <snip> > 2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device > 2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device > 2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device > 2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device > 2009-04-29 15:18:23 E [fuse-bridge.c:923:fuse_err_cbk] glusterfs-fuse: 4939521: FLUSH() ERR => -1 (No space left on device) > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
