On 01/17/2014 07:48 PM, Pat Haley wrote: > > Hi Franco, > > I checked using df -i on all 3 bricks. No brick is over > 1% inode usage. > It might be worth a quick inode allocation test on the fs for each brick, regardless. There are other non-obvious scenarios that can cause inode allocation to fail, at least on xfs (i.e., contiguous block allocation). Ideally, you'll have the ability to do this in a subdirectory outside the actual glusterfs brick. Brian > Thanks. > > Pat > >> Have you run out of inodes on the underlying filesystems? >> >> On 18 Jan 2014 05:41, Pat Haley <phaley@xxxxxxx> wrote: >> >> Latest updates: >> >> no error messages were found on the log files of the bricks. >> >> The error messages appear on the client log files. Writing >> from a second client also has the same errors. >> >> Note that if I try to write a directory with 480 6Mb files >> to /projects, over 3/4 of the files are written. It is >> random which files are not written (i.e. it is not the >> last 1/4 of the files which fail) >> >>> >>> Hi, >>> >>> Some additional data >>> >>> >>> [root@mseas-data save]# gluster volume info >>> >>> Volume Name: gdata >>> Type: Distribute >>> Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d >>> Status: Started >>> Number of Bricks: 3 >>> Transport-type: tcp >>> Bricks: >>> Brick1: gluster-0-0:/mseas-data-0-0 >>> Brick2: gluster-0-1:/mseas-data-0-1 >>> Brick3: gluster-data:/data >>> >>> >>> >>> [root@mseas-data save]# gluster volume status >>> Status of volume: gdata >>> Gluster process Port >> Online Pid >>> >> ------------------------------------------------------------------------------ >> >>> >>> Brick gluster-0-0:/mseas-data-0-0 24009 Y 27006 >>> Brick gluster-0-1:/mseas-data-0-1 24009 >>> Y 7063 >>> Brick gluster-data:/data 24010 >>> N 8007 >>> NFS Server on localhost 38467 >>> Y 8013 >>> NFS Server on gluster-0-1 38467 Y 10228 >>> NFS Server on 10.1.1.10 38467 >>> Y 3867 >>> >>> >>> Noticing that the brick gluster-data:/data was appearing as N >>> in the "online" column, I tried (1) detaching gluster-data (using >>> gluster peer detach gluster-data issued from gluster-0-0), (2) removing >>> /var/lib/glusterd, (3) restarting glusterd on gluster-data, >>> (4) reattaching /gluster-data (using gluster peer probe gluster-data >>> issued from gluster-0-0) then (5) restart glusterd one more time on all >>> 3 bricks. The brick gluster-data:/data still appears as N in the >>> Online >>> column. >>> >>> [root@mseas-data save]# gluster peer status >>> Number of Peers: 2 >>> >>> Hostname: gluster-0-1 >>> Uuid: 393fc4a6-1573-4564-971e-1b1aec434167 >>> State: Peer in Cluster (Connected) >>> >>> Hostname: 10.1.1.10 >>> Uuid: 3619440a-4ca3-4151-b62e-d4d6bf2e0c03 >>> State: Peer in Cluster (Connected) >>> >>> (similarly from the other bricks) >>> >>> Ping works between all bricks too. >>> >>> > >>> > Hi, >>> > >>> > We are using gluster to present 3 bricks as a single name space. >>> > We appear to have a situation in which gluster thinks there >>> > is no disk space when there is actually plenty. I have restarted >>> > the glusterd deamons on all three bricks and I still get the >>> > following message >>> > >>> > /bin/cp: cannot create regular file >>> > `./Bottom_Gravity_Current_25/344.mat': No space left on device >>> > >>> > This is a 6Mbyte file. The total space available on >>> > gluster is 3.6T >>> > >>> > Filesystem Size Used Avail Use% Mounted on >>> > mseas-data:/gdata 55T 51T 3.6T 94% /gdata >>> > >>> > Also, no single brick is full: >>> > >>> > Filesystem Size Used Avail Use% Mounted on >>> > /dev/mapper/the_raid-lv_data >>> > 15T 14T 804G 95% /data >>> > >>> > Filesystem Size Used Avail Use% Mounted on >>> > /dev/sdb1 21T 18T 2.1T 90% /mseas-data-0-0 >>> > >>> > Filesystem Size Used Avail Use% Mounted on >>> > /dev/sdb1 21T 20T 784G 97% /mseas-data-0-1 >>> > >>> > What should we do to fix this problem or look at to diagnose >>> > this problem? >>> > >>> > Thanks. >>> > >>> > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>> > Pat Haley Email: phaley@xxxxxxx >>> > Center for Ocean Engineering Phone: (617) 253-6824 >>> > Dept. of Mechanical Engineering Fax: (617) 253-8125 >>> > MIT, Room 5-213 http://web.mit.edu/phaley/www/ >>> > 77 Massachusetts Avenue >>> > Cambridge, MA 02139-4301 >>> > _______________________________________________ >>> > Gluster-users mailing list >>> > Gluster-users@xxxxxxxxxxx >>> > http://supercolony.gluster.org/mailman/listinfo/gluster-users >>> >>> >> >> >> -- >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley Email: phaley@xxxxxxx >> Center for Ocean Engineering Phone: (617) 253-6824 >> Dept. of Mechanical Engineering Fax: (617) 253-8125 >> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users@xxxxxxxxxxx >> http://supercolony.gluster.org/mailman/listinfo/gluster-users >> >> ------------------------------------------------------------------------ >> >> >> This email and any files transmitted with it are confidential and are >> intended solely for the use of the individual or entity to whom they >> are addressed. If you are not the original recipient or the person >> responsible for delivering the email to the intended recipient, be >> advised that you have received this email in error, and that any use, >> dissemination, forwarding, printing, or copying of this email is >> strictly prohibited. If you received this email in error, please >> immediately notify the sender and delete the original. >> > > _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
