hi fabricio, many thanks for your input. indeed i am using xfs - but that seems to be mentioned in the gluster docs without any mention of problems. we benchmarked xfs vs ext4 - and found that xfs to be much better at dealing with the bulk of our data - hi-def frames ~3-10M each - and large geometry/particle/volume files. 10M-200M. so, i'm keen to hear from anyone abotu xfs's suitability for gluster storage... as for file size; my understanding is that a distributed file system performance only really kicks in when your dealing with large >1M files. however, is dealing with small files meant to be unreliable with locking/access errors? again thanks - and i look forward to hearing if gluster is able to reliably serve svn working directories and cope with locks... regards, paul On Mon, Feb 21, 2011 at 2:42 PM, Fabricio Cannini <fcannini at gmail.com>wrote: > Em Sexta-feira 18 Fevereiro 2011, ?s 23:24:10, paul simpson escreveu: > > hello all, > > > > i have been testing gluster as a central file server for a small > animation > > studio/post production company. my initial experiments were using the > fuse > > glusterfs protocol - but that ran extremely slowly for home dirs and > > general file sharing. we have since switched to using NFS over > glusterfs. > > NFS has certainly seemed more responsive re. stat and dir traversal. > > however, i'm now being plagued with three different types of errors: > > > > 1/ Stale NFS file handle > > 2/ input/output errors > > 3/ and a new one: > > $ l -l /n/auto/gv1/production/conan/hda/published/OLD/ > > ls: cannot access /n/auto/gv1/production/conan/hda/published/OLD/shot: > > Remote I/O error > > total 0 > > d????????? ? ? ? ? ? shot > > > > ...so it's a bit all over the place. i've tried rebooting both servers > and > > clients. these issues are very erratic - they come and go. > > > > some information on my setup: glusterfs 3.1.2 > > > > g1:~ # gluster volume info > > > > Volume Name: glustervol1 > > Type: Distributed-Replicate > > Status: Started > > Number of Bricks: 4 x 2 = 8 > > Transport-type: tcp > > Bricks: > > Brick1: g1:/mnt/glus1 > > Brick2: g2:/mnt/glus1 > > Brick3: g3:/mnt/glus1 > > Brick4: g4:/mnt/glus1 > > Brick5: g1:/mnt/glus2 > > Brick6: g2:/mnt/glus2 > > Brick7: g3:/mnt/glus2 > > Brick8: g4:/mnt/glus2 > > Options Reconfigured: > > > > > > performance.write-behind-window-size: 1mb > > > > > > performance.cache-size: 1gb > > > > > > performance.stat-prefetch: 1 > > > > > > network.ping-timeout: 20 > > > > > > diagnostics.latency-measurement: off > > > > > > diagnostics.dump-fd-stats: on > > > > > > that is 4 servers - serving ~30 clients - 95% linux, 5% mac. all NFS. > > other points: > > - i'm automounting using NFS via autofs (with ldap). ie: > > gus:/glustervol1 on /n/auto/gv1 type nfs > > (rw,vers=3,rsize=32768,wsize=32768,intr,sloppy,addr=10.0.0.13) > > gus is pointing to rr dns machines (g1,g2,g3,g4). that all seems to be > > working. > > > > - backend files system on g[1-4] is xfs. ie, > > > > g1:/var/log/glusterfs # xfs_info /mnt/glus1 > > meta-data=/dev/sdb1 isize=256 agcount=7, agsize=268435200 > > blks > > = sectsz=512 attr=2 > > data = bsize=4096 blocks=1627196928, > imaxpct=5 > > = sunit=256 swidth=2560 blks > > naming =version 2 bsize=4096 ascii-ci=0 > > log =internal bsize=4096 blocks=32768, version=2 > > = sectsz=512 sunit=8 blks, lazy-count=0 > > realtime =none extsz=4096 blocks=0, rtextents=0 > > > > > > - sometimes root can stat/read the file in question while the user > cannot! > > i can remount the same NFS share to another mount point - and i can then > > see that with the same user. > > > > - sample output of g1 nfs.log file: > > > > [2011-02-18 15:27:07.201433] I [io-stats.c:338:io_stats_dump_fd] > > glustervol1: Filename : > > /production/conan/hda/published/shot/backup/.svn/tmp/entries > > [2011-02-18 15:27:07.201445] I [io-stats.c:353:io_stats_dump_fd] > > glustervol1: BytesWritten : 1414 bytes > > [2011-02-18 15:27:07.201455] I [io-stats.c:365:io_stats_dump_fd] > > glustervol1: Write 001024b+ : 1 > > [2011-02-18 15:27:07.205999] I [io-stats.c:333:io_stats_dump_fd] > > glustervol1: --- fd stats --- > > [2011-02-18 15:27:07.206032] I [io-stats.c:338:io_stats_dump_fd] > > glustervol1: Filename : > > /production/conan/hda/published/shot/backup/.svn/props/tempfile.tmp > > [2011-02-18 15:27:07.210799] I [io-stats.c:333:io_stats_dump_fd] > > glustervol1: --- fd stats --- > > [2011-02-18 15:27:07.210824] I [io-stats.c:338:io_stats_dump_fd] > > glustervol1: Filename : > > /production/conan/hda/published/shot/backup/.svn/tmp/log > > [2011-02-18 15:27:07.211904] I [io-stats.c:333:io_stats_dump_fd] > > glustervol1: --- fd stats --- > > [2011-02-18 15:27:07.211928] I [io-stats.c:338:io_stats_dump_fd] > > glustervol1: Filename : > > > /prod_data/xmas/lgl/pic/mr_all_PBR_HIGHNO_DF/035/1920x1080/mr_all_PBR_HIGHN > > O_DF.6084.exr [2011-02-18 15:27:07.211940] I > > [io-stats.c:343:io_stats_dump_fd] > > glustervol1: Lifetime : 8731secs, 610796usecs > > [2011-02-18 15:27:07.211951] I [io-stats.c:353:io_stats_dump_fd] > > glustervol1: BytesWritten : 2321370 bytes > > [2011-02-18 15:27:07.211962] I [io-stats.c:365:io_stats_dump_fd] > > glustervol1: Write 000512b+ : 1 > > [2011-02-18 15:27:07.211972] I [io-stats.c:365:io_stats_dump_fd] > > glustervol1: Write 002048b+ : 1 > > [2011-02-18 15:27:07.211983] I [io-stats.c:365:io_stats_dump_fd] > > glustervol1: Write 004096b+ : 4 > > [2011-02-18 15:27:07.212009] I [io-stats.c:365:io_stats_dump_fd] > > glustervol1: Write 008192b+ : 4 > > [2011-02-18 15:27:07.212019] I [io-stats.c:365:io_stats_dump_fd] > > glustervol1: Write 016384b+ : 20 > > [2011-02-18 15:27:07.212030] I [io-stats.c:365:io_stats_dump_fd] > > glustervol1: Write 032768b+ : 54 > > [2011-02-18 15:27:07.228051] I [io-stats.c:333:io_stats_dump_fd] > > glustervol1: --- fd stats --- > > [2011-02-18 15:27:07.228078] I [io-stats.c:338:io_stats_dump_fd] > > glustervol1: Filename : > > /production/conan/hda/published/shot/backup/.svn/tmp/entries > > > > ...so, the files not working don't have lifetime, read/written lines > after > > their log entry. > > > > all very perplexing - and scary. one thing that reliably fails is using > > svn working dirs on the gluster filesystem. nfs locks keep being > dropped. > > this is temporarily fixed when i view the file as root (on a client) - > > but then re-appears very quickly. i assume that gluster is upto > something > > as simple as having svn working dirs? > > > > i'm hoping i've done something stupid which is easily fixed. we seem so > > close - but right now, i'm at a loss and loosing confidence. i would > > greatly appreciate any help/pointers out there. > > > > regards, > > > > paul > > Hi Paul. > > I've been using gluster for ~6 months now, so i'm by no means an expert, > but i > can see that you're doing two things that are dicouraged by the devels: > > - Using xfs as a backend filesystem > - Serving small files ( < 1MB of size ) > ( I'm assuming that because of log messages like this ) > > > [2011-02-18 15:27:07.201433] I [io-stats.c:338:io_stats_dump_fd] > > glustervol1: Filename : > > /production/conan/hda/published/shot/backup/.svn/tmp/entries > > [2011-02-18 15:27:07.201445] I [io-stats.c:353:io_stats_dump_fd] > > glustervol1: BytesWritten : 1414 bytes > > Can any one please confirm or correct my assumptions ? > > TIA > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >
