Em Sexta-feira 18 Fevereiro 2011, ?s 23:24:10, paul simpson escreveu: > hello all, > > i have been testing gluster as a central file server for a small animation > studio/post production company. my initial experiments were using the fuse > glusterfs protocol - but that ran extremely slowly for home dirs and > general file sharing. we have since switched to using NFS over glusterfs. > NFS has certainly seemed more responsive re. stat and dir traversal. > however, i'm now being plagued with three different types of errors: > > 1/ Stale NFS file handle > 2/ input/output errors > 3/ and a new one: > $ l -l /n/auto/gv1/production/conan/hda/published/OLD/ > ls: cannot access /n/auto/gv1/production/conan/hda/published/OLD/shot: > Remote I/O error > total 0 > d????????? ? ? ? ? ? shot > > ...so it's a bit all over the place. i've tried rebooting both servers and > clients. these issues are very erratic - they come and go. > > some information on my setup: glusterfs 3.1.2 > > g1:~ # gluster volume info > > Volume Name: glustervol1 > Type: Distributed-Replicate > Status: Started > Number of Bricks: 4 x 2 = 8 > Transport-type: tcp > Bricks: > Brick1: g1:/mnt/glus1 > Brick2: g2:/mnt/glus1 > Brick3: g3:/mnt/glus1 > Brick4: g4:/mnt/glus1 > Brick5: g1:/mnt/glus2 > Brick6: g2:/mnt/glus2 > Brick7: g3:/mnt/glus2 > Brick8: g4:/mnt/glus2 > Options Reconfigured: > > > performance.write-behind-window-size: 1mb > > > performance.cache-size: 1gb > > > performance.stat-prefetch: 1 > > > network.ping-timeout: 20 > > > diagnostics.latency-measurement: off > > > diagnostics.dump-fd-stats: on > > > that is 4 servers - serving ~30 clients - 95% linux, 5% mac. all NFS. > other points: > - i'm automounting using NFS via autofs (with ldap). ie: > gus:/glustervol1 on /n/auto/gv1 type nfs > (rw,vers=3,rsize=32768,wsize=32768,intr,sloppy,addr=10.0.0.13) > gus is pointing to rr dns machines (g1,g2,g3,g4). that all seems to be > working. > > - backend files system on g[1-4] is xfs. ie, > > g1:/var/log/glusterfs # xfs_info /mnt/glus1 > meta-data=/dev/sdb1 isize=256 agcount=7, agsize=268435200 > blks > = sectsz=512 attr=2 > data = bsize=4096 blocks=1627196928, imaxpct=5 > = sunit=256 swidth=2560 blks > naming =version 2 bsize=4096 ascii-ci=0 > log =internal bsize=4096 blocks=32768, version=2 > = sectsz=512 sunit=8 blks, lazy-count=0 > realtime =none extsz=4096 blocks=0, rtextents=0 > > > - sometimes root can stat/read the file in question while the user cannot! > i can remount the same NFS share to another mount point - and i can then > see that with the same user. > > - sample output of g1 nfs.log file: > > [2011-02-18 15:27:07.201433] I [io-stats.c:338:io_stats_dump_fd] > glustervol1: Filename : > /production/conan/hda/published/shot/backup/.svn/tmp/entries > [2011-02-18 15:27:07.201445] I [io-stats.c:353:io_stats_dump_fd] > glustervol1: BytesWritten : 1414 bytes > [2011-02-18 15:27:07.201455] I [io-stats.c:365:io_stats_dump_fd] > glustervol1: Write 001024b+ : 1 > [2011-02-18 15:27:07.205999] I [io-stats.c:333:io_stats_dump_fd] > glustervol1: --- fd stats --- > [2011-02-18 15:27:07.206032] I [io-stats.c:338:io_stats_dump_fd] > glustervol1: Filename : > /production/conan/hda/published/shot/backup/.svn/props/tempfile.tmp > [2011-02-18 15:27:07.210799] I [io-stats.c:333:io_stats_dump_fd] > glustervol1: --- fd stats --- > [2011-02-18 15:27:07.210824] I [io-stats.c:338:io_stats_dump_fd] > glustervol1: Filename : > /production/conan/hda/published/shot/backup/.svn/tmp/log > [2011-02-18 15:27:07.211904] I [io-stats.c:333:io_stats_dump_fd] > glustervol1: --- fd stats --- > [2011-02-18 15:27:07.211928] I [io-stats.c:338:io_stats_dump_fd] > glustervol1: Filename : > /prod_data/xmas/lgl/pic/mr_all_PBR_HIGHNO_DF/035/1920x1080/mr_all_PBR_HIGHN > O_DF.6084.exr [2011-02-18 15:27:07.211940] I > [io-stats.c:343:io_stats_dump_fd] > glustervol1: Lifetime : 8731secs, 610796usecs > [2011-02-18 15:27:07.211951] I [io-stats.c:353:io_stats_dump_fd] > glustervol1: BytesWritten : 2321370 bytes > [2011-02-18 15:27:07.211962] I [io-stats.c:365:io_stats_dump_fd] > glustervol1: Write 000512b+ : 1 > [2011-02-18 15:27:07.211972] I [io-stats.c:365:io_stats_dump_fd] > glustervol1: Write 002048b+ : 1 > [2011-02-18 15:27:07.211983] I [io-stats.c:365:io_stats_dump_fd] > glustervol1: Write 004096b+ : 4 > [2011-02-18 15:27:07.212009] I [io-stats.c:365:io_stats_dump_fd] > glustervol1: Write 008192b+ : 4 > [2011-02-18 15:27:07.212019] I [io-stats.c:365:io_stats_dump_fd] > glustervol1: Write 016384b+ : 20 > [2011-02-18 15:27:07.212030] I [io-stats.c:365:io_stats_dump_fd] > glustervol1: Write 032768b+ : 54 > [2011-02-18 15:27:07.228051] I [io-stats.c:333:io_stats_dump_fd] > glustervol1: --- fd stats --- > [2011-02-18 15:27:07.228078] I [io-stats.c:338:io_stats_dump_fd] > glustervol1: Filename : > /production/conan/hda/published/shot/backup/.svn/tmp/entries > > ...so, the files not working don't have lifetime, read/written lines after > their log entry. > > all very perplexing - and scary. one thing that reliably fails is using > svn working dirs on the gluster filesystem. nfs locks keep being dropped. > this is temporarily fixed when i view the file as root (on a client) - > but then re-appears very quickly. i assume that gluster is upto something > as simple as having svn working dirs? > > i'm hoping i've done something stupid which is easily fixed. we seem so > close - but right now, i'm at a loss and loosing confidence. i would > greatly appreciate any help/pointers out there. > > regards, > > paul Hi Paul. I've been using gluster for ~6 months now, so i'm by no means an expert, but i can see that you're doing two things that are dicouraged by the devels: - Using xfs as a backend filesystem - Serving small files ( < 1MB of size ) ( I'm assuming that because of log messages like this ) > [2011-02-18 15:27:07.201433] I [io-stats.c:338:io_stats_dump_fd] > glustervol1: Filename : > /production/conan/hda/published/shot/backup/.svn/tmp/entries > [2011-02-18 15:27:07.201445] I [io-stats.c:353:io_stats_dump_fd] > glustervol1: BytesWritten : 1414 bytes Can any one please confirm or correct my assumptions ? TIA
