Yes, my teammate is working on it, and will get back. I will update once i have some info. On Thu, May 8, 2008 at 9:03 AM, Dan Parsons <dparsons@xxxxxxxx> wrote: > Amar, any update on this issue? > > > Dan Parsons > > > > On May 6, 2008, at 11:21 PM, Dan Parsons wrote: > > Amar, quick question. I've switched to readahead but really wish I could >> use io-cache. How likely do you think changing block-size from 128kb to 1MB >> (the same as what stripe uses, based on your advice) would fix the crash >> issue? >> >> >> Dan Parsons >> >> >> On May 6, 2008, at 12:43 PM, Amar S. Tumballi wrote: >> >> Mostly give the same block-size (and page-size) in stripe and io-cache. >>> Just >>> for checking it. But currently you can fall back to read-ahead. >>> >>> Regards, >>> Amar >>> >>> On Tue, May 6, 2008 at 12:38 PM, Dan Parsons <dparsons@xxxxxxxx> wrote: >>> >>> Ah, so it's not something I'm doing wrong? Do you think changing >>>> cache-size back to 32MB will prevent the problem from happening? >>>> >>>> Perhaps I should switch to readahead until fix? >>>> >>>> >>>> Dan Parsons >>>> >>>> >>>> >>>> On May 6, 2008, at 12:37 PM, Amar S. Tumballi wrote: >>>> >>>> Thanks for the bug report, We will get back to you in another 2-3 days >>>> >>>>> about it. Mostly with a fix :) >>>>> >>>>> Regards, >>>>> Amar >>>>> >>>>> On Tue, May 6, 2008 at 10:14 AM, Dan Parsons <dparsons@xxxxxxxx> >>>>> wrote: >>>>> Oh, one more useful bit of information, I see lines like the below a >>>>> lot >>>>> in glusterfs log files, what do they mean? >>>>> >>>>> 2008-05-05 21:20:11 W [fuse-bridge.c:402:fuse_entry_cbk] >>>>> glusterfs-fuse: >>>>> 18054459: (34) >>>>> >>>>> /bio/data/fast-hmmsearch-all/tmpDCex3b_fast-hmmsearch-all_job/result.tigrfam.TIGR02736.hmmhits >>>>> => 610503040 Rehashing because st_nlink less than dentry maps >>>>> >>>>> Dan Parsons >>>>> >>>>> >>>>> >>>>> On May 6, 2008, at 10:13 AM, Dan Parsons wrote: >>>>> >>>>> I'm experiencing a glusterfs client crash, signal 11, under the >>>>> io-cache >>>>> xlator. This is on our bioinformatics cluster- the crash happened on 2 >>>>> out >>>>> of 33 machines. I've verified the hardware stability of the machines. >>>>> >>>>> Running v1.3.8 built May 5th, 2008 from latest downloadable version. >>>>> >>>>> Here is the crash message: >>>>> >>>>> [0xffffe420] >>>>> >>>>> >>>>> /usr/local/lib/glusterfs/1.3.8/xlator/performance/io-cache.so(ioc_page_wakeup+0x67)[0xb76c5f67] >>>>> >>>>> >>>>> /usr/local/lib/glusterfs/1.3.8/xlator/performance/io-cache.so(ioc_inode_wakeup+0xb2)[0xb76c6902] >>>>> >>>>> >>>>> /usr/local/lib/glusterfs/1.3.8/xlator/performance/io-cache.so(ioc_cache_validate_cbk+0xae)[0xb76c1e5e] >>>>> >>>>> >>>>> /usr/local/lib/glusterfs/1.3.8/xlator/cluster/stripe.so(stripe_stack_unwind_buf_cbk+0x98)[0xb76cd038] >>>>> >>>>> >>>>> /usr/local/lib/glusterfs/1.3.8/xlator/protocol/client.so(client_fstat_cbk+0xcc)[0xb76dd13c] >>>>> >>>>> >>>>> /usr/local/lib/glusterfs/1.3.8/xlator/protocol/client.so(notify+0xa97)[0xb76db117] >>>>> /usr/local/lib/libglusterfs.so.0(transport_notify+0x38)[0xb7efe978] >>>>> /usr/local/lib/libglusterfs.so.0(sys_epoll_iteration+0xd6)[0xb7eff906] >>>>> /usr/local/lib/libglusterfs.so.0(poll_iteration+0x98)[0xb7efeb28] >>>>> [glusterfs](main+0x85e)[0x804a14e] >>>>> /lib/libc.so.6(__libc_start_main+0xdc)[0x7b1dec] >>>>> [glusterfs][0x8049391] >>>>> >>>>> And here is my config file. The only thing I can think of is maybe my >>>>> cache-size is too big. I want a lot of cache, we have big files, and >>>>> the >>>>> boxes have the RAM. Anyway, below is the config. If you see any >>>>> problems >>>>> with it, please let me know. There are no errors on the glusterfsd >>>>> servers, >>>>> except for an EOF from the machines where glusterfs client segfaulted. >>>>> >>>>> volume fuse >>>>> type mount/fuse >>>>> option direct-io-mode 1 >>>>> option entry-timeout 1 >>>>> option attr-timeout 1 >>>>> option mount-point /glusterfs >>>>> subvolumes ioc >>>>> end-volume >>>>> >>>>> volume ioc >>>>> type performance/io-cache >>>>> option priority *.psiblast:3,*.seq:2,*:1 >>>>> option force-revalidate-timeout 5 >>>>> option cache-size 1200MB >>>>> option page-size 128KB >>>>> subvolumes stripe0 >>>>> end-volume >>>>> >>>>> volume stripe0 >>>>> type cluster/stripe >>>>> option alu.disk-usage.exit-threshold 100MB >>>>> option alu.disk-usage.entry-threshold 2GB >>>>> option alu.write-usage.exit-threshold 4% >>>>> option alu.write-usage.entry-threshold 20% >>>>> option alu.read-usage.exit-threshold 4% >>>>> option alu.read-usage.entry-threshold 20% >>>>> option alu.order read-usage:write-usage:disk-usage >>>>> option scheduler alu >>>>> option block-size *:1MB >>>>> subvolumes distfs01 distfs02 distfs03 distfs04 >>>>> end-volume >>>>> >>>>> volume distfs04 >>>>> type protocol/client >>>>> option remote-subvolume brick >>>>> option remote-host 10.8.101.54 >>>>> option transport-type tcp/client >>>>> end-volume >>>>> >>>>> volume distfs03 >>>>> type protocol/client >>>>> option remote-subvolume brick >>>>> option remote-host 10.8.101.53 >>>>> option transport-type tcp/client >>>>> end-volume >>>>> >>>>> volume distfs02 >>>>> type protocol/client >>>>> option remote-subvolume brick >>>>> option remote-host 10.8.101.52 >>>>> option transport-type tcp/client >>>>> end-volume >>>>> >>>>> volume distfs01 >>>>> type protocol/client >>>>> option remote-subvolume brick >>>>> option remote-host 10.8.101.51 >>>>> option transport-type tcp/client >>>>> end-volume >>>>> >>>>> >>>>> Dan Parsons >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Gluster-devel mailing list >>>>> Gluster-devel@xxxxxxxxxx >>>>> http://lists.nongnu.org/mailman/listinfo/gluster-devel >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Gluster-devel mailing list >>>>> Gluster-devel@xxxxxxxxxx >>>>> http://lists.nongnu.org/mailman/listinfo/gluster-devel >>>>> >>>>> >>>>> >>>>> -- >>>>> Amar Tumballi >>>>> Gluster/GlusterFS Hacker >>>>> [bulde on #gluster/irc.gnu.org] >>>>> http://www.zresearch.com - Commoditizing Super Storage! >>>>> >>>>> >>>> >>>> >>>> >>> >>> -- >>> Amar Tumballi >>> Gluster/GlusterFS Hacker >>> [bulde on #gluster/irc.gnu.org] >>> http://www.zresearch.com - Commoditizing Super Storage! >>> >> >> >> >> _______________________________________________ >> Gluster-devel mailing list >> Gluster-devel@xxxxxxxxxx >> http://lists.nongnu.org/mailman/listinfo/gluster-devel >> >> > > -- Amar Tumballi Gluster/GlusterFS Hacker [bulde on #gluster/irc.gnu.org] http://www.zresearch.com - Commoditizing Super Storage!
