Brick crashes

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



This is the core file from the crash just now

[root at psanaoss213 /]# ls -al core*
-rw------- 1 root root 4073594880 Jun  8 15:05 core.22682

 From yesterday:
[root at psanaoss214 /]# ls -al core*
-rw------- 1 root root 4362727424 Jun  8 00:58 core.13483
-rw------- 1 root root 4624773120 Jun  8 03:21 core.8792


On 06/08/2012 04:34 PM, Anand Avati wrote:
> Is it possible the system was running low on memory? I see you have 
> 48GB, but memory registration failure typically would be because the 
> system limit on the number of pinnable pages in RAM was hit. Can you 
> tell us the size of your core dump files after the crash?
>
> Avati
>
> On Fri, Jun 8, 2012 at 4:22 PM, Ling Ho <ling at slac.stanford.edu 
> <mailto:ling at slac.stanford.edu>> wrote:
>
>     Hello,
>
>     I have a brick that crashed twice today, and another different
>     brick that crashed just a while a go.
>
>     This is what I see in one of the brick logs:
>
>     patchset: git://git.gluster.com/glusterfs.git
>     <http://git.gluster.com/glusterfs.git>
>     patchset: git://git.gluster.com/glusterfs.git
>     <http://git.gluster.com/glusterfs.git>
>     signal received: 6
>     signal received: 6
>     time of crash: 2012-06-08 15:05:11
>     configuration details:
>     argp 1
>     backtrace 1
>     dlfcn 1
>     fdatasync 1
>     libpthread 1
>     llistxattr 1
>     setfsid 1
>     spinlock 1
>     epoll.h 1
>     xattr.h 1
>     st_atim.tv_nsec 1
>     package-string: glusterfs 3.2.6
>     /lib64/libc.so.6[0x34bc032900]
>     /lib64/libc.so.6(gsignal+0x35)[0x34bc032885]
>     /lib64/libc.so.6(abort+0x175)[0x34bc034065]
>     /lib64/libc.so.6[0x34bc06f977]
>     /lib64/libc.so.6[0x34bc075296]
>     /opt/glusterfs/3.2.6/lib64/libglusterfs.so.0(__gf_free+0x44)[0x7f1740ba25e4]
>     /opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(rpc_transport_destroy+0x47)[0x7f1740956967]
>     /opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(rpc_transport_unref+0x62)[0x7f1740956a32]
>     /opt/glusterfs/3.2.6/lib64/glusterfs/3.2.6/rpc-transport/rdma.so(+0xc135)[0x7f173ca27135]
>     /lib64/libpthread.so.0[0x34bc8077f1]
>     /lib64/libc.so.6(clone+0x6d)[0x34bc0e5ccd]
>     ---------
>
>     And somewhere before these, there is also
>     [2012-06-08 15:05:07.512604] E [rdma.c:198:rdma_new_post]
>     0-rpc-transport/rdma: memory registration failed
>
>     I have 48GB of memory on the system:
>
>     # free
>                 total       used       free     shared    buffers    
>     cached
>     Mem:      49416716   34496648   14920068          0      31692  
>     28209612
>     -/+ buffers/cache:    6255344   43161372
>     Swap:      4194296 1740 4192556 <tel:1740%20%20%20%204192556>
>
>     # uname -a
>     Linux psanaoss213 2.6.32-220.7.1.el6.x86_64 #1 SMP Fri Feb 10
>     15:22:22 EST 2012 x86_64 x86_64 x86_64 GNU/Linux
>
>     The server gluster versions is 3.2.6-1. I am using have both rdma
>     clients and tcp clients over 10Gb/s network.
>
>     Any suggestion what I should look for?
>
>     Is there a way to just restart the brick, and not glusterd on the
>     server? I have 8 bricks on the server.
>
>     Thanks,
>     ...
>     ling
>
>
>     Here's the volume info:
>
>     # gluster volume info
>
>     Volume Name: ana12
>     Type: Distribute
>     Status: Started
>     Number of Bricks: 40
>     Transport-type: tcp,rdma
>     Bricks:
>     Brick1: psanaoss214:/brick1
>     Brick2: psanaoss214:/brick2
>     Brick3: psanaoss214:/brick3
>     Brick4: psanaoss214:/brick4
>     Brick5: psanaoss214:/brick5
>     Brick6: psanaoss214:/brick6
>     Brick7: psanaoss214:/brick7
>     Brick8: psanaoss214:/brick8
>     Brick9: psanaoss211:/brick1
>     Brick10: psanaoss211:/brick2
>     Brick11: psanaoss211:/brick3
>     Brick12: psanaoss211:/brick4
>     Brick13: psanaoss211:/brick5
>     Brick14: psanaoss211:/brick6
>     Brick15: psanaoss211:/brick7
>     Brick16: psanaoss211:/brick8
>     Brick17: psanaoss212:/brick1
>     Brick18: psanaoss212:/brick2
>     Brick19: psanaoss212:/brick3
>     Brick20: psanaoss212:/brick4
>     Brick21: psanaoss212:/brick5
>     Brick22: psanaoss212:/brick6
>     Brick23: psanaoss212:/brick7
>     Brick24: psanaoss212:/brick8
>     Brick25: psanaoss213:/brick1
>     Brick26: psanaoss213:/brick2
>     Brick27: psanaoss213:/brick3
>     Brick28: psanaoss213:/brick4
>     Brick29: psanaoss213:/brick5
>     Brick30: psanaoss213:/brick6
>     Brick31: psanaoss213:/brick7
>     Brick32: psanaoss213:/brick8
>     Brick33: psanaoss215:/brick1
>     Brick34: psanaoss215:/brick2
>     Brick35: psanaoss215:/brick4
>     Brick36: psanaoss215:/brick5
>     Brick37: psanaoss215:/brick7
>     Brick38: psanaoss215:/brick8
>     Brick39: psanaoss215:/brick3
>     Brick40: psanaoss215:/brick6
>     Options Reconfigured:
>     performance.io-thread-count: 16
>     performance.write-behind-window-size: 16MB
>     performance.cache-size: 1GB
>     nfs.disable: on
>     performance.cache-refresh-timeout: 1
>     network.ping-timeout: 42
>     performance.cache-max-file-size: 1PB
>
>     _______________________________________________
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>     http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gluster.org/pipermail/gluster-users/attachments/20120608/9562d07f/attachment-0001.htm>


[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux