On Mon, 2016-06-27 at 15:05 +0200, Yann LEMARIE wrote: > @Anoop, > > Where can I find the coredump file ? > You will get hints about the crash from entries inside /var/log/messages(for example pid of the process, location of coredump etc). > The crash occurs 2 times last 7 days, each time a sunday morning with > no reason, no increase of traffic or something like this, the volume > was mounted since 15 days. > > The bricks are used as a CDN like, distributting small images and css > files with a nginx https service (with a load balancer and 2 EC2), on > a sunday morning there is not a lot of activity ... > >From the very minimal back trace that we have from brick logs I would assume that a truncate operation was being handled by trash translator and it crashed. > Volume infos: > > root@nfs05 /var/log/glusterfs # gluster volume info cdn > > > > Volume Name: cdn > > Type: Replicate > > Volume ID: c53b9bae-5e12-4f13-8217-53d8c96c302c > > Status: Started > > Number of Bricks: 1 x 2 = 2 > > Transport-type: tcp > > Bricks: > > Brick1: nfs05:/srv/cdn > > Brick2: nfs06:/srv/cdn > > Options Reconfigured: > > performance.readdir-ahead: on > > features.trash: on > > features.trash-max-filesize: 20MB > > I don't know if there is a link with this crash problem, but I have > another problem with my 2 servers that make GluserFS's clients > disconnected (from another volume) : > > Jun 24 02:28:04 nfs05 kernel: [2039468.818617] xen_netfront: > > xennet: skb rides the rocket: 19 slots > > Jun 24 02:28:11 nfs05 kernel: [2039475.744086] net_ratelimit: 66 > > callbacks suppressed > It seem to be a network interface problem : > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811 > > Yann > > Le 27/06/2016 12:59, Anoop C S a écrit : > > On Mon, 2016-06-27 at 09:47 +0200, Yann LEMARIE wrote: > > > Hi, > > > > > > I'm using GlusterFS since many years and never see this problem, > > > but > > > this is the second time in one week ... > > > > > > I have 3 volumes with 2 bricks and 1 volume crash with no reason, > > Did you observe the crash while mounting the volume? Or can you be > > more > > specific on what were you doing just before you saw the crash? Can > > you > > please share the output of `gluster volume info <VOLNAME>`? > > > > > I just have to stop/start the volume to make it up again. > > > The only logs I can find are in syslog : > > > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: pending frames: > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: frame : type(0) op(10) > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: patchset: > > > > git://git.gluster.com/glusterfs.git > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: signal received: 11 > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: time of crash: > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: 2016-06-26 09:27:44 > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: configuration details: > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: argp 1 > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: backtrace 1 > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: dlfcn 1 > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: libpthread 1 > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: llistxattr 1 > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: setfsid 1 > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: spinlock 1 > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: epoll.h 1 > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: xattr.h 1 > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: st_atim.tv_nsec 1 > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: package-string: glusterfs > > > > 3.7.11 > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: --------- > > > > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: pending frames: > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: frame : type(0) op(10) > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: patchset: > > > > git://git.gluster.com/glusterfs.git > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: signal received: 11 > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: time of crash: > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: 2016-06-26 09:27:44 > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: configuration details: > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: argp 1 > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: backtrace 1 > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: dlfcn 1 > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: libpthread 1 > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: llistxattr 1 > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: setfsid 1 > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: spinlock 1 > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: epoll.h 1 > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: xattr.h 1 > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: st_atim.tv_nsec 1 > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: package-string: glusterfs > > > > 3.7.11 > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: --------- > > > > > > > > > > Thanks for your help > > > > > > > > > Regards > > > -- > > > Yann Lemarié > > > iRaiser - Support Technique > > > > > > ylemarie@xxxxxxxxxx > > > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users@xxxxxxxxxxx > > > http://www.gluster.org/mailman/listinfo/gluster-users > > -- > Yann Lemarié > iRaiser - Support Technique > > ylemarie@xxxxxxxxxx > > > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
