Hi! It seems that my crashes not connected with DHT and write-behind. One of the servers crashed yesterday with the following backtrace: SERVER n54 (Jun 29 15:51) ------------------------ Core was generated by `/usr/sbin/glusterfsd -f /etc/glusterfs/server.vol -l /var/log/glusterfs/server.'. Program terminated with signal 6, Aborted. #0 0x00002b600b19db45 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00002b600b19db45 in raise () from /lib64/libc.so.6 #1 0x00002b600b19f0e0 in abort () from /lib64/libc.so.6 #2 0x00002b600b1d5fbb in ?? () from /lib64/libc.so.6 #3 0x00002b600b1db21d in ?? () from /lib64/libc.so.6 #4 0x00002b600b1dcf76 in free () from /lib64/libc.so.6 #5 0x00002b600ab3cefd in mem_put (pool=0x612760, ptr=0x637340) at mem-pool.c:191 #6 0x00002b600c1422ce in iot_dequeue_ordered (worker=0x6126f0) at io-threads.c:2407 #7 0x00002b600c142326 in iot_worker_ordered (arg=<value optimized out>) at io-threads.c:2421 #8 0x00002b600af57020 in start_thread () from /lib64/libpthread.so.0 #9 0x00002b600b231f8d in clone () from /lib64/libc.so.6 #10 0x0000000000000000 in ?? () ------------------------- Also glusterFS servers continue crashing with other backtrace, which I've already reported: SERVER n45 (Jun 29 19:54) ---------------------- Core was generated by `/usr/sbin/glusterfsd -f /etc/glusterfs/server.vol -l /var/log/glusterfs/server.'. Program terminated with signal 11, Segmentation fault. #0 0x00002b1395813860 in ?? () from /lib64/libc.so.6 (gdb) bt #0 0x00002b1395813860 in ?? () from /lib64/libc.so.6 #1 0x00002b139581492b in calloc () from /lib64/libc.so.6 #2 0x00002b13969841b3 in get_frame_for_call (trans=0x61ea70, hdr=0x2aaaac01cde0) at server-protocol.c:7315 #3 0x00002b1396984496 in protocol_server_interpret (this=0x60f8c0, trans=0x61ea70, hdr_p=0x2aaaac01cde0 "", hdrlen=60, iobuf=0x2aaaac03dea8) at server-protocol.c:7481 #4 0x00002b1396984720 in protocol_server_pollin (this=0x60f8c0, trans=0x61ea70) at server-protocol.c:7763 #5 0x00002b13969847b2 in notify (this=0x60f8c0, event=<value optimized out>, data=0x6124b9) at server-protocol.c:7819 #6 0x00002b139515a183 in xlator_notify (xl=0x60f8c0, event=2, data=0x61ea70) at xlator.c:820 #7 0x00002aaaaaaaff0b in socket_event_handler (fd=<value optimized out>, idx=41, data=0x61ea70, poll_in=1, poll_out=0, poll_err=0) at socket.c:813 #8 0x00002b13951722aa in event_dispatch_epoll (event_pool=0x6094f0) at event.c:804 #9 0x0000000000403f34 in main (argc=5, argv=0x7fff15b7f878) at glusterfsd.c:1223 ------------------ Probably this bug is not connected with the first one. But it's very annoying, as it's triggered more frequently. Best wishes, Andrey Shehjar Tikoo (shehjart@xxxxxxxxxxx) wrote: > Hi > > You can keep track of the progress of this crash at the > following URL: > > http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=102 > > Thanks > Shehjar > > > NovA wrote: >> >> Hi everybody!