Re: df command shows transport endpoint mount error on gluster client v.3.10.5 + core dump

[Mauro Tridici <mauro.tridici@xxxxxxx>]

Hi Ashish,

in attachment you can find the gdb output (with bt and thread outputs) and the complete log file until the crash happened.

Thank you for your support.

Mauro Tridici

Attachment: tier2.log-20170924.gz
Description: GNU Zip compressed data

[athena_login2][/var/log/glusterfs]> gdb /usr/local/sbin/glusterfs core.15795 
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
/usr/local/sbin/glusterfs: No such file or directory.
Missing separate debuginfo for the main executable file
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/fb/4d988b681faa09bb74becacd7a24f4186e8185
[New Thread 15802]
[New Thread 15804]
[New Thread 15805]
[New Thread 6856]
[New Thread 30432]
[New Thread 30486]
[New Thread 1619]
[New Thread 15806]
[New Thread 15810]
[New Thread 30412]
[New Thread 15809]
[New Thread 15799]
[New Thread 30487]
[New Thread 15795]
[New Thread 15797]
[New Thread 15798]
[New Thread 15800]
[New Thread 15801]
Core was generated by `/usr/sbin/glusterfs --volfile-server=s01-stg --volfile-id=/tier2 /tier2'.
Program terminated with signal 6, Aborted.
#0  0x00000032a74328a5 in ?? ()
(gdb) bt
#0  0x00000032a74328a5 in ?? ()
#1  0x00000032a7434085 in ?? ()
#2  0x00007fd8ce3bbb58 in ?? ()
#3  0x00007fd8ce3bbb40 in ?? ()
#4  0x00007fd8d87284b8 in ?? ()
#5  0x0000000000000010 in ?? ()
#6  0x00000032a7554b6a in ?? ()
#7  0x0000000000000001 in ?? ()
#8  0x00000032a7413b8a in ?? ()
#9  0x0000000000000005 in ?? ()
#10 0x00000032a75559b5 in ?? ()
#11 0x0000000000000003 in ?? ()
#12 0x00007fd8ce3bbb3e in ?? ()
#13 0x0000000000000002 in ?? ()
#14 0x00000032a7554b84 in ?? ()
#15 0x0000000000000001 in ?? ()
#16 0x00000032a75565c0 in ?? ()
#17 0x0000000000000003 in ?? ()
#18 0x00007fd8ce3bbb46 in ?? ()
#19 0x000000000000000a in ?? ()
#20 0x00000032a75565c4 in ?? ()
#21 0x0000000000000002 in ?? ()
#22 0x0000000000000020 in ?? ()
#23 0x0000000000000000 in ?? ()
(gdb) thread apply all bt

Thread 18 (Thread 15801):
#0  0x00000032a800b7bb in ?? ()
#1  0x0000000200000000 in ?? ()
#2  0x0000000000baafa8 in ?? ()
#3  0x0000000000baaf80 in ?? ()
#4  0x0000000000000105 in ?? ()
#5  0x0000000059c61d51 in ?? ()
#6  0x000000000000003e in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 17 (Thread 15800):
#0  0x00000032a800b7bb in ?? ()
#1  0x0000000200000000 in ?? ()
#2  0x0000000000baafa8 in ?? ()
#3  0x0000000000baaf80 in ?? ()
#4  0x0000000000000104 in ?? ()
#5  0x0000000059c61d51 in ?? ()
#6  0x0000000000000039 in ?? ()
#7  0x00007fd8ac0008c0 in ?? ()
#8  0x00007fd8cfc00e10 in ?? ()
#9  0x0000000000baaf48 in ?? ()
#10 0x0000000000ba7340 in ?? ()
#11 0x0000000000ba7340 in ?? ()
#12 0x00007fd8d8bc5afc in ?? ()
#13 0x00000032a8011200 in ?? ()
#14 0x0000000000baaf60 in ?? ()
#15 0x0000000059c61fa9 in ?? ()
#16 0x0000000000000000 in ?? ()

Thread 16 (Thread 15798):
#0  0x00000032a800f2a5 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 15 (Thread 15797):
#0  0x00000032a800ed2d in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 14 (Thread 15795):
#0  0x00000032a80080ad in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x00000032a7014695 in ?? ()
#3  0x00000032a8007f80 in ?? ()
#4  0x00007fd8ce3bdd28 in ?? ()
#5  0x0000000000b8d628 in ?? ()
#6  0x0000000000000000 in ?? ()

Thread 13 (Thread 30487):
#0  0x00000032a800b7bb in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 12 (Thread 15799):
#0  0x00000032a74ac3bd in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 11 (Thread 15809):
#0  0x00000032a74e02d7 in ?? ()
#1  0x0000000000b91c80 in ?? ()
#2  0x0000000000000002 in ?? ()
#3  0x00007fd8c4d97e10 in ?? ()
#4  0x00007fd800000008 in ?? ()
#5  0x0000000000000001 in ?? ()
#6  0x00007fd8d1a23d88 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 10 (Thread 30412):
#0  0x00000032a800b7bb in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 9 (Thread 15810):
#0  0x00000032a800b43c in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 8 (Thread 15806):
#0  0x00000032a74e8713 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 7 (Thread 1619):
#0  0x00000032a800b7bb in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 6 (Thread 30486):
#0  0x00000032a800b7bb in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 5 (Thread 30432):
#0  0x00000032a800b7bb in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 4 (Thread 6856):
#0  0x00000032a800b7bb in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 3 (Thread 15805):
#0  0x00000032a74e8713 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 2 (Thread 15804):
#0  0x00007fd8cd51e750 in ?? ()
#1  0x00007fd8cd52dc98 in ?? ()
#2  0x0000000000000001 in ?? ()
#3  0x00007fd8c6ed3520 in ?? ()
#4  0x00007fd8c80b22f0 in ?? ()
#5  0x00007fd8c80b0fd8 in ?? ()
#6  0x00007fd8c6ed3520 in ?? ()
#7  0x00007fd8cd51bdf8 in ?? ()
#8  0x00007fd8ab59f000 in ?? ()
#9  0x0000000000020000 in ?? ()
#10 0x00007fd8c80b0e90 in ?? ()
#11 0x00007fd89c09a580 in ?? ()
#12 0x00007fd8c6ed3570 in ?? ()
#13 0x0000000000000000 in ?? ()

Thread 1 (Thread 15802):
#0  0x00000032a74328a5 in ?? ()
#1  0x00000032a7434085 in ?? ()
#2  0x00007fd8ce3bbb58 in ?? ()
#3  0x00007fd8ce3bbb40 in ?? ()
#4  0x00007fd8d87284b8 in ?? ()
#5  0x0000000000000010 in ?? ()
#6  0x00000032a7554b6a in ?? ()
#7  0x0000000000000001 in ?? ()
#8  0x00000032a7413b8a in ?? ()
#9  0x0000000000000005 in ?? ()
#10 0x00000032a75559b5 in ?? ()
#11 0x0000000000000003 in ?? ()
#12 0x00007fd8ce3bbb3e in ?? ()
#13 0x0000000000000002 in ?? ()
#14 0x00000032a7554b84 in ?? ()
#15 0x0000000000000001 in ?? ()
#16 0x00000032a75565c0 in ?? ()
#17 0x0000000000000003 in ?? ()
#18 0x00007fd8ce3bbb46 in ?? ()
#19 0x000000000000000a in ?? ()
#20 0x00000032a75565c4 in ?? ()
#21 0x0000000000000002 in ?? ()
#22 0x0000000000000020 in ?? ()
#23 0x0000000000000000 in ?? ()
(gdb)

Il giorno 26 set 2017, alle ore 10:11, Ashish Pandey <aspandey@xxxxxxxxxx> ha scritto:

Hi,

Following are the command to get the debug info for gluster -

gdb /usr/local/sbin/glusterfs  <core file path> 
Then on gdb prompt you need to type bt or backtrace 
(gdb) bt
You can also provide the output of "thread apply all bt".
(gdb) thread apply all bt 

Above commands should be executed on client node on which you have mounted the gluster volume and it crashed.

However, I am not sure if you have enabled the generation of core dump on your system and have setup path to coredump.

https://stackoverflow.com/questions/8305866/how-to-analyze-a-programs-core-dump-file-with-gdb

https://unix.stackexchange.com/questions/192716/how-to-set-the-core-dump-file-location-and-name

https://stackoverflow.com/questions/2065912/core-dumped-but-core-file-is-not-in-current-directory

--

Ashish

---

From: "Mauro Tridici" <mauro.tridici@xxxxxxx>
To: "Ashish Pandey" <aspandey@xxxxxxxxxx>
Cc: "Gluster Users" <gluster-users@xxxxxxxxxxx>
Sent: Tuesday, September 26, 2017 1:21:42 PM
Subject: Re: df command shows transport endpoint mount error        on        gluster client v.3.10.5 + core dump

Hi Ashish,

thank you for your answer.

Do you need complete client log file only or something else in particular?

Unfortunately, I never used “bt” command. Could you please provide me an usage example string?

I will provide all logs you need.

Thank you again,

Mauro

Il giorno 26 set 2017, alle ore 09:30, Ashish Pandey <aspandey@xxxxxxxxxx> ha scritto:

Hi Mauro,

We would require complete log file to debug this issue.

Also, could you please provide some more information of the core after attaching to gdb and using command "bt".

---

Ashish

---

From: "Mauro Tridici" <mauro.tridici@xxxxxxx>
To: "Gluster Users" <gluster-users@xxxxxxxxxxx>
Sent: Monday, September 25, 2017 5:59:10 PM
Subject: df command shows transport endpoint mount error on        gluster client v.3.10.5 + core dump

Dear Gluster Users,

I implemented a distributed disperse 6x(4+2) gluster (v.3.10.5) volume with the following options:

[root@s01 tier2]# gluster volume info
 
Volume Name: tier2
Type: Distributed-Disperse
Volume ID: a28d88c5-3295-4e35-98d4-210b3af9358c
Status: Started
Snapshot Count: 0
Number of Bricks: 6 x (4 + 2) = 36
Transport-type: tcp
Bricks:
Brick1: s01-stg:/gluster/mnt1/brick
Brick2: s02-stg:/gluster/mnt1/brick
Brick3: s03-stg:/gluster/mnt1/brick
Brick4: s01-stg:/gluster/mnt2/brick
Brick5: s02-stg:/gluster/mnt2/brick
Brick6: s03-stg:/gluster/mnt2/brick
Brick7: s01-stg:/gluster/mnt3/brick
Brick8: s02-stg:/gluster/mnt3/brick
Brick9: s03-stg:/gluster/mnt3/brick
Brick10: s01-stg:/gluster/mnt4/brick
Brick11: s02-stg:/gluster/mnt4/brick
Brick12: s03-stg:/gluster/mnt4/brick
Brick13: s01-stg:/gluster/mnt5/brick
Brick14: s02-stg:/gluster/mnt5/brick
Brick15: s03-stg:/gluster/mnt5/brick
Brick16: s01-stg:/gluster/mnt6/brick
Brick17: s02-stg:/gluster/mnt6/brick
Brick18: s03-stg:/gluster/mnt6/brick
Brick19: s01-stg:/gluster/mnt7/brick
Brick20: s02-stg:/gluster/mnt7/brick
Brick21: s03-stg:/gluster/mnt7/brick
Brick22: s01-stg:/gluster/mnt8/brick
Brick23: s02-stg:/gluster/mnt8/brick
Brick24: s03-stg:/gluster/mnt8/brick
Brick25: s01-stg:/gluster/mnt9/brick
Brick26: s02-stg:/gluster/mnt9/brick
Brick27: s03-stg:/gluster/mnt9/brick
Brick28: s01-stg:/gluster/mnt10/brick
Brick29: s02-stg:/gluster/mnt10/brick
Brick30: s03-stg:/gluster/mnt10/brick
Brick31: s01-stg:/gluster/mnt11/brick
Brick32: s02-stg:/gluster/mnt11/brick
Brick33: s03-stg:/gluster/mnt11/brick
Brick34: s01-stg:/gluster/mnt12/brick
Brick35: s02-stg:/gluster/mnt12/brick
Brick36: s03-stg:/gluster/mnt12/brick
Options Reconfigured:
features.scrub: Active
features.bitrot: on
features.inode-quota: on
features.quota: on
performance.client-io-threads: on
cluster.min-free-disk: 10
cluster.quorum-type: auto
transport.address-family: inet
nfs.disable: on
server.event-threads: 4
client.event-threads: 4
cluster.lookup-optimize: on
performance.readdir-ahead: on
performance.parallel-readdir: off
cluster.readdir-optimize: on
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
network.inode-lru-limit: 50000
performance.io-cache: off
disperse.cpu-extensions: auto
performance.io-thread-count: 16
features.quota-deem-statfs: on
features.default-soft-limit: 90
cluster.server-quorum-type: server
cluster.brick-multiplex: on
cluster.server-quorum-ratio: 51%

I also started a long write test (about 69 TB to be written) from different gluster clients.
One of this client returned the error "Transport Endpoint is not connected" during the rsync copy process.
A core dump file has been generated by the issue; this is the core dump content:

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Missing separate debuginfo for the main executable file
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/fb/4d988b681faa09bb74becacd7a24f4186e8185
[New Thread 15802]
[New Thread 15804]
[New Thread 15805]
[New Thread 6856]
[New Thread 30432]
[New Thread 30486]
[New Thread 1619]
[New Thread 15806]
[New Thread 15810]
[New Thread 30412]
[New Thread 15809]
[New Thread 15799]
[New Thread 30487]
[New Thread 15795]
[New Thread 15797]
[New Thread 15798]
[New Thread 15800]
[New Thread 15801]
Core was generated by `/usr/sbin/glusterfs --volfile-server=s01-stg --volfile-id=/tier2 /tier2'.
Program terminated with signal 6, Aborted.
#0  0x00000032a74328a5 in ?? ()
"/core.15795" is a core file.
Please specify an executable to debug.

No problem detected on the gluster servers, no brick down and so on...

Anyone of us experienced the same problem?
If yes, how did you resolve it?

You can find the client "/var/log/messages" and "/var/log/glusterfs" log files content.
Thank you in advance.
Mauro Tridici

--------------------------------------

In /var/log/syslog-ng/messages on the client (OS=centos 6.2, 16 cores, 64GB RAM, gluster client v.3.10.5)

Sep 23 10:42:43 login2 tier2[15795]: pending frames:
Sep 23 10:42:43 login2 tier2[15795]: frame : type(1) op(WRITE)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(1) op(WRITE)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(1) op(WRITE)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(1) op(WRITE)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)
Sep 23 10:42:43 login2 tier2[15795]: patchset: git://git.gluster.org/glusterfs.git
Sep 23 10:42:43 login2 tier2[15795]: signal received: 6
Sep 23 10:42:43 login2 tier2[15795]: time of crash:
Sep 23 10:42:43 login2 tier2[15795]: 2017-09-23 08:42:43
Sep 23 10:42:43 login2 tier2[15795]: configuration details:
Sep 23 10:42:43 login2 tier2[15795]: argp 1
Sep 23 10:42:43 login2 tier2[15795]: backtrace 1
Sep 23 10:42:43 login2 tier2[15795]: dlfcn 1
Sep 23 10:42:43 login2 tier2[15795]: libpthread 1
Sep 23 10:42:43 login2 tier2[15795]: llistxattr 1
Sep 23 10:42:43 login2 tier2[15795]: setfsid 1
Sep 23 10:42:43 login2 tier2[15795]: spinlock 1
Sep 23 10:42:43 login2 tier2[15795]: epoll.h 1
Sep 23 10:42:43 login2 tier2[15795]: xattr.h 1
Sep 23 10:42:43 login2 tier2[15795]: st_atim.tv_nsec 1
Sep 23 10:42:43 login2 tier2[15795]: package-string: glusterfs 3.10.5
Sep 23 10:42:43 login2 tier2[15795]: ---------
Sep 23 10:42:43 login2 abrt[8605]: file /usr/sbin/glusterfsd seems to be deleted

In /var/log/glusterfs file:

pending frames:
frame : type(1) op(WRITE)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(1) op(WRITE)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(1) op(WRITE)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(1) op(WRITE)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
patchset: git://git.gluster.org/glusterfs.git
signal received: 6
time of crash:
2017-09-23 08:42:43
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.10.5
/usr/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x9c)[0x7fd8d8b8af3c]
/usr/lib64/libglusterfs.so.0(gf_print_trace+0x338)[0x7fd8d8b96538]
/lib64/libc.so.6[0x32a7432920]
/lib64/libc.so.6(gsignal+0x35)[0x32a74328a5]
/lib64/libc.so.6(abort+0x175)[0x32a7434085]
/lib64/libc.so.6[0x32a746ffe7]
/lib64/libc.so.6[0x32a7475916]
/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x3181f)[0x7fd8cd51681f]
/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x31dad)[0x7fd8cd516dad]
/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x12027)[0x7fd8cd4f7027]
/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x11dc5)[0x7fd8cd4f6dc5]
/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x1475f)[0x7fd8cd4f975f]
/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x15178)[0x7fd8cd4fa178]
/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x31dc8)[0x7fd8cd516dc8]
/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x12027)[0x7fd8cd4f7027]
/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x11dc5)[0x7fd8cd4f6dc5]
/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x13f24)[0x7fd8cd4f8f24]
/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x33e12)[0x7fd8cd518e12]
/usr/lib64/glusterfs/3.10.5/xlator/protocol/client.so(+0x2c420)[0x7fd8cd793420]
/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x7fd8d8955ad5]
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x195)[0x7fd8d8956c85]
/usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7fd8d8951d68]
/usr/lib64/glusterfs/3.10.5/rpc-transport/socket.so(+0x9ccd)[0x7fd8ce5f1ccd]
/usr/lib64/glusterfs/3.10.5/rpc-transport/socket.so(+0xaffe)[0x7fd8ce5f2ffe]
/usr/lib64/libglusterfs.so.0(+0x87806)[0x7fd8d8be8806]
/lib64/libpthread.so.0[0x32a8007851]
/lib64/libc.so.6(clone+0x6d)[0x32a74e811d]
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users