(gdb) up
#1 0x00007f308ca06098 in abort () from /lib64/libc.so.6
(gdb) up
#2 0x00007f308ca45197 in __libc_message () from /lib64/libc.so.6
(gdb) up
#3 0x00007f308ca4c56d in _int_free () from /lib64/libc.so.6
(gdb) up
#4 0x00007f308096ebc1 in dht_local_wipe (this=0x7f308f306460, local=0x7f3080042880) at dht-helper.c:475
475 iobref_unref (local->rebalance.iobref);
(gdb) p local->rebalance.iobref
$1 = (struct iobref *) 0x7f30740022c0
(gdb) p *local->rebalance.iobref
$2 = {lock = 1, ref = 2, iobrefs = 0x7f307402f240, alloced = 16, used = 1}
(gdb)
Thanks for your response.
We use glusterfs 3.6.7.
Sure ,We use Centos7.0.
Related log show bellow:
143 [2016-04-13 06:33:54.236013] W [glusterfsd.c:1211:cleanup_and_exit] (--> 0-: received signum (15), shutting down
144 [2016-04-13 06:33:54.236081] I [fuse-bridge.c:5607:fini] 0-fuse: Unmounting '/var/lib/nova/mnt/a77401a594b06b2b56cc52ee61bb4 def'.
145 [2016-04-13 06:55:11.365558] I [MSGID: 100030] [glusterfsd.c:2035:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glu sterfs version 3.6.7 (args: /usr/sbin/glusterfs --volfile-server=10.74.125.254 --volfile-id=/xitos-volume /var/lib/nova/mnt/ a77401a594b06b2b56cc52ee61bb4def)
146 [2016-04-13 06:55:11.402762] I [dht-shared.c:337:dht_init_regex] 0-xitos-volume-dht: using regex rsync-hash-regex = ^\.(.+)\ .[^.]+$
147 [2016-04-13 06:55:11.404087] I [client.c:2268:notify] 0-xitos-volume-client-0: parent translators are ready, attempting conn ect on transport
148 Final graph:
149 +------------------------------------------------------------------------------+
150 1: volume xitos-volume-client-0
151 2: type protocol/client
152 3: option ping-timeout 42
153 4: option remote-host 10.74.125.247
154 5: option remote-subvolume /mnt/dht/volume
155 6: option transport-type socket
156 7: option send-gids true
157 8: end-volume
158 9:
159 10: volume xitos-volume-dht
160 11: type cluster/distribute
161 12: subvolumes xitos-volume-client-0
162 13: end-volume
163 14:
164 15: volume xitos-volume-write-behind
165 16: type performance/write-behind
166 17: subvolumes xitos-volume-dht
167 18: end-volume
168 19:
169 20: volume xitos-volume-read-ahead
170 21: type performance/read-ahead
171 22: subvolumes xitos-volume-write-behind
172 23: end-volume
173 24:
174 25: volume xitos-volume-io-cache
175 26: type performance/io-cache
176 27: subvolumes xitos-volume-read-ahead
177 28: end-volume
178 29:
179 30: volume xitos-volume-quick-read
180 31: type performance/quick-read
181 32: subvolumes xitos-volume-io-cache
182 33: end-volume
183 34:
184 35: volume xitos-volume-open-behind
185 36: type performance/open-behind
186 37: subvolumes xitos-volume-quick-read
187 38: end-volume
188 39:
189 40: volume xitos-volume-md-cache
190 41: type performance/md-cache
191 42: subvolumes xitos-volume-open-behind
192 43: end-volume
193 44:
194 45: volume xitos-volume
195 46: type debug/io-stats
196 47: option latency-measurement off
197 48: option count-fop-hits off
198 49: subvolumes xitos-volume-md-cache
199 50: end-volume
200 51:
201 52: volume meta-autoload
202 53: type meta
203 54: subvolumes xitos-volume
204 55: end-volume
205 56:
206 +------------------------------------------------------------------------------+
207 [2016-04-13 06:55:11.408891] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-xitos-volume-client-0: changing port to 49152 (from 0)
208 [2016-04-13 06:55:11.413254] I [client-handshake.c:1413:select_server_supported_programs] 0-xitos-volume-client-0: Using Pro gram GlusterFS 3.3, Num (1298437), Version (330)
209 [2016-04-13 06:55:11.413766] I [client-handshake.c:1200:client_setvolume_cbk] 0-xitos-volume-client-0: Connected to xitos-vo lume-client-0, attached to remote volume '/mnt/dht/volume'.
210 [2016-04-13 06:55:11.413795] I [client-handshake.c:1210:client_setvolume_cbk] 0-xitos-volume-client-0: Server and Client lk- version numbers are not same, reopening the fds
211 [2016-04-13 06:55:11.420494] I [fuse-bridge.c:5086:fuse_graph_setup] 0-fuse: switched to graph 0
212 [2016-04-13 06:55:11.420691] I [client-handshake.c:188:client_set_lk_version_cbk] 0-xitos-volume-client-0: Server lk version = 1
213 [2016-04-13 06:55:11.420921] I [fuse-bridge.c:4015:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterf s 7.22 kernel 7.22
214 pending frames:
215 frame : type(0) op(0)
216 frame : type(0) op(0)
217 frame : type(0) op(0)
218 frame : type(0) op(0)
219 frame : type(0) op(0)
220 frame : type(0) op(0)
221 frame : type(0) op(0)
222 patchset: git://git.gluster.com/glusterfs.git
223 signal received: 6
224 time of crash:
225 2016-04-13 08:15:37
226 configuration details:
227 argp 1
228 backtrace 1
229 dlfcn 1
230 libpthread 1
231 llistxattr 1
232 setfsid 1
233 spinlock 1
234 epoll.h 1
235 xattr.h 1
236 st_atim.tv_nsec 1
237 package-string: glusterfs 3.6.7
238 /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb2)[0x7f308d9e93d2]
239 /lib64/libglusterfs.so.0(gf_print_trace+0x32d)[0x7f308da008bd]
240 /lib64/libc.so.6(+0x35a00)[0x7f308ca04a00]
241 /lib64/libc.so.6(gsignal+0x39)[0x7f308ca04989]
242 /lib64/libc.so.6(abort+0x148)[0x7f308ca06098]
243 /lib64/libc.so.6(+0x76197)[0x7f308ca45197]
244 /lib64/libc.so.6(+0x7d56d)[0x7f308ca4c56d]
245 /usr/lib64/glusterfs/3.6.7/xlator/cluster/distribute.so(dht_local_wipe+0x151)[0x7f308096ebc1]
246 /usr/lib64/glusterfs/3.6.7/xlator/cluster/distribute.so(dht_writev_cbk+0x19d)[0x7f308099b9fd]
247 /usr/lib64/glusterfs/3.6.7/xlator/protocol/client.so(client3_3_writev_cbk+0x672)[0x7f3080be4512]
248 /lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0x90)[0x7f308d7bd100]
249 /lib64/libgfrpc.so.0(rpc_clnt_notify+0x174)[0x7f308d7bd374]
250 /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f308d7b92c3]
251 /usr/lib64/glusterfs/3.6.7/rpc-transport/socket.so(+0x87a0)[0x7f3082ac17a0]
252 /usr/lib64/glusterfs/3.6.7/rpc-transport/socket.so(+0xaf94)[0x7f3082ac3f94]
253 /lib64/libglusterfs.so.0(+0x768c2)[0x7f308da3e8c2]
254 /usr/sbin/glusterfs(main+0x502)[0x7f308de92fe2]
255 /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f308c9f0af5]
256 /usr/sbin/glusterfs(+0x6381)[0x7f308de93381]
257 ---------
258 [2016-04-13 08:36:22.385866] I [MSGID: 100030] [glusterfsd.c:2035:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glu sterfs version 3.6.7 (args: /usr/sbin/glusterfs --volfile-server=10.74.125.254 --volfile-id=/xitos-volume /var/lib/nova/mnt/ a77401a594b06b2b56cc52ee61bb4def)
259 [2016-04-13 08:36:22.409856] I [dht-shared.c:337:dht_init_regex] 0-xitos-volume-dht: using regex rsync-hash-regex = ^\.(.+)\ .[^.]+$
260 [2016-04-13 08:36:22.411477] I [client.c:2268:notify] 0-xitos-volume-client-0: parent translators are ready, attempting conn ect on transport
261 Final graph:
262 +------------------------------------------------------------------------------+
263 1: volume xitos-volume-client-0
264 2: type protocol/client
265 3: option ping-timeout 42
On Fri, Apr 15, 2016 at 2:43 AM, yang.bin18@xxxxxxxxxx <yang.bin18@xxxxxxxxxx> wrote:
Glusterfs core when mounting. here is the backtree.
Program terminated with signal 6, Aborted.
#0 0x00007f308ca04989 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-55.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.11.3-49.el7.x86_64 libcom_err-1.42.9-4.el7.x86_64 libgcc-4.8.2-16.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 openssl-libs-1.0.1e-34.el7.7.x86_64 pcre-8.32-12.el7.x86_64 sssd-client-1.11.2-65.el7.x86_64 xz-libs-5.1.2-8alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64
(gdb) bt
#0 0x00007f308ca04989 in raise () from /lib64/libc.so.6
#1 0x00007f308ca06098 in abort () from /lib64/libc.so.6
#2 0x00007f308ca45197 in __libc_message () from /lib64/libc.so.6
#3 0x00007f308ca4c56d in _int_free () from /lib64/libc.so.6
#4 0x00007f308096ebc1 in dht_local_wipe (this=0x7f308f306460, local=0x7f3080042880) at dht-helper.c:475
#5 0x00007f308099b9fd in dht_writev_cbk (frame=0x7f308bc11e5c, cookie=<optimized out>, this=<optimized out>, op_ret=131072,
op_errno=0, prebuf=<optimized out>, postbuf=0x7fff2020c870, xdata=0x0) at dht-inode-write.c:84
#6 0x00007f3080be4512 in client3_3_writev_cbk (req=<optimized out>, iov=<optimized out>, count=<optimized out>,
myframe=0x7f308bc116f8) at client-rpc-fops.c:856
#7 0x00007f308d7bd100 in rpc_clnt_handle_reply (clnt=clnt@entry=0x7f308f3292d0, pollin=pollin@entry=0x7f308f39bb10)
at rpc-clnt.c:763
#8 0x00007f308d7bd374 in rpc_clnt_notify (trans=<optimized out>, mydata=0x7f308f329300, event=<optimized out>,
data="" at rpc-clnt.c:891
#9 0x00007f308d7b92c3 in rpc_transport_notify (this=this@entry=0x7f308f35f3f0, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED,
data="" at rpc-transport.c:516
#10 0x00007f3082ac17a0 in socket_event_poll_in (this=this@entry=0x7f308f35f3f0) at socket.c:2234
#11 0x00007f3082ac3f94 in socket_event_handler (fd=<optimized out>, idx=1, data="" poll_in=1, poll_out=0,
poll_err=0) at socket.c:2347
#12 0x00007f308da3e8c2 in event_dispatch_epoll_handler (i=<optimized out>, events=0x7f308f2febd0, event_pool=0x7f308f2b76c0)
at event-epoll.c:384
#13 event_dispatch_epoll (event_pool=0x7f308f2b76c0) at event-epoll.c:445
#14 0x00007f308de92fe2 in main (argc=4, argv=0x7fff2020de78) at glusterfsd.c:2060
Thanks for the report. What version of gluster is being used here? I assume this is on Centos 7.
Would it be possible to share the client log file?
Regards,
Vijay
-------------------------------------------------------- ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s). If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited. If you have received this mail in error, please delete it and notify us immediately.
_______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
