On Monday, November 10, 2014, Vijay Bellur <vbellur@xxxxxxxxxx> wrote:
On 11/08/2014 03:50 PM, Jaden Liang wrote:
Hi all,
We are testing BD xlator to verify the KVM running with gluster. After some
simple tests, we encountered a coredump of glusterfs lead by liblvm2app.so.
Hope some one here might give some advises about this issue.
We have debug for some time, and found out this coredump is triggered by a
thread-safe issue. From the core file, the top function is _update_mda()
with a invailid pointer which is from lvmcache_foreach_mda(). As we
know, the glusterfsd
has some io threads to simulate the async io. That will make more than 1
thread run into
bd_statfs_cbk(). And in liblvm2app.so, _text_read() will look up an info
in a hash
table named _pvid_hash. If no info item exist, it will allocate a new
one. However,
there isn't any lock to protect this operations! liblvm2app.so will get
crashed with
multi-thread like this precedures:
Thread A and thread B go into bd_statfs_cbk() at the same time:
1. A allocate an new info node, and put it into _pvid_hash, call
lvmcache_foreach_mda().
2. B looks up and get the info generaed by A in _pvid_hash, pass it to
lvmcache_del_mdas(), this will free the info node.
3. A keep using the info node which has been freed by B.
4. Memory crash...
Thanks for the report and the steps to recreate the problem.
#9 0x00007f83b599753f in _lvm_vg_open (mode=0x7f83b5c8971e "r",
vgname=0x11d2c50 "bd-vg", libh=0x11d3c40,
flags=<optimized out>) at lvm_vg.c:221
#10 lvm_vg_open (libh=0x11d3c40, vgname=0x11d2c50 "bd-vg",
mode=mode@entry=0x7f83b5c8971e "r", flags=flags@entry=0)
at lvm_vg.c:238
#11 0x00007f83b5c7ee36 in bd_statfs_cbk (frame=0x7f83b95416e4,
cookie=<optimized out>, this=0x119eb90, op_ret=0, op_errno=0,
buff=0x7f83b1d0ac70, xdata=0x0) at bd.c:353
One quick fix would be to serialize calls to lvm_vg_open() by holding a lock in bd xlator. Have you tried attempting that?
-Vijay
Thanks for reviewing this issue.
Yes, we also noticed this spot and had added a lock to lvm_vg_open() to test. It looks find by now. We are
going to do some more tests with a real KVM to verify this modification. Then submit a patch.
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
