Fwd: [Gluster-Maintainers] Build failed in Jenkins: regression-test-with-multiplex #831

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



This is the first nightly job failure since we reopened the master branch. Crash seems to be from fini () code path. Need investigation and RCA here.

---------- Forwarded message ---------
From: <jenkins@xxxxxxxxxxxxxxxxx>
Date: Fri, 17 Aug 2018 at 23:54
Subject: [Gluster-Maintainers] Build failed in Jenkins: regression-test-with-multiplex #831
To: <maintainers@xxxxxxxxxxx>, <amarts@xxxxxxxxxx>


See <https://build.gluster.org/job/regression-test-with-multiplex/831/display/redirect?page=changes>

Changes:

[Amar Tumballi] jbr : fix coverity issues in jbr

[Amar Tumballi] statedump : fix coverity issues

[Amar Tumballi] glusterd: coverity defects fix introduced by commit 1f3bfe7

[Amar Tumballi] features/acl: Fix a possible null dereference

[Amar Tumballi] meta : fix coverity in meta-helpers.c

[Amar Tumballi] nfs-server-mount : fix coverity issues in mount3.c

[Amar Tumballi] posix: FORWARD_NULL coverity fix

[Amar Tumballi] locks: FORWARD_NULL coverity fix

[Amar Tumballi] doc: Add details around xlator categories

[Amar Tumballi] features/changelog: Fix missing unlocks

------------------------------------------
[...truncated 1.01 MB...]
        top = 0x0
        victim = 0x0
        trav_p = 0x0
        count = 0
        victim_found = false
        ctx = 0x16cc010
        __FUNCTION__ = "posix_health_check_thread_proc"
#3  0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fba8d753bad in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 8 (Thread 0x7fba2b5f0700 (LWP 31062)):
#0  0x00007fba8d71a56d in nanosleep () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007fba8d71a404 in sleep () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007fba816f2081 in posix_disk_space_check_thread_proc (data="" at <https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/storage/posix/src/posix-helpers.c>:2286
        this = 0x7fba505beff0
        priv = 0x7fba53fe9490
        interval = 5
        ret = 0
        __FUNCTION__ = "posix_disk_space_check_thread_proc"
#3  0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007fba8d753bad in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x7fba703b3700 (LWP 31055)):
#0  0x00007fba8e092d42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fba7b345737 in iot_worker (data="" at <https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/performance/io-threads/src/io-threads.c>:195
        conf = 0x7fba53ba86b0
        this = 0x7fba53277620
        stub = 0x0
        sleep_till = {tv_sec = 1534518836, tv_nsec = 345143975}
        ret = 0
        pri = -1
        bye = false
        __FUNCTION__ = "iot_worker"
#2  0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007fba8d753bad in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 6 (Thread 0x7fba704f5700 (LWP 31054)):
#0  0x00007fba8e092995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fba7aae8308 in index_worker (data="" at <https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/features/index/src/index.c>:218
        priv = 0x7fba53b980f0
        this = 0x7fba5327d8a0
        stub = 0x0
        bye = false
#2  0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007fba8d753bad in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 5 (Thread 0x7fba1d4d5700 (LWP 30825)):
#0  0x00007fba8e092995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fba8edf9da0 in rpcsvc_request_handler (arg=0x7fba7c046210) at <https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c>:1983
        program = 0x7fba7c046210
        req = 0x0
        actor = 0x0
        done = false
        ret = 0
        __FUNCTION__ = "rpcsvc_request_handler"
#2  0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007fba8d753bad in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 4 (Thread 0x7fba1dcd6700 (LWP 30824)):
#0  0x00007fba8e092995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fba8edf9da0 in rpcsvc_request_handler (arg=0x7fba7c0465d0) at <https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c>:1983
        program = 0x7fba7c0465d0
        req = 0x0
        actor = 0x7fba7a449700 <glusterfs4_0_fop_actors+1792>
        done = false
        ret = 0
        __FUNCTION__ = "rpcsvc_request_handler"
#2  0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007fba8d753bad in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x7fba1f4d9700 (LWP 30313)):
#0  0x00007fba8e092995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fba8edf9da0 in rpcsvc_request_handler (arg=0x7fba7c0465d0) at <https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c>:1983
        program = 0x7fba7c0465d0
        req = 0x0
        actor = 0x7fba7a449700 <glusterfs4_0_fop_actors+1792>
        done = false
        ret = 0
        __FUNCTION__ = "rpcsvc_request_handler"
#2  0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007fba8d753bad in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x7fba795b7700 (LWP 29638)):
#0  0x00007fba8e092995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007fba8edf9da0 in rpcsvc_request_handler (arg=0x7fba7c0465d0) at <https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c>:1983
        program = 0x7fba7c0465d0
        req = 0x0
        actor = 0x7fba7a449a80 <glusterfs4_0_fop_actors+2688>
        done = false
        ret = 0
        __FUNCTION__ = "rpcsvc_request_handler"
#2  0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007fba8d753bad in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 1 (Thread 0x7fba82147700 (LWP 29636)):
#0  0x00007fba7a67bf95 in ios_stat_unref (iosstat=0xffffffffffffffff) at <https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/debug/io-stats/src/io-stats.c>:350
        cleanup = 0
        refcnt = 0
#1  0x00007fba7a69f766 in ios_destroy_top_stats (conf=0x7fba5005fff0) at <https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/debug/io-stats/src/io-stats.c>:3717
        i = 4
        list_head = 0x7fba50061848
        entry = 0x7fba50486400
        tmp = 0xffffffffffffff00
        list = 0x7fba50486400
        stat = 0xffffffffffffffff
        __PRETTY_FUNCTION__ = "ios_destroy_top_stats"
#2  0x00007fba7a6a0504 in ios_conf_destroy (conf=0x7fba5005fff0) at <https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/debug/io-stats/src/io-stats.c>:3966
No locals.
#3  0x00007fba7a6a0ee4 in fini (this=0x7fba5005cf30) at <https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/debug/io-stats/src/io-stats.c>:4145
        conf = 0x7fba5005fff0
        __FUNCTION__ = "fini"
#4  0x00007fba8f03822d in xlator_call_fini (this=0x7fba5005cf30) at <https://build.gluster.org/job/regression-test-with-multiplex/ws/libglusterfs/src/xlator.c>:1015
No locals.
#5  0x00007fba8f0382ce in xlator_mem_cleanup (this=0x7fba5005df70) at <https://build.gluster.org/job/regression-test-with-multiplex/ws/libglusterfs/src/xlator.c>:1036
        list = 0x7fba50486780
        trav = 0x7fba5005cf30
        inode_table = 0x0
        prev = 0x7fba5005cf30
        ctx = 0x16cc010
        trav_p = 0x0
        top = 0x0
        victim = 0x0
#6  0x00007fba7a1cd5f9 in server_rpc_notify (rpc=0x7fba7c044540, xl=0x7fba7c02f940, event=RPCSVC_EVENT_TRANSPORT_DESTROY, data="" at <https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/protocol/server/src/server.c>:565
        detached = false
        this = 0x7fba7c02f940
        trans = 0x7fba64009380
        conf = 0x7fba7c037020
        client = 0x7fba6438d8a0
        auth_path = 0x0
        ret = -1
        victim_found = true
        xlator_name = 0x7fba7c017170 "/d/backends/patchy2"
        ctx = 0x16cc010
        top = 0x7fba7c02f940
        trav_p = 0x7fba7c02f970
        travxl = 0x7fba5005df70
        xprtrefcount = 0
        tmp = 0x7fba50486cd0
        __FUNCTION__ = "server_rpc_notify"
#7  0x00007fba8edf6a59 in rpcsvc_program_notify (listener=0x7fba7c046110, event=RPCSVC_EVENT_TRANSPORT_DESTROY, data="" at <https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c>:345
        wrapper = 0x7fba7c045360
#8  0x00007fba8edf7c3f in rpcsvc_notify (trans=0x7fba64009380, mydata=0x7fba7c044540, event=RPC_TRANSPORT_CLEANUP, data="" at <https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c>:838
        ret = -1
        msg = 0x0
        new_trans = 0x0
        svc = 0x7fba7c044540
        listener = 0x7fba7c046110
        __FUNCTION__ = "rpcsvc_notify"
#9  0x00007fba8edfd6e8 in rpc_transport_unref (this=0x7fba64009380) at <https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpc-transport.c>:526
        refcount = 0
        ret = -1
        __FUNCTION__ = "rpc_transport_unref"
#10 0x00007fba83c104cd in socket_event_handler (fd=39, idx=18, gen=10, data="" poll_in=1, poll_out=0, poll_err=16) at <https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-transport/socket/src/socket.c>:3026
        sa = 0x7fba64009440
        this = 0x7fba64009380
        priv = 0x7fba64007880
        ret = -1
        ctx = 0x16cc010
        socket_closed = true
        notify_handled = false
        __FUNCTION__ = "socket_event_handler"
#11 0x00007fba8f0b5bff in event_dispatch_epoll_handler (event_pool=0x1703c30, event=0x7fba82146ea0) at <https://build.gluster.org/job/regression-test-with-multiplex/ws/libglusterfs/src/event-epoll.c>:587
        ev_data = 0x7fba82146ea4
        slot = 0x1735510
        handler = 0x7fba83c0feba <socket_event_handler>
        data = "">         idx = 18
        gen = 10
        ret = -1
        fd = 39
        handled_error_previously = false
        __FUNCTION__ = "event_dispatch_epoll_handler"
#12 0x00007fba8f0b5ef2 in event_dispatch_epoll_worker (data="" at <https://build.gluster.org/job/regression-test-with-multiplex/ws/libglusterfs/src/event-epoll.c>:663
        event = {events = 17, data = "" = 0xa00000012, fd = 18, u32 = 18, u64 = 42949672978}}
        ret = 1
        ev_data = 0x174f120
        event_pool = 0x1703c30
        myindex = 1
        timetodie = 0
        __FUNCTION__ = "event_dispatch_epoll_worker"
#13 0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#14 0x00007fba8d753bad in clone () from /lib64/libc.so.6
No symbol table info available.
=========================================================
              Finish backtrace
         program name : /build/install/sbin/glusterfsd
         corefile     : /glfs_epoll000-29630.core
=========================================================

+ rm -f /build/install/cores/gdbout.txt
+ sort /build/install/cores/liblist.txt
+ uniq
+ cat /build/install/cores/liblist.txt.tmp
+ grep -v /build/install
+ tar -cf /archives/archived_builds/build-install-regression-test-with-multiplex-831.tar /build/install/sbin /build/install/bin /build/install/lib /build/install/libexec /build/install/cores
tar: Removing leading `/' from member names
+ tar -rhf /archives/archived_builds/build-install-regression-test-with-multiplex-831.tar -T /build/install/cores/liblist.txt
tar: Removing leading `/' from member names
+ bzip2 /archives/archived_builds/build-install-regression-test-with-multiplex-831.tar
+ rm -f /build/install/cores/liblist.txt
+ rm -f /build/install/cores/liblist.txt.tmp
+ find /archives -size +1G -delete -type f
+ echo 'Cores and build archived in http://builder102.cloud.gluster.org/archived_builds/build-install-regression-test-with-multiplex-831.tar.bz2'
Cores and build archived in http://builder102.cloud.gluster.org/archived_builds/build-install-regression-test-with-multiplex-831.tar.bz2
+ echo 'Open core using the following command to get a proper stack'
Open core using the following command to get a proper stack
+ echo 'Example: From root of extracted tarball'
Example: From root of extracted tarball
+ echo '\t\tgdb -ex '\''set sysroot ./'\'' -ex '\''core-file ./build/install/cores/xxx.core'\'' <target, say ./build/install/sbin/glusterd>'
\t\tgdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/xxx.core' <target, say ./build/install/sbin/glusterd>
+ RET=1
+ '[' 1 -ne 0 ']'
+ tar -czf <https://build.gluster.org/job/regression-test-with-multiplex/831/artifact/glusterfs-logs.tgz> /var/log/glusterfs /var/log/messages /var/log/messages-20180722 /var/log/messages-20180729 /var/log/messages-20180805 /var/log/messages-20180812
tar: Removing leading `/' from member names
+ scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i <https://build.gluster.org/job/regression-test-with-multiplex/ws/> glusterfs-logs.tgz _logs_collector@xxxxxxxxxxxxxxxxxxxxxxxx:/var/www/glusterfs-logs/regression-test-with-multiplex-831.tgz
ssh: connect to host http.int.rht.gluster.org port 22: Connection timed out
lost connection
+ true
+ case $(uname -s) in
++ uname -s
+ /sbin/sysctl -w kernel.core_pattern=/%e-%p.core
kernel.core_pattern = /%e-%p.core
+ exit 1
Build step 'Execute shell' marked build as failure
_______________________________________________
maintainers mailing list
maintainers@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/maintainers
--
- Atin (atinm)
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-devel

[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux