Fwd: [Gluster-Maintainers] Build failed in Jenkins: regression-test-burn-in #4710

[Atin Mukherjee <amukherj@xxxxxxxxxx>]

Since last few days I was trying to understand the nightly failures we were seeing even after addressing the port already in use issue. So here's the analysis:

From console output of https://build.gluster.org/job/regression-test-burn-in/4710/consoleFull

<snip>

19:51:56 Started by upstream project "nightly-master" build number 843
19:51:56 originally caused by:
19:51:56  Started by timer
19:51:56 Running as SYSTEM
19:51:57 Building remotely on builder209.aws.gluster.org (centos7) in workspace /home/jenkins/root/workspace/regression-test-burn-in
19:51:58 No credentials specified
19:51:58  > git rev-parse --is-inside-work-tree # timeout=10
19:51:58 Fetching changes from the remote Git repository
19:51:58  > git config remote.origin.url git://review.gluster.org/glusterfs.git # timeout=10
19:51:58 Fetching upstream changes from git://review.gluster.org/glusterfs.git
19:51:58  > git --version # timeout=10
19:51:58  > git fetch --tags --progress git://review.gluster.org/glusterfs.git refs/heads/master # timeout=10
19:52:01  > git rev-parse origin/master^{commit} # timeout=10
19:52:01 Checking out Revision a31fad885c30cbc1bea652349c7d52bac1414c08 (origin/master)
19:52:01  > git config core.sparsecheckout # timeout=10
19:52:01  > git checkout -f a31fad885c30cbc1bea652349c7d52bac1414c08 # timeout=10
19:52:02 Commit message: "tests: heal-info add --xml option for more coverage"
19:52:02  > git rev-list --no-walk a31fad885c30cbc1bea652349c7d52bac1414c08 # timeout=10
19:52:02 [regression-test-burn-in] $ /bin/bash /tmp/jenkins7274529097702336737.sh
19:52:02 Start time Mon Aug 26 14:22:02 UTC 2019

</snip>

The latest commit which it picked up as part of git checkout is quite old and hence we continue to see the similar failures in the latest nightly runs which has been already addressed by commit c370c70

commit c370c70f77079339e2cfb7f284f3a2fb13fd2f97
Author: Mohit Agrawal <moagrawal@xxxxxxxxxx>
Date:   Tue Aug 13 18:45:43 2019 +0530

    rpc: glusterd start is failed and throwing an error Address already in use
   
    Problem: Some of the .t are failed due to bind is throwing
             an error EADDRINUSE
   
    Solution: After killing all gluster processes .t is trying
              to start glusterd but somehow if kernel has not cleaned
              up resources(socket) then glusterd startup is failed due to
              bind system call failure.To avoid the issue retries to call
              bind 10 times to execute system call succesfully
   
    Change-Id: Ia5fd6b788f7b211c1508c1b7304fc08a32266629
    Fixes: bz#1743020
    Signed-off-by: Mohit Agrawal <moagrawal@xxxxxxxxxx>

So the (puzzling) question is - why are we picking up old commit?

In my local setup when I run the following command I do see the latest commit id being picked up:

atin@dhcp35-96:~/codebase/upstream/glusterfs_master/glusterfs$ git rev-parse origin/master^{commit} # timeout=10
7926992e65d0a07fdc784a6e45740306d9b4a9f2

atin@dhcp35-96:~/codebase/upstream/glusterfs_master/glusterfs$ git show 7926992e65d0a07fdc784a6e45740306d9b4a9f2
commit 7926992e65d0a07fdc784a6e45740306d9b4a9f2 (origin/master, origin/HEAD, master)
Author: Sanju Rakonde <srakonde@xxxxxxxxxx>
Date:   Mon Aug 26 12:38:40 2019 +0530

    glusterd: Unused value coverity fix
   
    CID: 1288765
    updates: bz#789278
   
    Change-Id: Ie6b01f81339769f44d82fd7c32ad0ed1a697c69c
    Signed-off-by: Sanju Rakonde <srakonde@xxxxxxxxxx>

---------- Forwarded message ---------
From: <jenkins@xxxxxxxxxxxxxxxxx>
Date: Mon, Aug 26, 2019 at 11:32 PM
Subject: [Gluster-Maintainers] Build failed in Jenkins: regression-test-burn-in #4710
To: <maintainers@xxxxxxxxxxx>

See <https://build.gluster.org/job/regression-test-burn-in/4710/display/redirect>

------------------------------------------
[...truncated 4.18 MB...]
./tests/features/lock-migration/lkmigration-set-option.t  -  7 second
./tests/bugs/upcall/bug-1458127.t  -  7 second
./tests/bugs/transport/bug-873367.t  -  7 second
./tests/bugs/snapshot/bug-1260848.t  -  7 second
./tests/bugs/shard/shard-inode-refcount-test.t  -  7 second
./tests/bugs/replicate/bug-986905.t  -  7 second
./tests/bugs/replicate/bug-921231.t  -  7 second
./tests/bugs/replicate/bug-1132102.t  -  7 second
./tests/bugs/replicate/bug-1037501.t  -  7 second
./tests/bugs/posix/bug-1175711.t  -  7 second
./tests/bugs/posix/bug-1122028.t  -  7 second
./tests/bugs/glusterfs/bug-861015-log.t  -  7 second
./tests/bugs/fuse/bug-983477.t  -  7 second
./tests/bugs/ec/bug-1227869.t  -  7 second
./tests/bugs/distribute/bug-1086228.t  -  7 second
./tests/bugs/cli/bug-1087487.t  -  7 second
./tests/bitrot/br-stub.t  -  7 second
./tests/basic/ctime/ctime-noatime.t  -  7 second
./tests/basic/afr/ta-write-on-bad-brick.t  -  7 second
./tests/basic/afr/ta.t  -  7 second
./tests/basic/afr/ta-shd.t  -  7 second
./tests/basic/afr/root-squash-self-heal.t  -  7 second
./tests/basic/afr/granular-esh/add-brick.t  -  7 second
./tests/bugs/upcall/bug-1369430.t  -  6 second
./tests/bugs/snapshot/bug-1064768.t  -  6 second
./tests/bugs/shard/bug-1258334.t  -  6 second
./tests/bugs/replicate/bug-1250170-fsync.t  -  6 second
./tests/bugs/quota/bug-1250582-volume-reset-should-not-remove-quota-quota-deem-statfs.t  -  6 second
./tests/bugs/quota/bug-1243798.t  -  6 second
./tests/bugs/quota/bug-1104692.t  -  6 second
./tests/bugs/protocol/bug-1321578.t  -  6 second
./tests/bugs/nfs/bug-915280.t  -  6 second
./tests/bugs/io-cache/bug-858242.t  -  6 second
./tests/bugs/glusterfs-server/bug-877992.t  -  6 second
./tests/bugs/glusterfs/bug-902610.t  -  6 second
./tests/bugs/distribute/bug-884597.t  -  6 second
./tests/bugs/core/bug-1699025-brick-mux-detach-brick-fd-issue.t  -  6 second
./tests/bugs/core/bug-1168803-snapd-option-validation-fix.t  -  6 second
./tests/bugs/bug-1702299.t  -  6 second
./tests/bugs/bug-1371806_2.t  -  6 second
./tests/bugs/bug-1258069.t  -  6 second
./tests/bugs/bitrot/1209818-vol-info-show-scrub-process-properly.t  -  6 second
./tests/bugs/bitrot/1209751-bitrot-scrub-tunable-reset.t  -  6 second
./tests/basic/glusterd/thin-arbiter-volume-probe.t  -  6 second
./tests/basic/gfapi/libgfapi-fini-hang.t  -  6 second
./tests/basic/fencing/fencing-crash-conistency.t  -  6 second
./tests/basic/ec/statedump.t  -  6 second
./tests/basic/distribute/file-create.t  -  6 second
./tests/basic/afr/tarissue.t  -  6 second
./tests/basic/afr/gfid-heal.t  -  6 second
./tests/basic/afr/afr-read-hash-mode.t  -  6 second
./tests/basic/afr/add-brick-self-heal.t  -  6 second
./tests/gfid2path/gfid2path_fuse.t  -  5 second
./tests/bugs/shard/bug-1259651.t  -  5 second
./tests/bugs/replicate/bug-767585-gfid.t  -  5 second
./tests/bugs/replicate/bug-1686568-send-truncate-on-arbiter-from-shd.t  -  5 second
./tests/bugs/replicate/bug-1626994-info-split-brain.t  -  5 second
./tests/bugs/replicate/bug-1365455.t  -  5 second
./tests/bugs/replicate/bug-1101647.t  -  5 second
./tests/bugs/nfs/bug-877885.t  -  5 second
./tests/bugs/nfs/bug-847622.t  -  5 second
./tests/bugs/nfs/bug-1116503.t  -  5 second
./tests/bugs/md-cache/setxattr-prepoststat.t  -  5 second
./tests/bugs/md-cache/bug-1211863_unlink.t  -  5 second
./tests/bugs/md-cache/afr-stale-read.t  -  5 second
./tests/bugs/io-stats/bug-1598548.t  -  5 second
./tests/bugs/glusterfs/bug-895235.t  -  5 second
./tests/bugs/glusterfs/bug-856455.t  -  5 second
./tests/bugs/glusterfs/bug-848251.t  -  5 second
./tests/bugs/glusterd/quorum-value-check.t  -  5 second
./tests/bugs/glusterd/bug-1242875-do-not-pass-volinfo-quota.t  -  5 second
./tests/bugs/ec/bug-1179050.t  -  5 second
./tests/bugs/distribute/bug-912564.t  -  5 second
./tests/bugs/distribute/bug-1368012.t  -  5 second
./tests/bugs/core/bug-986429.t  -  5 second
./tests/bugs/core/bug-908146.t  -  5 second
./tests/bugs/bug-1371806_1.t  -  5 second
./tests/bugs/bitrot/bug-1229134-bitd-not-support-vol-set.t  -  5 second
./tests/bugs/bitrot/1207029-bitrot-daemon-should-start-on-valid-node.t  -  5 second
./tests/basic/playground/template-xlator-sanity.t  -  5 second
./tests/basic/hardlink-limit.t  -  5 second
./tests/basic/glusterd/arbiter-volume-probe.t  -  5 second
./tests/basic/ec/nfs.t  -  5 second
./tests/basic/ec/ec-read-policy.t  -  5 second
./tests/basic/ec/ec-anonymous-fd.t  -  5 second
./tests/basic/afr/arbiter-remove-brick.t  -  5 second
./tests/gfid2path/gfid2path_nfs.t  -  4 second
./tests/gfid2path/get-gfid-to-path.t  -  4 second
./tests/gfid2path/block-mount-access.t  -  4 second
./tests/bugs/upcall/bug-upcall-stat.t  -  4 second
./tests/bugs/trace/bug-797171.t  -  4 second
./tests/bugs/snapshot/bug-1178079.t  -  4 second
./tests/bugs/shard/bug-1342298.t  -  4 second
./tests/bugs/shard/bug-1272986.t  -  4 second
./tests/bugs/rpc/bug-954057.t  -  4 second
./tests/bugs/replicate/bug-886998.t  -  4 second
./tests/bugs/replicate/bug-1480525.t  -  4 second
./tests/bugs/replicate/bug-1448804-check-quorum-type-values.t  -  4 second
./tests/bugs/replicate/bug-1325792.t  -  4 second
./tests/bugs/readdir-ahead/bug-1670253-consistent-metadata.t  -  4 second
./tests/bugs/posix/bug-gfid-path.t  -  4 second
./tests/bugs/posix/bug-765380.t  -  4 second
./tests/bugs/posix/bug-1619720.t  -  4 second
./tests/bugs/nfs/zero-atime.t  -  4 second
./tests/bugs/nfs/subdir-trailing-slash.t  -  4 second
./tests/bugs/nfs/socket-as-fifo.t  -  4 second
./tests/bugs/nfs/showmount-many-clients.t  -  4 second
./tests/bugs/nfs/bug-1161092-nfs-acls.t  -  4 second
./tests/bugs/nfs/bug-1143880-fix-gNFSd-auth-crash.t  -  4 second
./tests/bugs/glusterfs-server/bug-873549.t  -  4 second
./tests/bugs/glusterfs-server/bug-864222.t  -  4 second
./tests/bugs/glusterfs/bug-893378.t  -  4 second
./tests/bugs/glusterd/bug-948729/bug-948729-force.t  -  4 second
./tests/bugs/glusterd/bug-1482906-peer-file-blank-line.t  -  4 second
./tests/bugs/glusterd/bug-1091935-brick-order-check-from-cli-to-glusterd.t  -  4 second
./tests/bugs/geo-replication/bug-1296496.t  -  4 second
./tests/bugs/ec/bug-1161621.t  -  4 second
./tests/bugs/distribute/bug-1088231.t  -  4 second
./tests/bugs/cli/bug-977246.t  -  4 second
./tests/bugs/cli/bug-1004218.t  -  4 second
./tests/bugs/bug-1138841.t  -  4 second
./tests/bugs/bitrot/bug-1210684-scrub-pause-resume-error-handling.t  -  4 second
./tests/bugs/access-control/bug-1051896.t  -  4 second
./tests/bitrot/bug-1221914.t  -  4 second
./tests/basic/ec/ec-internal-xattrs.t  -  4 second
./tests/basic/distribute/non-root-unlink-stale-linkto.t  -  4 second
./tests/basic/distribute/bug-1265677-use-readdirp.t  -  4 second
./tests/basic/changelog/changelog-rename.t  -  4 second
./tests/basic/afr/ta-check-locks.t  -  4 second
./tests/basic/afr/heal-info.t  -  4 second
./tests/performance/quick-read.t  -  3 second
./tests/line-coverage/meta-max-coverage.t  -  3 second
./tests/bugs/upcall/bug-1422776.t  -  3 second
./tests/bugs/upcall/bug-1394131.t  -  3 second
./tests/bugs/unclassified/bug-1034085.t  -  3 second
./tests/bugs/snapshot/bug-1111041.t  -  3 second
./tests/bugs/shard/bug-1256580.t  -  3 second
./tests/bugs/shard/bug-1250855.t  -  3 second
./tests/bugs/replicate/bug-976800.t  -  3 second
./tests/bugs/replicate/bug-880898.t  -  3 second
./tests/bugs/read-only/bug-1134822-read-only-default-in-graph.t  -  3 second
./tests/bugs/readdir-ahead/bug-1446516.t  -  3 second
./tests/bugs/readdir-ahead/bug-1439640.t  -  3 second
./tests/bugs/readdir-ahead/bug-1390050.t  -  3 second
./tests/bugs/quota/bug-1287996.t  -  3 second
./tests/bugs/quick-read/bug-846240.t  -  3 second
./tests/bugs/nl-cache/bug-1451588.t  -  3 second
./tests/bugs/nfs/bug-1210338.t  -  3 second
./tests/bugs/nfs/bug-1166862.t  -  3 second
./tests/bugs/md-cache/bug-1632503.t  -  3 second
./tests/bugs/md-cache/bug-1476324.t  -  3 second
./tests/bugs/glusterfs-server/bug-861542.t  -  3 second
./tests/bugs/glusterfs/bug-869724.t  -  3 second
./tests/bugs/glusterfs/bug-844688.t  -  3 second
./tests/bugs/glusterfs/bug-1482528.t  -  3 second
./tests/bugs/glusterd/bug-948729/bug-948729.t  -  3 second
./tests/bugs/glusterd/bug-948729/bug-948729-mode-script.t  -  3 second
./tests/bugs/fuse/bug-1336818.t  -  3 second
./tests/bugs/fuse/bug-1126048.t  -  3 second
./tests/bugs/distribute/bug-907072.t  -  3 second
./tests/bugs/core/log-bug-1362520.t  -  3 second
./tests/bugs/core/io-stats-1322825.t  -  3 second
./tests/bugs/core/bug-913544.t  -  3 second
./tests/bugs/core/bug-845213.t  -  3 second
./tests/bugs/core/bug-834465.t  -  3 second
./tests/bugs/core/bug-1421721-mpx-toggle.t  -  3 second
./tests/bugs/core/bug-1135514-allow-setxattr-with-null-value.t  -  3 second
./tests/bugs/core/bug-1117951.t  -  3 second
./tests/bugs/core/949327.t  -  3 second
./tests/bugs/cli/bug-983317-volume-get.t  -  3 second
./tests/bugs/cli/bug-961307.t  -  3 second
./tests/bugs/access-control/bug-1387241.t  -  3 second
./tests/bitrot/bug-internal-xattrs-check-1243391.t  -  3 second
./tests/basic/quota-rename.t  -  3 second
./tests/basic/glusterd/check-cloudsync-ancestry.t  -  3 second
./tests/basic/fops-sanity.t  -  3 second
./tests/basic/fencing/test-fence-option.t  -  3 second
./tests/basic/ec/ec-fallocate.t  -  3 second
./tests/basic/ec/dht-rename.t  -  3 second
./tests/basic/distribute/lookup.t  -  3 second
./tests/basic/distribute/debug-xattrs.t  -  3 second
./tests/line-coverage/some-features-in-libglusterfs.t  -  2 second
./tests/bugs/unclassified/bug-991622.t  -  2 second
./tests/bugs/shard/bug-1245547.t  -  2 second
./tests/bugs/replicate/bug-884328.t  -  2 second
./tests/bugs/readdir-ahead/bug-1512437.t  -  2 second
./tests/bugs/posix/disallow-gfid-volumeid-removexattr.t  -  2 second
./tests/bugs/nfs/bug-970070.t  -  2 second
./tests/bugs/nfs/bug-1302948.t  -  2 second
./tests/bugs/logging/bug-823081.t  -  2 second
./tests/bugs/glusterfs-server/bug-889996.t  -  2 second
./tests/bugs/glusterfs/bug-860297.t  -  2 second
./tests/bugs/glusterfs/bug-811493.t  -  2 second
./tests/bugs/glusterd/bug-1085330-and-bug-916549.t  -  2 second
./tests/bugs/fuse/bug-1283103.t  -  2 second
./tests/bugs/distribute/bug-924265.t  -  2 second
./tests/bugs/distribute/bug-1204140.t  -  2 second
./tests/bugs/core/bug-924075.t  -  2 second
./tests/bugs/core/bug-903336.t  -  2 second
./tests/bugs/core/bug-1119582.t  -  2 second
./tests/bugs/core/bug-1111557.t  -  2 second
./tests/bugs/cli/bug-969193.t  -  2 second
./tests/bugs/cli/bug-949298.t  -  2 second
./tests/bugs/cli/bug-1378842-volume-get-all.t  -  2 second
./tests/basic/md-cache/bug-1418249.t  -  2 second
./tests/basic/afr/arbiter-cli.t  -  2 second
./tests/line-coverage/volfile-with-all-graph-syntax.t  -  1 second
./tests/bugs/shard/bug-1261773.t  -  1 second
./tests/bugs/replicate/ta-inode-refresh-read.t  -  1 second
./tests/bugs/glusterfs/bug-892730.t  -  1 second
./tests/bugs/glusterfs/bug-853690.t  -  1 second
./tests/bugs/cli/bug-921215.t  -  1 second
./tests/bugs/cli/bug-867252.t  -  1 second
./tests/bugs/cli/bug-764638.t  -  1 second
./tests/bugs/cli/bug-1047378.t  -  1 second
./tests/basic/posixonly.t  -  1 second
./tests/basic/peer-parsing.t  -  1 second
./tests/basic/netgroup_parsing.t  -  1 second
./tests/basic/gfapi/sink.t  -  1 second
./tests/basic/exports_parsing.t  -  1 second
./tests/basic/glusterfsd-args.t  -  0 second


4 test(s) failed
./tests/bugs/core/multiplex-limit-issue-151.t
./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t
./tests/bugs/glusterd/brick-mux-validation.t
./tests/bugs/glusterd/bug-1595320.t

0 test(s) generated core


10 test(s) needed retry
./tests/bugs/core/bug-1119582.t
./tests/bugs/core/multiplex-limit-issue-151.t
./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t
./tests/bugs/glusterd/brick-mux-validation.t
./tests/bugs/glusterd/bug-1595320.t
./tests/bugs/glusterd/bug-1696046.t
./tests/bugs/glusterd/optimized-basic-testcases.t
./tests/bugs/replicate/bug-1134691-afr-lookup-metadata-heal.t
./tests/bugs/replicate/bug-976800.t
./tests/bugs/snapshot/bug-1111041.t

Result is 1

tar: Removing leading `/' from member names
kernel.core_pattern = /%e-%p.core
Build step 'Execute shell' marked build as failure
_______________________________________________
maintainers mailing list
maintainers@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/maintainers

_______________________________________________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/836554017

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/486278655

Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-devel