On Fri, Jun 7, 2019 at 10:07 AM Amar Tumballi Suryanarayan <atumball@xxxxxxxxxx> wrote:
Got time to test subdir-mount.t failing in brick-mux scenario.I noticed some issues, where I need further help from glusterd team.subdir-mount.t expects 'hook' script to run after add-brick to make sure the required subdirectories are healed and are present in new bricks. This is important as subdir mount expects the subdirs to exist for successful mount.But in case of brick-mux setup, I see that in some cases (6/10), hook script (add-brick/post-hook/S13-create-subdir-mount.sh) started getting executed after 20second of finishing the add-brick command. Due to this, the mount which we execute after add-brick failed.My question is, what is making post hook script to run so late ??
It's not only the add-brick in the post hook. Given post hook scripts are async in nature, I see the respective hook scripts of create/start/set volume operation have executed quite a late which is very surprising until and unless some thread has been stuck for quite a while. Unfortunately for both Mohit and I, the issue isn't reproducible locally. Mohit would give it a try in softserve infra but at this point of time, there's no conclusive evidence, the analysis continues.
Amar - would it be possible for you to do a git blame given you can reproduce this? May 31 nightly (https://build.gluster.org/job/regression-test-with-multiplex/1359/) is when this test started failing.
I can recreate the issues locally on my laptop too.On Sat, Jun 1, 2019 at 4:55 PM Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:subdir-mount.t has started failing in brick mux regression nightly. This needs to be fixed.Raghavendra - did we manage to get any further clue on uss.t failure?------------ Forwarded message ---------
From: <jenkins@xxxxxxxxxxxxxxxxx>
Date: Fri, 31 May 2019 at 23:34
Subject: [Gluster-Maintainers] Build failed in Jenkins: regression-test-with-multiplex #1359
To: <maintainers@xxxxxxxxxxx>, <rkavunga@xxxxxxxxxx>, <amarts@xxxxxxxxxx>, <amukherj@xxxxxxxxxx>, <khiremat@xxxxxxxxxx>
See <https://build.gluster.org/job/regression-test-with-multiplex/1359/display/redirect?page=changes>
Changes:
[atin] glusterd: add an op-version check
[atin] glusterd/svc: glusterd_svcs_stop should call individual wrapper function
[atin] glusterd/svc: Stop stale process using the glusterd_proc_stop
[Amar Tumballi] lcov: more coverage to shard, old-protocol, sdfs
[Kotresh H R] tests/geo-rep: Add EC volume test case
[Amar Tumballi] glusterfsd/cleanup: Protect graph object under a lock
[Mohammed Rafi KC] glusterd/shd: Optimize the glustershd manager to send reconfigure
[Kotresh H R] tests/geo-rep: Add tests to cover glusterd geo-rep
[atin] glusterd: Optimize code to copy dictionary in handshake code path
------------------------------------------
[...truncated 3.18 MB...]
./tests/basic/afr/stale-file-lookup.t - 9 second
./tests/basic/afr/granular-esh/replace-brick.t - 9 second
./tests/basic/afr/granular-esh/add-brick.t - 9 second
./tests/basic/afr/gfid-mismatch.t - 9 second
./tests/performance/open-behind.t - 8 second
./tests/features/ssl-authz.t - 8 second
./tests/features/readdir-ahead.t - 8 second
./tests/bugs/upcall/bug-1458127.t - 8 second
./tests/bugs/transport/bug-873367.t - 8 second
./tests/bugs/replicate/bug-1498570-client-iot-graph-check.t - 8 second
./tests/bugs/replicate/bug-1132102.t - 8 second
./tests/bugs/quota/bug-1250582-volume-reset-should-not-remove-quota-quota-deem-statfs.t - 8 second
./tests/bugs/quota/bug-1104692.t - 8 second
./tests/bugs/posix/bug-1360679.t - 8 second
./tests/bugs/posix/bug-1122028.t - 8 second
./tests/bugs/nfs/bug-1157223-symlink-mounting.t - 8 second
./tests/bugs/glusterfs/bug-861015-log.t - 8 second
./tests/bugs/glusterd/sync-post-glusterd-restart.t - 8 second
./tests/bugs/glusterd/bug-1696046.t - 8 second
./tests/bugs/fuse/bug-983477.t - 8 second
./tests/bugs/ec/bug-1227869.t - 8 second
./tests/bugs/distribute/bug-1088231.t - 8 second
./tests/bugs/distribute/bug-1086228.t - 8 second
./tests/bugs/cli/bug-1087487.t - 8 second
./tests/bugs/cli/bug-1022905.t - 8 second
./tests/bugs/bug-1258069.t - 8 second
./tests/bugs/bitrot/1209752-volume-status-should-show-bitrot-scrub-info.t - 8 second
./tests/basic/xlator-pass-through-sanity.t - 8 second
./tests/basic/quota-nfs.t - 8 second
./tests/basic/glusterd/arbiter-volume.t - 8 second
./tests/basic/ctime/ctime-noatime.t - 8 second
./tests/line-coverage/cli-peer-and-volume-operations.t - 7 second
./tests/gfid2path/get-gfid-to-path.t - 7 second
./tests/bugs/upcall/bug-1369430.t - 7 second
./tests/bugs/snapshot/bug-1260848.t - 7 second
./tests/bugs/shard/shard-inode-refcount-test.t - 7 second
./tests/bugs/shard/bug-1258334.t - 7 second
./tests/bugs/replicate/bug-767585-gfid.t - 7 second
./tests/bugs/replicate/bug-1448804-check-quorum-type-values.t - 7 second
./tests/bugs/replicate/bug-1250170-fsync.t - 7 second
./tests/bugs/posix/bug-1175711.t - 7 second
./tests/bugs/nfs/bug-915280.t - 7 second
./tests/bugs/md-cache/setxattr-prepoststat.t - 7 second
./tests/bugs/md-cache/bug-1211863_unlink.t - 7 second
./tests/bugs/glusterfs/bug-848251.t - 7 second
./tests/bugs/distribute/bug-1122443.t - 7 second
./tests/bugs/changelog/bug-1208470.t - 7 second
./tests/bugs/bug-1702299.t - 7 second
./tests/bugs/bug-1371806_2.t - 7 second
./tests/bugs/bitrot/1209818-vol-info-show-scrub-process-properly.t - 7 second
./tests/bugs/bitrot/1209751-bitrot-scrub-tunable-reset.t - 7 second
./tests/bugs/bitrot/1207029-bitrot-daemon-should-start-on-valid-node.t - 7 second
./tests/bitrot/br-stub.t - 7 second
./tests/basic/glusterd/arbiter-volume-probe.t - 7 second
./tests/basic/gfapi/libgfapi-fini-hang.t - 7 second
./tests/basic/fencing/fencing-crash-conistency.t - 7 second
./tests/basic/distribute/file-create.t - 7 second
./tests/basic/afr/tarissue.t - 7 second
./tests/basic/afr/gfid-heal.t - 7 second
./tests/bugs/snapshot/bug-1178079.t - 6 second
./tests/bugs/snapshot/bug-1064768.t - 6 second
./tests/bugs/shard/bug-1342298.t - 6 second
./tests/bugs/shard/bug-1259651.t - 6 second
./tests/bugs/replicate/bug-1686568-send-truncate-on-arbiter-from-shd.t - 6 second
./tests/bugs/replicate/bug-1626994-info-split-brain.t - 6 second
./tests/bugs/replicate/bug-1325792.t - 6 second
./tests/bugs/replicate/bug-1101647.t - 6 second
./tests/bugs/quota/bug-1243798.t - 6 second
./tests/bugs/protocol/bug-1321578.t - 6 second
./tests/bugs/nfs/bug-877885.t - 6 second
./tests/bugs/nfs/bug-1143880-fix-gNFSd-auth-crash.t - 6 second
./tests/bugs/md-cache/bug-1476324.t - 6 second
./tests/bugs/md-cache/afr-stale-read.t - 6 second
./tests/bugs/io-cache/bug-858242.t - 6 second
./tests/bugs/glusterfs/bug-893378.t - 6 second
./tests/bugs/glusterfs/bug-856455.t - 6 second
./tests/bugs/glusterd/quorum-value-check.t - 6 second
./tests/bugs/ec/bug-1179050.t - 6 second
./tests/bugs/distribute/bug-912564.t - 6 second
./tests/bugs/distribute/bug-884597.t - 6 second
./tests/bugs/distribute/bug-1368012.t - 6 second
./tests/bugs/core/bug-986429.t - 6 second
./tests/bugs/core/bug-1699025-brick-mux-detach-brick-fd-issue.t - 6 second
./tests/bugs/core/bug-1168803-snapd-option-validation-fix.t - 6 second
./tests/bugs/bug-1371806_1.t - 6 second
./tests/bugs/bitrot/bug-1229134-bitd-not-support-vol-set.t - 6 second
./tests/bugs/bitrot/bug-1210684-scrub-pause-resume-error-handling.t - 6 second
./tests/bitrot/bug-1221914.t - 6 second
./tests/basic/trace.t - 6 second
./tests/basic/playground/template-xlator-sanity.t - 6 second
./tests/basic/ec/nfs.t - 6 second
./tests/basic/ec/ec-read-policy.t - 6 second
./tests/basic/ec/ec-anonymous-fd.t - 6 second
./tests/basic/distribute/non-root-unlink-stale-linkto.t - 6 second
./tests/basic/changelog/changelog-rename.t - 6 second
./tests/basic/afr/heal-info.t - 6 second
./tests/basic/afr/afr-read-hash-mode.t - 6 second
./tests/gfid2path/gfid2path_nfs.t - 5 second
./tests/bugs/upcall/bug-1422776.t - 5 second
./tests/bugs/replicate/bug-886998.t - 5 second
./tests/bugs/replicate/bug-1365455.t - 5 second
./tests/bugs/readdir-ahead/bug-1670253-consistent-metadata.t - 5 second
./tests/bugs/posix/bug-gfid-path.t - 5 second
./tests/bugs/posix/bug-765380.t - 5 second
./tests/bugs/nfs/bug-847622.t - 5 second
./tests/bugs/nfs/bug-1116503.t - 5 second
./tests/bugs/io-stats/bug-1598548.t - 5 second
./tests/bugs/glusterfs-server/bug-877992.t - 5 second
./tests/bugs/glusterfs-server/bug-873549.t - 5 second
./tests/bugs/glusterfs/bug-895235.t - 5 second
./tests/bugs/fuse/bug-1126048.t - 5 second
./tests/bugs/distribute/bug-907072.t - 5 second
./tests/bugs/core/bug-913544.t - 5 second
./tests/bugs/core/bug-908146.t - 5 second
./tests/bugs/access-control/bug-1051896.t - 5 second
./tests/basic/ec/ec-internal-xattrs.t - 5 second
./tests/basic/ec/ec-fallocate.t - 5 second
./tests/basic/distribute/bug-1265677-use-readdirp.t - 5 second
./tests/basic/afr/arbiter-remove-brick.t - 5 second
./tests/performance/quick-read.t - 4 second
./tests/gfid2path/block-mount-access.t - 4 second
./tests/features/delay-gen.t - 4 second
./tests/bugs/upcall/bug-upcall-stat.t - 4 second
./tests/bugs/upcall/bug-1394131.t - 4 second
./tests/bugs/unclassified/bug-1034085.t - 4 second
./tests/bugs/snapshot/bug-1111041.t - 4 second
./tests/bugs/shard/bug-1272986.t - 4 second
./tests/bugs/shard/bug-1256580.t - 4 second
./tests/bugs/shard/bug-1250855.t - 4 second
./tests/bugs/shard/bug-1245547.t - 4 second
./tests/bugs/rpc/bug-954057.t - 4 second
./tests/bugs/replicate/bug-976800.t - 4 second
./tests/bugs/replicate/bug-880898.t - 4 second
./tests/bugs/replicate/bug-1480525.t - 4 second
./tests/bugs/read-only/bug-1134822-read-only-default-in-graph.t - 4 second
./tests/bugs/readdir-ahead/bug-1446516.t - 4 second
./tests/bugs/readdir-ahead/bug-1439640.t - 4 second
./tests/bugs/readdir-ahead/bug-1390050.t - 4 second
./tests/bugs/quota/bug-1287996.t - 4 second
./tests/bugs/quick-read/bug-846240.t - 4 second
./tests/bugs/posix/disallow-gfid-volumeid-removexattr.t - 4 second
./tests/bugs/posix/bug-1619720.t - 4 second
./tests/bugs/nl-cache/bug-1451588.t - 4 second
./tests/bugs/nfs/zero-atime.t - 4 second
./tests/bugs/nfs/subdir-trailing-slash.t - 4 second
./tests/bugs/nfs/socket-as-fifo.t - 4 second
./tests/bugs/nfs/showmount-many-clients.t - 4 second
./tests/bugs/nfs/bug-1210338.t - 4 second
./tests/bugs/nfs/bug-1166862.t - 4 second
./tests/bugs/nfs/bug-1161092-nfs-acls.t - 4 second
./tests/bugs/md-cache/bug-1632503.t - 4 second
./tests/bugs/glusterfs-server/bug-864222.t - 4 second
./tests/bugs/glusterfs/bug-1482528.t - 4 second
./tests/bugs/glusterd/bug-948729/bug-948729-mode-script.t - 4 second
./tests/bugs/glusterd/bug-948729/bug-948729-force.t - 4 second
./tests/bugs/glusterd/bug-1482906-peer-file-blank-line.t - 4 second
./tests/bugs/glusterd/bug-1091935-brick-order-check-from-cli-to-glusterd.t - 4 second
./tests/bugs/geo-replication/bug-1296496.t - 4 second
./tests/bugs/fuse/bug-1336818.t - 4 second
./tests/bugs/fuse/bug-1283103.t - 4 second
./tests/bugs/core/io-stats-1322825.t - 4 second
./tests/bugs/core/bug-834465.t - 4 second
./tests/bugs/core/bug-1135514-allow-setxattr-with-null-value.t - 4 second
./tests/bugs/core/949327.t - 4 second
./tests/bugs/cli/bug-977246.t - 4 second
./tests/bugs/cli/bug-961307.t - 4 second
./tests/bugs/cli/bug-1004218.t - 4 second
./tests/bugs/bug-1138841.t - 4 second
./tests/bugs/access-control/bug-1387241.t - 4 second
./tests/bitrot/bug-internal-xattrs-check-1243391.t - 4 second
./tests/basic/quota-rename.t - 4 second
./tests/basic/hardlink-limit.t - 4 second
./tests/basic/ec/dht-rename.t - 4 second
./tests/basic/distribute/lookup.t - 4 second
./tests/line-coverage/meta-max-coverage.t - 3 second
./tests/gfid2path/gfid2path_fuse.t - 3 second
./tests/bugs/unclassified/bug-991622.t - 3 second
./tests/bugs/trace/bug-797171.t - 3 second
./tests/bugs/glusterfs-server/bug-861542.t - 3 second
./tests/bugs/glusterfs/bug-869724.t - 3 second
./tests/bugs/glusterfs/bug-860297.t - 3 second
./tests/bugs/glusterfs/bug-844688.t - 3 second
./tests/bugs/glusterd/bug-948729/bug-948729.t - 3 second
./tests/bugs/distribute/bug-1204140.t - 3 second
./tests/bugs/core/bug-924075.t - 3 second
./tests/bugs/core/bug-845213.t - 3 second
./tests/bugs/core/bug-1421721-mpx-toggle.t - 3 second
./tests/bugs/core/bug-1119582.t - 3 second
./tests/bugs/core/bug-1117951.t - 3 second
./tests/bugs/cli/bug-983317-volume-get.t - 3 second
./tests/bugs/cli/bug-867252.t - 3 second
./tests/basic/glusterd/check-cloudsync-ancestry.t - 3 second
./tests/basic/fops-sanity.t - 3 second
./tests/basic/fencing/test-fence-option.t - 3 second
./tests/basic/distribute/debug-xattrs.t - 3 second
./tests/basic/afr/ta-check-locks.t - 3 second
./tests/line-coverage/volfile-with-all-graph-syntax.t - 2 second
./tests/line-coverage/some-features-in-libglusterfs.t - 2 second
./tests/bugs/shard/bug-1261773.t - 2 second
./tests/bugs/replicate/bug-884328.t - 2 second
./tests/bugs/readdir-ahead/bug-1512437.t - 2 second
./tests/bugs/nfs/bug-970070.t - 2 second
./tests/bugs/nfs/bug-1302948.t - 2 second
./tests/bugs/logging/bug-823081.t - 2 second
./tests/bugs/glusterfs-server/bug-889996.t - 2 second
./tests/bugs/glusterfs/bug-892730.t - 2 second
./tests/bugs/glusterfs/bug-811493.t - 2 second
./tests/bugs/glusterd/bug-1085330-and-bug-916549.t - 2 second
./tests/bugs/distribute/bug-924265.t - 2 second
./tests/bugs/core/log-bug-1362520.t - 2 second
./tests/bugs/core/bug-903336.t - 2 second
./tests/bugs/core/bug-1111557.t - 2 second
./tests/bugs/cli/bug-969193.t - 2 second
./tests/bugs/cli/bug-949298.t - 2 second
./tests/bugs/cli/bug-921215.t - 2 second
./tests/bugs/cli/bug-1378842-volume-get-all.t - 2 second
./tests/basic/peer-parsing.t - 2 second
./tests/basic/md-cache/bug-1418249.t - 2 second
./tests/basic/afr/arbiter-cli.t - 2 second
./tests/bugs/replicate/ta-inode-refresh-read.t - 1 second
./tests/bugs/glusterfs/bug-853690.t - 1 second
./tests/bugs/cli/bug-764638.t - 1 second
./tests/bugs/cli/bug-1047378.t - 1 second
./tests/basic/netgroup_parsing.t - 1 second
./tests/basic/gfapi/sink.t - 1 second
./tests/basic/exports_parsing.t - 1 second
./tests/basic/posixonly.t - 0 second
./tests/basic/glusterfsd-args.t - 0 second
2 test(s) failed
./tests/basic/uss.t
./tests/features/subdir-mount.t
0 test(s) generated core
5 test(s) needed retry
./tests/basic/afr/split-brain-favorite-child-policy.t
./tests/basic/ec/self-heal.t
./tests/basic/uss.t
./tests/basic/volfile-sanity.t
./tests/features/subdir-mount.t
Result is 1
tar: Removing leading `/' from member names
kernel.core_pattern = /%e-%p.core
Build step 'Execute shell' marked build as failure
_______________________________________________
maintainers mailing list
maintainers@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/maintainers
- Atin (atinm)_______________________________________________
maintainers mailing list
maintainers@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/maintainers
--Amar Tumballi (amarts)
_______________________________________________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/836554017 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/486278655 Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-devel
