Hi All,
This is the same issue as the previous tiering regression failure.
Volume brick not able to start brick because port is busy
[2015-07-02 10:20:20.601372] [run.c:190:runner_log] (--> /build/install/lib/libglusterfs.so.0(_gf_log_callingfn+0x240)[0x7f05e080bc32] (--> /build/install/lib/libglusterfs.so.0(runner_log+0x192)[0x7f05e08754ce] (--> /build/install/lib/glusterfs/3.8dev/xlator/mgmt/glusterd.so(glusterd_volume_start_glusterfs+0xae7)[0x7f05d5c935d7] (--> /build/install/lib/glusterfs/3.8dev/xlator/mgmt/glusterd.so(glusterd_brick_start+0x151)[0x7f05d5c9d4e3] (--> /build/install/lib/glusterfs/3.8dev/xlator/mgmt/glusterd.so(glusterd_op_perform_add_bricks+0x8fe)[0x7f05d5d10661] ))))) 0-: Starting GlusterFS: /build/install/sbin/glusterfsd -s slave33.cloud.gluster.org --volfile-id patchy.slave33.cloud.gluster.org.d-backends-patchy5 -p /var/lib/glusterd/vols/patchy/run/slave33.cloud.gluster.org-d-backends-patchy5.pid -S /var/run/gluster/ca5f5a89aa3a24f0a54852590ab82ad5.socket --brick-name /d/backends/patchy5 -l /var/log/glusterfs/bricks/d-backends-patchy5.log --xlator-option *-posix.glusterd-uuid=da011de8-9103-4cf2-9f4b-03707d0019d0 --brick-port 49167 --xlator-option patchy-server.listen-port=49167
[2015-07-02 10:20:20.624297] I [MSGID: 106144] [glusterd-pmap.c:269:pmap_registry_remove] 0-pmap: removing brick (null) on port 49167
[2015-07-02 10:20:20.625315] E [MSGID: 106005] [glusterd-utils.c:4448:glusterd_brick_start] 0-management: Unable to start brick slave33.cloud.gluster.org:/d/backends/patchy5
[2015-07-02 10:20:20.625354] E [MSGID: 106074] [glusterd-brick-ops.c:2096:glusterd_op_add_brick] 0-glusterd: Unable to add bricks
[2015-07-02 10:20:20.625368] E [MSGID: 106123] [glusterd-syncop.c:1416:gd_commit_op_phase] 0-management: Commit of operation 'Volume Add brick' failed on localhost
Brick Log:
[2015-07-02 10:20:20.608547] I [MSGID: 100030] [glusterfsd.c:2296:main] 0-/build/install/sbin/glusterfsd: Started running /build/install/sbin/glusterfsd version 3.8dev (args: /build/install/sbin/glusterfsd -s slave33.cloud.gluster.org --volfile-id patchy.slave33.cloud.gluster.org.d-backends-patchy5 -p /var/lib/glusterd/vols/patchy/run/slave33.cloud.gluster.org-d-backends-patchy5.pid -S /var/run/gluster/ca5f5a89aa3a24f0a54852590ab82ad5.socket --brick-name /d/backends/patchy5 -l /var/log/glusterfs/bricks/d-backends-patchy5.log --xlator-option *-posix.glusterd-uuid=da011de8-9103-4cf2-9f4b-03707d0019d0 --brick-port 49167 --xlator-option patchy-server.listen-port=49167)
[2015-07-02 10:20:20.617113] I [MSGID: 101190] [event-epoll.c:627:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-07-02 10:20:20.623097] I [MSGID: 101173] [graph.c:268:gf_add_cmdline_options] 0-patchy-server: adding option 'listen-port' for volume 'patchy-server' with value '49167'
[2015-07-02 10:20:20.623135] I [MSGID: 101173] [graph.c:268:gf_add_cmdline_options] 0-patchy-posix: adding option 'glusterd-uuid' for volume 'patchy-posix' with value 'da011de8-9103-4cf2-9f4b-03707d0019d0'
[2015-07-02 10:20:20.623358] I [MSGID: 115034] [server.c:392:_check_for_auth_option] 0-/d/backends/patchy5: skip format check for non-addr auth option auth.login./d/backends/patchy5.allow
[2015-07-02 10:20:20.623374] I [MSGID: 115034] [server.c:392:_check_for_auth_option] 0-/d/backends/patchy5: skip format check for non-addr auth option auth.login.96bcb872-559b-4f19-84ad-a735dc6068f6.password
[2015-07-02 10:20:20.623568] I [rpcsvc.c:2210:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2015-07-02 10:20:20.623633] W [MSGID: 101002] [options.c:952:xl_opt_validate] 0-patchy-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
[2015-07-02 10:20:20.623707] E [socket.c:818:__socket_server_bind] 0-tcp.patchy-server: binding to failed: Address already in use
[2015-07-02 10:20:20.623720] E [socket.c:821:__socket_server_bind] 0-tcp.patchy-server: Port is already in use
[2015-07-02 10:20:20.623746] W [rpcsvc.c:1599:rpcsvc_transport_create] 0-rpc-service: listening on transport failed
[2015-07-02 10:20:20.623758] W [MSGID: 115045] [server.c:998:init] 0-patchy-server: creation of listener failed
[2015-07-02 10:20:20.623772] E [MSGID: 101019] [xlator.c:423:xlator_init] 0-patchy-server: Initialization of volume 'patchy-server' failed, review your volfile again
[2015-07-02 10:20:20.623783] E [MSGID: 101066] [graph.c:323:glusterfs_graph_init] 0-patchy-server: initializing translator failed
[2015-07-02 10:20:20.623792] E [MSGID: 101176] [graph.c:669:glusterfs_graph_activate] 0-graph: init failed
[2015-07-02 10:20:20.624203] W [glusterfsd.c:1214:cleanup_and_exit] (--> 0-: received signum (0), shutting down
Regards,
Joe
----- Original Message -----
From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx>
To: "Dan Lambright" <dlambrig@xxxxxxxxxx>
Cc: "Gluster Devel" <gluster-devel@xxxxxxxxxxx>, "Joseph Fernandes" <josferna@xxxxxxxxxx>
Sent: Thursday, July 2, 2015 6:16:44 PM
Subject: Re: Failure in tests/basic/tier/bug-1214222-directories_miising_after_attach_tier.t
Thanks Dan!.
Pranith
On 07/02/2015 06:14 PM, Dan Lambright wrote:
> I'll check on this.
>
> ----- Original Message -----
>> From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx>
>> To: "Gluster Devel" <gluster-devel@xxxxxxxxxxx>, "Joseph Fernandes" <josferna@xxxxxxxxxx>
>> Sent: Thursday, July 2, 2015 5:40:34 AM
>> Subject: Failure in tests/basic/tier/bug-1214222-directories_miising_after_attach_tier.t
>>
>> hi Joseph,
>> Could you take a look at
>> http://build.gluster.org/job/rackspace-regression-2GB-triggered/11842/consoleFull
>>
>> Pranith
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel@xxxxxxxxxxx
>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>
--- Begin Message ---
Hi All,
TEST 4-5 are failing i.e the following
TEST $CLI volume start $V0
TEST $CLI volume attach-tier $V0 replica 2 $H0:$B0/${V0}$CACHE_BRICK_FIRST $H0:$B0/${V0}$CACHE_BRICK_LAST
Glusterd Logs say:
[2015-07-01 07:33:25.053412] I [rpc-clnt.c:965:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-07-01 07:33:25.053851] [run.c:190:runner_log] (--> /build/install/lib/libglusterfs.so.0(_gf_log_callingfn+0x240)[0x7fe8349bfb82] (--> /build/install/lib/libglusterfs.so.0(runner_log+0x192)[0x7fe834a29426] (--> /build/install/lib/glusterfs/3.8dev/xlator/mgmt/glusterd.so(glusterd_volume_start_glusterfs+0xae7)[0x7fe829e475d7] (--> /build/install/lib/glusterfs/3.8dev/xlator/mgmt/glusterd.so(glusterd_brick_start+0x151)[0x7fe829e514e3] (--> /build/install/lib/glusterfs/3.8dev/xlator/mgmt/glusterd.so(glusterd_start_volume+0xba)[0x7fe829ebd534] ))))) 0-: Starting GlusterFS: /build/install/sbin/glusterfsd -s slave26.cloud.gluster.org --volfile-id patchy.slave26.cloud.gluster.org.d-backends-patchy3 -p /var/lib/glusterd/vols/patchy/run/slave26.cloud.gluster.org-d-backends-patchy3.pid -S /var/run/gluster/e511d04af0bd91bfc3b030969b789d95.socket --brick-name /d/backends/patchy3 -l /var/log/glusterfs/bricks/d-backends-patchy3.log --xlator-option *-posix.glusterd-uuid=aff38c34-7744-4cc0-9aa4-a9fab5a71b2f --brick-port 49172 --xlator-option patchy-server.listen-port=49172
[2015-07-01 07:33:25.070284] I [MSGID: 106144] [glusterd-pmap.c:269:pmap_registry_remove] 0-pmap: removing brick (null) on port 49172
[2015-07-01 07:33:25.071022] E [MSGID: 106005] [glusterd-utils.c:4448:glusterd_brick_start] 0-management: Unable to start brick slave26.cloud.gluster.org:/d/backends/patchy3
[2015-07-01 07:33:25.071053] E [MSGID: 106123] [glusterd-syncop.c:1416:gd_commit_op_phase] 0-management: Commit of operation 'Volume Start' failed on localhost
The volume is 2x2 :
LAST_BRICK=3
TEST $CLI volume create $V0 replica 2 $H0:$B0/${V0}{0..$LAST_BRICK}
When looked into the 3 bricks are fine but when looked at the 4th brick log:
[2015-07-01 07:33:25.056463] I [MSGID: 100030] [glusterfsd.c:2296:main] 0-/build/install/sbin/glusterfsd: Started running /build/install/sbin/glusterfsd version 3.8dev (args: /build/install/sbin/glusterfsd -s slave26.cloud.gluster.org --volfile-id patchy.slave26.cloud.gluster.org.d-backends-patchy3 -p /var/lib/glusterd/vols/patchy/run/slave26.cloud.gluster.org-d-backends-patchy3.pid -S /var/run/gluster/e511d04af0bd91bfc3b030969b789d95.socket --brick-name /d/backends/patchy3 -l /var/log/glusterfs/bricks/d-backends-patchy3.log --xlator-option *-posix.glusterd-uuid=aff38c34-7744-4cc0-9aa4-a9fab5a71b2f --brick-port 49172 --xlator-option patchy-server.listen-port=49172)
[2015-07-01 07:33:25.064879] I [MSGID: 101190] [event-epoll.c:627:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-07-01 07:33:25.068992] I [MSGID: 101173] [graph.c:268:gf_add_cmdline_options] 0-patchy-server: adding option 'listen-port' for volume 'patchy-server' with value '49172'
[2015-07-01 07:33:25.069034] I [MSGID: 101173] [graph.c:268:gf_add_cmdline_options] 0-patchy-posix: adding option 'glusterd-uuid' for volume 'patchy-posix' with value 'aff38c34-7744-4cc0-9aa4-a9fab5a71b2f'
[2015-07-01 07:33:25.069313] I [MSGID: 115034] [server.c:392:_check_for_auth_option] 0-/d/backends/patchy3: skip format check for non-addr auth option auth.login./d/backends/patchy3.allow
[2015-07-01 07:33:25.069316] I [MSGID: 101190] [event-epoll.c:627:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2015-07-01 07:33:25.069330] I [MSGID: 115034] [server.c:392:_check_for_auth_option] 0-/d/backends/patchy3: skip format check for non-addr auth option auth.login.18b50c0d-38fb-4b49-bb5e-b203f4217223.password
[2015-07-01 07:33:25.069580] I [rpcsvc.c:2210:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2015-07-01 07:33:25.069647] W [MSGID: 101002] [options.c:952:xl_opt_validate] 0-patchy-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
[2015-07-01 07:33:25.069736] E [socket.c:818:__socket_server_bind] 0-tcp.patchy-server: binding to failed: Address already in use
[2015-07-01 07:33:25.069750] E [socket.c:821:__socket_server_bind] 0-tcp.patchy-server: Port is already in use
[2015-07-01 07:33:25.069763] W [rpcsvc.c:1599:rpcsvc_transport_create] 0-rpc-service: listening on transport failed
[2015-07-01 07:33:25.069774] W [MSGID: 115045] [server.c:996:init] 0-patchy-server: creation of listener failed
[2015-07-01 07:33:25.069788] E [MSGID: 101019] [xlator.c:423:xlator_init] 0-patchy-server: Initialization of volume 'patchy-server' failed, review your volfile again
[2015-07-01 07:33:25.069798] E [MSGID: 101066] [graph.c:323:glusterfs_graph_init] 0-patchy-server: initializing translator failed
[2015-07-01 07:33:25.069808] E [MSGID: 101176] [graph.c:669:glusterfs_graph_activate] 0-graph: init failed
[2015-07-01 07:33:25.070183] W [glusterfsd.c:1214:cleanup_and_exit] (--> 0-: received signum (0), shutting down
Looks like it is assigned a port which is already in used.
The status of the volume in glusterd is not started, as a result attach-tier command fails, i.e tiering rebalancer cannot run.
[2015-07-01 07:33:25.275092] E [MSGID: 106301] [glusterd-op-sm.c:4086:glusterd_op_ac_send_stage_op] 0-management: Staging of operation 'Volume Rebalance' failed on localhost : Volume patchy needs to be started to perform rebalance
but the volume is running in the crippled mode, as a result mount works fine.
i.e TEST $GFS --volfile-id=/$V0 --volfile-server=$H0 $M0; works fine
TEST 9-12 failed as attach has failed.
Regards,
Joe
----- Original Message -----
From: "Joseph Fernandes" <josferna@xxxxxxxxxx>
To: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx>
Cc: "Gluster Devel" <gluster-devel@xxxxxxxxxxx>
Sent: Wednesday, July 1, 2015 1:59:41 PM
Subject: Re: spurious failures tests/bugs/tier/bug-1205545-CTR-and-trash-integration.t
Yep will have a look
----- Original Message -----
From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx>
To: "Joseph Fernandes" <josferna@xxxxxxxxxx>, "Gluster Devel" <gluster-devel@xxxxxxxxxxx>
Sent: Wednesday, July 1, 2015 1:44:44 PM
Subject: spurious failures tests/bugs/tier/bug-1205545-CTR-and-trash-integration.t
hi,
http://build.gluster.org/job/rackspace-regression-2GB-triggered/11757/consoleFull
has the logs. Could you please look into it.
Pranith
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel
--- End Message ---
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel
