答复: 答复: 答复: 答复: Gluster peer rejected and failed to start

[vyyy杨雨阳 <yuyangyang@xxxxxxxxx>]

Both sides are not work, from inside the trusted to invite new node, the error message is as following:

 

 

[2015-06-05 02:38:55.995296] I [glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glust

erfs05.sh2.ctripcorp.com 24007

[2015-06-05 02:38:55.995762] I [glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: g

lusterfs05.sh2.ctripcorp.com (24007)

[2015-06-05 02:38:56.018494] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600

[2015-06-05 02:38:56.021761] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0

[2015-06-05 02:38:56.037771] E [glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the

'versions' from peer (10.8.230.209:24007)

[2015-06-05 02:39:42.275385] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:39:43.327381] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:39:45.536731] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:39:46.258616] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:38:56.037819] I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer 000000

00-0000-0000-0000-000000000000, in Establishing Connection state, has disconnected from glusterd.

[2015-06-05 02:40:42.519877] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:40:43.659864] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:40:45.749804] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:40:46.570764] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:41:36.704914] I [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-vers

ion 2

[2015-06-05 02:41:36.706599] I [glusterd-handler.c:2523:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid:

2cb15023-28b0-4d0d-8a43-b8c6e570776f

[2015-06-05 02:41:36.706672] I [glusterd-handler.c:2551:__glusterd_handle_probe_query] 0-glusterd: Unable to find peerinfo fo

r host: 10.8.230.210 (24007)

[2015-06-05 02:41:36.707650] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600

[2015-06-05 02:41:36.710842] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0

[2015-06-05 02:41:36.710887] I [glusterd-handler.c:2575:__glusterd_handle_probe_query] 0-glusterd: Responded to 10.8.230.210,

op_ret: 0, op_errno: 0, ret: 0

[2015-06-05 02:41:36.713242] I [glusterd-handler.c:2216:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe fro

m uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f

[2015-06-05 02:41:36.717927] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick

glusterfs05.sh2.ctripcorp.com:/export/sdb/brick resides

[2015-06-05 02:41:36.717960] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs05.sh2.ct

ripcorp.com:/export/sdb/brick for JQStore2

[2015-06-05 02:41:36.736253] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick

glusterfs05.sh2.ctripcorp.com:/export/sdb/brick3 resides

[2015-06-05 02:41:36.736274] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs05.sh2.ct

ripcorp.com:/export/sdb/brick3 for Webresource

[2015-06-05 02:41:36.754597] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick

glusterfs05.sh2.ctripcorp.com:/export/sdb/brick2 resides

[2015-06-05 02:41:36.754618] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs05.sh2.ct

ripcorp.com:/export/sdb/brick2 for ccim

[2015-06-05 02:41:36.772127] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick

glusterfs07.sh2.ctripcorp.com:/export/sdb/brick4 resides

[2015-06-05 02:41:36.772148] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs07.sh2.ct

ripcorp.com:/export/sdb/brick4 for cloudimage

[2015-06-05 02:41:36.785585] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick

glusterfs07.sh2.ctripcorp.com:/export/sdb/brick1 resides

[2015-06-05 02:41:36.785613] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs07.sh2.ct

ripcorp.com:/export/sdb/brick1 for filews

[2015-06-05 02:41:36.798333] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick

glusterfs05.sh2.ctripcorp.com:/export/sdb/brick1 resides

[2015-06-05 02:41:36.798353] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs05.sh2.ct

ripcorp.com:/export/sdb/brick1 for filex

[2015-06-05 02:41:36.807443] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick

glusterfs05.sh2.ctripcorp.com:/export/sdb/brick4 resides

[2015-06-05 02:41:36.807462] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs05.sh2.ct

ripcorp.com:/export/sdb/brick4 for pubmail

[2015-06-05 02:41:36.831335] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600

[2015-06-05 02:41:36.837596] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600

[2015-06-05 02:41:36.837858] I [glusterd-handler.c:3334:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.8.230.210

(0), ret: 0

[2015-06-05 02:41:36.851547] I [glusterd-rpc-ops.c:245:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 2cb

15023-28b0-4d0d-8a43-b8c6e570776f, host: 10.8.230.210

[2015-06-05 02:41:36.853231] I [glusterd-rpc-ops.c:387:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req

[2015-06-05 02:41:36.853833] I [glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: 2cb1502

3-28b0-4d0d-8a43-b8c6e570776f, host: 10.8.230.210, port: 0

[2015-06-05 02:41:42.433916] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:41:42.435146] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:41:42.436189] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:41:42.437029] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:41:42.437909] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:41:42.438683] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:41:42.439463] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:41:42.440199] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:41:43.592906] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:41:43.593817] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:41:43.594798] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:41:43.595756] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:41:43.596548] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:41:43.597246] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:41:43.597975] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:41:43.598783] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:41:44.665688] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:41:45.724663] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:41:46.739705] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:41:46.741074] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:41:46.741903] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:41:46.742747] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:41:46.743364] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:41:46.743907] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:41:46.744502] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:41:46.745020] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:41:57.869703] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:42:02.453058] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:42:03.333862] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:42:42.713885] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:42:42.714508] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:42:42.715378] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:42:42.716251] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:42:42.717072] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:42:42.717877] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:42:42.718771] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:42:42.719606] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:42:43.752726] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:42:43.753414] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:42:43.754334] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:42:43.755216] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:42:43.756063] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:42:43.756933] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:42:43.757851] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:42:43.758667] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:42:44.854449] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:42:45.874503] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:42:46.918639] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:42:46.919484] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:42:46.920105] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:42:46.920881] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:42:46.921602] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:42:46.922174] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:42:46.922755] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:42:46.923288] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:43:42.600915] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:43:42.601670] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:43:42.602719] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:43:42.603634] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:43:42.604380] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:43:42.605099] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:43:42.605910] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:43:42.606698] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:43:43.686849] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:43:43.688073] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:43:43.689113] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:43:43.690029] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:43:43.691034] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:43:43.691920] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:43:43.692814] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:43:43.693619] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:43:44.712225] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:43:45.832058] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:43:46.835629] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:43:46.836511] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:43:46.837130] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:43:46.837775] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:43:46.838325] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:43:46.838859] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:43:46.839415] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:43:46.839914] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:44:42.881894] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:44:42.882568] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:44:42.884089] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:44:42.885076] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:44:42.886050] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:44:42.886969] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:44:42.887786] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:44:42.888593] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:44:43.964725] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:44:43.965426] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:44:43.966304] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:44:43.967172] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:44:43.968021] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:44:43.968916] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:44:43.969815] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:44:43.970628] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:44:44.216615] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:44:45.229045] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:44:46.320259] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:44:46.320928] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:44:46.321593] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:44:46.322185] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:44:46.322693] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:44:46.323219] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:44:46.323705] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:44:46.324208] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:45:42.979863] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:45:42.980449] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:45:42.981336] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:45:42.982186] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:45:42.982969] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:45:42.983793] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:45:42.984687] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:45:42.985503] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:45:43.770839] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:45:43.771639] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:45:43.773052] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:45:43.773992] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:45:43.774922] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:45:43.775784] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:45:43.776600] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:45:43.777346] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:45:44.820664] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:45:45.921119] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:45:46.241215] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:45:46.242124] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:45:46.242938] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:45:46.243649] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:45:46.244241] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:45:46.244863] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:45:46.245414] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:45:46.245913] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:46:42.792418] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:46:42.793154] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:46:42.793937] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:46:42.794760] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:46:42.795684] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:46:42.796540] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:46:42.797285] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:46:42.798119] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:46:44.012405] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:46:44.013126] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:46:44.014101] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:46:44.015042] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:46:44.016040] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:46:44.016847] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:46:44.017613] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:46:44.018320] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:46:44.337732] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:46:45.211598] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:46:46.409761] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:46:46.410668] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:46:46.411291] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:46:46.411925] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:46:46.412655] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:46:46.413251] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:46:46.413825] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:46:46.414344] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:47:42.808905] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:47:42.809651] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:47:42.810677] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:47:42.811566] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:47:42.812321] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:47:42.813129] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:47:42.814020] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:47:42.814851] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:47:43.071647] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:47:43.072843] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:47:43.073781] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:47:43.074627] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:47:43.075332] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:47:43.076053] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:47:43.076925] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:47:43.077794] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:47:44.684657] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:47:45.857070] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:47:46.889607] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:47:46.890387] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:47:46.890953] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:47:46.891498] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:47:46.892030] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:47:46.892671] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:47:46.893202] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:47:46.893781] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:48:42.320307] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:48:42.321041] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:48:42.322102] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:48:42.322970] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:48:42.323779] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:48:42.324583] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:48:42.325287] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:48:42.326075] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:48:43.115674] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-06-05 02:48:43.116339] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume JQStore2

[2015-06-05 02:48:43.117275] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume Webresource

[2015-06-05 02:48:43.118098] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume ccim

[2015-06-05 02:48:43.119014] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume cloudimage

[2015-06-05 02:48:43.119929] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filews

[2015-06-05 02:48:43.120829] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume filex

[2015-06-05 02:48:43.121615] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume

req for volume pubmail

[2015-06-05 02:48:44.656698] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:48:45.724179] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2015-06-05 02:48:46.780692] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:48:46.781567] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:48:46.782187] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:48:46.782917] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:48:46.783658] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:48:46.784251] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:48:46.784839] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[2015-06-05 02:48:46.785392] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

[root@SVR6993HW2285 glusterfs]#       

Best Regards

杨雨阳

Site Operations Center

Ext.15554  Seat.1#3F049

We are the F.I.R.S.T.

Fast, Innovation, Reliability , Secure, Team

 

 

发件人: Joe Julian [mailto:joe@xxxxxxxxxxxxxxxx]
发送时间: Friday, June 05, 2015 10:44 AM
收件人: vyyy杨雨阳; Atin Mukherjee; Gluster-users@xxxxxxxxxxx
主题: Re: [Gluster-users] 答复: 答复: 答复: Gluster peer rejected and failed to start

 

You cannot join a trusted pool, you must be invited.

In other words, a server inside the trusted pool must probe the new server, and that new server must not be part of another pool.

On June 4, 2015 7:24:28 PM PDT, "vyyy杨雨阳" <yuyangyang@xxxxxxxxx> wrote:

Can anyone help me?

New nodes still can't join the cluster.
The new nodes can peer each other, just not exist cluster nodes.
I can not find any network problem, there were no firewall, port 24007 is connected, there were all at the same subnet...

Thanks 



[root@SVR6993HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log
[2015-06-05 01:59:02.276107] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glust
erd version 3.6.3 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2015-06-05 01:59:02.318477] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536
[2015-06-05 01:59:02.318524] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory
[2015-06-05 01:59:02.329603] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans
port/rdma.so: cannot open shared object file: No such file or directory
[2015-06-05 01:59:02.329625] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-
type 'rdma' is not valid or not found on this machine
[2015-06-05 01:59:02.329637] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the tra
nsport failed
[2015-06-05 01:59:02.343346] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed
 in the system
[2015-06-05 01:59:02.351861] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2
[2015-06-05 01:59:02.363138] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snap
s list.
Final graph:
+

+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option transport.socket.listen-backlog 128
  7:     option ping-timeout 30
  8:     option transport.socket.read-fail-log off
  9:     option transport.socket.keepalive-interval 2
 10:     option transport.socket.keepalive-time 10
 11:     option transport-type rdma
 12:     option working-directory /var/lib/glusterd
 13: end-volume
 14:  
+

+
[2015-06-05 02:00:43.025378] I [glusterd.c:146:glusterd_uuid_init] 0-management: retrieved UUID: 1b98af34-08f9-492f-b54e-7608
2588d63b
[2015-06-05 02:00:43.025462] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:00:44.025069] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:00:44.599862] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]

0-glusterd: Received cli list req
[2015-06-05 02:00:46.029823] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:00:46.701845] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-06-05 02:01:43.119612] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:01:44.165527] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:01:44.851903] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:01:46.190401] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:01:46.722394] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-06-05 02:01:58.719968] I

[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:02:32.584999] I [glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glust
erfs05.sh2.ctripcorp.com 24007
[2015-06-05 02:02:32.586281] I [glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: g
lusterfs05.sh2.ctripcorp.com (24007)
[2015-06-05 02:02:32.602192] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-06-05 02:02:32.606053] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0
[2015-06-05 02:02:32.620958] E [glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the 
'versions' from peer (10.8.230.209:24007)
[2015-06-05

02:02:32.621020] I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer 000000
00-0000-0000-0000-000000000000, in Establishing Connection state, has disconnected from glusterd.
[2015-06-05 02:02:43.264141] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:02:44.312918] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:02:44.781434] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:02:46.176888] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:02:46.799916] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-06-05 02:03:42.720994] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:03:43.771987] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:03:44.659775] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:03:45.956807] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:03:46.688787] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-06-05 02:04:42.891131] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:04:43.957423] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:04:44.995638] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:04:46.062171] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]

0-glusterd: Received cli list req
[2015-06-05 02:04:46.957327] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-06-05 02:05:42.966651] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:05:44.008498] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:05:45.049936] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:05:46.087007] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:05:47.029325] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-06-05 02:06:42.874089] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:06:43.910998] W

[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:06:44.932802] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:06:46.053731] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:06:47.055859] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-06-05 02:07:43.099647] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:07:44.146454] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:07:45.218982] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:07:46.246303] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list

req
[2015-06-05 02:07:47.106954] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-06-05 02:08:43.226090] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:08:44.292041] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-06-05 02:08:45.351830] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:08:45.688349] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-06-05 02:08:46.651768] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-06-05 02:09:42.748628] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed



Best Regards 


-----邮件原件-----
发件人:

vyyy杨雨阳 
发送时间: Thursday, June 04, 2015 11:00 AM
收件人: 'Atin Mukherjee'; Gluster-users@xxxxxxxxxxx
主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start

　　
The problem 2 has resolved.

From glusterd log file, I find Cksums error as following:
[2015-06-04 01:25:03.137798] E [MSGID: 106010] [glusterd-utils.c:3297:glusterd_compare_friend_volume] 0-management: Version of Cksums JQStore2 differ. local cksum = 2617028277, remote cksum = 2210669223 on peer glusterfs09.sh2.ctripcorp.com

The solution for this problem was to do the following, on the BAD node:
/etc/init.d/glusterd stop
rsync -havP --delete 172.26.177.115:/var/lib/glusterd/vols/ /var/lib/glusterd/vols/
/etc/init.d/glusterd start


 
The problem 1 is still under inverstigate:

New nodes

can not join the cluster, log file shown: failed to get the 'versions' from peer
I have checked the file /var/lib/glusterd/ glusterd.info, all nodes are " operating-version=2"

　　
　　[2015-06-04 02:13:49.199416] E [glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the 
　　'versions' from peer (10.8.230.211:24007)
　　[2015-06-04 02:14:07.112577] I [glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glust
　　erfs05.sh2.ctripcorp.com 24007
　　[2015-06-04 02:14:07.113778] I [glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: g
　　lusterfs05.sh2.ctripcorp.com (24007)
　　[2015-06-04 02:14:07.131023] I

[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
　　[2015-06-04 02:14:07.134829] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0
　　[




Best Regards 
杨雨阳
Site Operations Center
Ext.15554  Seat.1#3F049
We are the F.I.R.S.T. 
Fast, Innovation, Reliability , Secure, Team


-----邮件原件-----
发件人: vyyy杨雨阳 
发送时间: Wednesday, June 03, 2015 5:37 PM
收件人: 'Atin Mukherjee'; Gluster-users@xxxxxxxxxxx
主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start

Hi Atin and everybody,

 I have forced peer detach the news nodes ,The peer status seems even strange, glusterfs05 and glusterfs10 are peer in cluster, Can you help me? Any advice is appreciated.

Glusterfs05 
　　
　　[root@sh02svr5956 ~]# gluster peer

status
　　Number of Peers: 5
　　
　　Hostname: glusterfs06.sh2.ctripcorp.com
　　Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
　　State: Peer Rejected (Connected)
　　
　　Hostname: glusterfs07.sh2.ctripcorp.com
　　Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
　　State: Peer Rejected (Connected)
　　
　　Hostname: glusterfs08.sh2.ctripcorp.com
　　Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
　　State: Peer Rejected (Connected)
　　
　　Hostname: glusterfs10.sh2.ctripcorp.com
　　Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
　　State: Peer in Cluster (Connected)
　　
　　Hostname: glusterfs09.sh2.ctripcorp.com
　　Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
　　State: Peer Rejected (Connected)
　　[root@sh02svr5956 ~]#
　　
　　
　　Glusterfs06 
　　
　　[root@SH02SVR5955 ~]# gluster peer status
　　Number of Peers: 5
　　
　　Hostname: glusterfs10.sh2.ctripcorp.com
　　Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
　　State: Peer Rejected (Connected)
　　
　　Hostname: glusterfs08.sh2.ctripcorp.com
　　Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
　　State: Peer in Cluster (Connected)
　　
　　Hostname: glusterfs07.sh2.ctripcorp.com
　　Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
　　State: Peer in Cluster (Connected)
　　
　　Hostname: 10.8.230.209
　　Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
　　State: Peer Rejected (Connected)
　　
　　Hostname: glusterfs09.sh2.ctripcorp.com
　　Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
　　State: Peer in Cluster (Connected)
　　[root@SH02SVR5955 ~]#
　　

Glusterfs07

[root@SH02SVR5954 ~]# gluster peer status
Number of Peers: 5

Hostname: glusterfs10.sh2.ctripcorp.com
Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
State: Peer Rejected (Connected)

Hostname: 10.8.230.209
Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
State: Peer Rejected (Connected)

Hostname: glusterfs06.sh2.ctripcorp.com
Uuid:

2cb15023-28b0-4d0d-8a43-b8c6e570776f
State: Peer in Cluster (Connected)

Hostname: glusterfs08.sh2.ctripcorp.com
Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
State: Peer in Cluster (Connected)

Hostname: glusterfs09.sh2.ctripcorp.com
Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
State: Peer in Cluster (Connected)
[root@SH02SVR5954 ~]#

 
 
Glusterfs08

[root@SH02SVR5953 ~]# gluster peer status
Number of Peers: 5

Hostname: glusterfs07.sh2.ctripcorp.com
Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
State: Peer in Cluster (Connected)

Hostname: glusterfs10.sh2.ctripcorp.com
Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
State: Peer Rejected (Connected)

Hostname:

glusterfs06.sh2.ctripcorp.com
Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
State: Peer in Cluster (Connected)

Hostname: 10.8.230.209
Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
State: Peer Rejected (Connected)

Hostname: glusterfs09.sh2.ctripcorp.com
Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
State: Peer in Cluster (Connected)
[root@SH02SVR5953 ~]#


[root@SH02SVR5952 ~]# gluster peer status
Number of Peers: 5

Hostname: glusterfs06.sh2.ctripcorp.com
Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
State: Peer in Cluster (Connected)

Hostname: glusterfs07.sh2.ctripcorp.com
Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
State: Peer in Cluster

(Connected)

Hostname: glusterfs08.sh2.ctripcorp.com
Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
State: Peer in Cluster (Connected)

Hostname: 10.8.230.209
Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
State: Peer Rejected (Connected)

Hostname: glusterfs10.sh2.ctripcorp.com
Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
State: Peer Rejected (Connected)
[root@SH02SVR5952 ~]#


Glusterfs10

[root@SH02SVR5951 ~]# gluster peer status
Number of Peers: 6

Hostname: glusterfs07.sh2.ctripcorp.com
Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
State: Peer Rejected (Connected)

Hostname: glusterfs11.sh2.ctripcorp.com
Uuid:

2d703550-92b5-4f5e-af90-ff2fbf3366f0
State: Peer Rejected (Connected)

Hostname: glusterfs09.sh2.ctripcorp.com
Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
State: Peer Rejected (Connected)

Hostname: glusterfs06.sh2.ctripcorp.com
Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
State: Peer Rejected (Connected)

Hostname: 10.8.230.209
Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
State: Peer in Cluster (Connected)

Hostname: glusterfs08.sh2.ctripcorp.com
Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
State: Peer Rejected (Connected)
[root@SH02SVR5951 ~]#







Best Regards 
杨雨阳
Site Operations Center
Ext.15554  Seat.1#3F049
We are the F.I.R.S.T. 
Fast,

Innovation, Reliability , Secure, Team



-----邮件原件-----
发件人: vyyy杨雨阳 
发送时间: Wednesday, June 03, 2015 2:55 PM
收件人: 'Atin Mukherjee'
主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start

Can I delete the files in /var/lib/glusterd/peers which state is not =3 on all nodes, then restart glusterd?


[root@sh02svr5956 peers]# more 0475e11a-3c43-4b76-99dd-1d2e814c5f15 
uuid=0475e11a-3c43-4b76-99dd-1d2e814c5f15
state=6
hostname1=glusterfs12.sh2.ctripcorp.com




Best Regards 
Yang


-----邮件原件-----
发件人: vyyy杨雨阳 
发送时间: Wednesday, June 03, 2015 2:39 PM
收件人: 'Atin Mukherjee'
主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start

Since it's intranet, You

can only control my desktop, and then access these node.
Should I apply a webex room?


Best Regards 
杨雨阳
Site Operations Center
Ext.15554  Seat.1#3F049
We are the F.I.R.S.T. 
Fast, Innovation, Reliability , Secure, Team



-----邮件原件-----
发件人: Atin Mukherjee [mailto:amukherj@xxxxxxxxxx] 
发送时间: Wednesday, June 03, 2015 2:36 PM
收件人: vyyy杨雨阳
主题: Re: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start


Is it possible to share your ip/user/passwd for few of these nodes?

On 06/03/2015 11:50 AM, vyyy杨雨阳 wrote:

 Can you remote control my desktop (like webex)?
 The nodes host bricks. And on the new nodes, I did "clean /var/lib/glusterd and restart" several times, but have no

effect.
 Is there any other work around?
 
 
 Best Regards 
 Yang
 
 -----邮件原件-----
 发件人: Atin Mukherjee [mailto:amukherj@xxxxxxxxxx] 
 发送时间: Wednesday, June 03, 2015 12:33 PM
 收件人: vyyy杨雨阳; Gluster-users@xxxxxxxxxxx
 主题: Re: 答复: [Gluster-users] Gluster peer rejected and failed to start
 
 I can't comment on what went wrong till I can have visibility on the
 entire cluster. Does the node which fails to start host any brick? If
 not then work around could be to clean /var/lib/glusterd and restart
 glusterd, that should sync up data and bring this node in consistency.
 
 HTH,
 Atin
 
 On 06/03/2015 08:34 AM, vyyy杨雨阳 wrote:

 I found that /var/lib/glusterd/peers can't include node's own peer, I delete this file.

Glusterd started finally.
 But the peer status still not normal.
 Seems that glusterfs10 leave the cluster and join the new nodes alone.

 Peer status on glusterfs10
 [root@SH02SVR5951 ~]# gluster peer status
 Number of Peers: 7

 Hostname: glusterfs07.sh2.ctripcorp.com
 Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
 State: Peer Rejected (Connected)

 Hostname: glusterfs11.sh2.ctripcorp.com
 Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
 State: Accepted peer request (Connected)

 Hostname: glusterfs09.sh2.ctripcorp.com
 Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
 State: Peer Rejected (Connected)

 Hostname: glusterfs06.sh2.ctripcorp.com
 Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f

State: Peer Rejected (Connected)

 Hostname: 10.8.230.209
 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
 State: Peer Rejected (Connected)

 Hostname: glusterfs08.sh2.ctripcorp.com
 Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
 State: Peer Rejected (Connected)

 Hostname: glusterfs12.sh2.ctripcorp.com
 Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15
 State: Peer in Cluster (Connected)


  
 Peer status on glusterfs10

 [root@SVR6993HW2285 ~]# gluster peer status
 Number of Peers: 6

 Hostname: glusterfs05.sh2.ctripcorp.com
 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
 State: Peer Rejected (Connected)

 Hostname: glusterfs09.sh2.ctripcorp.com
 Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
 State: Peer Rejected (Connected)

 Hostname: glusterfs07.sh2.ctripcorp.com
 Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
 State: Peer Rejected (Connected)

 Hostname: glusterfs08.sh2.ctripcorp.com
 Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
 State: Peer Rejected (Connected)

 Hostname: glusterfs06.sh2.ctripcorp.com
 Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
 State: Peer Rejected (Connected)

 Hostname: glusterfs10.sh2.ctripcorp.com
 Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
 State: Peer in Cluster (Connected)


 Peer status on glusterfs05, this is the node

that I tried to peer probe new nodes (glusterfs11 to glusterfs14)

 [root@sh02svr5956 ~]# gluster peer status
 Number of Peers: 9

 Hostname: glusterfs06.sh2.ctripcorp.com
 Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
 State: Peer in Cluster (Connected)

 Hostname: glusterfs07.sh2.ctripcorp.com
 Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
 State: Peer in Cluster (Connected)

 Hostname: glusterfs08.sh2.ctripcorp.com
 Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
 State: Peer in Cluster (Connected)

 Hostname: glusterfs10.sh2.ctripcorp.com
 Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
 State: Peer Rejected (Connected)

 Hostname: glusterfs09.sh2.ctripcorp.com
 Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
 State: Peer in Cluster (Connected)

 Hostname: glusterfs11.sh2.ctripcorp.com
 Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
 State: Peer Rejected (Connected)

 Hostname: glusterfs12.sh2.ctripcorp.com
 Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15
 State: Peer in Cluster (Connected)

 Hostname: glusterfs13.sh2.ctripcorp.com
 Uuid: 43e96d31-6c6d-4096-832a-440e544554ca
 State: Peer Rejected (Connected)

 Hostname: glusterfs14.sh2.ctripcorp.com
 Uuid: 45c11af9-7850-4598-ad62-4bb343b62524
 State: Peer Rejected (Connected)
 [root@sh02svr5956 ~]#


  

Best Regards 
 Yang 


 -----邮件原件-----
 发件人: vyyy杨雨阳 
 发送时间: Wednesday, June 03, 2015 10:04 AM
 收件人: 'Atin Mukherjee'; 'Gluster-users@xxxxxxxxxxx'
 主题: [Gluster-users] Gluster peer rejected and failed to start

 I am still messed, can you help me figure out problem 2 (A node of cluster failed to start glusterd)  priority

 The glusterd log on the node gluster10 shown ' resolve brick failed in restore'
 I checked /var/lib/glusterd/vols, seems ok

 Glusterd log file Error message:

 [2015-06-03 01:31:32.564010] E [glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore
 [2015-06-03 01:31:32.564036] E [xlator.c:425:xlator_init] 0-management: Initialization of volume 'management' failed, review 
 your volfile again




 Best Regards 
 Yang

 -----邮件原件-----

发件人: vyyy杨雨阳 
 发送时间: Tuesday, June 02, 2015 6:27 PM
 收件人: 'Atin Mukherjee'; Gluster-users@xxxxxxxxxxx
 主题: 答复: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start

 Gluster11 to gluster14 are new nodes I tried to join exit cluster but failed 
 　　
 　　
 　　On gluster12:
 [root@SVR6994HW2285 ~]# gluster peer status
 Number of Peers: 1

 Hostname: glusterfs05.sh2.ctripcorp.com
 Uuid: 04f22ee8-8e0

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users