Abhishek,
Here is the plan of action I suggest:3. check if you end up having multiple entries in gluster peer status output, if so share both the glusterd logs along with content of /var/lib/glusterd from both the nodes.
On Mon, Nov 21, 2016 at 3:40 PM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:
AbhishekRegards,I just want to know why this another peer file is getting created when there is not entry in logs for it.I have another set of logs for this problem and in this we don't have time stamp problem for this but we are getting same UUID duplicate entries on BoardBPreviously also I provided logs to you. Again I am attaching logs for you to analyze.
ls -lart
total 16
drwxrwxr-x 12 abhishek abhishek 4096 Oct 3 08:58 ..
drwxrwxr-x 2 abhishek abhishek 4096 Oct 3 08:58 .
-rw-rw-r-- 1 abhishek abhishek 71 Oct 3 13:36 d8f66ce8-4154-4246-8084-c63b9cfc1af4
-rw-rw-r-- 1 abhishek abhishek 71 Oct 3 13:36 28d37300-425a-4781-b1e0-c13efa2ceee6 On Mon, Nov 21, 2016 at 2:47 PM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:On Mon, Nov 21, 2016 at 2:28 PM, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:On Mon, Nov 21, 2016 at 10:00 AM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:Is it possible for gluster if we fall in this situation then instead of manually doing the steps which you mentioned above gluster will take care of this?Yes, I have also seen these two files in peers directory on 002500 board and I want to know the reason why gluster creates the second file when there is old file is exist. Even when you see the content of the these file are same.Hi Atin,
System is the embedded system and these dates are before the system get in timer sync.We shouldn't have any unwanted data in /var/lib/glusterd at first place and that's a prerequisite of gluster installation failing which inconsistencies of configuration data can't be handled automatically until manual intervention.
it means before starting of gluster installation /var/lib/glusterd always we empty? because in this case nothing is unwanted before installing the glusterd.I have some questions:1. based on the logs can we find out the reason for having two peers files with same contents.No we can't as the log file doesn't have any entry of 26ae19a6-b58f-446a-b079-411d4ee57450 which indicates that this entry is a stale one and was (is) there since long time and the log files are the latest. I agreed this 26ae19a6-b58f-446a-b079-411d4ee57450 entry is not there but as we checked this file is newer in peer and 5be8603b-18d0-4333-8590-38f918 a22857 is the older file. Also, below are some more logs in etc-glusterfs-glusterd.log file from 002500 board file
The message "I [MSGID: 106004] [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer <10.32.0.48> (<5be8603b-18d0-4333-8590-38f9 18a22857>), in state <Peer in Cluster>, has disconnected from glusterd." repeated 3 times between [2016-11-17 22:01:23.542556] and [2016-11-17 22:01:36.993584]
The message "W [MSGID: 106118] [glusterd-handler.c:5087:__glusterd_peer_rpc_notify] 0-management: Lock not released for c_glusterfs" repeated 3 times between [2016-11-17 22:01:23.542973] and [2016-11-17 22:01:36.993855]
[2016-11-17 22:01:48.860555] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2016-11-17 22:01:49.137733] I [MSGID: 106163] [glusterd-handshake.c:1193:__glusterd_mgmt_hndsk_versions_ac k] 0-management: using the op-version 30706
[2016-11-17 22:01:49.240986] I [MSGID: 106493] [glusterd-rpc-ops.c:694:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5be8603b-18d0-4333-8590-38f918 a22857
[2016-11-17 22:11:58.658884] E [rpc-clnt.c:201:call_bail] 0-management: bailing out frame type(glusterd mgmt) op(--(3)) xid = 0x15 sent = 2016-11-17 22:01:48.945424. timeout = 600 for 10.32.0.48:24007
[2016-11-17 22:11:58.658987] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on 10.32.0.48. Please check log file for details.
[2016-11-17 22:11:58.659243] I [socket.c:3382:socket_submit_reply] 0-socket.management: not connected (priv->connected = 255)
[2016-11-17 22:11:58.659265] E [rpcsvc.c:1314:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1, Program: GlusterD svc cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.management)
[2016-11-17 22:11:58.659305] E [MSGID: 106430] [glusterd-utils.c:400:glusterd_submit_reply] 0-glusterd: Reply submission failed
[2016-11-17 22:13:58.674343] E [rpc-clnt.c:201:call_bail] 0-management: bailing out frame type(glusterd mgmt) op(--(3)) xid = 0x11 sent = 2016-11-17 22:03:50.268751. timeout = 600 for 10.32.0.48:24007
[2016-11-17 22:13:58.674414] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on 10.32.0.48. Please check log file for details.
[2016-11-17 22:13:58.674604] I [socket.c:3382:socket_submit_reply] 0-socket.management: not connected (priv->connected = 255)
[2016-11-17 22:13:58.674627] E [rpcsvc.c:1314:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1, Program: GlusterD svc cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.management)
[2016-11-17 22:13:58.674667] E [MSGID: 106430] [glusterd-utils.c:400:glusterd_submit_reply] 0-glusterd: Reply submission failed
[2016-11-17 22:15:58.687737] E [rpc-clnt.c:201:call_bail] 0-management: bailing out frame type(glusterd mgmt) op(--(3)) xid = 0x17 sent = 2016-11-17 22:05:51.341614. timeout = 600 for 10.32.0.48:24007is these logs causing duplicate UUID or duplicate UUID causing this?
2. is there any way to do it from gluster code.Ditto as above.AbhishekRegards,AbhishekRegards,--On Mon, Nov 21, 2016 at 9:52 AM, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:atin@dhcp35-96:~/Downloads/gluIn board 2500 look at the date of the file 5be8603b-18d0-4333-8590-38f918ster_users/abhishek_dup_uuid/d uplicate_uuid/glusterd_2500/pe ers$ ls -lrt
total 8
-rw-------. 1 atin wheel 71 Jan 1 1970 5be8603b-18d0-4333-8590-38f918a22857
-rw-------. 1 atin wheel 71 Nov 18 03:31 26ae19a6-b58f-446a-b079-411d4ee57450 a22857 (marked in bold). Not sure how did you end up having this file in such time stamp. I am guessing this could be because of the set up been not cleaned properly at the time of re-installation. Here is the steps what I'd recommend for now:1. rename 26ae19a6-b58f-446a-b079-411d4ee57450 to 5be8603b-18d0-4333-8590-38f918 a22857, you should have only one entry in the peers folder in board 2500. 2. Bring down both glusterd instances3. Bring back one by oneAnd then restart glusterd to see if the issue persists.--On Mon, Nov 21, 2016 at 9:34 AM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:Hope you will see in the logs......--On Mon, Nov 21, 2016 at 9:17 AM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:So, they will remain as same as previous.Hi Atin,It is not getting wipe off we have changed the configuration path from /var/lib/glusterd to /system/glusterd.--On Mon, Nov 21, 2016 at 9:15 AM, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:Abhishek,rebooting the board does wipe of /var/lib/glusterd contents in your set up right (as per my earlier conversation with you) ? In that case, how are you ensuring that the same node gets back the older UUID? If you don't then this is bound to happen.On Mon, Nov 21, 2016 at 9:11 AM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:______________________________Also, we are not replacing any board from setup just rebooting.So could you please check what is the reason to get in this situation as it is very frequent in multiple case.I am attaching all logs from both the boards and the command outputs as well.We are having the setup of replicate volume setup with two brick but after restarting the second board I am getting the duplicate entry in "gluster peer status" command like below:Hi Team,Please lookinto this problem as this is very widely seen problem in our system.
# gluster peer status
Number of Peers: 2
Hostname: 10.32.0.48
Uuid: 5be8603b-18d0-4333-8590-38f918a22857
State: Peer in Cluster (Connected)
Hostname: 10.32.0.48
Uuid: 5be8603b-18d0-4333-8590-38f918a22857
State: Peer in Cluster (Connected)
#
--
Regards
Abhishek Paliwal
_________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
--~ Atin (atinm)
Regards
Abhishek Paliwal
Regards
Abhishek Paliwal
~ Atin (atinm)
Regards
Abhishek Paliwal
--~ Atin (atinm)
--
Regards
Abhishek Paliwal
--
~ Atin (atinm)
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
