Thanks Vijay & Prashant. I will look the integrity of the state file you mentioned. But It was failing to resolve the bricks's IP address which I had got on my laptop after logging from home. I corrected that and could work now, although I have some queries around ? IMO, it should not stop "glusterd" from starting ? and may allow to leave not-reachable volumes in "STOPPED" state as it supports "start /stop" states. I may be overlooking the bigger picture/usecase here. please correct. Thanks, Chetan Risbud. ----- Original Message ----- From: "Vijay Bellur" <vbellur at redhat.com> To: "Chetan Risbud" <crisbud at redhat.com> Cc: "gluster-users Discussion List" <Gluster-users at gluster.org> Sent: Friday, August 16, 2013 11:20:37 AM Subject: Re: Need help understanding the glusterd issue On 08/16/2013 10:32 AM, Chetan Risbud wrote: > HI All, > > I am init related failures while restarting glusterd. I did restart a glusterd as I had changed the ring files for some other swift related activity after adding a new volume. Is there any workaround for this problem? CC'ing gluster-users as this is the relevant mailer for this. > /var/log/glusterfs/etc-glusterfs-glusterd.vol.log > > > > > [2013-08-16 04:55:24.399286] I [glusterfsd.c:1910:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.4.0 (/usr/sbin/glusterd -p /run/glusterd.pid) > [2013-08-16 04:55:24.404097] I [glusterd.c:962:init] 0-management: Using /var/lib/glusterd as working directory > [2013-08-16 04:55:24.407802] I [socket.c:3480:socket_init] 0-socket.management: SSL support is NOT enabled > [2013-08-16 04:55:24.407835] I [socket.c:3495:socket_init] 0-socket.management: using system polling thread > [2013-08-16 04:55:24.407972] E [rpc-transport.c:253:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.4.0/rpc-transport/rdma.so: cannot open shared object file: No such file or directory > [2013-08-16 04:55:24.407995] W [rpc-transport.c:257:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine > [2013-08-16 04:55:24.408009] W [rpcsvc.c:1387:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed > [2013-08-16 04:55:25.867973] I [glusterd-store.c:1328:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2 > [2013-08-16 04:55:25.884692] E [glusterd-store.c:1845:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 > [2013-08-16 04:55:25.884771] E [glusterd-store.c:1845:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 > [2013-08-16 04:55:26.110537] E [glusterd-store.c:1845:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 > [2013-08-16 04:55:26.110617] E [glusterd-store.c:1845:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 > [2013-08-16 04:55:26.185491] E [glusterd-store.c:1845:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 > [2013-08-16 04:55:26.185571] E [glusterd-store.c:1845:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 > [2013-08-16 04:55:29.250542] E [glusterd-store.c:2472:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore You seem to have an incomplete state file in /var/lib/glusterd/vols/<volname> and hence initialization of glusterd seems to have failed. Can you please check that out? Regards, Vijay > [2013-08-16 04:55:29.250615] E [xlator.c:390:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again > [2013-08-16 04:55:29.250634] E [graph.c:292:glusterfs_graph_init] 0-management: initializing translator failed