Re: volume start: data0: failed: Commit failed on localhost.

[Deepak Naidu <dnaidu@xxxxxxxxxx>]

Thanks Rafi for workaround.

>> To find the root cause we need to get logs for the first failure of volume start or volume stop .

Below, exact steps to re-produce the issue and attached log file contents from /varlog/gluster folder(parsed)

 

STEPS to re-produce the issue

 

root@hostname:~# gluster volume create home-folder transport tcp,rdma storageN1:/gluster/disk1/home-folder storageN2:/gluster/disk1/home-folder

volume create: home-folder: success: please start the volume to access data

root@hostname:~#

root@hostname:~# gluster volume info home-folder

Volume Name: home-folder

Type: Distribute

Volume ID: 09abd02a-b760-459f-afde-95b374eafc53

Status: Created

Snapshot Count: 0

Number of Bricks: 2

Transport-type: tcp,rdma

Bricks:

Brick1: storageN1:/gluster/disk1/home-folder

Brick2: storageN2:/gluster/disk1/home-folder

Options Reconfigured:

performance.readdir-ahead: on

nfs.disable: on

 

root@hostname:~# gluster volume status home-folder

Volume home-folder is not started

root@hostname:~# gluster volume start home-folder

volume start: home-folder: failed: Commit failed on localhost. Please check log file for details.

root@hostname:~# gluster volume status home-folder

Volume home-folder is not started

 

root@hostname:~# gluster volume start home-folder force

volume start: home-folder: success

root@hostname:~# gluster volume status home-folder

Status of volume: home-folder

Gluster process                             TCP Port  RDMA Port  Online  Pid

------------------------------------------------------------------------------

Brick storageN1:/gluster/disk1/home-folder  N/A       N/A        Y       17737

Brick storageN2:/gluster/disk1/home-folder  N/A       N/A        N       N/A 

 

Task Status of Volume home-folder

------------------------------------------------------------------------------

There are no active volume tasks

root@hostname:~# gluster volume info home-folder

Volume Name: home-folder

Type: Distribute

Volume ID: 09abd02a-b760-459f-afde-95b374eafc53

Status: Started

Snapshot Count: 0

Number of Bricks: 2

Transport-type: tcp,rdma

Bricks:

Brick1: storageN1:/gluster/disk1/home-folder

Brick2: storageN2:/gluster/disk1/home-folder

Options Reconfigured:

performance.readdir-ahead: on

nfs.disable: on

root@hostname:~#

 

 

--

Deepak

 

 

From: Mohammed Rafi K C [mailto:rkavunga@xxxxxxxxxx]
Sent: Friday, February 24, 2017 1:08 AM
To: Deepak Naidu; gluster-users@xxxxxxxxxxx
Subject: Re: volume start: data0: failed: Commit failed on localhost.

 

It looks like it is ended up in split brain kind of situation. To find the root cause we need to get logs for the first failure of volume start or volume stop .

Or to work around it, you can do a volume start force.

 

Regards

Rafi KC

 

On 02/24/2017 01:36 PM, Deepak Naidu wrote:

I keep on getting this error when my config.transport is set to both tcp,rdma. The volume doesn’t start. I get the below error during volume start.

 

To get around this, I end up delete the volume, then configure either only rdma or tcp. May be I am missing something, just trying to get the volume up.

 

root@hostname:~# gluster volume start data0

volume start: data0: failed: Commit failed on localhost. Please check log file for details.

root@hostname:~#

 

root@ hostname:~# gluster volume status data0

Staging failed on storageN2. Error: Volume data0 is not started

root@ hostname:~

 

=============

[2017-02-24 08:00:29.923516] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume data0

[2017-02-24 08:00:29.926140] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on storageN2. Error: Volume data0 is not started

[2017-02-24 08:00:33.770505] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume data0

[2017-02-24 08:00:33.772824] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on storageN2. Error: Volume data0 is not started

=============

[2017-02-24 08:01:36.305165] E [MSGID: 106537] [glusterd-volume-ops.c:1660:glusterd_op_stage_start_volume] 0-management: Volume data0 already started

[2017-02-24 08:01:36.305191] W [MSGID: 106122] [glusterd-mgmt.c:198:gd_mgmt_v3_pre_validate_fn] 0-management: Volume start prevalidation failed.

[2017-02-24 08:01:36.305198] E [MSGID: 106122] [glusterd-mgmt.c:884:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Start on local node

[2017-02-24 08:01:36.305205] E [MSGID: 106122] [glusterd-mgmt.c:2009:glusterd_mgmt_v3_initiate_all_phases] 0-management: Pre Validation Failed

 

 

--

Deepak

 

---

This email message is for the sole use of the intended recipient(s) and may contain confidential information.  Any unauthorized review, use, disclosure or distribution is prohibited.  If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

---

_______________________________________________

Gluster-users mailing list

Gluster-users@xxxxxxxxxxx

http://lists.gluster.org/mailman/listinfo/gluster-users

 

Attachment: Commit-failed.log
Description: Commit-failed.log

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users