Re: op-version for reset-brick (Was: Re: [ovirt-users] Upgrading HC from 4.0 to 4.1)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





On Fri, Jul 7, 2017 at 2:23 PM, Gianluca Cecchi <gianluca.cecchi@xxxxxxxxx> wrote:
On Thu, Jul 6, 2017 at 3:22 PM, Gianluca Cecchi <gianluca.cecchi@xxxxxxxxx> wrote:
On Thu, Jul 6, 2017 at 2:16 PM, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:


On Thu, Jul 6, 2017 at 5:26 PM, Gianluca Cecchi <gianluca.cecchi@xxxxxxxxx> wrote:
On Thu, Jul 6, 2017 at 8:38 AM, Gianluca Cecchi <gianluca.cecchi@xxxxxxxxx> wrote:

Eventually I can destroy and recreate this "export" volume again with the old names (ovirt0N.localdomain.local) if you give me the sequence of commands, then enable debug and retry the reset-brick command

Gianluca


So it seems I was able to destroy and re-create.
Now I see that the volume creation uses by default the new ip, so I reverted the hostnames roles in the commands after putting glusterd in debug mode on the host where I execute the reset-brick command (do I have to set debug for the the nodes too?)

You have to set the log level to debug for glusterd instance where the commit fails and share the glusterd log of that particular node.
 

Ok, done.

Command executed on ovirt01 with timestamp "2017-07-06 13:04:12" in glusterd log files

[root@ovirt01 export]# gluster volume reset-brick export gl01.localdomain.local:/gluster/brick3/export start
volume reset-brick: success: reset-brick start operation successful

[root@ovirt01 export]# gluster volume reset-brick export gl01.localdomain.local:/gluster/brick3/export ovirt01.localdomain.local:/gluster/brick3/export commit force
volume reset-brick: failed: Commit failed on ovirt02.localdomain.local. Please check log file for details.
Commit failed on ovirt03.localdomain.local. Please check log file for details.
[root@ovirt01 export]#

See glusterd log files for the 3 nodes in debug mode here:

HIH debugging
Gianluca


Hi Atin,
did you have time to see the logs?
Comparing debug enabled messages with previous ones, I see these added lines on nodes where commit failed after running the commands

gluster volume reset-brick export gl01.localdomain.local:/gluster/brick3/export start
gluster volume reset-brick export gl01.localdomain.local:/gluster/brick3/export ovirt01.localdomain.local:/gluster/brick3/export commit force


[2017-07-06 13:04:30.221872] D [MSGID: 0] [glusterd-peer-utils.c:674:gd_peerinfo_find_from_hostname] 0-management: Friend ovirt01.localdomain.local found.. state: 3
[2017-07-06 13:04:30.221882] D [MSGID: 0] [glusterd-peer-utils.c:167:glusterd_hostname_to_uuid] 0-management: returning 0
[2017-07-06 13:04:30.221888] D [MSGID: 0] [glusterd-utils.c:1039:glusterd_resolve_brick] 0-management: Returning 0
[2017-07-06 13:04:30.221908] D [MSGID: 0] [glusterd-utils.c:998:glusterd_brickinfo_new] 0-management: Returning 0
[2017-07-06 13:04:30.221915] D [MSGID: 0] [glusterd-utils.c:1195:glusterd_brickinfo_new_from_brick] 0-management: Returning 0
[2017-07-06 13:04:30.222187] D [MSGID: 0] [glusterd-peer-utils.c:167:glusterd_hostname_to_uuid] 0-management: returning 0
[2017-07-06 13:04:30.222201] D [MSGID: 0] [glusterd-utils.c:1486:glusterd_volume_brickinfo_get] 0-management: Returning -1

The above log entry is the reason of the failure. GlusterD is unable to find the old brick (src_brick) from its volinfo structure. FWIW, would you be able to share the content of 'gluster get-state' output & gluster volume info output after running reset-brick start? I'd need to check why glusterd is unable to find out the old brick's details from its volinfo post reset-brick start.

 
[2017-07-06 13:04:30.222207] D [MSGID: 0] [store.c:459:gf_store_handle_destroy] 0-: Returning 0
[2017-07-06 13:04:30.222242] D [MSGID: 0] [glusterd-utils.c:1512:glusterd_volume_brickinfo_get_by_brick] 0-glusterd: Returning -1
[2017-07-06 13:04:30.222250] D [MSGID: 0] [glusterd-replace-brick.c:416:glusterd_op_perform_replace_brick] 0-glusterd: Returning -1
[2017-07-06 13:04:30.222257] C [MSGID: 106074] [glusterd-reset-brick.c:372:glusterd_op_reset_brick] 0-management: Unable to add dst-brick: ovirt01.localdomain.local:/gluster/brick3/export to volume: export


Does it share up more light?

Thanks,
Gianluca

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux