Re: Adding a 3.6.3 peer to a 3.6.2 trusted pool fails

[Adrián Santos Marrero <asmarre@xxxxxxxxxx>]

Hi,

Finally, this was a network problem. A switch was mangling jumbo frames.

Regards.

2015-05-13 9:07 GMT+01:00 Adrián Santos Marrero <asmarre@xxxxxxxxxx>:

Hi Kaushal,

Volume info:

 

gluster02:~# gluster volume info all
 
Volume Name: ovirt1
Type: Replicate
Volume ID: 05171231-a4a5-48ae-bcfe-63408bc0fc0c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gluster01.stic.ull.es:/gluster/ovirt1_brick_01/brick
Brick2: gluster02.stic.ull.es:/gluster/ovirt1_brick_01/brick
Options Reconfigured:
storage.owner-gid: 36
storage.owner-uid: 36
cluster.server-quorum-type: none
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
 
Volume Name: ovirt2
Type: Replicate
Volume ID: 0c0be558-9d98-4403-82d4-faafa2a3630a
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gluster01.stic.ull.es:/gluster/ovirt2_brick_01/brick
Brick2: gluster02.stic.ull.es:/gluster/ovirt2_brick_01/brick
Options Reconfigured:
storage.owner-gid: 36
storage.owner-uid: 36
cluster.server-quorum-type: none
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off

And attached you can find the debug log for gluster03.

Gluster01 is still re-healing so I am not sure if it is safe to restart glusterd in DEBUG mode to obtain logs....

Regards.

 

2015-05-13 8:41 GMT+01:00 Kaushal M <kshlmster@xxxxxxxxx>:

Hey Adrian,
Can you list your volume infos? Most of the times the reason for peer state being stuck is because of some volume information being incorrect and the volume infos can help debug the problem. Could you also start glusterd in debug mode (`glusterd -LDEBUG`) and give logs?

~kaushal

On May 13, 2015 9:13 AM, "Adrián Santos Marrero" <asmarre@xxxxxxxxxx> wrote:

Hi again,

I upgraded to 3.6.3 all my gluster servers but peer probes still get stuck in "Accepted peer request".

Any hint?

Regards.

2015-05-11 13:55 GMT+01:00 Adrián Santos Marrero <asmarre@xxxxxxxxxx>:

Hi, I've the same error with gluster03 (first installed with 3.6.2 version, but upgraded to 3.6.3 before any peer probe) and gluster04 (fresh install with Gluster 3.6.3). I attach the whole content of /var/log/glusterfs/ directory from gluster03.

Regards.

2015-05-11 12:06 GMT+01:00 Gaurav Garg <ggarg@xxxxxxxxxx>:

Hi Adrian,

Did you try to upgrade/downgrade gluster03 before performing any operation. If you attach fresh node (3.6.3 version) to (3.6.2) glusterfs version then this problems is not coming. we need to know what operation you performed on gluster03 node before peer probe. if possible could you attach cmd_history.log and glusterd log.

~ Gaurav

----- Original Message -----
From: "Adrián Santos Marrero" <asmarre@xxxxxxxxxx>
To: "Gluster-users" <gluster-users@xxxxxxxxxxx>
Sent: Monday, May 11, 2015 12:39:33 PM
Subject: Adding a 3.6.3 peer to a 3.6.2 trusted pool fails

Hi,

I've been trying to add a third node to my two-node trusted pool with gluster peer probe:



gluster02:~# gluster peer probe gluster03.stic.ull.es
peer probe: success.

The problem is that this node remains in "Accepted peer request" state:



Hostname: gluster03.stic.ull.es
Uuid: 9ca23fb2-cd76-4d1c-8a5f-f93306119539
State: Accepted peer request (Connected)

If I restart glusterd in gluster02, the state changes to "Sent and Received peer request":



Hostname: gluster03.stic.ull.es
Uuid: 9ca23fb2-cd76-4d1c-8a5f-f93306119539
State: Sent and Received peer request (Connected)

And if I restart glusterd in gluster03 it doesn't start because is "Unable to find friend: gluster01.stic.ull.es ":



[2015-05-11 06:59:39.035184] D [glusterd-peer-utils.c:121:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: gluster01.stic.ull.es
[2015-05-11 06:59:39.037942] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.107.34.17
[2015-05-11 06:59:39.039392] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.107.34.17
[2015-05-11 06:59:39.040584] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.107.34.17
[2015-05-11 06:59:39.041790] D [common-utils.c:2913:gf_is_local_addr] 0-management: gluster01.stic.ull.es is not local

In fact, the only peer that gluster03 knows is gluster02:



gluster03:~# ls -l /var/lib/glusterd/peers/
total 4
-rw-------. 1 root root 73 may 11 07:48 739d7c3f-99d5-4c8b-ac28-f85163e26322
gluster03:~# cat /var/lib/glusterd/peers/739d7c3f-99d5-4c8b-ac28-f85163e26322
uuid=739d7c3f-99d5-4c8b-ac28-f85163e26322
state=5
hostname1=10.107.34.18

10.107.34.18 == gluster02



gluster03:~# gluster --version
glusterfs 3.6.3 built on Apr 23 2015 16:11:45
Repository revision: git:// git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. < http://www.gluster.com >
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.

All servers are running Centos 7, gluster01 and gluster02 have Gluster 3.6.2 version and gluster03 3.6.3.

Am I forced to upgrade gluster01 and gluster02 to 3.6.3?

Regards.

--
----------------------------------------------------------------
Adrián Santos Marrero
Técnico de Sistemas - Área de Infraestructuras TIC
Servicios de Tecnologías de la Información y Comunicación (STIC)
Universidad de La Laguna (ULL)
Teléfono/Phone: +34 922 845089
----------------------------------------------------------------

Este mensaje puede contener información confidencial y/o privilegiada.
Si usted no es el destinatario o lo ha recibido por error debe borrarlo
inmediatamente.
Está estrictamente prohibido por la legislación vigente realizar
cualquier copia, revelación o distribución del contenido de este mensaje
sin autorización expresa.

This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient or have received this e-mail in
error you must destroy it.
Any unauthorised copying, disclosure or distribution of the material in
this e-mail is strictly forbidden by current legislation.

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

--

----------------------------------------------------------------
 Adrián Santos Marrero
 Técnico de Sistemas - Área de Infraestructuras TIC
 Servicios de Tecnologías de la Información y Comunicación (STIC)
 Universidad de La Laguna (ULL)
 Teléfono/Phone: +34 922 845089
----------------------------------------------------------------

Este mensaje puede contener información confidencial y/o privilegiada.
Si usted no es el destinatario o lo ha recibido por error debe borrarlo
inmediatamente.
Está estrictamente prohibido por la legislación vigente realizar
cualquier copia, revelación o distribución del contenido de este mensaje
sin autorización expresa.

This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient or have received this e-mail in
error you must destroy it.
Any unauthorised copying, disclosure or distribution of the material in
this e-mail is strictly forbidden by current legislation.

--

----------------------------------------------------------------
 Adrián Santos Marrero
 Técnico de Sistemas - Área de Infraestructuras TIC
 Servicios de Tecnologías de la Información y Comunicación (STIC)
 Universidad de La Laguna (ULL)
 Teléfono/Phone: +34 922 845089
----------------------------------------------------------------

Este mensaje puede contener información confidencial y/o privilegiada.
Si usted no es el destinatario o lo ha recibido por error debe borrarlo
inmediatamente.
Está estrictamente prohibido por la legislación vigente realizar
cualquier copia, revelación o distribución del contenido de este mensaje
sin autorización expresa.

This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient or have received this e-mail in
error you must destroy it.
Any unauthorised copying, disclosure or distribution of the material in
this e-mail is strictly forbidden by current legislation.

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

--

----------------------------------------------------------------
 Adrián Santos Marrero
 Técnico de Sistemas - Área de Infraestructuras TIC
 Servicios de Tecnologías de la Información y Comunicación (STIC)
 Universidad de La Laguna (ULL)
 Teléfono/Phone: +34 922 845089
----------------------------------------------------------------

Este mensaje puede contener información confidencial y/o privilegiada.
Si usted no es el destinatario o lo ha recibido por error debe borrarlo
inmediatamente.
Está estrictamente prohibido por la legislación vigente realizar
cualquier copia, revelación o distribución del contenido de este mensaje
sin autorización expresa.

This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient or have received this e-mail in
error you must destroy it.
Any unauthorised copying, disclosure or distribution of the material in
this e-mail is strictly forbidden by current legislation.

--

----------------------------------------------------------------
 Adrián Santos Marrero
 Técnico de Sistemas - Área de Infraestructuras TIC
 Servicios de Tecnologías de la Información y Comunicación (STIC)
 Universidad de La Laguna (ULL)
 Teléfono/Phone: +34 922 845089
----------------------------------------------------------------

Este mensaje puede contener información confidencial y/o privilegiada.
Si usted no es el destinatario o lo ha recibido por error debe borrarlo
inmediatamente.
Está estrictamente prohibido por la legislación vigente realizar
cualquier copia, revelación o distribución del contenido de este mensaje
sin autorización expresa.

This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient or have received this e-mail in
error you must destroy it.
Any unauthorised copying, disclosure or distribution of the material in
this e-mail is strictly forbidden by current legislation.

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users