Re: [ovirt-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 07/20/2017 03:42 PM, yayo (j) wrote:

2017-07-20 11:34 GMT+02:00 Ravishankar N <ravishankar@xxxxxxxxxx>:

Could you check if the self-heal daemon on all nodes is connected to the 3 bricks? You will need to check the glustershd.log for that.
 If it is not connected, try restarting the shd using `gluster volume start engine force`, then launch the heal command like you did earlier and see if heals happen.


I've executed the command on all 3 nodes (Know is enougth only one) , after that the "heal" command report elements between 6 and 10 ... (sometime 6, sometime 8, sometime 10)


Log on glustershd.log don't say anything :

But it does  say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case.
Hope this helps,
Ravi


[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1  sinks=2
[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81
[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1  sinks=2

 
If it doesn't, please provide the getfattr outputs of the 12 files from all 3 nodes using `getfattr -d -m . -e hex /gluster/engine/brick/path-to-file` ?


NODE01:
getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-1=0x000000000000000000000000
trusted.afr.engine-client-2=0x000000120000000000000000
trusted.bit-rot.version=0x090000000000000059647d5b000447e9
trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-1=0x000000000000000000000000
trusted.afr.engine-client-2=0x0000000e0000000000000000
trusted.bit-rot.version=0x090000000000000059647d5b000447e9
trusted.gfid=0x676067891f344c1586b8c0d05b07f187

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-1=0x000000000000000000000000
trusted.afr.engine-client-2=0x000000550000000000000000
trusted.bit-rot.version=0x090000000000000059647d5b000447e9
trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7
trusted.glusterfs.shard.block-size=0x0000000020000000
trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-1=0x000000000000000000000000
trusted.afr.engine-client-2=0x000000070000000000000000
trusted.bit-rot.version=0x090000000000000059647d5b000447e9
trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-1=0x000000000000000000000000
trusted.afr.engine-client-2=0x000000000000000000000000
trusted.bit-rot.version=0x0f0000000000000059647d5b000447e9
trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3
trusted.glusterfs.shard.block-size=0x0000000020000000
trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/__DIRECT_IO_TEST__
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-1=0x000000000000000000000000
trusted.afr.engine-client-2=0x000000000000000000000000
trusted.gfid=0xf05b97422771484a85fc5b6974bcef81
trusted.glusterfs.shard.block-size=0x0000000020000000
trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-1=0x000000000000000000000000
trusted.afr.engine-client-2=0x000000010000000000000000
trusted.bit-rot.version=0x0f0000000000000059647d5b000447e9
trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327
trusted.glusterfs.shard.block-size=0x0000000020000000
trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-1=0x000000000000000000000000
trusted.afr.engine-client-2=0x0000000a0000000000000000
trusted.bit-rot.version=0x090000000000000059647d5b000447e9
trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0


NODE02:
getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-0=0x000000000000000000000000
trusted.afr.engine-client-2=0x0000001a0000000000000000
trusted.bit-rot.version=0x08000000000000005965ede0000c352d
trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48
trusted.afr.dirty=0x000000010000000000000000
trusted.afr.engine-client-0=0x000000000000000000000000
trusted.afr.engine-client-2=0x0000000c0000000000000000
trusted.bit-rot.version=0x08000000000000005965ede0000c352d
trusted.gfid=0x676067891f344c1586b8c0d05b07f187

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-0=0x000000000000000000000000
trusted.afr.engine-client-1=0x000000000000000000000000
trusted.afr.engine-client-2=0x0000008e0000000000000000
trusted.bit-rot.version=0x08000000000000005965ede0000c352d
trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7
trusted.glusterfs.shard.block-size=0x0000000020000000
trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-0=0x000000000000000000000000
trusted.afr.engine-client-2=0x000000090000000000000000
trusted.bit-rot.version=0x08000000000000005965ede0000c352d
trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-0=0x000000000000000000000000
trusted.afr.engine-client-2=0x000000010000000000000000
trusted.bit-rot.version=0x08000000000000005965ede0000c352d
trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3
trusted.glusterfs.shard.block-size=0x0000000020000000
trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/__DIRECT_IO_TEST__
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-0=0x000000000000000000000000
trusted.afr.engine-client-2=0x000000000000000000000000
trusted.gfid=0xf05b97422771484a85fc5b6974bcef81
trusted.glusterfs.shard.block-size=0x0000000020000000
trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-0=0x000000000000000000000000
trusted.afr.engine-client-2=0x000000020000000000000000
trusted.bit-rot.version=0x08000000000000005965ede0000c352d
trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327
trusted.glusterfs.shard.block-size=0x0000000020000000
trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-0=0x000000000000000000000000
trusted.afr.engine-client-2=0x000000120000000000000000
trusted.bit-rot.version=0x08000000000000005965ede0000c352d
trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0




NODE04:
getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.bit-rot.version=0x050000000000000059662c390006b836
trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.bit-rot.version=0x050000000000000059662c390006b836
trusted.gfid=0x676067891f344c1586b8c0d05b07f187

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.bit-rot.version=0x050000000000000059662c390006b836
trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7
trusted.glusterfs.shard.block-size=0x0000000020000000
trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.bit-rot.version=0x050000000000000059662c390006b836
trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x050000000000000059662c390006b836
trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3
trusted.glusterfs.shard.block-size=0x0000000020000000
trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/__DIRECT_IO_TEST__
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.bit-rot.version=0x0200000000000000596484e20006237b
trusted.gfid=0xf05b97422771484a85fc5b6974bcef81
trusted.glusterfs.shard.block-size=0x0000000020000000
trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x050000000000000059662c390006b836
trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327
trusted.glusterfs.shard.block-size=0x0000000020000000
trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000

getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.bit-rot.version=0x050000000000000059662c390006b836
trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0




hum.... Is selinux the problem? but on node04 was disabled (AFTER GLUSTER JOIN, I hope to remember) ... You think I needs to relabel? how?

[root@node01 ~]# sestatus
SELinux status:                 disabled

[root@node02 ~]# sestatus
SELinux status:                 disabled

[root@node04 ~]# sestatus
SELinux status:                 disabled


Thank you 

 
Thanks,
Ravi


 
2. Are these 12 files also present in the 3rd data brick?

I've checked right now: all files exists in all 3 nodes 
 
3. Can you provide the output of `gluster volume info` for the this volume?


Volume Name: engine
Type: Replicate
Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: node01:/gluster/engine/brick
Brick2: node02:/gluster/engine/brick
Brick3: node04:/gluster/engine/brick
Options Reconfigured:
nfs.disable: on
performance.readdir-ahead: on
transport.address-family: inet
storage.owner-uid: 36
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
performance.low-prio-threads: 32
network.remote-dio: off
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
storage.owner-gid: 36
features.shard-block-size: 512MB
network.ping-timeout: 30
performance.strict-o-direct: on
cluster.granular-entry-heal: on
auth.allow: *
          server.allow-insecure: on



 

Some extra info:

We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster

Just curious, how did you do this? `remove-brick` of arbiter brick  followed by an `add-brick` to increase to replica-3?


Yes


#gluster volume remove-brick engine replica 2 node03:/gluster/data/brick force (OK!)

#gluster volume heal engine info (no entries!)

#gluster volume add-brick engine replica 3 node04:/gluster/engine/brick (OK!)

After some minutes

[root@node01 ~]#  gluster volume heal engine info
Brick node01:/gluster/engine/brick
Status: Connected
Number of entries: 0

Brick node02:/gluster/engine/brick
Status: Connected
Number of entries: 0

Brick node04:/gluster/engine/brick
Status: Connected
Number of entries: 0

 
Thanks,
Ravi

Another extra info (I don't know if this can be the problem): Five days ago A black out has suddenly shut down the networks switch (also gluster network) of node 03 and 04 ... But I don't know this problem is in place after this black out  

Thank you!





--
Linux User: 369739 http://counter.li.org

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux