Re: add-brick: failed: Commit failed

David Cunningham <dcunningham@xxxxxxxxxxxxx> · Sat, 18 May 2019 11:01:24 +1200

Hi Ravi,

The existing two nodes aren't in split-brain, at least that I'm aware of. Running "gluster volume status all" doesn't show any problem.

I'm not sure what "in metadata" means. Can you please explain that one?

On Fri, 17 May 2019 at 22:43, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:

    On 17/05/19 5:59 AM, David Cunningham
      wrote:

                      Hello,

                      We're adding an arbiter node to an existing
                        volume and having an issue. Can anyone help? The
                        root cause error appears to be
                        "00000000-0000-0000-0000-000000000001: failed to
                        resolve (Transport endpoint is not connected)",
                        as below.

    Was your root directory of the replica 2 volume  in metadata or
      entry split-brain? If yes, you need to resolve it before
      proceeding with the add-brick.

    -Ravi

                      We are running glusterfs 5.6.1. Thanks in
                        advance for any assistance!

                      On existing node gfs1, trying to add new
                        arbiter node gfs3:

                      # gluster volume add-brick gvol0 replica 3
                        arbiter 1 gfs3:/nodirectwritedata/gluster/gvol0

                        volume add-brick: failed: Commit failed on gfs3.
                        Please check log file for details.

                      On new node gfs3 in
                        gvol0-add-brick-mount.log:

                      [2019-05-17 01:20:22.689721] I
                        [fuse-bridge.c:4267:fuse_init] 0-glusterfs-fuse:
                        FUSE inited with protocol versions: glusterfs
                        7.24 kernel 7.22

                        [2019-05-17 01:20:22.689778] I
                        [fuse-bridge.c:4878:fuse_graph_sync] 0-fuse:
                        switched to graph 0

                        [2019-05-17 01:20:22.694897] E
                        [fuse-bridge.c:4336:fuse_first_lookup] 0-fuse:
                        first lookup on root failed (Transport endpoint
                        is not connected)

                        [2019-05-17 01:20:22.699770] W
                        [fuse-resolve.c:127:fuse_resolve_gfid_cbk]
                        0-fuse: 00000000-0000-0000-0000-000000000001:
                        failed to resolve (Transport endpoint is not
                        connected)

                        [2019-05-17 01:20:22.699834] W
                        [fuse-bridge.c:3294:fuse_setxattr_resume]
                        0-glusterfs-fuse: 2: SETXATTR
                        00000000-0000-0000-0000-000000000001/1
                        (trusted.add-brick) resolution failed

                        [2019-05-17 01:20:22.715656] I
                        [fuse-bridge.c:5144:fuse_thread_proc] 0-fuse:
                        initating unmount of /tmp/mntQAtu3f

                        [2019-05-17 01:20:22.715865] W
                        [glusterfsd.c:1500:cleanup_and_exit]
                        (-->/lib64/libpthread.so.0(+0x7dd5)
                        [0x7fb223bf6dd5]
                        -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5)
                        [0x560886581e75]
                        -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b)
                        [0x560886581ceb] ) 0-: received signum (15),
                        shutting down

                        [2019-05-17 01:20:22.715926] I
                        [fuse-bridge.c:5914:fini] 0-fuse: Unmounting
                        '/tmp/mntQAtu3f'.

                        [2019-05-17 01:20:22.715953] I
                        [fuse-bridge.c:5919:fini] 0-fuse: Closing fuse
                        connection to '/tmp/mntQAtu3f'.

                      Processes running on new node gfs3:

                      # ps -ef | grep gluster

                        root      6832     1  0 20:17 ?        00:00:00
                        /usr/sbin/glusterd -p /var/run/glusterd.pid
                        --log-level INFO

                        root     15799     1  0 20:17 ?        00:00:00
                        /usr/sbin/glusterfs -s localhost --volfile-id
                        gluster/glustershd -p
                        /var/run/gluster/glustershd/glustershd.pid -l
                        /var/log/glusterfs/glustershd.log -S
                        /var/run/gluster/24c12b09f93eec8e.socket
                        --xlator-option
                        *replicate*.node-uuid=2069cfb3-c798-47e3-8cf8-3c584cf7c412
                        --process-name glustershd

                        root     16856 16735  0 21:21 pts/0    00:00:00
                        grep --color=auto gluster

                      -- 

                                          David Cunningham,
                                            Voisonics Limited

                                            http://voisonics.com/

                                            USA: +1 213 221 1092

                                            New Zealand: +64 (0)28 2558
                                            3782

      _______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

-- 
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users