Re: Problem with add-brick

Dennis Michael <dennis.michael@xxxxxxxxx> · Tue, 27 Sep 2016 10:03:05 -0700

and on fs1:
[root@fs1 ~]# gluster volume status
Status of volume: cees-data
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick fs1:/data/brick                       49152     49153      Y       1878 
Brick fs2:/data/brick                       49152     0          Y       1707 
Brick fs3:/data/brick                       49152     0          Y       4696 
NFS Server on fs4                           2049      0          Y       13968
NFS Server on localhost                     2049      0          Y       6722 
Quota Daemon on localhost                   N/A       N/A        Y       6730 
NFS Server on fs2                           2049      0          Y       11702
Quota Daemon on fs2                         N/A       N/A        Y       11710
NFS Server on fs3                           2049      0          Y       12553
Quota Daemon on fs3                         N/A       N/A        Y       12561
Quota Daemon on fs4                         N/A       N/A        Y       13976

Task Status of Volume cees-data
------------------------------------------------------------------------------
There are no active volume tasks

[root@fs1 ~]# gluster volume info

Volume Name: cees-data
Type: Distribute
Volume ID: 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2
Status: Started
Number of Bricks: 4
Transport-type: tcp,rdma
Bricks:
Brick1: fs1:/data/brick
Brick2: fs2:/data/brick
Brick3: fs3:/data/brick
Brick4: fs4:/data/brick
Options Reconfigured:
performance.readdir-ahead: on
features.quota: on
features.inode-quota: on
features.quota-deem-statfs: on

On Tue, Sep 27, 2016 at 9:40 AM, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:

    On 09/27/2016 09:53 PM, Dennis Michael
      wrote:

      Yes, you are right.  I mixed up the logs.  I just
        ran the add-brick command again after cleaning up fs4 and
        re-installing gluster.  This is the complete fs4 data-brick.log.

          [root@fs1 ~]# gluster volume add-brick cees-data
            fs4:/data/brick
          volume add-brick: failed: Commit failed on fs4. Please
            check log file for details.

            [root@fs4 bricks]# pwd
            /var/log/glusterfs/bricks
            [root@fs4 bricks]# cat data-brick.log 
            [2016-09-27 16:16:28.095661] I [MSGID: 100030]
              [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfsd: Started
              running /usr/sbin/glusterfsd version 3.7.14 (args:
              /usr/sbin/glusterfsd -s fs4 --volfile-id
              cees-data.fs4.data-brick -p
              /var/lib/glusterd/vols/cees-data/run/fs4-data-brick.pid -S
              /var/run/gluster/5203ab38be21e1d37c04f6bdfee77d4a.socket
              --brick-name /data/brick -l
              /var/log/glusterfs/bricks/data-brick.log --xlator-option
              *-posix.glusterd-uuid=f04b231e-63f8-4374-91ae-17c0c623f165
              --brick-port 49152 --xlator-option
              cees-data-server.transport.rdma.listen-port=49153
              --xlator-option cees-data-server.listen-port=49152
              --volfile-server-transport=socket,rdma)
            [2016-09-27 16:16:28.101547] I [MSGID: 101190]
              [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll:
              Started thread with index 1
            [2016-09-27 16:16:28.104637] I
              [graph.c:269:gf_add_cmdline_options] 0-cees-data-server:
              adding option 'listen-port' for volume 'cees-data-server'
              with value '49152'
            [2016-09-27 16:16:28.104646] I
              [graph.c:269:gf_add_cmdline_options] 0-cees-data-server:
              adding option 'transport.rdma.listen-port' for volume
              'cees-data-server' with value '49153'
            [2016-09-27 16:16:28.104662] I
              [graph.c:269:gf_add_cmdline_options] 0-cees-data-posix:
              adding option 'glusterd-uuid' for volume 'cees-data-posix'
              with value 'f04b231e-63f8-4374-91ae-17c0c623f165'
            [2016-09-27 16:16:28.104808] I [MSGID: 115034]
              [server.c:403:_check_for_auth_option] 0-/data/brick: skip
              format check for non-addr auth option
              auth.login./data/brick.allow
            [2016-09-27 16:16:28.104814] I [MSGID: 115034]
              [server.c:403:_check_for_auth_option] 0-/data/brick: skip
              format check for non-addr auth option
              auth.login.18ddaf4c-ad98-4155-9372-717eae718b4c.password
            [2016-09-27 16:16:28.104883] I [MSGID: 101190]
              [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll:
              Started thread with index 2
            [2016-09-27 16:16:28.105479] I
              [rpcsvc.c:2196:rpcsvc_set_outstanding_rpc_limit]
              0-rpc-service: Configured rpc.outstanding-rpc-limit with
              value 64
            [2016-09-27 16:16:28.105532] W [MSGID: 101002]
              [options.c:957:xl_opt_validate] 0-cees-data-server: option
              'listen-port' is deprecated, preferred is
              'transport.socket.listen-port', continuing with correction
            [2016-09-27 16:16:28.109456] W
              [socket.c:3665:reconfigure] 0-cees-data-quota: NBIO on -1
              failed (Bad file descriptor)
            [2016-09-27 16:16:28.489255] I [MSGID: 121050]
              [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore:
              CTR Xlator is disabled.
            [2016-09-27 16:16:28.489272] W [MSGID: 101105]
              [gfdb_sqlite3.h:239:gfdb_set_sql_params]
              0-cees-data-changetimerecorder: Failed to retrieve
              sql-db-pagesize from params.Assigning default value: 4096
            [2016-09-27 16:16:28.489278] W [MSGID: 101105]
              [gfdb_sqlite3.h:239:gfdb_set_sql_params]
              0-cees-data-changetimerecorder: Failed to retrieve
              sql-db-journalmode from params.Assigning default value:
              wal
            [2016-09-27 16:16:28.489284] W [MSGID: 101105]
              [gfdb_sqlite3.h:239:gfdb_set_sql_params]
              0-cees-data-changetimerecorder: Failed to retrieve
              sql-db-sync from params.Assigning default value: off
            [2016-09-27 16:16:28.489288] W [MSGID: 101105]
              [gfdb_sqlite3.h:239:gfdb_set_sql_params]
              0-cees-data-changetimerecorder: Failed to retrieve
              sql-db-autovacuum from params.Assigning default value:
              none
            [2016-09-27 16:16:28.490431] I [trash.c:2412:init]
              0-cees-data-trash: no option specified for 'eliminate',
              using NULL
            [2016-09-27 16:16:28.672814] W
              [graph.c:357:_log_if_unknown_option] 0-cees-data-server:
              option 'rpc-auth.auth-glusterfs' is not recognized
            [2016-09-27 16:16:28.672854] W
              [graph.c:357:_log_if_unknown_option] 0-cees-data-server:
              option 'rpc-auth.auth-unix' is not recognized
            [2016-09-27 16:16:28.672872] W
              [graph.c:357:_log_if_unknown_option] 0-cees-data-server:
              option 'rpc-auth.auth-null' is not recognized
            [2016-09-27 16:16:28.672924] W
              [graph.c:357:_log_if_unknown_option] 0-cees-data-quota:
              option 'timeout' is not recognized
            [2016-09-27 16:16:28.672955] W
              [graph.c:357:_log_if_unknown_option] 0-cees-data-trash:
              option 'brick-path' is not recognized
            Final graph:
            +------------------------------------------------------------------------------+
              1: volume cees-data-posix
              2:     type storage/posix
              3:     option glusterd-uuid
              f04b231e-63f8-4374-91ae-17c0c623f165
              4:     option directory /data/brick
              5:     option volume-id
              27d2a59c-bdac-4f66-bcd8-e6124e53a4a2
              6:     option update-link-count-parent on
              7: end-volume
              8:  
              9: volume cees-data-trash
             10:     type features/trash
             11:     option trash-dir .trashcan
             12:     option brick-path /data/brick
             13:     option trash-internal-op off
             14:     subvolumes cees-data-posix
             15: end-volume
             16:  
             17: volume cees-data-changetimerecorder
             18:     type features/changetimerecorder
             19:     option db-type sqlite3
             20:     option hot-brick off
             21:     option db-name brick.db
             22:     option db-path /data/brick/.glusterfs/
             23:     option record-exit off
             24:     option ctr_link_consistency off
             25:     option ctr_lookupheal_link_timeout 300
             26:     option ctr_lookupheal_inode_timeout 300
             27:     option record-entry on
             28:     option ctr-enabled off
             29:     option record-counters off
             30:     option ctr-record-metadata-heat off
             31:     option sql-db-cachesize 1000
             32:     option sql-db-wal-autocheckpoint 1000
             33:     subvolumes cees-data-trash
             34: end-volume
             35:  
             36: volume cees-data-changelog
             37:     type features/changelog
             38:     option changelog-brick /data/brick
             39:     option changelog-dir
              /data/brick/.glusterfs/changelogs
             40:     option changelog-barrier-timeout 120
             41:     subvolumes cees-data-changetimerecorder
             42: end-volume
             43:  
             44: volume cees-data-bitrot-stub
             45:     type features/bitrot-stub
             46:     option export /data/brick
             47:     subvolumes cees-data-changelog
             48: end-volume
             49:  
             50: volume cees-data-access-control
             51:     type features/access-control
             52:     subvolumes cees-data-bitrot-stub
             53: end-volume
             54:  
             55: volume cees-data-locks
             56:     type features/locks
             57:     subvolumes cees-data-access-control
             58: end-volume
             59:  
             60: volume cees-data-upcall
             61:     type features/upcall
             62:     option cache-invalidation off
             63:     subvolumes cees-data-locks
             64: end-volume
             65:  
             66: volume cees-data-io-threads
             67:     type performance/io-threads
             68:     subvolumes cees-data-upcall
             69: end-volume
             70:  
             71: volume cees-data-marker
             72:     type features/marker
             73:     option volume-uuid
              27d2a59c-bdac-4f66-bcd8-e6124e53a4a2
             74:     option timestamp-file
              /var/lib/glusterd/vols/cees-data/marker.tstamp
             75:     option quota-version 1
             76:     option xtime off
             77:     option gsync-force-xtime off
             78:     option quota on
             79:     option inode-quota on
             80:     subvolumes cees-data-io-threads
             81: end-volume
             82:  
             83: volume cees-data-barrier
             84:     type features/barrier
             85:     option barrier disable
             86:     option barrier-timeout 120
             87:     subvolumes cees-data-marker
             88: end-volume
             89:  
             90: volume cees-data-index
             91:     type features/index
             92:     option index-base
              /data/brick/.glusterfs/indices
             93:     subvolumes cees-data-barrier
             94: end-volume
             95:  
             96: volume cees-data-quota
             97:     type features/quota
             98:     option transport.socket.connect-path
              /var/run/gluster/quotad.socket
             99:     option transport-type socket
            100:     option transport.address-family unix
            101:     option volume-uuid cees-data
            102:     option server-quota on
            103:     option timeout 0
            104:     option deem-statfs on
            105:     subvolumes cees-data-index
            106: end-volume
            107:  
            108: volume cees-data-worm
            109:     type features/worm
            110:     option worm off
            111:     subvolumes cees-data-quota
            112: end-volume
            113:  
            114: volume cees-data-read-only
            115:     type features/read-only
            116:     option read-only off
            117:     subvolumes cees-data-worm
            118: end-volume
            119:  
            120: volume /data/brick
            121:     type debug/io-stats
            122:     option log-level INFO
            123:     option latency-measurement off
            124:     option count-fop-hits off
            125:     subvolumes cees-data-read-only
            126: end-volume
            127:  
            128: volume cees-data-server
            129:     type protocol/server
            130:     option transport.socket.listen-port 49152
            131:     option rpc-auth.auth-glusterfs on
            132:     option rpc-auth.auth-unix on
            133:     option rpc-auth.auth-null on
            134:     option rpc-auth-allow-insecure on
            135:     option transport.rdma.listen-port 49153
            136:     option transport-type tcp,rdma
            137:     option auth.login./data/brick.allow
              18ddaf4c-ad98-4155-9372-717eae718b4c
            138:     option
              auth.login.18ddaf4c-ad98-4155-9372-717eae718b4c.password
              9e913e92-7de0-47f9-94ed-d08cbb130d23
            139:     option auth.addr./data/brick.allow *
            140:     subvolumes /data/brick
            141: end-volume
            142:  
            +------------------------------------------------------------------------------+
            [2016-09-27 16:16:30.079541] I [login.c:81:gf_auth]
              0-auth/login: allowed user names:
              18ddaf4c-ad98-4155-9372-717eae718b4c
            [2016-09-27 16:16:30.079567] I [MSGID: 115029]
              [server-handshake.c:690:server_setvolume]
              0-cees-data-server: accepted client from
              fs3-12560-2016/09/27-16:16:30:47674-cees-data-client-3-0-0
              (version: 3.7.14)
            [2016-09-27 16:16:30.081487] I [login.c:81:gf_auth]
              0-auth/login: allowed user names:
              18ddaf4c-ad98-4155-9372-717eae718b4c
            [2016-09-27 16:16:30.081505] I [MSGID: 115029]
              [server-handshake.c:690:server_setvolume]
              0-cees-data-server: accepted client from
              fs2-11709-2016/09/27-16:16:30:50047-cees-data-client-3-0-0
              (version: 3.7.14)
            [2016-09-27 16:16:30.111091] I [login.c:81:gf_auth]
              0-auth/login: allowed user names:
              18ddaf4c-ad98-4155-9372-717eae718b4c
            [2016-09-27 16:16:30.111113] I [MSGID: 115029]
              [server-handshake.c:690:server_setvolume]
              0-cees-data-server: accepted client from
              fs2-11701-2016/09/27-16:16:29:24060-cees-data-client-3-0-0
              (version: 3.7.14)
            [2016-09-27 16:16:30.112822] I [login.c:81:gf_auth]
              0-auth/login: allowed user names:
              18ddaf4c-ad98-4155-9372-717eae718b4c
            [2016-09-27 16:16:30.112836] I [MSGID: 115029]
              [server-handshake.c:690:server_setvolume]
              0-cees-data-server: accepted client from
              fs3-12552-2016/09/27-16:16:29:23041-cees-data-client-3-0-0
              (version: 3.7.14)
            [2016-09-27 16:16:31.950978] I [login.c:81:gf_auth]
              0-auth/login: allowed user names:
              18ddaf4c-ad98-4155-9372-717eae718b4c
            [2016-09-27 16:16:31.950998] I [MSGID: 115029]
              [server-handshake.c:690:server_setvolume]
              0-cees-data-server: accepted client from
              fs1-6721-2016/09/27-16:16:26:939991-cees-data-client-3-0-0
              (version: 3.7.14)
            [2016-09-27 16:16:31.981977] I [login.c:81:gf_auth]
              0-auth/login: allowed user names:
              18ddaf4c-ad98-4155-9372-717eae718b4c
            [2016-09-27 16:16:31.981994] I [MSGID: 115029]
              [server-handshake.c:690:server_setvolume]
              0-cees-data-server: accepted client from
              fs1-6729-2016/09/27-16:16:27:971228-cees-data-client-3-0-0
              (version: 3.7.14)

    Hmm, this shows the brick has started. 

    Does gluster volume info on fs4 shows all 4 bricks? (I guess it does
    based on your first email).

    Does gluster volume status on fs4  (or ps aux|grep glusterfsd) show
    the brick as running? 

    Does gluster peer status on all nodes list the other 3 nodes as
    connected? 

    If yes, you could try `service glusterd restart` on fs4 and see if
    if brings up the brick? I'm just shooting in the dark here for
    possible clues.

    -Ravi

        On Tue, Sep 27, 2016 at 8:46 AM,
          Ravishankar N <ravishankar@xxxxxxxxxx>
          wrote:

                On 09/27/2016 09:06 PM, Dennis Michael wrote:

                  Yes, the brick log
                    /var/log/glusterfs/bricks/data-brick.log is
                    created on fs4, and the snippets showing the errors
                    were from that log.

               Unless I'm missing something, the snippet below is
              from glusterd's log and not the brick's as is evident from
              the function names.

              -Ravi

                   Dennis 

                    On Mon, Sep 26, 2016 at
                      5:58 PM, Ravishankar N <ravishankar@xxxxxxxxxx>
                      wrote:

                      On 09/27/2016
                          05:25 AM, Dennis Michael wrote:

                           [2016-09-26
                            22:44:39.254921] E [MSGID: 106005]
                            [glusterd-utils.c:4771:glusterd_brick_start]
                            0-management: Unable to start brick
                            fs4:/data/brick

                            [2016-09-26 22:44:39.254949] E [MSGID:
                            106074] [glusterd-brick-ops.c:2372:glusterd_op_add_brick]
                            0-glusterd: Unable to add bricks

                         Is the brick log created on fs4? Does it
                        contain warnings/errors?

                        -Ravi

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users