Re: Problem with add-brick

[Mohammed Rafi K C <rkavunga@xxxxxxxxxx>]

On 09/30/2016 02:35 AM, Dennis Michael wrote:

Are there any workarounds to this?  RDMA is configured on my servers.

By this, I assume your rdma setup/configuration over IPoIB is working fine.

Can you tell us what machine you are using and whether SELinux is configured on the machine or not.

Also I couldn't see any logs attached here.

Rafi KC

Dennis

On Thu, Sep 29, 2016 at 7:19 AM, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:

Dennis,

Thanks for sharing the logs.

It seems like a volume configured created with tcp,rdma transport fails to start (atleast in my local set up). The issue here is although the brick process comes up, but glusterd receives a non zero ret code from the runner interface which spawns the brick process(es).

Raghavendra Talur/Rafi,

Is this an intended behaviour if rdma device is not configured? Please chime in with your thoughts

On Wed, Sep 28, 2016 at 10:22 AM, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:

Dennis,

It seems like that add-brick has definitely failed and the entry is not committed into glusterd store. volume status and volume info commands are referring the in-memory data for fs4 (which exist) but post a restart they are no longer available. Could you run glusterd with debug log enabled (systemctl stop glusterd; glusterd -LDEBUG) and provide us cmd_history.log, glusterd log along with fs4 brick log files to further analyze the issue? Regarding the missing RDMA ports for fs2, fs3 brick can you cross check if glusterfs-rdma package is installed on both the nodes?

On Wed, Sep 28, 2016 at 7:14 AM, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:

On 09/27/2016 10:29 PM, Dennis Michael wrote:

[root@fs4 bricks]# gluster volume info

 

Volume Name: cees-data

Type: Distribute

Volume ID: 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2

Status: Started

Number of Bricks: 4

Transport-type: tcp,rdma

Bricks:

Brick1: fs1:/data/brick

Brick2: fs2:/data/brick

Brick3: fs3:/data/brick

Brick4: fs4:/data/brick

Options Reconfigured:

features.quota-deem-statfs: on

features.inode-quota: on

features.quota: on

performance.readdir-ahead: on

[root@fs4 bricks]# gluster volume status

Status of volume: cees-data

Gluster process                             TCP Port  RDMA Port  Online  Pid

------------------------------------------------------------------------------

Brick fs1:/data/brick                       49152     49153      Y       1878 

Brick fs2:/data/brick                       49152     0          Y       1707 

Brick fs3:/data/brick                       49152     0          Y       4696 

Brick fs4:/data/brick                       N/A       N/A        N       N/A  

NFS Server on localhost                     2049      0          Y       13808

Quota Daemon on localhost                   N/A       N/A        Y       13813

NFS Server on fs1                           2049      0          Y       6722 

Quota Daemon on fs1                         N/A       N/A        Y       6730 

NFS Server on fs3                           2049      0          Y       12553

Quota Daemon on fs3                         N/A       N/A        Y       12561

NFS Server on fs2                           2049      0          Y       11702

Quota Daemon on fs2                         N/A       N/A        Y       11710

 

Task Status of Volume cees-data

------------------------------------------------------------------------------

There are no active volume tasks

 

[root@fs4 bricks]# ps auxww | grep gluster

root     13791  0.0  0.0 701472 19768 ?        Ssl  09:06   0:00 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO

root     13808  0.0  0.0 560236 41420 ?        Ssl  09:07   0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/gluster/01c61523374369658a62b75c582b5ac2.socket

root     13813  0.0  0.0 443164 17908 ?        Ssl  09:07   0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l /var/log/glusterfs/quotad.log -S /var/run/gluster/3753def90f5c34f656513dba6a544f7d.socket --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off

root     13874  0.0  0.0 1200472 31700 ?       Ssl  09:16   0:00 /usr/sbin/glusterfsd -s fs4 --volfile-id cees-data.fs4.data-brick -p /var/lib/glusterd/vols/cees-data/run/fs4-data-brick.pid -S /var/run/gluster/5203ab38be21e1d37c04f6bdfee77d4a.socket --brick-name /data/brick -l /var/log/glusterfs/bricks/data-brick.log --xlator-option *-posix.glusterd-uuid=f04b231e-63f8-4374-91ae-17c0c623f165 --brick-port 49152 49153 --xlator-option cees-data-server.transport.rdma.listen-port=49153 --xlator-option cees-data-server.listen-port=49152 --volfile-server-transport=socket,rdma

root     13941  0.0  0.0 112648   976 pts/0    S+   09:50   0:00 grep --color=auto gluster

[root@fs4 bricks]# systemctl restart glusterfsd glusterd

[root@fs4 bricks]# ps auxww | grep gluster

root     13808  0.0  0.0 560236 41420 ?        Ssl  09:07   0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/gluster/01c61523374369658a62b75c582b5ac2.socket

root     13813  0.0  0.0 443164 17908 ?        Ssl  09:07   0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l /var/log/glusterfs/quotad.log -S /var/run/gluster/3753def90f5c34f656513dba6a544f7d.socket --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off

root     13953  0.1  0.0 570740 14988 ?        Ssl  09:51   0:00 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO

root     13965  0.0  0.0 112648   976 pts/0    S+   09:51   0:00 grep --color=auto gluster

[root@fs4 bricks]# gluster volume info

 

Volume Name: cees-data

Type: Distribute

Volume ID: 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2

Status: Started

Number of Bricks: 3

Transport-type: tcp,rdma

Bricks:

Brick1: fs1:/data/brick

Brick2: fs2:/data/brick

Brick3: fs3:/data/brick

Options Reconfigured:

performance.readdir-ahead: on

features.quota: on

features.inode-quota: on

features.quota-deem-statfs: on

I'm not sure what's going on here. Restarting glusterd seems to change the output of gluster volume info?  I also see you are using RDMA. Not sure why the RDMA ports for fs2 and fs3 are not shown in the volume status output. CC'ing some glusterd/rdma devs for pointers.

-Ravi

[root@fs4 bricks]# gluster volume status

Status of volume: cees-data

Gluster process                             TCP Port  RDMA Port  Online  Pid

------------------------------------------------------------------------------

Brick fs1:/data/brick                       49152     49153      Y       1878 

Brick fs2:/data/brick                       49152     0          Y       1707 

Brick fs3:/data/brick                       49152     0          Y       4696 

NFS Server on localhost                     2049      0          Y       13968

Quota Daemon on localhost                   N/A       N/A        Y       13976

NFS Server on fs2                           2049      0          Y       11702

Quota Daemon on fs2                         N/A       N/A        Y       11710

NFS Server on fs3                           2049      0          Y       12553

Quota Daemon on fs3                         N/A       N/A        Y       12561

NFS Server on fs1                           2049      0          Y       6722 

 

Task Status of Volume cees-data

------------------------------------------------------------------------------

There are no active volume tasks

[root@fs4 bricks]# gluster peer status

Number of Peers: 3

Hostname: fs1

Uuid: ddc0a23e-05e5-48f7-993e-a37e43b21605

State: Peer in Cluster (Connected)

Hostname: fs2

Uuid: e37108f8-d2f1-4f28-adc8-0b3d3401df29

State: Peer in Cluster (Connected)

Hostname: fs3

Uuid: 19a42201-c932-44db-b1a7-8b5b1af32a36

State: Peer in Cluster (Connected)

Dennis

On Tue, Sep 27, 2016 at 9:40 AM, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:

On 09/27/2016 09:53 PM, Dennis Michael wrote:

Yes, you are right.  I mixed up the logs.  I just ran the add-brick command again after cleaning up fs4 and re-installing gluster.  This is the complete fs4 data-brick.log.

[root@fs1 ~]# gluster volume add-brick cees-data fs4:/data/brick

volume add-brick: failed: Commit failed on fs4. Please check log file for details.

[root@fs4 bricks]# pwd

/var/log/glusterfs/bricks

[root@fs4 bricks]# cat data-brick.log 

[2016-09-27 16:16:28.095661] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.14 (args: /usr/sbin/glusterfsd -s fs4 --volfile-id cees-data.fs4.data-brick -p /var/lib/glusterd/vols/cees-data/run/fs4-data-brick.pid -S /var/run/gluster/5203ab38be21e1d37c04f6bdfee77d4a.socket --brick-name /data/brick -l /var/log/glusterfs/bricks/data-brick.log --xlator-option *-posix.glusterd-uuid=f04b231e-63f8-4374-91ae-17c0c623f165 --brick-port 49152 --xlator-option cees-data-server.transport.rdma.listen-port=49153 --xlator-option cees-data-server.listen-port=49152 --volfile-server-transport=socket,rdma)

[2016-09-27 16:16:28.101547] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1

[2016-09-27 16:16:28.104637] I [graph.c:269:gf_add_cmdline_options] 0-cees-data-server: adding option 'listen-port' for volume 'cees-data-server' with value '49152'

[2016-09-27 16:16:28.104646] I [graph.c:269:gf_add_cmdline_options] 0-cees-data-server: adding option 'transport.rdma.listen-port' for volume 'cees-data-server' with value '49153'

[2016-09-27 16:16:28.104662] I [graph.c:269:gf_add_cmdline_options] 0-cees-data-posix: adding option 'glusterd-uuid' for volume 'cees-data-posix' with value 'f04b231e-63f8-4374-91ae-17c0c623f165'

[2016-09-27 16:16:28.104808] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/data/brick: skip format check for non-addr auth option auth.login./data/brick.allow

[2016-09-27 16:16:28.104814] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/data/brick: skip format check for non-addr auth option auth.login.18ddaf4c-ad98-4155-9372-717eae718b4c.password

[2016-09-27 16:16:28.104883] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2

[2016-09-27 16:16:28.105479] I [rpcsvc.c:2196:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64

[2016-09-27 16:16:28.105532] W [MSGID: 101002] [options.c:957:xl_opt_validate] 0-cees-data-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction

[2016-09-27 16:16:28.109456] W [socket.c:3665:reconfigure] 0-cees-data-quota: NBIO on -1 failed (Bad file descriptor)

[2016-09-27 16:16:28.489255] I [MSGID: 121050] [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled.

[2016-09-27 16:16:28.489272] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: Failed to retrieve sql-db-pagesize from params.Assigning default value: 4096

[2016-09-27 16:16:28.489278] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: Failed to retrieve sql-db-journalmode from params.Assigning default value: wal

[2016-09-27 16:16:28.489284] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: Failed to retrieve sql-db-sync from params.Assigning default value: off

[2016-09-27 16:16:28.489288] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: Failed to retrieve sql-db-autovacuum from params.Assigning default value: none

[2016-09-27 16:16:28.490431] I [trash.c:2412:init] 0-cees-data-trash: no option specified for 'eliminate', using NULL

[2016-09-27 16:16:28.672814] W [graph.c:357:_log_if_unknown_option] 0-cees-data-server: option 'rpc-auth.auth-glusterfs' is not recognized

[2016-09-27 16:16:28.672854] W [graph.c:357:_log_if_unknown_option] 0-cees-data-server: option 'rpc-auth.auth-unix' is not recognized

[2016-09-27 16:16:28.672872] W [graph.c:357:_log_if_unknown_option] 0-cees-data-server: option 'rpc-auth.auth-null' is not recognized

[2016-09-27 16:16:28.672924] W [graph.c:357:_log_if_unknown_option] 0-cees-data-quota: option 'timeout' is not recognized

[2016-09-27 16:16:28.672955] W [graph.c:357:_log_if_unknown_option] 0-cees-data-trash: option 'brick-path' is not recognized

Final graph:

+------------------------------------------------------------------------------+

  1: volume cees-data-posix

  2:     type storage/posix

  3:     option glusterd-uuid f04b231e-63f8-4374-91ae-17c0c623f165

  4:     option directory /data/brick

  5:     option volume-id 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2

  6:     option update-link-count-parent on

  7: end-volume

  8:  

  9: volume cees-data-trash

 10:     type features/trash

 11:     option trash-dir .trashcan

 12:     option brick-path /data/brick

 13:     option trash-internal-op off

 14:     subvolumes cees-data-posix

 15: end-volume

 16:  

 17: volume cees-data-changetimerecorder

 18:     type features/changetimerecorder

 19:     option db-type sqlite3

 20:     option hot-brick off

 21:     option db-name brick.db

 22:     option db-path /data/brick/.glusterfs/

 23:     option record-exit off

 24:     option ctr_link_consistency off

 25:     option ctr_lookupheal_link_timeout 300

 26:     option ctr_lookupheal_inode_timeout 300

 27:     option record-entry on

 28:     option ctr-enabled off

 29:     option record-counters off

 30:     option ctr-record-metadata-heat off

 31:     option sql-db-cachesize 1000

 32:     option sql-db-wal-autocheckpoint 1000

 33:     subvolumes cees-data-trash

 34: end-volume

 35:  

 36: volume cees-data-changelog

 37:     type features/changelog

 38:     option changelog-brick /data/brick

 39:     option changelog-dir /data/brick/.glusterfs/changelogs

 40:     option changelog-barrier-timeout 120

 41:     subvolumes cees-data-changetimerecorder

 42: end-volume

 43:  

 44: volume cees-data-bitrot-stub

 45:     type features/bitrot-stub

 46:     option export /data/brick

 47:     subvolumes cees-data-changelog

 48: end-volume

 49:  

 50: volume cees-data-access-control

 51:     type features/access-control

 52:     subvolumes cees-data-bitrot-stub

 53: end-volume

 54:  

 55: volume cees-data-locks

 56:     type features/locks

 57:     subvolumes cees-data-access-control

 58: end-volume

 59:  

 60: volume cees-data-upcall

 61:     type features/upcall

 62:     option cache-invalidation off

 63:     subvolumes cees-data-locks

 64: end-volume

 65:  

 66: volume cees-data-io-threads

 67:     type performance/io-threads

 68:     subvolumes cees-data-upcall

 69: end-volume

 70:  

 71: volume cees-data-marker

 72:     type features/marker

 73:     option volume-uuid 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2

 74:     option timestamp-file /var/lib/glusterd/vols/cees-data/marker.tstamp

 75:     option quota-version 1

 76:     option xtime off

 77:     option gsync-force-xtime off

 78:     option quota on

 79:     option inode-quota on

 80:     subvolumes cees-data-io-threads

 81: end-volume

 82:  

 83: volume cees-data-barrier

 84:     type features/barrier

 85:     option barrier disable

 86:     option barrier-timeout 120

 87:     subvolumes cees-data-marker

 88: end-volume

 89:  

 90: volume cees-data-index

 91:     type features/index

 92:     option index-base /data/brick/.glusterfs/indices

 93:     subvolumes cees-data-barrier

 94: end-volume

 95:  

 96: volume cees-data-quota

 97:     type features/quota

 98:     option transport.socket.connect-path /var/run/gluster/quotad.socket

 99:     option transport-type socket

100:     option transport.address-family unix

101:     option volume-uuid cees-data

102:     option server-quota on

103:     option timeout 0

104:     option deem-statfs on

105:     subvolumes cees-data-index

106: end-volume

107:  

108: volume cees-data-worm

109:     type features/worm

110:     option worm off

111:     subvolumes cees-data-quota

112: end-volume

113:  

114: volume cees-data-read-only

115:     type features/read-only

116:     option read-only off

117:     subvolumes cees-data-worm

118: end-volume

119:  

120: volume /data/brick

121:     type debug/io-stats

122:     option log-level INFO

123:     option latency-measurement off

124:     option count-fop-hits off

125:     subvolumes cees-data-read-only

126: end-volume

127:  

128: volume cees-data-server

129:     type protocol/server

130:     option transport.socket.listen-port 49152

131:     option rpc-auth.auth-glusterfs on

132:     option rpc-auth.auth-unix on

133:     option rpc-auth.auth-null on

134:     option rpc-auth-allow-insecure on

135:     option transport.rdma.listen-port 49153

136:     option transport-type tcp,rdma

137:     option auth.login./data/brick.allow 18ddaf4c-ad98-4155-9372-717eae718b4c

138:     option auth.login.18ddaf4c-ad98-4155-9372-717eae718b4c.password 9e913e92-7de0-47f9-94ed-d08cbb130d23

139:     option auth.addr./data/brick.allow *

140:     subvolumes /data/brick

141: end-volume

142:  

+------------------------------------------------------------------------------+

[2016-09-27 16:16:30.079541] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c

[2016-09-27 16:16:30.079567] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs3-12560-2016/09/27-16:16:30:47674-cees-data-client-3-0-0 (version: 3.7.14)

[2016-09-27 16:16:30.081487] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c

[2016-09-27 16:16:30.081505] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs2-11709-2016/09/27-16:16:30:50047-cees-data-client-3-0-0 (version: 3.7.14)

[2016-09-27 16:16:30.111091] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c

[2016-09-27 16:16:30.111113] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs2-11701-2016/09/27-16:16:29:24060-cees-data-client-3-0-0 (version: 3.7.14)

[2016-09-27 16:16:30.112822] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c

[2016-09-27 16:16:30.112836] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs3-12552-2016/09/27-16:16:29:23041-cees-data-client-3-0-0 (version: 3.7.14)

[2016-09-27 16:16:31.950978] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c

[2016-09-27 16:16:31.950998] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs1-6721-2016/09/27-16:16:26:939991-cees-data-client-3-0-0 (version: 3.7.14)

[2016-09-27 16:16:31.981977] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c

[2016-09-27 16:16:31.981994] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs1-6729-2016/09/27-16:16:27:971228-cees-data-client-3-0-0 (version: 3.7.14)

Hmm, this shows the brick has started.
Does gluster volume info on fs4 shows all 4 bricks? (I guess it does based on your first email).
Does gluster volume status on fs4  (or ps aux|grep glusterfsd) show the brick as running?
Does gluster peer status on all nodes list the other 3 nodes as connected?

If yes, you could try `service glusterd restart` on fs4 and see if if brings up the brick? I'm just shooting in the dark here for possible clues.
-Ravi

On Tue, Sep 27, 2016 at 8:46 AM, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:

On 09/27/2016 09:06 PM, Dennis Michael wrote:

Yes, the brick log /var/log/glusterfs/bricks/data-brick.log is created on fs4, and the snippets showing the errors were from that log.

Unless I'm missing something, the snippet below is from glusterd's log and not the brick's as is evident from the function names.
-Ravi

Dennis 

On Mon, Sep 26, 2016 at 5:58 PM, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:

On 09/27/2016 05:25 AM, Dennis Michael wrote:

[2016-09-26 22:44:39.254921] E [MSGID: 106005] [glusterd-utils.c:4771:glusterd_brick_start] 0-management: Unable to start brick fs4:/data/brick
[2016-09-26 22:44:39.254949] E [MSGID: 106074] [glusterd-brick-ops.c:2372:glusterd_op_add_brick] 0-glusterd: Unable to add bricks

Is the brick log created on fs4? Does it contain warnings/errors?

-Ravi

--

--Atin

--

--Atin

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users