Re: Problem with add-brick

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Top posting because there are multiple questions....

1. Atin, it is expected to fail if you don't have RDMA device or if it is not configured.

2. Rafi and Dennis,
I was not able to determine from logs if it really is a RDMA bug. The brick logs suggest that brick started and even accepted clients. We should look at the brick log more deeply to see if there is a config issue somewhere.

As to why RDMA listener did not initiate on fs2 and fs3, we need to get brick logs for them too.

Thanks,
Raghavendra Talur


On Fri, Sep 30, 2016 at 3:43 PM, Mohammed Rafi K C <rkavunga@xxxxxxxxxx> wrote:

It seems like an actual bug, if you can file a bug in bugzilla, that would be great.


At least I don't see workaround for this issue, may  be till the next update is available with fix, you can use either rdma alone or tcp alone volume.

Let me know whether this is acceptable, if so I can give you the steps to change the transport of an existing volume.


Regards

Rafi KC


On 09/30/2016 10:35 AM, Mohammed Rafi K C wrote:



On 09/30/2016 02:35 AM, Dennis Michael wrote:

Are there any workarounds to this?  RDMA is configured on my servers.


By this, I assume your rdma setup/configuration over IPoIB is working fine.

Can you tell us what machine you are using and whether SELinux is configured on the machine or not.

Also I couldn't see any logs attached here.

Rafi KC



Dennis

On Thu, Sep 29, 2016 at 7:19 AM, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:
Dennis,

Thanks for sharing the logs.

It seems like a volume configured created with tcp,rdma transport fails to start (atleast in my local set up). The issue here is although the brick process comes up, but glusterd receives a non zero ret code from the runner interface which spawns the brick process(es).

Raghavendra Talur/Rafi,

Is this an intended behaviour if rdma device is not configured? Please chime in with your thoughts


On Wed, Sep 28, 2016 at 10:22 AM, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:
Dennis,

It seems like that add-brick has definitely failed and the entry is not committed into glusterd store. volume status and volume info commands are referring the in-memory data for fs4 (which exist) but post a restart they are no longer available. Could you run glusterd with debug log enabled (systemctl stop glusterd; glusterd -LDEBUG) and provide us cmd_history.log, glusterd log along with fs4 brick log files to further analyze the issue? Regarding the missing RDMA ports for fs2, fs3 brick can you cross check if glusterfs-rdma package is installed on both the nodes?

On Wed, Sep 28, 2016 at 7:14 AM, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:
On 09/27/2016 10:29 PM, Dennis Michael wrote:


[root@fs4 bricks]# gluster volume info
 
Volume Name: cees-data
Type: Distribute
Volume ID: 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2
Status: Started
Number of Bricks: 4
Transport-type: tcp,rdma
Bricks:
Brick1: fs1:/data/brick
Brick2: fs2:/data/brick
Brick3: fs3:/data/brick
Brick4: fs4:/data/brick
Options Reconfigured:
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
performance.readdir-ahead: on
[root@fs4 bricks]# gluster volume status
Status of volume: cees-data
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick fs1:/data/brick                       49152     49153      Y       1878 
Brick fs2:/data/brick                       49152     0          Y       1707 
Brick fs3:/data/brick                       49152     0          Y       4696 
Brick fs4:/data/brick                       N/A       N/A        N       N/A  
NFS Server on localhost                     2049      0          Y       13808
Quota Daemon on localhost                   N/A       N/A        Y       13813
NFS Server on fs1                           2049      0          Y       6722 
Quota Daemon on fs1                         N/A       N/A        Y       6730 
NFS Server on fs3                           2049      0          Y       12553
Quota Daemon on fs3                         N/A       N/A        Y       12561
NFS Server on fs2                           2049      0          Y       11702
Quota Daemon on fs2                         N/A       N/A        Y       11710
 
Task Status of Volume cees-data
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@fs4 bricks]# ps auxww | grep gluster
root     13791  0.0  0.0 701472 19768 ?        Ssl  09:06   0:00 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
root     13808  0.0  0.0 560236 41420 ?        Ssl  09:07   0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/gluster/01c61523374369658a62b75c582b5ac2.socket
root     13813  0.0  0.0 443164 17908 ?        Ssl  09:07   0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l /var/log/glusterfs/quotad.log -S /var/run/gluster/3753def90f5c34f656513dba6a544f7d.socket --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off
root     13874  0.0  0.0 1200472 31700 ?       Ssl  09:16   0:00 /usr/sbin/glusterfsd -s fs4 --volfile-id cees-data.fs4.data-brick -p /var/lib/glusterd/vols/cees-data/run/fs4-data-brick.pid -S /var/run/gluster/5203ab38be21e1d37c04f6bdfee77d4a.socket --brick-name /data/brick -l /var/log/glusterfs/bricks/data-brick.log --xlator-option *-posix.glusterd-uuid=f04b231e-63f8-4374-91ae-17c0c623f165 --brick-port 49152 49153 --xlator-option cees-data-server.transport.rdma.listen-port=49153 --xlator-option cees-data-server.listen-port=49152 --volfile-server-transport=socket,rdma
root     13941  0.0  0.0 112648   976 pts/0    S+   09:50   0:00 grep --color=auto gluster

[root@fs4 bricks]# systemctl restart glusterfsd glusterd

[root@fs4 bricks]# ps auxww | grep gluster
root     13808  0.0  0.0 560236 41420 ?        Ssl  09:07   0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/gluster/01c61523374369658a62b75c582b5ac2.socket
root     13813  0.0  0.0 443164 17908 ?        Ssl  09:07   0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l /var/log/glusterfs/quotad.log -S /var/run/gluster/3753def90f5c34f656513dba6a544f7d.socket --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off
root     13953  0.1  0.0 570740 14988 ?        Ssl  09:51   0:00 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
root     13965  0.0  0.0 112648   976 pts/0    S+   09:51   0:00 grep --color=auto gluster

[root@fs4 bricks]# gluster volume info
 
Volume Name: cees-data
Type: Distribute
Volume ID: 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2
Status: Started
Number of Bricks: 3
Transport-type: tcp,rdma
Bricks:
Brick1: fs1:/data/brick
Brick2: fs2:/data/brick
Brick3: fs3:/data/brick
Options Reconfigured:
performance.readdir-ahead: on
features.quota: on
features.inode-quota: on
features.quota-deem-statfs: on


I'm not sure what's going on here. Restarting glusterd seems to change the output of gluster volume info?  I also see you are using RDMA. Not sure why the RDMA ports for fs2 and fs3 are not shown in the volume status output. CC'ing some glusterd/rdma devs for pointers.

-Ravi



[root@fs4 bricks]# gluster volume status
Status of volume: cees-data
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick fs1:/data/brick                       49152     49153      Y       1878 
Brick fs2:/data/brick                       49152     0          Y       1707 
Brick fs3:/data/brick                       49152     0          Y       4696 
NFS Server on localhost                     2049      0          Y       13968
Quota Daemon on localhost                   N/A       N/A        Y       13976
NFS Server on fs2                           2049      0          Y       11702
Quota Daemon on fs2                         N/A       N/A        Y       11710
NFS Server on fs3                           2049      0          Y       12553
Quota Daemon on fs3                         N/A       N/A        Y       12561
NFS Server on fs1                           2049      0          Y       6722 
 
Task Status of Volume cees-data
------------------------------------------------------------------------------
There are no active volume tasks

[root@fs4 bricks]# gluster peer status
Number of Peers: 3

Hostname: fs1
Uuid: ddc0a23e-05e5-48f7-993e-a37e43b21605
State: Peer in Cluster (Connected)

Hostname: fs2
Uuid: e37108f8-d2f1-4f28-adc8-0b3d3401df29
State: Peer in Cluster (Connected)

Hostname: fs3
Uuid: 19a42201-c932-44db-b1a7-8b5b1af32a36
State: Peer in Cluster (Connected)

Dennis


On Tue, Sep 27, 2016 at 9:40 AM, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:
On 09/27/2016 09:53 PM, Dennis Michael wrote:
Yes, you are right.  I mixed up the logs.  I just ran the add-brick command again after cleaning up fs4 and re-installing gluster.  This is the complete fs4 data-brick.log.

[root@fs1 ~]# gluster volume add-brick cees-data fs4:/data/brick
volume add-brick: failed: Commit failed on fs4. Please check log file for details.

[root@fs4 bricks]# pwd
/var/log/glusterfs/bricks
[root@fs4 bricks]# cat data-brick.log 
[2016-09-27 16:16:28.095661] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.14 (args: /usr/sbin/glusterfsd -s fs4 --volfile-id cees-data.fs4.data-brick -p /var/lib/glusterd/vols/cees-data/run/fs4-data-brick.pid -S /var/run/gluster/5203ab38be21e1d37c04f6bdfee77d4a.socket --brick-name /data/brick -l /var/log/glusterfs/bricks/data-brick.log --xlator-option *-posix.glusterd-uuid=f04b231e-63f8-4374-91ae-17c0c623f165 --brick-port 49152 --xlator-option cees-data-server.transport.rdma.listen-port=49153 --xlator-option cees-data-server.listen-port=49152 --volfile-server-transport=socket,rdma)
[2016-09-27 16:16:28.101547] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-09-27 16:16:28.104637] I [graph.c:269:gf_add_cmdline_options] 0-cees-data-server: adding option 'listen-port' for volume 'cees-data-server' with value '49152'
[2016-09-27 16:16:28.104646] I [graph.c:269:gf_add_cmdline_options] 0-cees-data-server: adding option 'transport.rdma.listen-port' for volume 'cees-data-server' with value '49153'
[2016-09-27 16:16:28.104662] I [graph.c:269:gf_add_cmdline_options] 0-cees-data-posix: adding option 'glusterd-uuid' for volume 'cees-data-posix' with value 'f04b231e-63f8-4374-91ae-17c0c623f165'
[2016-09-27 16:16:28.104808] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/data/brick: skip format check for non-addr auth option auth.login./data/brick.allow
[2016-09-27 16:16:28.104814] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/data/brick: skip format check for non-addr auth option auth.login.18ddaf4c-ad98-4155-9372-717eae718b4c.password
[2016-09-27 16:16:28.104883] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-09-27 16:16:28.105479] I [rpcsvc.c:2196:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2016-09-27 16:16:28.105532] W [MSGID: 101002] [options.c:957:xl_opt_validate] 0-cees-data-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
[2016-09-27 16:16:28.109456] W [socket.c:3665:reconfigure] 0-cees-data-quota: NBIO on -1 failed (Bad file descriptor)
[2016-09-27 16:16:28.489255] I [MSGID: 121050] [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled.
[2016-09-27 16:16:28.489272] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: Failed to retrieve sql-db-pagesize from params.Assigning default value: 4096
[2016-09-27 16:16:28.489278] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: Failed to retrieve sql-db-journalmode from params.Assigning default value: wal
[2016-09-27 16:16:28.489284] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: Failed to retrieve sql-db-sync from params.Assigning default value: off
[2016-09-27 16:16:28.489288] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: Failed to retrieve sql-db-autovacuum from params.Assigning default value: none
[2016-09-27 16:16:28.490431] I [trash.c:2412:init] 0-cees-data-trash: no option specified for 'eliminate', using NULL
[2016-09-27 16:16:28.672814] W [graph.c:357:_log_if_unknown_option] 0-cees-data-server: option 'rpc-auth.auth-glusterfs' is not recognized
[2016-09-27 16:16:28.672854] W [graph.c:357:_log_if_unknown_option] 0-cees-data-server: option 'rpc-auth.auth-unix' is not recognized
[2016-09-27 16:16:28.672872] W [graph.c:357:_log_if_unknown_option] 0-cees-data-server: option 'rpc-auth.auth-null' is not recognized
[2016-09-27 16:16:28.672924] W [graph.c:357:_log_if_unknown_option] 0-cees-data-quota: option 'timeout' is not recognized
[2016-09-27 16:16:28.672955] W [graph.c:357:_log_if_unknown_option] 0-cees-data-trash: option 'brick-path' is not recognized
Final graph:
+------------------------------------------------------------------------------+
  1: volume cees-data-posix
  2:     type storage/posix
  3:     option glusterd-uuid f04b231e-63f8-4374-91ae-17c0c623f165
  4:     option directory /data/brick
  5:     option volume-id 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2
  6:     option update-link-count-parent on
  7: end-volume
  8:  
  9: volume cees-data-trash
 10:     type features/trash
 11:     option trash-dir .trashcan
 12:     option brick-path /data/brick
 13:     option trash-internal-op off
 14:     subvolumes cees-data-posix
 15: end-volume
 16:  
 17: volume cees-data-changetimerecorder
 18:     type features/changetimerecorder
 19:     option db-type sqlite3
 20:     option hot-brick off
 21:     option db-name brick.db
 22:     option db-path /data/brick/.glusterfs/
 23:     option record-exit off
 24:     option ctr_link_consistency off
 25:     option ctr_lookupheal_link_timeout 300
 26:     option ctr_lookupheal_inode_timeout 300
 27:     option record-entry on
 28:     option ctr-enabled off
 29:     option record-counters off
 30:     option ctr-record-metadata-heat off
 31:     option sql-db-cachesize 1000
 32:     option sql-db-wal-autocheckpoint 1000
 33:     subvolumes cees-data-trash
 34: end-volume
 35:  
 36: volume cees-data-changelog
 37:     type features/changelog
 38:     option changelog-brick /data/brick
 39:     option changelog-dir /data/brick/.glusterfs/changelogs
 40:     option changelog-barrier-timeout 120
 41:     subvolumes cees-data-changetimerecorder
 42: end-volume
 43:  
 44: volume cees-data-bitrot-stub
 45:     type features/bitrot-stub
 46:     option export /data/brick
 47:     subvolumes cees-data-changelog
 48: end-volume
 49:  
 50: volume cees-data-access-control
 51:     type features/access-control
 52:     subvolumes cees-data-bitrot-stub
 53: end-volume
 54:  
 55: volume cees-data-locks
 56:     type features/locks
 57:     subvolumes cees-data-access-control
 58: end-volume
 59:  
 60: volume cees-data-upcall
 61:     type features/upcall
 62:     option cache-invalidation off
 63:     subvolumes cees-data-locks
 64: end-volume
 65:  
 66: volume cees-data-io-threads
 67:     type performance/io-threads
 68:     subvolumes cees-data-upcall
 69: end-volume
 70:  
 71: volume cees-data-marker
 72:     type features/marker
 73:     option volume-uuid 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2
 74:     option timestamp-file /var/lib/glusterd/vols/cees-data/marker.tstamp
 75:     option quota-version 1
 76:     option xtime off
 77:     option gsync-force-xtime off
 78:     option quota on
 79:     option inode-quota on
 80:     subvolumes cees-data-io-threads
 81: end-volume
 82:  
 83: volume cees-data-barrier
 84:     type features/barrier
 85:     option barrier disable
 86:     option barrier-timeout 120
 87:     subvolumes cees-data-marker
 88: end-volume
 89:  
 90: volume cees-data-index
 91:     type features/index
 92:     option index-base /data/brick/.glusterfs/indices
 93:     subvolumes cees-data-barrier
 94: end-volume
 95:  
 96: volume cees-data-quota
 97:     type features/quota
 98:     option transport.socket.connect-path /var/run/gluster/quotad.socket
 99:     option transport-type socket
100:     option transport.address-family unix
101:     option volume-uuid cees-data
102:     option server-quota on
103:     option timeout 0
104:     option deem-statfs on
105:     subvolumes cees-data-index
106: end-volume
107:  
108: volume cees-data-worm
109:     type features/worm
110:     option worm off
111:     subvolumes cees-data-quota
112: end-volume
113:  
114: volume cees-data-read-only
115:     type features/read-only
116:     option read-only off
117:     subvolumes cees-data-worm
118: end-volume
119:  
120: volume /data/brick
121:     type debug/io-stats
122:     option log-level INFO
123:     option latency-measurement off
124:     option count-fop-hits off
125:     subvolumes cees-data-read-only
126: end-volume
127:  
128: volume cees-data-server
129:     type protocol/server
130:     option transport.socket.listen-port 49152
131:     option rpc-auth.auth-glusterfs on
132:     option rpc-auth.auth-unix on
133:     option rpc-auth.auth-null on
134:     option rpc-auth-allow-insecure on
135:     option transport.rdma.listen-port 49153
136:     option transport-type tcp,rdma
137:     option auth.login./data/brick.allow 18ddaf4c-ad98-4155-9372-717eae718b4c
138:     option auth.login.18ddaf4c-ad98-4155-9372-717eae718b4c.password 9e913e92-7de0-47f9-94ed-d08cbb130d23
139:     option auth.addr./data/brick.allow *
140:     subvolumes /data/brick
141: end-volume
142:  
+------------------------------------------------------------------------------+
[2016-09-27 16:16:30.079541] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c
[2016-09-27 16:16:30.079567] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs3-12560-2016/09/27-16:16:30:47674-cees-data-client-3-0-0 (version: 3.7.14)
[2016-09-27 16:16:30.081487] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c
[2016-09-27 16:16:30.081505] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs2-11709-2016/09/27-16:16:30:50047-cees-data-client-3-0-0 (version: 3.7.14)
[2016-09-27 16:16:30.111091] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c
[2016-09-27 16:16:30.111113] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs2-11701-2016/09/27-16:16:29:24060-cees-data-client-3-0-0 (version: 3.7.14)
[2016-09-27 16:16:30.112822] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c
[2016-09-27 16:16:30.112836] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs3-12552-2016/09/27-16:16:29:23041-cees-data-client-3-0-0 (version: 3.7.14)
[2016-09-27 16:16:31.950978] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c
[2016-09-27 16:16:31.950998] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs1-6721-2016/09/27-16:16:26:939991-cees-data-client-3-0-0 (version: 3.7.14)
[2016-09-27 16:16:31.981977] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c
[2016-09-27 16:16:31.981994] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs1-6729-2016/09/27-16:16:27:971228-cees-data-client-3-0-0 (version: 3.7.14)


Hmm, this shows the brick has started.
Does gluster volume info on fs4 shows all 4 bricks? (I guess it does based on your first email).
Does gluster volume status on fs4  (or ps aux|grep glusterfsd) show the brick as running?
Does gluster peer status on all nodes list the other 3 nodes as connected?

If yes, you could try `service glusterd restart` on fs4 and see if if brings up the brick? I'm just shooting in the dark here for possible clues.
-Ravi

On Tue, Sep 27, 2016 at 8:46 AM, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:
On 09/27/2016 09:06 PM, Dennis Michael wrote:
Yes, the brick log /var/log/glusterfs/bricks/data-brick.log is created on fs4, and the snippets showing the errors were from that log.

 Unless I'm missing something, the snippet below is from glusterd's log and not the brick's as is evident from the function names.
-Ravi
Dennis 

On Mon, Sep 26, 2016 at 5:58 PM, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:
On 09/27/2016 05:25 AM, Dennis Michael wrote:
[2016-09-26 22:44:39.254921] E [MSGID: 106005] [glusterd-utils.c:4771:glusterd_brick_start] 0-management: Unable to start brick fs4:/data/brick
[2016-09-26 22:44:39.254949] E [MSGID: 106074] [glusterd-brick-ops.c:2372:glusterd_op_add_brick] 0-glusterd: Unable to add bricks

 Is the brick log created on fs4? Does it contain warnings/errors?

-Ravi










--

--Atin



--

--Atin




_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux