Re: Gluster replicate-brick issues (Distrubuted-Replica)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



That makes sense, but now that the brick is missing and the data is not replicated, how don I get the brick back in to the trusted pool?

-------- Original message --------
From: Subrata Ghosh
Date:02/14/2015 9:55 PM (GMT-08:00)
To: Thomas Holkenbrink , "'gluster-users@xxxxxxxxxxx'"
Subject: Re: Gluster replicate-brick issues (Distrubuted-Replica)

Please try to use "commit force" option and see whether it works as gluster  suggested the other only replace-brick command has been depreciated.

[root@Server1 ~]# gluster volume replace-brick $volname $from $to start

All replace-brick commands except commit force are deprecated. Do you want to continue? (y/n) y

volume replace-brick: success: replace-brick started successfully

ID: 0062d555-e7eb-4ebe-a264-7e0baf6e7546

You could try following  options , used to resolve our replace-brcik scenario  looks like close to your case  "gluster heal info" has some issues. That I requested for a clarification fewdays back.

 

# gluster volume replace-bricks $vol_name $old_brick $new_brick commit force

# gluster volume heal $vol_name full

#gluster volume $vol_name info      --> this shows that the Numer of files=1, even though the file is already healed.


Regards,
Subrata

On 02/14/2015 02:28 PM, Thomas Holkenbrink wrote:

We have tried to migrate a Brick from one server to another using the following commands.   But the data is NOT being replicated… and the BRICK is not showing up anymore.

Gluster still appears to be working but the Bricks are not balanced and I need to add the other Brick for Server3 that I don’t want to do until after Server1:Brick2 gets replicated.

 

This is the command to create the Original Volume:

[root@Server1 ~]# gluster volume create Storage1 replica 2 transport tcp Server1:/exp/br01/brick1 Server2:/exp/br01/brick1 Server1:/exp/br02/brick2 Server2:/exp/br02/brick2

 

 

This is the Current configuration BEFORE the migration.. Server3 has been Peer Probed successfully but that has been it

[root@Server1 ~]# gluster --version

glusterfs 3.6.2 built on Jan 22 2015 12:58:11

 

[root@Server1 ~]# gluster volume status

Status of volume: Storage1

Gluster process                 Port    Online  Pid

------------------------------------------------------------------------------

Brick Server1:/exp/br01/brick1  49152   Y       2167

Brick Server2:/exp/br01/brick1  49152   Y       2192

Brick Server1:/exp/br02/brick2  49153   Y       2172   <--- this is the one that goes missing

Brick Server2:/exp/br02/brick2  49153   Y       2193

NFS Server on localhost         2049    Y       2181

Self-heal Daemon on localhost   N/A     Y       2186

NFS Server on Server2           2049    Y       2205

Self-heal Daemon on Server2     N/A     Y       2210

NFS Server on Server3           2049    Y       6015

Self-heal Daemon on Server3     N/A     Y       6016

 

Task Status of Volume Storage1

------------------------------------------------------------------------------

There are no active volume tasks

[root@Server1 ~]# gluster volume info

 

Volume Name: Storage1

Type: Distributed-Replicate

Volume ID: 9616ce42-48bd-4fe3-883f-decd6c4fcd00

Status: Started

Number of Bricks: 2 x 2 = 4

Transport-type: tcp

Bricks:

Brick1: Server1:/exp/br01/brick1

Brick2: Server2:/exp/br01/brick1

Brick3: Server1:/exp/br02/brick2

Brick4: Server2:/exp/br02/brick2

Options Reconfigured:

diagnostics.brick-log-level: WARNING

diagnostics.client-log-level: WARNING

cluster.entry-self-heal: off

cluster.data-self-heal: off

cluster.metadata-self-heal: off

performance.cache-size: 1024MB

performance.cache-max-file-size: 2MB

performance.cache-refresh-timeout: 1

performance.stat-prefetch: off

performance.read-ahead: on

performance.quick-read: off

performance.write-behind-window-size: 4MB

performance.flush-behind: on

performance.write-behind: on

performance.io-thread-count: 32

performance.io-cache: on

network.ping-timeout: 2

nfs.addr-namelookup: off

performance.strict-write-ordering: on

[root@Server1 ~]#

 

 

 

So we start the Migration of the Brick to the new server using the replace Brick command

[root@Server1 ~]# volname=Storage1

 

[root@Server1 ~]# from=Server1:/exp/br02/brick2

 

[root@Server1 ~]# to=Server3:/exp/br02/brick2

 

[root@Server1 ~]# gluster volume replace-brick $volname $from $to start

All replace-brick commands except commit force are deprecated. Do you want to continue? (y/n) y

volume replace-brick: success: replace-brick started successfully

ID: 0062d555-e7eb-4ebe-a264-7e0baf6e7546

 

 

[root@Server1 ~]# gluster volume replace-brick $volname $from $to status

All replace-brick commands except commit force are deprecated. Do you want to continue? (y/n) y

volume replace-brick: success: Number of files migrated = 281   Migration complete

 

At this point everything seems to be in order with no outstanding issues.

 

[root@Server1 ~]# gluster volume status

Status of volume: Storage1

Gluster process                 Port    Online  Pid

------------------------------------------------------------------------------

Brick Server1:/exp/br01/brick1  49152   Y       2167

Brick Server2:/exp/br01/brick1  49152   Y       2192

Brick Server1:/exp/br02/brick2  49153   Y       27557

Brick Server2:/exp/br02/brick2  49153   Y       2193

NFS Server on localhost         2049    Y       27562

Self-heal Daemon on localhost   N/A     Y       2186

NFS Server on Server2           2049    Y       2205

Self-heal Daemon on Server2     N/A     Y       2210

NFS Server on Server3           2049    Y       6015

Self-heal Daemon on Server3     N/A     Y       6016

 

Task Status of Volume Storage1

------------------------------------------------------------------------------

Task                 : Replace brick

ID                   : 0062d555-e7eb-4ebe-a264-7e0baf6e7546

Source Brick         : Server1:/exp/br02/brick2

Destination Brick    : Server3:/exp/br02/brick2

Status               : completed

 

The volume reports that the replace Brick command completed.. so the next step is to commit the change

 

[root@Server1 ~]# gluster volume replace-brick $volname $from $to commit

All replace-brick commands except commit force are deprecated. Do you want to continue? (y/n) y

volume replace-brick: success: replace-brick commit successful

 

At this point when I take a look at the status I see that the OLD brick is now missing (Server1:/exp/br02/brick2) AND I don’t see the new Brick… WTF… panic!

 

[root@Server1 ~]# gluster volume status

Status of volume: Storage1

Gluster process                                         Port    Online  Pid

------------------------------------------------------------------------------

Brick Server1:/exp/br01/brick1  49152   Y       2167

Brick Server2:/exp/br01/brick1  49152   Y       2192

Brick Server2:/exp/br02/brick2  49153   Y       2193

NFS Server on localhost         2049    Y       28906

Self-heal Daemon on localhost   N/A     Y       28911

NFS Server on Server2           2049    Y       2205

Self-heal Daemon on Server2     N/A     Y       2210

NFS Server on Server3           2049    Y       6015

Self-heal Daemon on Server3     N/A     Y       6016

 

Task Status of Volume Storage1

------------------------------------------------------------------------------

There are no active volume tasks

 

 

After the commit on Server1 it does not have the Tasks listed anymore… yet server2 and server3 see this

 

[root@Server2 ~]# gluster volume status

Status of volume: Storage1

Gluster process                 Port    Online  Pid

------------------------------------------------------------------------------

Brick Server1:/exp/br01/brick1  49152   Y       2167

Brick Server2:/exp/br01/brick1  49152   Y       2192

Brick Server2:/exp/br02/brick2  49153   Y       2193

NFS Server on localhost         2049    Y       2205

Self-heal Daemon on localhost   N/A     Y       2210

NFS Server on 10.45.16.17       2049    Y       28906

Self-heal Daemon on 10.45.16.17 N/A     Y       28911

NFS Server on server3           2049    Y       6015

Self-heal Daemon on server3     N/A     Y       6016

 

Task Status of Volume Storage1

------------------------------------------------------------------------------

Task                 : Replace brick

ID                   : 0062d555-e7eb-4ebe-a264-7e0baf6e7546

Source Brick         : Server1:/exp/br02/brick2

Destination Brick    : server3:/exp/br02/brick2

Status               : completed

 

 

If I navigate the brick on Server3 the brick is NOT empty.. but missing A LOT!  It’s like the replace brick stopped… and never restarted again.

The replace brick reported back “Number of files migrated = 281   Migration complete” but when I look on Server3 Brick I get:

       [root@Server3 brick2]# find . -type f -print | wc -l

16

 

I’m missing 265 files.. (they still exist on the OLD brick.. but how can I move it?)

 

If I try to add the old brick back with another brick on the new server as such

[root@Server1 ~]# gluster volume add-brick Storage1 Server1:/exp/br02/brick2 Server3:/exp/br01/brick1

volume add-brick: failed: /exp/br02/brick2 is already part of a volume

 

Im fearfull of running:

[root@Server1 ~]# setfattr -n trusted.glusterfs.volume-id -v 0x$(grep volume-id /var/lib/glusterd/vols/$volname/info | cut -d= -f2 | sed 's/-//g') /exp/br02/brick2

Although it should allow me to add the brick

 

Gluster Heal info returns:

[root@Server2 ~]# gluster volume heal Storage1 info

Brick Server1:/exp/br01/brick1/

Number of entries: 0

 

Brick Server2:/exp/br01/brick1/

Number of entries: 0

 

Brick Server1:/exp/br02/brick2

Status: Transport endpoint is not connected

 

Brick Server2:/exp/br02/brick2/

Number of entries: 0

 

I have restarted glusterd numerous times.

 

 

at this time I’m not sure where to go from here… I know that the Server1:/exp/br02/brick2 still has all the data.. and Server3:/exp/br01/brick1 is not complete

 

How do I actually get the brick to replicate?

How can I add Server1:/exp/br02/brick2 back into the trusted pool if I can’t replicate it, or re-add it?

How can I fix this to get it back into a replicated state between the three servers?

 

Thomas

 

 

 

 

----DATA----

 

Gluster volume info at this point

[root@Server1 ~]# gluster volume info

 

Volume Name: Storage1

Type: Distributed-Replicate

Volume ID: 9616ce42-48bd-4fe3-883f-decd6c4fcd00

Status: Started

Number of Bricks: 2 x 2 = 4

Transport-type: tcp

Bricks:

Brick1: Server1:/exp/br01/brick1

Brick2: Server2:/exp/br01/brick1

Brick3: server3:/exp/br02/brick2

Brick4: Server2:/exp/br02/brick2

Options Reconfigured:

diagnostics.brick-log-level: WARNING

diagnostics.client-log-level: WARNING

cluster.entry-self-heal: off

cluster.data-self-heal: off

cluster.metadata-self-heal: off

performance.cache-size: 1024MB

performance.cache-max-file-size: 2MB

performance.cache-refresh-timeout: 1

performance.stat-prefetch: off

performance.read-ahead: on

performance.quick-read: off

performance.write-behind-window-size: 4MB

performance.flush-behind: on

performance.write-behind: on

performance.io-thread-count: 32

performance.io-cache: on

network.ping-timeout: 2

nfs.addr-namelookup: off

performance.strict-write-ordering: on

[root@Server1 ~]#

 

[root@server3 brick2]# gluster volume heal Storage1 info

Brick Server1:/exp/br01/brick1/

Number of entries: 0

 

Brick Server2:/exp/br01/brick1/

Number of entries: 0

 

Brick Server3:/exp/br02/brick2/

Number of entries: 0

 

Brick Server2:/exp/br02/brick2/

Number of entries: 0

 

 

Gluster LOG ( there are a few errors but I’m not sure how to decipher them)

 

[2015-02-14 06:29:19.862809] I [MSGID: 106005] [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick Server1:/exp/br02/brick2 has disconnected from glusterd.

[2015-02-14 06:29:19.862836] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/7565ec897c6454bd3e2f4800250a7221.socket failed (Invalid argument)

[2015-02-14 06:29:19.862853] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd.

[2015-02-14 06:29:19.953762] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /exp/br02/brick2 on port 49153

[2015-02-14 06:31:12.977450] I [glusterd-replace-brick.c:99:__glusterd_handle_replace_brick] 0-management: Received replace brick req

[2015-02-14 06:31:12.977495] I [glusterd-replace-brick.c:154:__glusterd_handle_replace_brick] 0-management: Received replace brick status request

[2015-02-14 06:31:13.048852] I [glusterd-replace-brick.c:1412:rb_update_srcbrick_port] 0-: adding src-brick port no

[2015-02-14 06:31:19.588380] I [glusterd-replace-brick.c:99:__glusterd_handle_replace_brick] 0-management: Received replace brick req

[2015-02-14 06:31:19.588422] I [glusterd-replace-brick.c:154:__glusterd_handle_replace_brick] 0-management: Received replace brick status request

[2015-02-14 06:31:19.661101] I [glusterd-replace-brick.c:1412:rb_update_srcbrick_port] 0-: adding src-brick port no

[2015-02-14 06:31:45.115355] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-02-14 06:31:45.118597] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1

[2015-02-14 06:32:10.956357] I [glusterd-replace-brick.c:99:__glusterd_handle_replace_brick] 0-management: Received replace brick req

[2015-02-14 06:32:10.956385] I [glusterd-replace-brick.c:154:__glusterd_handle_replace_brick] 0-management: Received replace brick commit request

[2015-02-14 06:32:11.028472] I [glusterd-replace-brick.c:1412:rb_update_srcbrick_port] 0-: adding src-brick port no

[2015-02-14 06:32:12.122552] I [glusterd-utils.c:6276:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully

[2015-02-14 06:32:12.131836] I [glusterd-utils.c:6281:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully

[2015-02-14 06:32:12.141107] I [glusterd-utils.c:6286:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully

[2015-02-14 06:32:12.150375] I [glusterd-utils.c:6291:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully

[2015-02-14 06:32:12.159630] I [glusterd-utils.c:6296:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully

[2015-02-14 06:32:12.168889] I [glusterd-utils.c:6301:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully

[2015-02-14 06:32:13.254689] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600

[2015-02-14 06:32:13.254799] W [socket.c:2992:socket_connect] 0-management: Ignore failed connection attempt on , (No such file or directory)

[2015-02-14 06:32:13.257790] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600

[2015-02-14 06:32:13.257908] W [socket.c:2992:socket_connect] 0-management: Ignore failed connection attempt on , (No such file or directory)

[2015-02-14 06:32:13.258031] W [socket.c:611:__socket_rwv] 0-socket.management: writev on 127.0.0.1:1019 failed (Broken pipe)

[2015-02-14 06:32:13.258111] W [socket.c:611:__socket_rwv] 0-socket.management: writev on 127.0.0.1:1021 failed (Broken pipe)

[2015-02-14 06:32:13.258130] W [socket.c:611:__socket_rwv] 0-socket.management: writev on 10.45.16.17:1018 failed (Broken pipe)

[2015-02-14 06:32:13.711948] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=588 max=0 total=0

[2015-02-14 06:32:13.711967] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=124 max=0 total=0

[2015-02-14 06:32:13.712008] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=588 max=0 total=0

[2015-02-14 06:32:13.712021] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=124 max=0 total=0

[2015-02-14 06:32:13.731311] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=588 max=0 total=0

[2015-02-14 06:32:13.731326] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=124 max=0 total=0

[2015-02-14 06:32:13.731356] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /exp/br02/brick2 on port 49153

[2015-02-14 06:32:13.823129] I [socket.c:2344:socket_event_handler] 0-transport: disconnecting now

[2015-02-14 06:32:13.840668] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/7565ec897c6454bd3e2f4800250a7221.socket failed (Invalid argument)

[2015-02-14 06:32:13.840693] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd.

[2015-02-14 06:32:13.840712] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/ac4c043d3c6a2e5159c86e8c75c51829.socket failed (Invalid argument)

[2015-02-14 06:32:13.840728] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: glustershd has disconnected from glusterd.

[2015-02-14 06:32:14.729667] E [glusterd-rpc-ops.c:1169:__glusterd_commit_op_cbk] 0-management: Received commit RJT from uuid: 294aa603-ec24-44b9-864b-0fe743faa8d9

[2015-02-14 06:32:14.743623] E [glusterd-rpc-ops.c:1169:__glusterd_commit_op_cbk] 0-management: Received commit RJT from uuid: 92aabaf4-4b6c-48da-82b6-c465aff2ec6d

[2015-02-14 06:32:18.762975] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-02-14 06:32:18.764552] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1

[2015-02-14 06:32:18.769051] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not match. Not aggregating tasks status.

[2015-02-14 06:32:18.769070] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick

[2015-02-14 06:32:18.771095] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not match. Not aggregating tasks status.

[2015-02-14 06:32:18.771108] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick

[2015-02-14 06:32:48.570796] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-02-14 06:32:48.572352] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1

[2015-02-14 06:32:48.576899] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not match. Not aggregating tasks status.

[2015-02-14 06:32:48.576918] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick

[2015-02-14 06:32:48.578982] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not match. Not aggregating tasks status.

[2015-02-14 06:32:48.579001] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick

[2015-02-14 06:36:57.840738] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-02-14 06:36:57.842370] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1

[2015-02-14 06:36:57.846919] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not match. Not aggregating tasks status.

[2015-02-14 06:36:57.846941] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick

[2015-02-14 06:36:57.849026] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not match. Not aggregating tasks status.

[2015-02-14 06:36:57.849046] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick

[2015-02-14 06:37:20.208081] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed

[2015-02-14 06:37:20.211279] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1

[2015-02-14 06:37:20.215792] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not match. Not aggregating tasks status.

[2015-02-14 06:37:20.215809] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick

[2015-02-14 06:37:20.216295] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not match. Not aggregating tasks status.

[2015-02-14 06:37:20.216308] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick



_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux