Re: healing never ends (or never starts?) on replicated volume with virtual block device

[Roman <romeo.r@xxxxxxxxx>]

[2014-11-06 12:21:01.575385] I [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-HA-WIN-TT-1T-replicate-0:  foreground data self heal  is successfully completed,  data self heal from HA-WIN-TT-1T-client-1  to sinks  HA-WIN-TT-1T-client-0, with 966367641600 bytes on HA-WIN-TT-1T-client-0, 966367641600 bytes on HA-WIN-TT-1T-client-1,  data - Pending matrix:  [ [ 1 1 ] [ 14192 0 ] ]  on <gfid:a2d3dc2d-6572-4e14-b1bf-4f1027433117>

[2014-11-06 12:21:01.576454] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-HA-WIN-TT-1T-client-1: remote operation failed: No such file or directory. Path: <gfid:b8026077-696e-45a8-b5a5-9c55e3813d38> (b8026077-696e-45a8-b5a5-9c55e3813d38)

[2014-11-06 12:21:01.576517] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-HA-WIN-TT-1T-client-0: remote operation failed: No such file or directory. Path: <gfid:b8026077-696e-45a8-b5a5-9c55e3813d38> (b8026077-696e-45a8-b5a5-9c55e3813d38)

[2014-11-06 12:21:01.577284] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-HA-WIN-TT-1T-client-1: remote operation failed: No such file or directory. Path: <gfid:2f1ad6a8-11aa-483c-a561-75ce45e5f245> (2f1ad6a8-11aa-483c-a561-75ce45e5f245)

[2014-11-06 12:21:01.577349] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-HA-WIN-TT-1T-client-0: remote operation failed: No such file or directory. Path: <gfid:2f1ad6a8-11aa-483c-a561-75ce45e5f245> (2f1ad6a8-11aa-483c-a561-75ce45e5f245)

[2014-11-06 12:21:01.578105] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-HA-WIN-TT-1T-client-1: remote operation failed: No such file or directory. Path: <gfid:66b3e4de-aea7-4fa2-9c0d-93fc1846dc35> (66b3e4de-aea7-4fa2-9c0d-93fc1846dc35)

[2014-11-06 12:21:01.578170] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-HA-WIN-TT-1T-client-0: remote operation failed: No such file or directory. Path: <gfid:66b3e4de-aea7-4fa2-9c0d-93fc1846dc35> (66b3e4de-aea7-4fa2-9c0d-93fc1846dc35)

[2014-11-06 12:21:01.902786] I [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-HA-WIN-TT-1T-replicate-0:  metadata self heal  is successfully completed, foreground data self heal  is successfully completed,  data self heal from HA-WIN-TT-1T-client-1  to sinks  HA-WIN-TT-1T-client-0, with 0 bytes on HA-WIN-TT-1T-client-0, 4 bytes on HA-WIN-TT-1T-client-1,  data - Pending matrix:  [ [ 0 0 ] [ 3 0 ] ]  metadata self heal from source HA-WIN-TT-1T-client-1 to HA-WIN-TT-1T-client-0,  metadata - Pending matrix:  [ [ 0 0 ] [ 3 0 ] ], on <gfid:c7b624e4-1c66-4de7-aa89-95460ff098aa>

ehm, just found this. what could it mean? any way to gfid to file transformation?

2014-11-06 14:40 GMT+02:00 Roman <romeo.r@xxxxxxxxx>:

oh, never mind. it is synced now. took a LOT of time :)

2014-11-06 13:12 GMT+02:00 Roman <romeo.r@xxxxxxxxx>:

Hi,

another stupid/interesting situation:

root@stor1:~# gluster volume heal HA-WIN-TT-1T info

Brick stor1:/exports/NFS-WIN/1T/

/disk - Possibly undergoing heal

Number of entries: 1

Brick stor2:/exports/NFS-WIN/1T/

/test

/disk - Possibly undergoing heal

Number of entries: 2

due to testings I've brought down stor1 port on the switch and the made it up again.

then one of the volumes successfully  restored and healed (with virtual machines)

while other still (about 2 hours atm) says, there is a healing process, meanwhile there is no traffic between the servers and client/server.

the /test is simple new file, i've made while stor1 was down.

the /disk is a simple virtual block-device made of /dev/null which is 900GB and is mounted on windows server via iscsitarget :). and it seem it wont stop healing forever, as it can't decide which file is right?

gluster client machine, where is volume for iscsi target is monted logs:

[2014-11-06 08:19:36.949092] W [client-rpc-fops.c:1812:client3_3_fxattrop_cbk] 0-HA-WIN-TT-1T-client-0: remote operation failed: Transport endpoint is not connected

[2014-11-06 08:19:36.949148] W [client-rpc-fops.c:1812:client3_3_fxattrop_cbk] 0-HA-WIN-TT-1T-client-0: remote operation failed: Transport endpoint is not connected

[2014-11-06 08:19:36.951202] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-HA-WIN-TT-1T-client-0: remote operation failed: Transport endpoint is not connected

[2014-11-06 08:19:57.682937] W [socket.c:522:__socket_rwv] 0-glusterfs: readv on 10.250.0.1:24007 failed (Connection timed out)

[2014-11-06 08:20:17.950981] E [socket.c:2161:socket_connect_finish] 0-glusterfs: connection to 10.250.0.1:24007 failed (No route to host)

[2014-11-06 08:20:40.062928] E [socket.c:2161:socket_connect_finish] 0-HA-WIN-TT-1T-client-0: connection to 10.250.0.1:24007 failed (Connection timed out)

[2014-11-06 08:30:15.638197] W [dht-diskusage.c:232:dht_is_subvol_filled] 0-HA-WIN-TT-1T-dht: disk space on subvolume 'HA-WIN-TT-1T-replicate-0' is getting full (95.00 %), consider adding more nodes

[2014-11-06 08:36:18.385659] I [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing

[2014-11-06 08:36:18.386573] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-WIN-TT-1T-client-0: changing port to 49160 (from 0)

[2014-11-06 08:36:18.387182] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-WIN-TT-1T-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2014-11-06 08:36:18.387414] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-0: Connected to 10.250.0.1:49160, attached to remote volume '/exports/NFS-WIN/1T'.

[2014-11-06 08:36:18.387433] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2014-11-06 08:36:18.387446] I [client-handshake.c:1314:client_post_handshake] 0-HA-WIN-TT-1T-client-0: 1 fds open - Delaying child_up until they are re-opened

[2014-11-06 08:36:18.387730] I [client-handshake.c:936:client_child_up_reopen_done] 0-HA-WIN-TT-1T-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP

[2014-11-06 08:36:18.387862] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-WIN-TT-1T-client-0: Server lk version = 1

brick log on stor1:

[2014-11-06 08:38:04.269503] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-WIN-TT-1T-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2014-11-06 08:38:04.269908] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-1: Connected to 10.250.0.2:49160, attached to remote volume '/exports/NFS-WIN/1T'.

[2014-11-06 08:38:04.269962] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2014-11-06 08:38:04.270560] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-WIN-TT-1T-client-1: Server lk version = 1

[2014-11-06 08:39:33.277219] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

[2014-11-06 08:49:33.327786] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

[2014-11-06 08:59:33.375835] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

[2014-11-06 09:09:33.430726] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

[2014-11-06 09:19:33.486488] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

[2014-11-06 09:29:33.541596] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

[2014-11-06 09:39:33.595242] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

[2014-11-06 09:49:33.648526] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

[2014-11-06 09:59:33.702368] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

[2014-11-06 10:09:33.756633] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

[2014-11-06 10:19:33.810984] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

[2014-11-06 10:29:33.865172] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

[2014-11-06 10:39:33.918765] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

[2014-11-06 10:49:33.973283] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

[2014-11-06 10:59:34.028836] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0

same on stor2

--

Best regards,
Roman.

--

Best regards,
Roman.

--

Best regards,
Roman.

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users