Re: Using the host name of the volume, its related commands can become very slow

[Atin Mukherjee <amukherj@xxxxxxxxxx>]

On Mon, Jan 15, 2018 at 6:30 PM, 陈曦 <chenxi@xxxxxxxxxx> wrote:

Using the host name of the volume, its related gluster commands can become very slow .For example,create,start,stop volume,nfs related commands. and some time And in some cases, the command will return Error : Request timed out

but If using ip address to create the volume. The volume all gluster commands  are normal.

I have  configured /etc/hosts correctly,Because,SSH can normally use the hostname to access another node from one node. And,I can ping other nodes use hostanme。

this is some logs:

[root@f08n25glusterfs]# tail -n 50 data-gluster-test.log

[2018-02-03 13:53:22.777184] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20 (args: /usr/sbin/glusterfs --volfile-server=localhost --volfile-id=test /data/gluster/test)

[2018-02-03 13:53:22.810249] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1

[2018-02-03 13:53:22.811289] E [glusterfsd-mgmt.c:1590:mgmt_getspec_cbk] 0-glusterfs: failed to get the 'volume file' from server

[2018-02-03 13:53:22.811323] E [glusterfsd-mgmt.c:1690:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:test)

[2018-02-03 13:53:22.811847] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa0) [0xffffad0cff84] -->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x3f8) [0xaaaad6e1f734] -->/usr/sbin/glusterfs(cleanup_and_exit+0x78) [0xaaaad6e199fc] ) 0-: received signum (0), shutting down

[2018-02-03 13:53:22.811892] I [fuse-bridge.c:5720:fini] 0-fuse: Unmounting '/data/gluster/test'.

[2018-02-03 13:53:22.914173] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7bb0) [0xffffaceb7bb0] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0x11c) [0xaaaad6e19bc0] -->/usr/sbin/glusterfs(cleanup_and_exit+0x78) [0xaaaad6e199fc] ) 0-: received signum (15), shutting down

[root@f08n25glusterfs]# tail -n 50 etc-glusterfs-glusterd.vol.log

[2018-02-03 13:55:09.106663] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-4845-b8ba-a2ed01c04b41/W_W on port 49195

[2018-02-03 13:55:09.108871] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-4845-b8ba-a2ed01c04b41/QAQ on port 49183

[2018-02-03 13:55:09.111075] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-4845-b8ba-a2ed01c04b41/v_v on port 49171

[2018-02-03 13:55:09.113281] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-40ae-9732-9902bd2be614/QAQ on port 49184

[2018-02-03 13:55:09.205465] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-4b4b-b5d1-53e5652f1a8f/OTZ on port 49210

[2018-02-03 13:55:09.207655] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/c26580f1-e879-4225-8b3b-e3a3738130a0/A_s on port 49219

[2018-02-03 13:55:09.209824] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-40ae-9732-9902bd2be614/v_v on port 49172

[2018-02-03 13:55:09.212030] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-4960-a9b9-389a98424bce/W_W on port 49188

[2018-02-03 13:55:09.214255] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-40ae-9732-9902bd2be614/W_W on port 49196

[2018-02-03 13:55:09.216464] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-4d94-865c-99ad16953403/QAQ on port 49185

[2018-02-03 13:55:09.218656] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/37313921-a9b2-41f1-b2ff-493485c2b449/OTZ on port 49203

[2018-02-03 13:55:09.220827] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-4960-a9b9-389a98424bce/Q on port 49152

[2018-02-03 13:55:09.223005] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-4845-b8ba-a2ed01c04b41/Q on port 49159

[2018-02-03 13:55:09.225209] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-4d94-865c-99ad16953403/v_v on port 49173

[2018-02-03 13:55:09.227452] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-4b4b-b5d1-53e5652f1a8f/QAQ on port 49186

[2018-02-03 13:55:09.229654] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/2a5f45bd-28a8-44a1-bb0b-594294b9c5c4/W_W on port 49189

[2018-02-03 13:55:09.231855] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-4d94-865c-99ad16953403/W_W on port 49197

[2018-02-03 13:55:09.234073] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-4b4b-b5d1-53e5652f1a8f/v_v on port 49174

[2018-02-03 13:55:09.236285] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/f7b86039-c3e9-425f-8d9e-d063b380058e/OTZ on port 49211

[2018-02-03 13:55:09.238463] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-4960-a9b9-389a98424bce/A_s on port 49212

[2018-02-03 13:55:09.240933] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/2a5f45bd-28a8-44a1-bb0b-594294b9c5c4/Q on port 49153

[2018-02-03 13:55:09.243114] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-4845-b8ba-a2ed01c04b41/A_s on port 49220

[2018-02-03 13:55:09.245295] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/55e4259d-8fc9-456b-861f-617be3683cd0/OTZ on port 49204

[2018-02-03 13:55:09.247480] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/f7b86039-c3e9-425f-8d9e-d063b380058e/QAQ on port 49187

[2018-02-03 13:55:09.249670] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-40ae-9732-9902bd2be614/Q on port 49160

[2018-02-03 13:55:09.251854] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/f7b86039-c3e9-425f-8d9e-d063b380058e/v_v on port 49175

[2018-02-03 13:55:09.254376] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/2f74a970-372c-4fda-95e5-5d0de352966d/W_W on port 49190

[2018-02-03 13:55:09.256604] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/2a5f45bd-28a8-44a1-bb0b-594294b9c5c4/A_s on port 49213

[2018-02-03 13:55:09.258780] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-4b4b-b5d1-53e5652f1a8f/W_W on port 49198

[2018-02-03 13:55:09.260979] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/a2a12678-7a4e-4e0a-8e4e-a96dedc0217d/OTZ on port 49205

[2018-02-03 13:55:09.263155] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/2f74a970-372c-4fda-95e5-5d0de352966d/Q on port 49154

[2018-02-03 13:55:09.265360] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-4d94-865c-99ad16953403/Q on port 49161

[2018-02-03 13:55:09.267556] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-40ae-9732-9902bd2be614/A_s on port 49221

[2018-02-03 13:55:09.269730] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/37313921-a9b2-41f1-b2ff-493485c2b449/W_W on port 49191

[2018-02-03 13:55:09.271933] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-4960-a9b9-389a98424bce/QAQ on port 49176

[2018-02-03 13:55:09.274159] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/f7b86039-c3e9-425f-8d9e-d063b380058e/W_W on port 49199

[2018-02-03 13:55:09.276368] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-4960-a9b9-389a98424bce/v_v on port 49164

[2018-02-03 13:55:09.278578] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-4d94-865c-99ad16953403/A_s on port 49222

[2018-02-03 13:55:09.280747] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/2f74a970-372c-4fda-95e5-5d0de352966d/A_s on port 49215

[2018-02-03 13:55:09.282913] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/37313921-a9b2-41f1-b2ff-493485c2b449/Q on port 49155

[2018-02-03 13:55:09.285117] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/c26580f1-e879-4225-8b3b-e3a3738130a0/OTZ on port 49206

[2018-02-03 13:55:09.315499] I [MSGID: 106490] [glusterd-handler.c:2600:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: d8ee609f-d47e-46e4-96ac-ef78cddf45b8

[2018-02-03 13:55:09.325999] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to f08n29 (0), ret: 0, op_ret: 0

[2018-02-03 13:55:09.331799] I [MSGID: 106492] [glusterd-handler.c:2776:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d8ee609f-d47e-46e4-96ac-ef78cddf45b8

[2018-02-03 13:55:09.331876] I [MSGID: 106502] [glusterd-handler.c:2821:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend

[2018-02-03 13:55:09.333350] I [MSGID: 106493] [glusterd-rpc-ops.c:696:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: d8ee609f-d47e-46e4-96ac-ef78cddf45b8

[2018-02-03 13:55:38.833207] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2018-02-03 13:57:09.161466] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2018-02-03 13:58:39.566130] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

[2018-02-03 14:00:09.918411] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

The snippet of the glusterd log file what you pointed is during the handshaking. What we'd need to check the logs for a particular transaction. Would you be able to share the glusterd log files from all the nodes along with the cmd_history.log from the node where the commands were initiated?

[root@f08n25glusterfs]# tail -n 50 glustershd.log

2478:     option remote-host 10.33.0.8

2479:     option remote-subvolume /data/gluster/ec025105-5ef2-46f2-b1cf-01a05daa237e/QAQ

2480:     option transport-type socket

2481:     option username 2a31c3ba-696d-4d49-9dee-540e47791e4a

2482:     option password 3f751501-3148-42bf-8c55-bfe1b3ddc747

2483: end-volume

2484: 

2485: volume QAQ-replicate-11

2486:     type cluster/replicate

2487:     option node-uuid 194ca8ea-df5b-4d5e-9af6-dbf36c485334

2488:     option background-self-heal-count 0

2489:     option metadata-self-heal on

2490:     option data-self-heal on

2491:     option entry-self-heal on

2492:     option self-heal-daemon enable

2493:     option iam-self-heal-daemon yes

2494:     subvolumes QAQ-client-33 QAQ-client-34 QAQ-client-35

2495: end-volume

2496: 

2497: volume glustershd

2498:     type debug/io-stats

2499:     option log-level INFO

2500:     subvolumes A_s-replicate-0 A_s-replicate-1 A_s-replicate-2 A_s-replicate-3 A_s-replicate-4 A_s-replicate-5 A_s-replicate-6 A_s-replicate-7 A_s-replicate-8 A_s-replicate-9 A_s-replicate-10 A_s-replicate-11 OTZ-disperse-0 OTZ-disperse-1 OTZ-disperse-2 OTZ-disperse-3 OTZ-disperse-4 OTZ-disperse-5 OTZ-disperse-6 OTZ-disperse-7 OTZ-disperse-8 OTZ-disperse-9 OTZ-disperse-10 OTZ-disperse-11 Q-disperse-0 Q-disperse-1 Q-disperse-2 Q-disperse-3 Q-disperse-4 Q-disperse-5 Q-disperse-6 Q-disperse-7 Q-disperse-8 Q-disperse-9 Q-disperse-10 Q-disperse-11 QAQ-replicate-0 QAQ-replicate-1 QAQ-replicate-2 QAQ-replicate-3 QAQ-replicate-4 QAQ-replicate-5 QAQ-replicate-6 QAQ-replicate-7 QAQ-replicate-8 QAQ-replicate-9 QAQ-replicate-10 QAQ-replicate-11

2501: end-volume

2502: 

+------------------------------------------------------------------------------+

[2018-02-03 13:55:20.771993] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-0: Going UP

[2018-02-03 13:55:20.772100] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-1: Going UP

[2018-02-03 13:55:20.772157] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-2: Going UP

[2018-02-03 13:55:20.772208] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-3: Going UP

[2018-02-03 13:55:20.772260] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-4: Going UP

[2018-02-03 13:55:20.772311] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-5: Going UP

[2018-02-03 13:55:20.772364] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-6: Going UP

[2018-02-03 13:55:20.772418] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-7: Going UP

[2018-02-03 13:55:20.772468] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-8: Going UP

[2018-02-03 13:55:20.772521] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-9: Going UP

[2018-02-03 13:55:20.772572] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-10: Going UP

[2018-02-03 13:55:20.772626] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-11: Going UP

[2018-02-03 13:55:20.772679] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-0: Going UP

[2018-02-03 13:55:20.772732] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-1: Going UP

[2018-02-03 13:55:20.772786] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-2: Going UP

[2018-02-03 13:55:20.772838] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-3: Going UP

[2018-02-03 13:55:20.772891] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-4: Going UP

[2018-02-03 13:55:20.772941] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-5: Going UP

[2018-02-03 13:55:20.772994] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-6: Going UP

[2018-02-03 13:55:20.773046] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-7: Going UP

[2018-02-03 13:55:20.773099] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-8: Going UP

[2018-02-03 13:55:20.773150] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-9: Going UP

[2018-02-03 13:55:21.773401] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-10: Going UP

[2018-02-03 13:55:21.773496] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-11: Going UP

[root@f08n25glusterfs]#

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users