Faulty staus in geo-replication session of a sub-volume

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Dear Developers/Users,

A geo-rep session of a sub-volume is in 'faulty' status. Please find the setup and log details below.

Setup Details:
Gluster version - 7.0
Volume configuration - 2x3 (DxR)
gysncd permission(master)  - root
gysncd permission(slave)  - sas (non-root)
glusterd, glusterfsd permissions(master) - root
glusterd, glusterfsd permissions(slave) - root

Log details:
In the master gyncd log, this traceback is printed repeatedly.
[2020-05-22 12:09:43.838727] I [master(worker /home/sas/gluster/data/code-ide):1991:syncjob] Syncer: Sync Time Taken duration=0.4240 num_files=1 job=1 return_code=0
[2020-05-22 12:09:43.944392] E [repce(worker /home/sas/gluster/data/code-ide):214:__call__] RepceClient: call failed call=261471:140535761106752:1590149383.8 method=entry_ops error=OSError
[2020-05-22 12:09:43.944746] E [syncdutils(worker /home/sas/gluster/data/code-ide):338:log_raise_exception] <top>: FAIL:
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in main
    func(args)
  File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in subcmd_worker
    local.service_loop(remote)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1305, in service_loop
    g3.crawlwrap(_oneshot_=True)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 602, in crawlwrap
    self.crawl()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1592, in crawl
    self.changelogs_batch_process(changes)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1492, in changelogs_batch_process
    self.process(batch)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1327, in process
    self.process_change(change, done, retry)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1221, in process_change
    failures = self.slave.server.entry_ops(entries)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__
    return self.ins(self.meth, *a)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__
    raise res
OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
[2020-05-22 12:09:43.968710] I [repce(agent /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF.
[2020-05-22 12:09:44.912470] I [monitor(monitor):280:monitor] Monitor: worker died in startup phase brick=/home/sas/gluster/data/code-ide
[2020-05-22 12:09:44.913692] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty

In salve end, these are printed repeatedly

[2020-05-22 11:23:26.65115] W [gsyncd(slave 10.47.8.153/home/sas/gluster/data/code-ide):307:main] <top>: Session config file not exists, using the default config path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf
[2020-05-22 11:23:26.77414] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1105:connect] GLUSTER: Mounting gluster volume locally...
[2020-05-22 11:23:27.297466] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1128:connect] GLUSTER: Mounted gluster volume duration=1.2199
[2020-05-22 11:23:27.298125] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1155:service_loop] GLUSTER: slave listening
[2020-05-22 11:23:32.654939] E [repce(slave 10.47.8.153/home/sas/gluster/data/code-ide):122:worker] <top>: call failed:
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker
    res = getattr(self.obj, rmeth)(*in_data[2:])
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 706, in entry_ops
    collect_failure(e, cmd_ret, uid, gid)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 444, in collect_failure
    disk_gfid)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, in get_slv_dir_path
    [ENOENT], [ESTALE])
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
    return call(*arg)
OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
[2020-05-22 11:23:32.741317] I [repce(slave 10.47.8.153/home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF.


Additional info:
Parallel to this GFID(/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e) mentioned in master gyscnd log, there are some files with ---------T permission & trusted.glusterfs.dht.linkto extended attribute in the master subvolume for which geo-rep session is in faulty status. Not sure whether this is related to this geo-rep issue or not.

I have attached a few screenshots and log stmts for further info. Please let us know how we should solve this.
Thanks in advance.

Regards,
Naranderan R

sas@10.37.11.246/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide>tail -f gsyncd.log 
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 706, in entry_ops
    collect_failure(e, cmd_ret, uid, gid)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 444, in collect_failure
    disk_gfid)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, in get_slv_dir_path
    [ENOENT], [ESTALE])
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
    return call(*arg)
OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
[2020-05-22 11:23:14.762231] I [repce(slave 10.47.8.153/home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF.

















[2020-05-22 11:23:26.65115] W [gsyncd(slave 10.47.8.153/home/sas/gluster/data/code-ide):307:main] <top>: Session config file not exists, using the default config	path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf
[2020-05-22 11:23:26.77414] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1105:connect] GLUSTER: Mounting gluster volume locally...
[2020-05-22 11:23:27.297466] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1128:connect] GLUSTER: Mounted gluster volume	duration=1.2199
[2020-05-22 11:23:27.298125] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1155:service_loop] GLUSTER: slave listening
[2020-05-22 11:23:32.654939] E [repce(slave 10.47.8.153/home/sas/gluster/data/code-ide):122:worker] <top>: call failed: 
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker
    res = getattr(self.obj, rmeth)(*in_data[2:])
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 706, in entry_ops
    collect_failure(e, cmd_ret, uid, gid)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 444, in collect_failure
    disk_gfid)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, in get_slv_dir_path
    [ENOENT], [ESTALE])
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
    return call(*arg)
OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
[2020-05-22 11:23:32.741317] I [repce(slave 10.47.8.153/home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF.


[2020-05-22 11:23:44.797867] W [gsyncd(slave 10.47.8.153/home/sas/gluster/data/code-ide):307:main] <top>: Session config file not exists, using the default config	path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf

Attachment: Slave-mnt-10.47.8.153-home-sas-gluster-data-code-ide-log-tailf.png
Description: PNG image

root@10.37.11.246/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide#ls
gsyncd.log  mnt-10.47.8.153-home-sas-gluster-data-code-ide.log
root@10.37.11.246/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide#tail -f mnt-10.47.8.153-home-sas-gluster-data-code-ide.log
[2020-05-22 11:31:10.001901] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-4: Connected to code-ide-client-4, attached to remote volume '/home/sas/gluster/data/code-ide'. 
[2020-05-22 11:31:10.001955] I [MSGID: 108002] [afr-common.c:5647:afr_notify] 0-code-ide-replicate-1: Client-quorum is met 
[2020-05-22 11:31:10.002114] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-1: inet_pton failed with return code 0 [Invalid argument]
[2020-05-22 11:31:10.003158] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-5: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) 
[2020-05-22 11:31:10.003215] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f8ec76fb244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f8ed62782f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f8ed6274c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:10.003227] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-5: failed to set process-name in handshake msg 
[2020-05-22 11:31:10.003564] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-5: Connected to code-ide-client-5, attached to remote volume '/home/sas/gluster/data/code-ide'. 
[2020-05-22 11:31:10.004717] I [fuse-bridge.c:5166:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2020-05-22 11:31:10.004746] I [fuse-bridge.c:5777:fuse_graph_sync] 0-fuse: switched to graph 0
[2020-05-22 11:31:10.007121] I [MSGID: 108031] [afr-common.c:2580:afr_local_discovery_cbk] 0-code-ide-replicate-1: selecting local read_child code-ide-client-5 
[2020-05-22 11:31:16.385568] I [fuse-bridge.c:6083:fuse_thread_proc] 0-fuse: initiating unmount of /var/mountbroker-root/user1000/mtpt-sas-IgRBaT
[2020-05-22 11:31:16.385666] W [glusterfsd.c:1596:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f8ed50c2dd5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x56136fdfa625] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x56136fdfa48b] ) 0-: received signum (15), shutting down 
[2020-05-22 11:31:16.385687] I [fuse-bridge.c:6871:fini] 0-fuse: Unmounting '/var/mountbroker-root/user1000/mtpt-sas-IgRBaT'.
[2020-05-22 11:31:16.385698] I [fuse-bridge.c:6876:fini] 0-fuse: Closing fuse connection to '/var/mountbroker-root/user1000/mtpt-sas-IgRBaT'.












[2020-05-22 11:31:27.695486] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 7.0 (args: /usr/sbin/glusterfs --user-map-root=sas --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide/mnt-10.47.8.153-home-sas-gluster-data-code-ide.log --volfile-server=localhost --volfile-id=code-ide --client-pid=-1 /var/mountbroker-root/user1000/mtpt-sas-gXOnl6) 
[2020-05-22 11:31:27.698895] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 230635
[2020-05-22 11:31:27.707687] I [MSGID: 101190] [event-epoll.c:674:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0 
[2020-05-22 11:31:27.707741] I [MSGID: 101190] [event-epoll.c:674:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 
[2020-05-22 11:31:27.714024] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-0: parent translators are ready, attempting connect on transport 
[2020-05-22 11:31:27.716381] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-1: parent translators are ready, attempting connect on transport 
[2020-05-22 11:31:27.716834] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-0: changing port to 49153 (from 0)
[2020-05-22 11:31:27.716863] I [socket.c:864:__socket_shutdown] 0-code-ide-client-0: intentional socket shutdown(12)
[2020-05-22 11:31:27.718763] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-2: parent translators are ready, attempting connect on transport 
[2020-05-22 11:31:27.721166] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-0: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) 
[2020-05-22 11:31:27.721451] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:27.721476] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-0: failed to set process-name in handshake msg 
[2020-05-22 11:31:27.721587] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-1: changing port to 49152 (from 0)
[2020-05-22 11:31:27.721606] I [socket.c:864:__socket_shutdown] 0-code-ide-client-1: intentional socket shutdown(13)
[2020-05-22 11:31:27.723607] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-3: parent translators are ready, attempting connect on transport 
[2020-05-22 11:31:27.725797] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-0: Connected to code-ide-client-0, attached to remote volume '/home/sas/gluster/data/code-ide'. 
[2020-05-22 11:31:27.725826] I [MSGID: 108005] [afr-common.c:5280:__afr_handle_child_up_event] 0-code-ide-replicate-0: Subvolume 'code-ide-client-0' came back up; going online. 
[2020-05-22 11:31:27.725971] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-0: inet_pton failed with return code 0 [Invalid argument]
[2020-05-22 11:31:27.726190] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-1: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) 
[2020-05-22 11:31:27.726263] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:27.726279] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-1: failed to set process-name in handshake msg 
[2020-05-22 11:31:27.726404] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-2: changing port to 49152 (from 0)
[2020-05-22 11:31:27.726419] I [socket.c:864:__socket_shutdown] 0-code-ide-client-2: intentional socket shutdown(12)
[2020-05-22 11:31:27.728149] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-4: parent translators are ready, attempting connect on transport 
[2020-05-22 11:31:27.730264] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-1: Connected to code-ide-client-1, attached to remote volume '/home/sas/gluster/data/code-ide'. 
[2020-05-22 11:31:27.730284] I [MSGID: 108002] [afr-common.c:5647:afr_notify] 0-code-ide-replicate-0: Client-quorum is met 
[2020-05-22 11:31:27.730395] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-0: inet_pton failed with return code 0 [Invalid argument]
[2020-05-22 11:31:27.730634] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-2: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) 
[2020-05-22 11:31:27.730714] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:27.730731] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-2: failed to set process-name in handshake msg 
[2020-05-22 11:31:27.730867] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-3: changing port to 49153 (from 0)
[2020-05-22 11:31:27.730883] I [socket.c:864:__socket_shutdown] 0-code-ide-client-3: intentional socket shutdown(13)
[2020-05-22 11:31:27.732462] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-5: parent translators are ready, attempting connect on transport 
[2020-05-22 11:31:27.734593] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-2: Connected to code-ide-client-2, attached to remote volume '/home/sas/gluster/data/code-ide'. 
[2020-05-22 11:31:27.734935] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-3: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) 
[2020-05-22 11:31:27.735007] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:27.735026] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-3: failed to set process-name in handshake msg 
[2020-05-22 11:31:27.735187] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-4: changing port to 49152 (from 0)
[2020-05-22 11:31:27.735203] I [socket.c:864:__socket_shutdown] 0-code-ide-client-4: intentional socket shutdown(12)
Final graph:
+------------------------------------------------------------------------------+
  1: volume code-ide-client-0
  2:     type protocol/client
  3:     option opversion 70000
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key code-ide
  7:     option client-version 7.0
  8:     option process-uuid CTX_ID:c97ffcc5-c4de-4f4e-a13e-71c83e8ce7ac-GRAPH_ID:0-PID:230635-HOST:10.37.11.246-PC_NAME:code-ide-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host 10.37.11.253
 12:     option remote-subvolume /home/sas/gluster/data/code-ide
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 57f0d708-564c-4762-bcba-39bbbd32a314
 16:     option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1
 17:     option transport.socket.ssl-enabled off
 18:     option transport.tcp-user-timeout 0
 19:     option transport.socket.keepalive-time 20
 20:     option transport.socket.keepalive-interval 2
 21:     option transport.socket.keepalive-count 9
 22:     option send-gids true
 23: end-volume
 24:  
 25: volume code-ide-client-1
 26:     type protocol/client
 27:     option opversion 70000
 28:     option clnt-lk-version 1
 29:     option volfile-checksum 0
 30:     option volfile-key code-ide
 31:     option client-version 7.0
 32:     option process-uuid CTX_ID:c97ffcc5-c4de-4f4e-a13e-71c83e8ce7ac-GRAPH_ID:0-PID:230635-HOST:10.37.11.246-PC_NAME:code-ide-client-1-RECON_NO:-0
 33:     option fops-version 1298437
 34:     option ping-timeout 42
 35:     option remote-host 10.37.11.252
 36:     option remote-subvolume /home/sas/gluster/data/code-ide
 37:     option transport-type socket
 38:     option transport.address-family inet
 39:     option username 57f0d708-564c-4762-bcba-39bbbd32a314
 40:     option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1
 41:     option transport.socket.ssl-enabled off
 42:     option transport.tcp-user-timeout 0
 43:     option transport.socket.keepalive-time 20
 44:     option transport.socket.keepalive-interval 2
 45:     option transport.socket.keepalive-count 9
 46:     option send-gids true
 47: end-volume
 48:  
 49: volume code-ide-client-2
 50:     type protocol/client
 51:     option opversion 70000
 52:     option clnt-lk-version 1
 53:     option volfile-checksum 0
 54:     option volfile-key code-ide
 55:     option client-version 7.0
 56:     option process-uuid CTX_ID:c97ffcc5-c4de-4f4e-a13e-71c83e8ce7ac-GRAPH_ID:0-PID:230635-HOST:10.37.11.246-PC_NAME:code-ide-client-2-RECON_NO:-0
 57:     option fops-version 1298437
 58:     option ping-timeout 42
 59:     option remote-host 10.37.11.251
 60:     option remote-subvolume /home/sas/gluster/data/code-ide
 61:     option transport-type socket
 62:     option transport.address-family inet
 63:     option username 57f0d708-564c-4762-bcba-39bbbd32a314
 64:     option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1
 65:     option transport.socket.ssl-enabled off
 66:     option transport.tcp-user-timeout 0
 67:     option transport.socket.keepalive-time 20
 68:     option transport.socket.keepalive-interval 2
 69:     option transport.socket.keepalive-count 9
 70:     option send-gids true
 71: end-volume
 72:  
 73: volume code-ide-replicate-0
 74:     type cluster/replicate
 75:     option afr-pending-xattr code-ide-client-0,code-ide-client-1,code-ide-client-2
 76:     option use-compound-fops off
 77:     subvolumes code-ide-client-0 code-ide-client-1 code-ide-client-2
 78: end-volume
 79:  
 80: volume code-ide-client-3
 81:     type protocol/client
 82:     option opversion 70000
 83:     option clnt-lk-version 1
 84:     option volfile-checksum 0
 85:     option volfile-key code-ide
 86:     option client-version 7.0
 87:     option process-uuid CTX_ID:c97ffcc5-c4de-4f4e-a13e-71c83e8ce7ac-GRAPH_ID:0-PID:230635-HOST:10.37.11.246-PC_NAME:code-ide-client-3-RECON_NO:-0
 88:     option fops-version 1298437
 89:     option ping-timeout 42
 90:     option remote-host 10.37.11.250
 91:     option remote-subvolume /home/sas/gluster/data/code-ide
 92:     option transport-type socket
 93:     option transport.address-family inet
 94:     option username 57f0d708-564c-4762-bcba-39bbbd32a314
 95:     option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1
 96:     option transport.socket.ssl-enabled off
 97:     option transport.tcp-user-timeout 0
 98:     option transport.socket.keepalive-time 20
 99:     option transport.socket.keepalive-interval 2
100:     option transport.socket.keepalive-count 9
101:     option send-gids true
102: end-volume
103:  
104: volume code-ide-client-4
105:     type protocol/client
106:     option ping-timeout 42
107:     option remote-host 10.37.11.249
108:     option remote-subvolume /home/sas/gluster/data/code-ide
109:     option transport-type socket
110:     option transport.address-family inet
111:     option username 57f0d708-564c-4762-bcba-39bbbd32a314
112:     option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1
113:     option transport.socket.ssl-enabled off
114:     option transport.tcp-user-timeout 0
115:     option transport.socket.keepalive-time 20
116:     option transport.socket.keepalive-interval 2
117:     option transport.socket.keepalive-count 9
118:     option send-gids true
119: end-volume
120:  
121: volume code-ide-client-5
122:     type protocol/client
123:     option ping-timeout 42
124:     option remote-host 10.37.11.246
125:     option remote-subvolume /home/sas/gluster/data/code-ide
126:     option transport-type socket
127:     option transport.address-family inet
128:     option username 57f0d708-564c-4762-bcba-39bbbd32a314
129:     option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1
130:     option transport.socket.ssl-enabled off
131:     option transport.tcp-user-timeout 0
132:     option transport.socket.keepalive-time 20
133:     option transport.socket.keepalive-interval 2
134:     option transport.socket.keepalive-count 9
135:     option send-gids true
136: end-volume
137:  
138: volume code-ide-replicate-1
139:     type cluster/replicate
140:     option afr-pending-xattr code-ide-client-3,code-ide-client-4,code-ide-client-5
141:     option use-compound-fops off
142:     subvolumes code-ide-client-3 code-ide-client-4 code-ide-client-5
143: end-volume
144:  
145: volume code-ide-dht
146:     type cluster/distribute
147:     option lock-migration off
148:     option force-migration off
149:     subvolumes code-ide-replicate-0 code-ide-replicate-1
150: end-volume
151:  
152: volume code-ide-utime
153:     type features/utime
154:     option noatime on
155:     subvolumes code-ide-dht
156: end-volume
157:  
158: volume code-ide-write-behind
159:     type performance/write-behind
160:     subvolumes code-ide-utime
161: end-volume
162:  
163: volume code-ide-read-ahead
164:     type performance/read-ahead
165:     subvolumes code-ide-write-behind
166: end-volume
167:  
168: volume code-ide-readdir-ahead
169:     type performance/readdir-ahead
170:     option parallel-readdir off
171:     option rda-request-size 131072
172:     option rda-cache-limit 10MB
173:     subvolumes code-ide-read-ahead
174: end-volume
175:  
176: volume code-ide-io-cache
177:     type performance/io-cache
178:     subvolumes code-ide-readdir-ahead
179: end-volume
180:  
181: volume code-ide-open-behind
182:     type performance/open-behind
183:     subvolumes code-ide-io-cache
184: end-volume
185:  
186: volume code-ide-quick-read
187:     type performance/quick-read
188:     subvolumes code-ide-open-behind
189: end-volume
190:  
191: volume code-ide-md-cache
192:     type performance/md-cache
193:     option cache-posix-acl true
194:     subvolumes code-ide-quick-read
195: end-volume
196:  
197: volume code-ide
198:     type debug/io-stats
199:     option log-level INFO
200:     option threads 16
201:     option latency-measurement off
202:     option count-fop-hits off
203:     option global-threading off
204:     subvolumes code-ide-md-cache
205: end-volume
206:  
207: volume posix-acl-autoload
208:     type system/posix-acl
209:     subvolumes code-ide
210: end-volume
211:  
212: volume gfid-access-autoload
213:     type features/gfid-access
214:     subvolumes posix-acl-autoload
215: end-volume
216:  
217: volume meta-autoload
218:     type meta
219:     subvolumes gfid-access-autoload
220: end-volume
221:  
+------------------------------------------------------------------------------+
[2020-05-22 11:31:27.737910] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-3: Connected to code-ide-client-3, attached to remote volume '/home/sas/gluster/data/code-ide'. 
[2020-05-22 11:31:27.737926] I [MSGID: 108005] [afr-common.c:5280:__afr_handle_child_up_event] 0-code-ide-replicate-1: Subvolume 'code-ide-client-3' came back up; going online. 
[2020-05-22 11:31:27.738026] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-1: inet_pton failed with return code 0 [Invalid argument]
[2020-05-22 11:31:27.738195] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-5: changing port to 49152 (from 0)
[2020-05-22 11:31:27.738212] I [socket.c:864:__socket_shutdown] 0-code-ide-client-5: intentional socket shutdown(13)
[2020-05-22 11:31:27.739521] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-4: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) 
[2020-05-22 11:31:27.739592] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:27.739608] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-4: failed to set process-name in handshake msg 
[2020-05-22 11:31:27.740090] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-4: Connected to code-ide-client-4, attached to remote volume '/home/sas/gluster/data/code-ide'. 
[2020-05-22 11:31:27.740105] I [MSGID: 108002] [afr-common.c:5647:afr_notify] 0-code-ide-replicate-1: Client-quorum is met 
[2020-05-22 11:31:27.740222] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-1: inet_pton failed with return code 0 [Invalid argument]
[2020-05-22 11:31:27.741492] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-5: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) 
[2020-05-22 11:31:27.741547] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:27.741560] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-5: failed to set process-name in handshake msg 
[2020-05-22 11:31:27.741931] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-5: Connected to code-ide-client-5, attached to remote volume '/home/sas/gluster/data/code-ide'. 
[2020-05-22 11:31:27.743285] I [fuse-bridge.c:5166:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2020-05-22 11:31:27.743338] I [fuse-bridge.c:5777:fuse_graph_sync] 0-fuse: switched to graph 0
[2020-05-22 11:31:27.746010] I [MSGID: 108031] [afr-common.c:2580:afr_local_discovery_cbk] 0-code-ide-replicate-1: selecting local read_child code-ide-client-5 
[2020-05-22 11:31:34.214315] I [fuse-bridge.c:6083:fuse_thread_proc] 0-fuse: initiating unmount of /var/mountbroker-root/user1000/mtpt-sas-gXOnl6
[2020-05-22 11:31:34.214450] W [glusterfsd.c:1596:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f9309abfdd5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55dcea04e625] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55dcea04e48b] ) 0-: received signum (15), shutting down 
[2020-05-22 11:31:34.214485] I [fuse-bridge.c:6871:fini] 0-fuse: Unmounting '/var/mountbroker-root/user1000/mtpt-sas-gXOnl6'.
[2020-05-22 11:31:34.214499] I [fuse-bridge.c:6876:fini] 0-fuse: Closing fuse connection to '/var/mountbroker-root/user1000/mtpt-sas-gXOnl6'.
[2020-05-22 11:31:46.446624] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 7.0 (args: /usr/sbin/glusterfs --user-map-root=sas --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide/mnt-10.47.8.153-home-sas-gluster-data-code-ide.log --volfile-server=localhost --volfile-id=code-ide --client-pid=-1 /var/mountbroker-root/user1000/mtpt-sas-oq0aUM) 
[2020-05-22 11:31:46.450189] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 230980
[2020-05-22 11:31:46.458835] I [MSGID: 101190] [event-epoll.c:674:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 
[2020-05-22 11:31:46.458826] I [MSGID: 101190] [event-epoll.c:674:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0 
[2020-05-22 11:31:46.465054] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-0: parent translators are ready, attempting connect on transport 
[2020-05-22 11:31:46.467356] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-1: parent translators are ready, attempting connect on transport 
[2020-05-22 11:31:46.467849] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-0: changing port to 49153 (from 0)
[2020-05-22 11:31:46.467872] I [socket.c:864:__socket_shutdown] 0-code-ide-client-0: intentional socket shutdown(12)
[2020-05-22 11:31:46.469623] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-2: parent translators are ready, attempting connect on transport 
[2020-05-22 11:31:46.472059] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-1: changing port to 49152 (from 0)
[2020-05-22 11:31:46.472078] I [socket.c:864:__socket_shutdown] 0-code-ide-client-1: intentional socket shutdown(13)
[2020-05-22 11:31:46.473883] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-3: parent translators are ready, attempting connect on transport 
[2020-05-22 11:31:46.475942] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-0: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) 
[2020-05-22 11:31:46.476169] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f948df1a244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f949ca972f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f949ca93c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:46.476186] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-0: failed to set process-name in handshake msg 
[2020-05-22 11:31:46.476472] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-1: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) 
[2020-05-22 11:31:46.476555] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f948df1a244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f949ca972f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f949ca93c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:46.476574] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-1: failed to set process-name in handshake msg 
[2020-05-22 11:31:46.476837] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-0: Connected to code-ide-client-0, attached to remote volume '/home/sas/gluster/data/code-ide'. 
[2020-05-22 11:31:46.476858] I [MSGID: 108005] [afr-common.c:5280:__afr_handle_child_up_event] 0-code-ide-replicate-0: Subvolume 'code-ide-client-0' came back up; going online. 
[2020-05-22 11:31:46.476979] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-0: inet_pton failed with return code 0 [Invalid argument]
[2020-05-22 11:31:46.477009] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-2: changing port to 49152 (from 0)
[2020-05-22 11:31:46.477025] I [socket.c:864:__socket_shutdown] 0-code-ide-client-2: intentional socket shutdown(12)
[2020-05-22 11:31:46.478210] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-4: parent translators are ready, attempting connect on transport 
[2020-05-22 11:31:46.480275] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-1: Connected to code-ide-client-1, attached to remote volume '/home/sas/gluster/data/code-ide'. 
[2020-05-22 11:31:46.480294] I [MSGID: 108002] [afr-common.c:5647:afr_notify] 0-code-ide-replicate-0: Client-quorum is met 
[2020-05-22 11:31:46.480406] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-0: inet_pton failed with return code 0 [Invalid argument]
[2020-05-22 11:31:46.480644] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-2: Using Program GlusterFS 4.x v1, Num (1298437), Version (400) 
[2020-05-22 11:31:46.480719] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f948df1a244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f949ca972f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f949ca93c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:46.480744] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-2: failed to set process-name in handshake msg 
[2020-05-22 11:31:46.480870] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-3: changing port to 49153 (from 0)
[2020-05-22 11:31:46.480887] I [socket.c:864:__socket_shutdown] 0-code-ide-client-3: intentional socket shutdown(13)
[2020-05-22 11:31:46.482509] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-5: parent translators are ready, attempting connect on transport 
[2020-05-22 11:31:46.484519] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-2: Connected to code-ide-client-2, attached to remote volume '/home/sas/gluster/data/code-ide'. 
sas@10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>date
Fri May 22 05:09:23 PDT 2020
sas@10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>
sas@10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>tail -f gsyncd.log
OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
[2020-05-22 12:09:25.176702] I [repce(agent /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF.
[2020-05-22 12:09:26.144371] I [monitor(monitor):280:monitor] Monitor: worker died in startup phase	brick=/home/sas/gluster/data/code-ide
[2020-05-22 12:09:26.145390] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change	status=Faulty
[2020-05-22 12:09:36.496685] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change	status=Initializing...
[2020-05-22 12:09:36.496835] I [monitor(monitor):159:monitor] Monitor: starting gsyncd worker	brick=/home/sas/gluster/data/code-ide	slave_node=10.37.11.246
[2020-05-22 12:09:36.541825] I [gsyncd(agent /home/sas/gluster/data/code-ide):311:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf
[2020-05-22 12:09:36.545790] I [changelogagent(agent /home/sas/gluster/data/code-ide):72:__init__] ChangelogAgent: Agent listining...
[2020-05-22 12:09:36.554223] I [gsyncd(worker /home/sas/gluster/data/code-ide):311:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf
[2020-05-22 12:09:36.570023] I [resource(worker /home/sas/gluster/data/code-ide):1394:connect_remote] SSH: Initializing SSH connection between master and slave...
[2020-05-22 12:09:38.784018] I [resource(worker /home/sas/gluster/data/code-ide):1443:connect_remote] SSH: SSH connection between master and slave established.	duration=2.2138
[2020-05-22 12:09:38.784291] I [resource(worker /home/sas/gluster/data/code-ide):1113:connect] GLUSTER: Mounting gluster volume locally...

[2020-05-22 12:09:39.906485] I [resource(worker /home/sas/gluster/data/code-ide):1136:connect] GLUSTER: Mounted gluster volume	duration=1.1220
[2020-05-22 12:09:39.906748] I [subcmds(worker /home/sas/gluster/data/code-ide):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor




[2020-05-22 12:09:42.84068] I [master(worker /home/sas/gluster/data/code-ide):1640:register] _GMaster: Working dir	path=/home/sas/gluster/internals/misc/gluster/gsyncd/code-ide_10.37.11.252_code-ide/home-sas-gluster-data-code-ide
[2020-05-22 12:09:42.84353] I [resource(worker /home/sas/gluster/data/code-ide):1299:service_loop] GLUSTER: Register time	time=1590149382
[2020-05-22 12:09:42.88851] I [gsyncdstatus(worker /home/sas/gluster/data/code-ide):281:set_active] GeorepStatus: Worker Status Change	status=Active
[2020-05-22 12:09:42.149405] I [gsyncdstatus(worker /home/sas/gluster/data/code-ide):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change	status=History Crawl
[2020-05-22 12:09:42.149740] I [master(worker /home/sas/gluster/data/code-ide):1554:crawl] _GMaster: starting history crawl	turns=1	stime=(1588918750, 0)	entry_stime=(1588918779, 0)	etime=1590149382
[2020-05-22 12:09:43.163168] I [master(worker /home/sas/gluster/data/code-ide):1583:crawl] _GMaster: slave's time	stime=(1588918750, 0)




[2020-05-22 12:09:43.838727] I [master(worker /home/sas/gluster/data/code-ide):1991:syncjob] Syncer: Sync Time Taken	duration=0.4240	num_files=1	job=1	return_code=0
[2020-05-22 12:09:43.944392] E [repce(worker /home/sas/gluster/data/code-ide):214:__call__] RepceClient: call failed	call=261471:140535761106752:1590149383.8	method=entry_ops	error=OSError
[2020-05-22 12:09:43.944746] E [syncdutils(worker /home/sas/gluster/data/code-ide):338:log_raise_exception] <top>: FAIL: 
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in main
    func(args)
  File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in subcmd_worker
    local.service_loop(remote)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1305, in service_loop
    g3.crawlwrap(oneshot=True)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 602, in crawlwrap
    self.crawl()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1592, in crawl
    self.changelogs_batch_process(changes)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1492, in changelogs_batch_process
    self.process(batch)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1327, in process
    self.process_change(change, done, retry)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1221, in process_change
    failures = self.slave.server.entry_ops(entries)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__
    return self.ins(self.meth, *a)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__
    raise res
OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
[2020-05-22 12:09:43.968710] I [repce(agent /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF.
[2020-05-22 12:09:44.912470] I [monitor(monitor):280:monitor] Monitor: worker died in startup phase	brick=/home/sas/gluster/data/code-ide
[2020-05-22 12:09:44.913692] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change	status=Faulty
^C



sas@10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>
sas@10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>date
Fri May 22 05:09:59 PDT 2020
sas@10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>tail -f gsyncd.log
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1327, in process
    self.process_change(change, done, retry)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1221, in process_change
    failures = self.slave.server.entry_ops(entries)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__
    return self.ins(self.meth, *a)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__
    raise res
OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
[2020-05-22 12:10:02.710613] I [repce(agent /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF.
[2020-05-22 12:10:03.681055] I [monitor(monitor):280:monitor] Monitor: worker died in startup phase	brick=/home/sas/gluster/data/code-ide
[2020-05-22 12:10:03.682048] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change	status=Faulty

[2020-05-22 12:10:14.31925] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change	status=Initializing...
[2020-05-22 12:10:14.32105] I [monitor(monitor):159:monitor] Monitor: starting gsyncd worker	brick=/home/sas/gluster/data/code-ide	slave_node=10.37.11.246
[2020-05-22 12:10:14.76786] I [gsyncd(agent /home/sas/gluster/data/code-ide):311:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf
[2020-05-22 12:10:14.81006] I [changelogagent(agent /home/sas/gluster/data/code-ide):72:__init__] ChangelogAgent: Agent listining...
[2020-05-22 12:10:14.91418] I [gsyncd(worker /home/sas/gluster/data/code-ide):311:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf
[2020-05-22 12:10:14.109091] I [resource(worker /home/sas/gluster/data/code-ide):1394:connect_remote] SSH: Initializing SSH connection between master and slave...
[2020-05-22 12:10:16.334167] I [resource(worker /home/sas/gluster/data/code-ide):1443:connect_remote] SSH: SSH connection between master and slave established.	duration=2.2248
[2020-05-22 12:10:16.334379] I [resource(worker /home/sas/gluster/data/code-ide):1113:connect] GLUSTER: Mounting gluster volume locally...
[2020-05-22 12:10:17.444447] I [resource(worker /home/sas/gluster/data/code-ide):1136:connect] GLUSTER: Mounted gluster volume	duration=1.1099
[2020-05-22 12:10:17.444658] I [subcmds(worker /home/sas/gluster/data/code-ide):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor

[2020-05-22 12:10:19.612169] I [master(worker /home/sas/gluster/data/code-ide):1640:register] _GMaster: Working dir	path=/home/sas/gluster/internals/misc/gluster/gsyncd/code-ide_10.37.11.252_code-ide/home-sas-gluster-data-code-ide
[2020-05-22 12:10:19.612401] I [resource(worker /home/sas/gluster/data/code-ide):1299:service_loop] GLUSTER: Register time	time=1590149419
[2020-05-22 12:10:19.616991] I [gsyncdstatus(worker /home/sas/gluster/data/code-ide):281:set_active] GeorepStatus: Worker Status Change	status=Active
[2020-05-22 12:10:19.669262] I [gsyncdstatus(worker /home/sas/gluster/data/code-ide):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change	status=History Crawl
[2020-05-22 12:10:19.669564] I [master(worker /home/sas/gluster/data/code-ide):1554:crawl] _GMaster: starting history crawl	turns=1	stime=(1588918750, 0)	entry_stime=(1588918779, 0)	etime=1590149419
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux