Re: Was: Upgrade to 4.1.2 geo-replication does not work Now: Upgraded to 4.1.3 geo node Faulty

Marcus Pedersén <marcus.pedersen@xxxxxx> · Sat, 1 Sep 2018 11:38:45 +0000

Hi again,
I found another problem on the other master node.
The node toggles Active/Faulty and it is the same error over and over again.

[2018-09-01 11:23:02.94080] E [repce(worker /urd-gds/gluster):197:__call__] RepceClient: call failed    call=1226:139955262510912:1535800981.24 method=entry_ops        error=GsyncdError

[2018-09-01 11:23:02.94214] E [syncdutils(worker /urd-gds/gluster):300:log_raise_exception] <top>: execution of "gluster" failed with ENOENT (No such file or directory)

[2018-09-01 11:23:02.106194] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF.

[2018-09-01 11:23:02.124444] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change status=Faulty

I have also found a python error as well, I have only seen this once though.

[2018-09-01 11:16:45.907660] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time      stime=(1524101534, 0)

[2018-09-01 11:16:47.364109] E [syncdutils(worker /urd-gds/gluster):332:log_raise_exception] <top>: FAIL:

Traceback (most recent call last):

  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 362, in twrap

    tf(*aargs)

  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1939, in syncjob

    po = self.sync_engine(pb, self.log_err)

  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1442, in rsync

    rconf.ssh_ctl_args + \

AttributeError: 'NoneType' object has no attribute 'split'

[2018-09-01 11:16:47.384531] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF.

[2018-09-01 11:16:48.362987] I [monitor(monitor):279:monitor] Monitor: worker died in startup phase     brick=/urd-gds/gluster

[2018-09-01 11:16:48.370701] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change status=Faulty

[2018-09-01 11:16:58.390548] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker   brick=/urd-gds/gluster  slave_node=urd-gds-geo-000

I attach the logs as well.

Many thanks!

Best regards
Marcus Pedersén

Från: gluster-users-bounces@xxxxxxxxxxx <gluster-users-bounces@xxxxxxxxxxx> för Marcus Pedersén <marcus.pedersen@xxxxxx>

Skickat: den 31 augusti 2018 16:09

Till: khiremat@xxxxxxxxxx

Kopia: gluster-users@xxxxxxxxxxx

Ämne: Re:  Was: Upgrade to 4.1.2 geo-replication does not work Now: Upgraded to 4.1.3 geo node Faulty

I realy appologize, third try to make mail smaller.

/Marcus 

Från: Marcus Pedersén

Skickat: den 31 augusti 2018 16:03

Till: Kotresh Hiremath Ravishankar

Kopia: gluster-users@xxxxxxxxxxx

Ämne: SV:  Was: Upgrade to 4.1.2 geo-replication does not work Now: Upgraded to 4.1.3 geo node Faulty

Sorry, resend due to too large mail.

/Marcus

Från: Marcus Pedersén

Skickat: den 31 augusti 2018 15:19

Till: Kotresh Hiremath Ravishankar

Kopia: gluster-users@xxxxxxxxxxx

Ämne: SV:  Was: Upgrade to 4.1.2 geo-replication does not work Now: Upgraded to 4.1.3 geo node Faulty

Hi Kotresh,
Please find attached logs, only logs from today.
The python error was repeated over and over again until I disabled selinux.
After that the node bacame active again.
The return code 23 seems to be repeated over and over again.

rsync version 3.1.2

Thanks a lot!

Best regards
Marcus

Från: Kotresh Hiremath Ravishankar <khiremat@xxxxxxxxxx>

Skickat: den 31 augusti 2018 11:09

Till: Marcus Pedersén

Kopia: gluster-users@xxxxxxxxxxx

Ämne: Re:  Was: Upgrade to 4.1.2 geo-replication does not work Now: Upgraded to 4.1.3 geo node Faulty

Hi Marcus,

Could you attach full logs? Is the same trace back happening repeatedly? It will be helpful you attach the corresponding mount log as well.

What's the rsync version, you are using?

Thanks,

Kotresh HR

On Fri, Aug 31, 2018 at 12:16 PM, Marcus Pedersén 
<marcus.pedersen@xxxxxx> wrote:

Hi all,
I had problems with stopping sync after upgrade to 4.1.2.
I upgraded to 4.1.3 and it ran fine for one day, but now one of the master nodes shows faulty.
Most of the sync jobs have return code 23, how do I resolve this?
I see messages like: 
_GMaster: Sucessfully fixed all entry ops with gfid mismatch
Will this resolve error code 23?

There is also a python error.
The python error was a selinux problem, turning off selinux made node go to active again.

See log below.

CentOS 7, installed through SIG Gluster (OS updated to latest at the same time)

Master cluster: 2 x (2 + 1) distributed, replicated
Client cluster: 1 x (2 + 1) replicated

Many thanks in advance!

Best regards
Marcus Pedersén

gsyncd.log from Faulty node:
[2018-08-31 06:25:51.375267] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.8099 num_files=57    job=3   return_code=23

[2018-08-31 06:25:51.465895] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0904 num_files=3     job=3   return_code=23

[2018-08-31 06:25:52.562107] E [repce(worker /urd-gds/gluster):197:__call__] RepceClient: call failed   call=30069:139655665837888:1535696752.35        method=entry_ops        error=OSError

[2018-08-31 06:25:52.562346] E [syncdutils(worker /urd-gds/gluster):332:log_raise_exception] <top>: FAIL:

Traceback (most recent call last):

  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 311, in main

    func(args)

  File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 72, in subcmd_worker

    local.service_loop(remote)

  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1288, in service_loop

    g3.crawlwrap(_oneshot_=True)

  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 615, in crawlwrap

    self.crawl()

  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1545, in crawl

    self.changelogs_batch_process(changes)

  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1445, in changelogs_batch_process

    self.process(batch)

  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1280, in process

    self.process_change(change, done, retry)

  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1179, in process_change

    failures = self.slave.server.entry_ops(entries)

  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 216, in __call__

    return self.ins(self.meth, *a)

  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 198, in __call__

    raise res

OSError: [Errno 13] Permission denied

[2018-08-31 06:25:52.578367] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF.

[2018-08-31 06:25:53.558765] I [monitor(monitor):279:monitor] Monitor: worker died in startup phase     brick=/urd-gds/gluster

[2018-08-31 06:25:53.569777] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change status=Faulty

[2018-08-31 06:26:03.593161] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker   brick=/urd-gds/gluster  slave_node=urd-gds-geo-000

[2018-08-31 06:26:03.636452] I [gsyncd(agent /urd-gds/gluster):297:main] <top>: Using session config file       path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf

[2018-08-31 06:26:03.636810] I [gsyncd(worker /urd-gds/gluster):297:main] <top>: Using session config file      path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf

[2018-08-31 06:26:03.637486] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...

[2018-08-31 06:26:03.650330] I [resource(worker /urd-gds/gluster):1377:connect_remote] SSH: Initializing SSH connection between master and slave...

[2018-08-31 06:26:05.296473] I [resource(worker /urd-gds/gluster):1424:connect_remote] SSH: SSH connection between master and slave established.        duration=1.6457

[2018-08-31 06:26:05.297904] I [resource(worker /urd-gds/gluster):1096:connect] GLUSTER: Mounting gluster volume locally...

[2018-08-31 06:26:06.396939] I [resource(worker /urd-gds/gluster):1119:connect] GLUSTER: Mounted gluster volume duration=1.0985

[2018-08-31 06:26:06.397691] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor

[2018-08-31 06:26:16.815566] I [master(worker /urd-gds/gluster):1593:register] _GMaster: Working dir    path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster

[2018-08-31 06:26:16.816423] I [resource(worker /urd-gds/gluster):1282:service_loop] GLUSTER: Register time     time=1535696776

[2018-08-31 06:26:16.888772] I [gsyncdstatus(worker /urd-gds/gluster):277:set_active] GeorepStatus: Worker Status Change        status=Active

[2018-08-31 06:26:16.892049] I [gsyncdstatus(worker /urd-gds/gluster):249:set_worker_crawl_status] GeorepStatus: Crawl Status Change    status=History Crawl

[2018-08-31 06:26:16.892703] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl    turns=1 stime=(1525739167, 0)   entry_stime=(1525740143, 0)     etime=1535696776

[2018-08-31 06:26:17.914803] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time      stime=(1525739167, 0)

[2018-08-31 06:26:18.521718] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.1063 num_files=17    job=3   return_code=23

[2018-08-31 06:26:19.260137] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.3441 num_files=34    job=1   return_code=23

[2018-08-31 06:26:19.615191] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0923 num_files=7     job=3   return_code=23

[2018-08-31 06:26:19.891227] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.1302 num_files=12    job=1   return_code=23

[2018-08-31 06:26:19.922700] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.5024 num_files=50    job=2   return_code=23
[2018-08-31 06:26:21.639342] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=1.5233 num_files=5     job=3   return_code=23

[2018-08-31 06:26:22.12726] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken    duration=0.1191 num_files=7     job=1   return_code=23

[2018-08-31 06:26:22.86136] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken    duration=0.0731 num_files=4     job=1   return_code=23

[2018-08-31 06:26:22.503290] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0779 num_files=15    job=2   return_code=23

[2018-08-31 06:26:23.214704] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0738 num_files=9     job=3   return_code=23

[2018-08-31 06:26:23.251876] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.2478 num_files=33    job=2   return_code=23

[2018-08-31 06:26:23.802699] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0873 num_files=9     job=3   return_code=23

[2018-08-31 06:26:23.828176] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0758 num_files=3     job=2   return_code=23

[2018-08-31 06:26:23.854063] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.2662 num_files=34    job=1   return_code=23

[2018-08-31 06:26:24.403228] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0997 num_files=30    job=3   return_code=23

[2018-08-31 06:26:25.526] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken      duration=0.0965 num_files=8     job=3   return_code=23

[2018-08-31 06:26:25.438527] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0832 num_files=9     job=1   return_code=23

[2018-08-31 06:26:25.447256] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.6180 num_files=86    job=2   return_code=23

[2018-08-31 06:26:25.571913] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0706 num_files=2     job=3   return_code=0

[2018-08-31 06:26:27.21325] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken    duration=0.0814 num_files=1     job=1   return_code=23

[2018-08-31 06:26:27.615520] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0933 num_files=13    job=1   return_code=23

[2018-08-31 06:26:27.668323] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.2190 num_files=95    job=2   return_code=23

[2018-08-31 06:26:27.740139] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0716 num_files=11    job=2   return_code=23

[2018-08-31 06:26:28.191068] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.1167 num_files=38    job=3   return_code=23

[2018-08-31 06:26:28.268213] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0768 num_files=7     job=3   return_code=23

[2018-08-31 06:26:28.317909] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0770 num_files=4     job=2   return_code=23

[2018-08-31 06:26:28.710064] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0932 num_files=23    job=1   return_code=23

[2018-08-31 06:26:28.907250] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0886 num_files=26    job=2   return_code=23

[2018-08-31 06:26:28.976679] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0692 num_files=4     job=2   return_code=23

[2018-08-31 06:26:29.55774] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken    duration=0.0788 num_files=9     job=2   return_code=23

[2018-08-31 06:26:29.295576] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0847 num_files=16    job=1   return_code=23

[2018-08-31 06:26:29.665076] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.1087 num_files=25    job=2   return_code=23

[2018-08-31 06:26:30.277998] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.1122 num_files=40    job=2   return_code=23

[2018-08-31 06:26:31.153105] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.3822 num_files=74    job=3   return_code=23

[2018-08-31 06:26:31.227639] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0743 num_files=18    job=3   return_code=23

[2018-08-31 06:26:31.302660] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken   duration=0.0748 num_files=18    job=3   return_code=23

---

När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka

här 

E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click

here 

_______________________________________________

Gluster-users mailing list

Gluster-users@xxxxxxxxxxx

https://lists.gluster.org/mailman/listinfo/gluster-users

-- 

Thanks and Regards,

Kotresh H R

---

När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka
här 

E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click
here 

---

När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka
här 

E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click
here 

[2018-09-01 11:16:10.909337] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.8130	num_files=92	job=1	return_code=23
[2018-09-01 11:16:16.659548] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.7450	num_files=83	job=1	return_code=0
[2018-09-01 11:16:19.382323] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.3692	num_files=85	job=3	return_code=23
[2018-09-01 11:16:19.402731] E [repce(worker /urd-gds/gluster):197:__call__] RepceClient: call failed	call=3934:139973243336512:1535800578.96	method=entry_ops	error=GsyncdError
[2018-09-01 11:16:19.402901] E [syncdutils(worker /urd-gds/gluster):300:log_raise_exception] <top>: execution of "gluster" failed with ENOENT (No such file or directory)
[2018-09-01 11:16:19.418284] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF.
[2018-09-01 11:16:19.439542] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change	status=Faulty
[2018-09-01 11:16:29.456732] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker	brick=/urd-gds/gluster	slave_node=urd-gds-geo-000
[2018-09-01 11:16:29.526350] I [gsyncd(agent /urd-gds/gluster):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:16:29.527015] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...
[2018-09-01 11:16:29.529350] I [gsyncd(worker /urd-gds/gluster):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:16:29.538482] I [resource(worker /urd-gds/gluster):1377:connect_remote] SSH: Initializing SSH connection between master and slave...
[2018-09-01 11:16:31.212218] I [resource(worker /urd-gds/gluster):1424:connect_remote] SSH: SSH connection between master and slave established.	duration=1.6735
[2018-09-01 11:16:31.212785] I [resource(worker /urd-gds/gluster):1096:connect] GLUSTER: Mounting gluster volume locally...
[2018-09-01 11:16:32.344389] I [resource(worker /urd-gds/gluster):1119:connect] GLUSTER: Mounted gluster volume	duration=1.1312
[2018-09-01 11:16:32.344978] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
[2018-09-01 11:16:44.829792] I [master(worker /urd-gds/gluster):1593:register] _GMaster: Working dir	path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster
[2018-09-01 11:16:44.830971] I [resource(worker /urd-gds/gluster):1282:service_loop] GLUSTER: Register time	time=1535800604
[2018-09-01 11:16:44.884566] I [gsyncdstatus(worker /urd-gds/gluster):277:set_active] GeorepStatus: Worker Status Change	status=Active
[2018-09-01 11:16:44.886054] I [gsyncdstatus(worker /urd-gds/gluster):249:set_worker_crawl_status] GeorepStatus: Crawl Status Change	status=History Crawl
[2018-09-01 11:16:44.886224] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl	turns=1	stime=(1524101534, 0)	entry_stime=(1524101804, 0)	etime=1535800604
[2018-09-01 11:16:45.907660] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time	stime=(1524101534, 0)
[2018-09-01 11:16:47.364109] E [syncdutils(worker /urd-gds/gluster):332:log_raise_exception] <top>: FAIL: 
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 362, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1939, in syncjob
    po = self.sync_engine(pb, self.log_err)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1442, in rsync
    rconf.ssh_ctl_args + \
AttributeError: 'NoneType' object has no attribute 'split'
[2018-09-01 11:16:47.384531] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF.
[2018-09-01 11:16:48.362987] I [monitor(monitor):279:monitor] Monitor: worker died in startup phase	brick=/urd-gds/gluster
[2018-09-01 11:16:48.370701] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change	status=Faulty
[2018-09-01 11:16:58.390548] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker	brick=/urd-gds/gluster	slave_node=urd-gds-geo-000
[2018-09-01 11:16:58.462002] I [gsyncd(agent /urd-gds/gluster):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:16:58.462469] I [gsyncd(worker /urd-gds/gluster):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:16:58.462757] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...
[2018-09-01 11:16:58.471395] I [resource(worker /urd-gds/gluster):1377:connect_remote] SSH: Initializing SSH connection between master and slave...
[2018-09-01 11:17:00.62880] I [resource(worker /urd-gds/gluster):1424:connect_remote] SSH: SSH connection between master and slave established.	duration=1.5912
[2018-09-01 11:17:00.63438] I [resource(worker /urd-gds/gluster):1096:connect] GLUSTER: Mounting gluster volume locally...
[2018-09-01 11:17:01.164509] I [resource(worker /urd-gds/gluster):1119:connect] GLUSTER: Mounted gluster volume	duration=1.1006
[2018-09-01 11:17:01.165066] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
[2018-09-01 11:17:03.448198] I [master(worker /urd-gds/gluster):1593:register] _GMaster: Working dir	path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster
[2018-09-01 11:17:03.448826] I [resource(worker /urd-gds/gluster):1282:service_loop] GLUSTER: Register time	time=1535800623
[2018-09-01 11:17:03.491442] I [gsyncdstatus(worker /urd-gds/gluster):277:set_active] GeorepStatus: Worker Status Change	status=Active
[2018-09-01 11:17:03.493484] I [gsyncdstatus(worker /urd-gds/gluster):249:set_worker_crawl_status] GeorepStatus: Crawl Status Change	status=History Crawl
[2018-09-01 11:17:03.493901] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl	turns=1	stime=(1524101534, 0)	entry_stime=(1524101804, 0)	etime=1535800623
[2018-09-01 11:17:04.513282] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time	stime=(1524101534, 0)
[2018-09-01 11:17:08.725709] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.5496	num_files=113	job=1	return_code=23
[2018-09-01 11:17:13.759978] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=3.5809	num_files=107	job=2	return_code=23
[2018-09-01 11:17:17.215708] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.0307	num_files=116	job=3	return_code=23
[2018-09-01 11:17:22.900979] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.6799	num_files=116	job=3	return_code=23
[2018-09-01 11:17:23.799818] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.3979	num_files=100	job=3	return_code=23
[2018-09-01 11:17:25.914112] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.1130	num_files=97	job=3	return_code=23
[2018-09-01 11:17:30.17796] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.7712	num_files=122	job=1	return_code=23
[2018-09-01 11:17:31.464807] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.5453	num_files=59	job=3	return_code=23
[2018-09-01 11:17:34.449332] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.6688	num_files=124	job=2	return_code=23
[2018-09-01 11:17:35.528789] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.5787	num_files=82	job=2	return_code=23
[2018-09-01 11:17:43.626379] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=2.6518	num_files=131	job=3	return_code=23
[2018-09-01 11:17:45.60198] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.9330	num_files=68	job=3	return_code=0
[2018-09-01 11:17:52.924736] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=2.3856	num_files=123	job=1	return_code=0
[2018-09-01 11:17:55.471108] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.5440	num_files=131	job=1	return_code=23
[2018-09-01 11:17:56.567857] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.5962	num_files=64	job=1	return_code=0
[2018-09-01 11:18:05.184760] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.1271	num_files=125	job=2	return_code=0
[2018-09-01 11:18:06.305181] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.7270	num_files=92	job=1	return_code=23
[2018-09-01 11:18:10.277966] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.6934	num_files=83	job=3	return_code=0
[2018-09-01 11:18:15.348804] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.1544	num_files=85	job=2	return_code=23
[2018-09-01 11:18:17.695431] E [repce(worker /urd-gds/gluster):197:__call__] RepceClient: call failed	call=7352:140551148898112:1535800696.72	method=entry_ops	error=GsyncdError
[2018-09-01 11:18:17.695614] E [syncdutils(worker /urd-gds/gluster):300:log_raise_exception] <top>: execution of "gluster" failed with ENOENT (No such file or directory)
[2018-09-01 11:18:17.711199] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF.
[2018-09-01 11:18:17.738365] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change	status=Faulty
[2018-09-01 11:18:27.760206] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker	brick=/urd-gds/gluster	slave_node=urd-gds-geo-000
[2018-09-01 11:18:27.831969] I [gsyncd(worker /urd-gds/gluster):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:18:27.841169] I [gsyncd(agent /urd-gds/gluster):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:18:27.842202] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...
[2018-09-01 11:18:27.872488] I [resource(worker /urd-gds/gluster):1377:connect_remote] SSH: Initializing SSH connection between master and slave...
[2018-09-01 11:18:29.449333] I [resource(worker /urd-gds/gluster):1424:connect_remote] SSH: SSH connection between master and slave established.	duration=1.5766
[2018-09-01 11:18:29.449908] I [resource(worker /urd-gds/gluster):1096:connect] GLUSTER: Mounting gluster volume locally...
[2018-09-01 11:18:30.571775] I [resource(worker /urd-gds/gluster):1119:connect] GLUSTER: Mounted gluster volume	duration=1.1215
[2018-09-01 11:18:30.572541] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
[2018-09-01 11:18:41.219701] I [master(worker /urd-gds/gluster):1593:register] _GMaster: Working dir	path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster
[2018-09-01 11:18:41.220571] I [resource(worker /urd-gds/gluster):1282:service_loop] GLUSTER: Register time	time=1535800721
[2018-09-01 11:18:41.882281] I [gsyncdstatus(worker /urd-gds/gluster):277:set_active] GeorepStatus: Worker Status Change	status=Active
[2018-09-01 11:18:41.884654] I [gsyncdstatus(worker /urd-gds/gluster):249:set_worker_crawl_status] GeorepStatus: Crawl Status Change	status=History Crawl
[2018-09-01 11:18:41.884835] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl	turns=1	stime=(1524101534, 0)	entry_stime=(1524101804, 0)	etime=1535800721
[2018-09-01 11:18:42.904170] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time	stime=(1524101534, 0)
[2018-09-01 11:18:47.568198] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.4760	num_files=113	job=3	return_code=23
[2018-09-01 11:18:51.249831] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=2.6803	num_files=107	job=3	return_code=23
[2018-09-01 11:18:54.151258] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.0522	num_files=116	job=2	return_code=23
[2018-09-01 11:18:59.42026] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.4375	num_files=116	job=1	return_code=23
[2018-09-01 11:19:01.320400] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.0612	num_files=100	job=3	return_code=23
[2018-09-01 11:19:03.631659] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.8092	num_files=97	job=3	return_code=23
[2018-09-01 11:19:07.380134] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.8306	num_files=122	job=1	return_code=23
[2018-09-01 11:19:08.305370] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.4248	num_files=59	job=1	return_code=23
[2018-09-01 11:19:14.602133] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=3.9631	num_files=124	job=3	return_code=23
[2018-09-01 11:19:22.315196] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=3.2089	num_files=82	job=3	return_code=23
[2018-09-01 11:19:28.428549] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.7422	num_files=131	job=2	return_code=23
[2018-09-01 11:19:31.183011] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.8598	num_files=68	job=3	return_code=0
[2018-09-01 11:19:34.593025] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.7619	num_files=123	job=1	return_code=0
[2018-09-01 11:19:40.10251] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.0700	num_files=131	job=2	return_code=23
[2018-09-01 11:19:41.427376] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.3273	num_files=64	job=1	return_code=0
[2018-09-01 11:19:48.773497] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.2509	num_files=125	job=2	return_code=0
[2018-09-01 11:19:55.991848] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.7845	num_files=92	job=3	return_code=23
[2018-09-01 11:19:57.328516] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.5470	num_files=83	job=2	return_code=0
[2018-09-01 11:20:00.677280] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.1818	num_files=85	job=3	return_code=23
[2018-09-01 11:20:01.225960] E [repce(worker /urd-gds/gluster):197:__call__] RepceClient: call failed	call=31640:140394415150912:1535800800.37	method=entry_ops	error=GsyncdError
[2018-09-01 11:20:01.226191] E [syncdutils(worker /urd-gds/gluster):300:log_raise_exception] <top>: execution of "gluster" failed with ENOENT (No such file or directory)
[2018-09-01 11:20:01.239264] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF.
[2018-09-01 11:20:01.246619] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change	status=Faulty
[2018-09-01 11:20:11.265338] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker	brick=/urd-gds/gluster	slave_node=urd-gds-geo-000
[2018-09-01 11:20:11.336337] I [gsyncd(agent /urd-gds/gluster):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:20:11.336961] I [gsyncd(worker /urd-gds/gluster):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:20:11.337119] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...
[2018-09-01 11:20:11.346508] I [resource(worker /urd-gds/gluster):1377:connect_remote] SSH: Initializing SSH connection between master and slave...
[2018-09-01 11:20:12.882986] I [resource(worker /urd-gds/gluster):1424:connect_remote] SSH: SSH connection between master and slave established.	duration=1.5362
[2018-09-01 11:20:12.883548] I [resource(worker /urd-gds/gluster):1096:connect] GLUSTER: Mounting gluster volume locally...
[2018-09-01 11:20:14.585222] I [resource(worker /urd-gds/gluster):1119:connect] GLUSTER: Mounted gluster volume	duration=1.7013
[2018-09-01 11:20:14.585820] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
[2018-09-01 11:20:27.485044] I [master(worker /urd-gds/gluster):1593:register] _GMaster: Working dir	path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster
[2018-09-01 11:20:27.486057] I [resource(worker /urd-gds/gluster):1282:service_loop] GLUSTER: Register time	time=1535800827
[2018-09-01 11:20:27.522258] I [gsyncdstatus(worker /urd-gds/gluster):277:set_active] GeorepStatus: Worker Status Change	status=Active
[2018-09-01 11:20:27.523973] I [gsyncdstatus(worker /urd-gds/gluster):249:set_worker_crawl_status] GeorepStatus: Crawl Status Change	status=History Crawl
[2018-09-01 11:20:27.524256] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl	turns=1	stime=(1524101534, 0)	entry_stime=(1524101804, 0)	etime=1535800827
[2018-09-01 11:20:28.543723] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time	stime=(1524101534, 0)
[2018-09-01 11:20:31.499804] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.8938	num_files=113	job=2	return_code=23
[2018-09-01 11:20:34.94470] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.5934	num_files=107	job=2	return_code=23
[2018-09-01 11:20:37.100281] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.5032	num_files=116	job=2	return_code=23
[2018-09-01 11:20:39.27117] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.9256	num_files=116	job=2	return_code=23
[2018-09-01 11:20:40.534573] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.5062	num_files=100	job=2	return_code=23
[2018-09-01 11:20:45.553112] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.0141	num_files=97	job=2	return_code=23
[2018-09-01 11:20:46.769187] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.7157	num_files=122	job=2	return_code=23
[2018-09-01 11:20:47.585594] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.3161	num_files=59	job=2	return_code=23
[2018-09-01 11:20:50.111612] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.5236	num_files=124	job=2	return_code=23
[2018-09-01 11:20:51.388463] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.7754	num_files=82	job=2	return_code=23
[2018-09-01 11:20:56.437892] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.3078	num_files=131	job=3	return_code=23
[2018-09-01 11:20:58.168567] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.2740	num_files=68	job=2	return_code=0
[2018-09-01 11:21:00.717818] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.5828	num_files=123	job=1	return_code=0
[2018-09-01 11:21:07.217396] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.7691	num_files=131	job=3	return_code=23
[2018-09-01 11:21:08.989913] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.3103	num_files=64	job=2	return_code=0
[2018-09-01 11:21:13.825584] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=3.3339	num_files=125	job=2	return_code=0
[2018-09-01 11:21:18.233908] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.5051	num_files=92	job=3	return_code=23
[2018-09-01 11:21:22.143229] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=3.3122	num_files=83	job=2	return_code=0
[2018-09-01 11:21:29.814635] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.6416	num_files=85	job=2	return_code=23
[2018-09-01 11:21:30.107693] E [repce(worker /urd-gds/gluster):197:__call__] RepceClient: call failed	call=22967:140511351723840:1535800889.35	method=entry_ops	error=GsyncdError
[2018-09-01 11:21:30.107831] E [syncdutils(worker /urd-gds/gluster):300:log_raise_exception] <top>: execution of "gluster" failed with ENOENT (No such file or directory)
[2018-09-01 11:21:30.123255] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF.
[2018-09-01 11:21:30.151117] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change	status=Faulty
[2018-09-01 11:21:40.173165] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker	brick=/urd-gds/gluster	slave_node=urd-gds-geo-000
[2018-09-01 11:21:40.221231] I [gsyncd(agent /urd-gds/gluster):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:21:40.223439] I [gsyncd(worker /urd-gds/gluster):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:21:40.223797] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...
[2018-09-01 11:21:40.248348] I [resource(worker /urd-gds/gluster):1377:connect_remote] SSH: Initializing SSH connection between master and slave...
[2018-09-01 11:21:41.869673] I [resource(worker /urd-gds/gluster):1424:connect_remote] SSH: SSH connection between master and slave established.	duration=1.6210
[2018-09-01 11:21:41.870523] I [resource(worker /urd-gds/gluster):1096:connect] GLUSTER: Mounting gluster volume locally...
[2018-09-01 11:21:42.999773] I [resource(worker /urd-gds/gluster):1119:connect] GLUSTER: Mounted gluster volume	duration=1.1288
[2018-09-01 11:21:43.335] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
[2018-09-01 11:21:51.755662] I [master(worker /urd-gds/gluster):1593:register] _GMaster: Working dir	path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster
[2018-09-01 11:21:51.756623] I [resource(worker /urd-gds/gluster):1282:service_loop] GLUSTER: Register time	time=1535800911
[2018-09-01 11:21:51.774579] I [gsyncdstatus(worker /urd-gds/gluster):277:set_active] GeorepStatus: Worker Status Change	status=Active
[2018-09-01 11:21:51.776083] I [gsyncdstatus(worker /urd-gds/gluster):249:set_worker_crawl_status] GeorepStatus: Crawl Status Change	status=History Crawl
[2018-09-01 11:21:51.776257] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl	turns=1	stime=(1524101534, 0)	entry_stime=(1524101804, 0)	etime=1535800911
[2018-09-01 11:21:52.796566] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time	stime=(1524101534, 0)
[2018-09-01 11:21:55.587325] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.5713	num_files=113	job=2	return_code=23
[2018-09-01 11:21:57.472486] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.4549	num_files=107	job=1	return_code=23
[2018-09-01 11:21:59.762489] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.2890	num_files=116	job=1	return_code=23
[2018-09-01 11:22:02.462059] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.1976	num_files=116	job=1	return_code=23
[2018-09-01 11:22:04.363795] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.7686	num_files=100	job=2	return_code=23
[2018-09-01 11:22:06.338656] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.9736	num_files=97	job=2	return_code=23
[2018-09-01 11:22:08.780875] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.9404	num_files=122	job=2	return_code=23
[2018-09-01 11:22:11.106581] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.1371	num_files=59	job=1	return_code=23
[2018-09-01 11:22:16.526092] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.4895	num_files=124	job=3	return_code=23
[2018-09-01 11:22:18.71861] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.0447	num_files=82	job=3	return_code=23
[2018-09-01 11:22:22.375686] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.7998	num_files=131	job=3	return_code=23
[2018-09-01 11:22:25.157756] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=2.5396	num_files=68	job=1	return_code=0
[2018-09-01 11:22:33.463901] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.0774	num_files=123	job=3	return_code=0
[2018-09-01 11:22:37.682714] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.0129	num_files=131	job=1	return_code=23
[2018-09-01 11:22:39.574531] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.3898	num_files=64	job=1	return_code=0
[2018-09-01 11:22:48.546103] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.9546	num_files=125	job=1	return_code=0
[2018-09-01 11:22:55.26038] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.0415	num_files=92	job=3	return_code=23
[2018-09-01 11:22:58.212902] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.3818	num_files=83	job=2	return_code=0
[2018-09-01 11:23:00.388211] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.8306	num_files=85	job=1	return_code=23
[2018-09-01 11:23:02.94080] E [repce(worker /urd-gds/gluster):197:__call__] RepceClient: call failed	call=1226:139955262510912:1535800981.24	method=entry_ops	error=GsyncdError
[2018-09-01 11:23:02.94214] E [syncdutils(worker /urd-gds/gluster):300:log_raise_exception] <top>: execution of "gluster" failed with ENOENT (No such file or directory)
[2018-09-01 11:23:02.106194] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF.
[2018-09-01 11:23:02.124444] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change	status=Faulty
[2018-09-01 11:23:12.133107] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker	brick=/urd-gds/gluster	slave_node=urd-gds-geo-000
[2018-09-01 11:23:12.180077] I [gsyncd(worker /urd-gds/gluster):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:23:12.183568] I [gsyncd(agent /urd-gds/gluster):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:23:12.184608] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...
[2018-09-01 11:23:13.353393] I [resource(worker /urd-gds/gluster):1377:connect_remote] SSH: Initializing SSH connection between master and slave...
[2018-09-01 11:23:14.927974] I [resource(worker /urd-gds/gluster):1424:connect_remote] SSH: SSH connection between master and slave established.	duration=1.5740
[2018-09-01 11:23:14.928534] I [resource(worker /urd-gds/gluster):1096:connect] GLUSTER: Mounting gluster volume locally...
[2018-09-01 11:23:16.29083] I [resource(worker /urd-gds/gluster):1119:connect] GLUSTER: Mounted gluster volume	duration=1.1001
[2018-09-01 11:23:16.29679] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
[2018-09-01 11:23:24.394258] I [gsyncd(config-get):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:23:24.463059] I [gsyncd(status):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:23:29.892555] I [master(worker /urd-gds/gluster):1593:register] _GMaster: Working dir	path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster
[2018-09-01 11:23:29.893162] I [resource(worker /urd-gds/gluster):1282:service_loop] GLUSTER: Register time	time=1535801009
[2018-09-01 11:23:29.927710] I [gsyncdstatus(worker /urd-gds/gluster):277:set_active] GeorepStatus: Worker Status Change	status=Active
[2018-09-01 11:23:29.930145] I [gsyncdstatus(worker /urd-gds/gluster):249:set_worker_crawl_status] GeorepStatus: Crawl Status Change	status=History Crawl
[2018-09-01 11:23:29.930346] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl	turns=1	stime=(1524101534, 0)	entry_stime=(1524101804, 0)	etime=1535801009
[2018-09-01 11:23:30.951765] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time	stime=(1524101534, 0)
[2018-09-01 11:23:33.237665] I [gsyncd(config-get):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:23:33.314560] I [gsyncd(status):297:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
[2018-09-01 11:23:33.901304] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.8518	num_files=113	job=1	return_code=23
[2018-09-01 11:23:36.586030] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.0335	num_files=107	job=3	return_code=23
[2018-09-01 11:23:38.482296] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=0.5768	num_files=116	job=1	return_code=23
[2018-09-01 11:23:41.598927] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken	duration=1.1144	num_files=116	job=1	return_code=23
[2018-09-01 11:00:23.502315] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:00:44.276362] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-z9BrJ2
[2018-09-01 11:00:44.276488] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f59bd1bfe25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x5602d6dfed65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x5602d6dfeb8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:00:44.276502] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-z9BrJ2'.
[2018-09-01 11:00:44.276508] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-z9BrJ2'.
[2018-09-01 11:00:57.809740] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-zTScgg)
[2018-09-01 11:00:57.820138] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:00:57.822525] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:00:57.822556] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:00:57.824724] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:00:57.828242] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:00:57.828293] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.828411] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.828508] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:00:57.839862] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:00:57.845915] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.845943] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.846191] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fc03570331a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc04325b266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fc043257ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:00:57.846202] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:00:57.846224] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.846319] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.846585] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:00:57.847584] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:00:57.852693] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.852949] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.853005] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:00:57.853016] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:00:57.854558] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:00:57.855972] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.856034] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fc03570331a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc04325b266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fc043257ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:00:57.856043] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:00:57.856056] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.856077] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.856506] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.856533] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:00:57.858687] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:00:57.860302] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:00:57.860331] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.863857] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:7021a46b-86d7-4d11-8bd3-e3af79d5ddd8-GRAPH_ID:0-PID:11495-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:7021a46b-86d7-4d11-8bd3-e3af79d5ddd8-GRAPH_ID:0-PID:11495-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option ping-timeout 42
 50:     option remote-host urd-gds-000
 51:     option remote-subvolume /urd-gds/gluster1
 52:     option transport-type socket
 53:     option transport.address-family inet
 54:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 55:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 56:     option transport.tcp-user-timeout 0
 57:     option transport.socket.keepalive-time 20
 58:     option transport.socket.keepalive-interval 2
 59:     option transport.socket.keepalive-count 9
 60:     option send-gids true
 61: end-volume
 62:  
 63: volume urd-gds-volume-replicate-0
 64:     type cluster/replicate
 65:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 66:     option arbiter-count 1
 67:     option use-compound-fops off
 68:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 69: end-volume
 70:  
 71: volume urd-gds-volume-client-3
 72:     type protocol/client
 73:     option ping-timeout 42
 74:     option remote-host urd-gds-003
 75:     option remote-subvolume /urd-gds/gluster
 76:     option transport-type socket
 77:     option transport.address-family inet
 78:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 79:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 80:     option transport.tcp-user-timeout 0
 81:     option transport.socket.keepalive-time 20
 82:     option transport.socket.keepalive-interval 2
 83:     option transport.socket.keepalive-count 9
 84:     option send-gids true
 85: end-volume
 86:  
 87: volume urd-gds-volume-client-4
 88:     type protocol/client
 89:     option ping-timeout 42
 90:     option remote-host urd-gds-004
 91:     option remote-subvolume /urd-gds/gluster
 92:     option transport-type socket
 93:     option transport.address-family inet
 94:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 95:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 96:     option transport.tcp-user-timeout 0
 97:     option transport.socket.keepalive-time 20
 98:     option transport.socket.keepalive-interval 2
 99:     option transport.socket.keepalive-count 9
100:     option send-gids true
101: end-volume
102:  
103: volume urd-gds-volume-client-5
104:     type protocol/client
105:     option ping-timeout 42
106:     option remote-host urd-gds-000
107:     option remote-subvolume /urd-gds/gluster2
108:     option transport-type socket
109:     option transport.address-family inet
110:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
111:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
112:     option transport.tcp-user-timeout 0
113:     option transport.socket.keepalive-time 20
114:     option transport.socket.keepalive-interval 2
115:     option transport.socket.keepalive-count 9
116:     option send-gids true
117: end-volume
118:  
119: volume urd-gds-volume-replicate-1
120:     type cluster/replicate
121:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
122:     option arbiter-count 1
123:     option use-compound-fops off
124:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
125: end-volume
126:  
127: volume urd-gds-volume-dht
128:     type cluster/distribute
129:     option lock-migration off
130:     option force-migration off
131:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
132: end-volume
133:  
134: volume urd-gds-volume-write-behind
135:     type performance/write-behind
136:     subvolumes urd-gds-volume-dht
137: end-volume
138:  
139: volume urd-gds-volume-read-ahead
140:     type performance/read-ahead
141:     subvolumes urd-gds-volume-write-behind
142: end-volume
143:  
144: volume urd-gds-volume-readdir-ahead
145:     type performance/readdir-ahead
146:     option parallel-readdir off
147:     option rda-request-size 131072
148:     option rda-cache-limit 10MB
149:     subvolumes urd-gds-volume-read-ahead
150: end-volume
151:  
152: volume urd-gds-volume-io-cache
153:     type performance/io-cache
154:     subvolumes urd-gds-volume-readdir-ahead
155: end-volume
156:  
157: volume urd-gds-volume-quick-read
158:     type performance/quick-read
159:     subvolumes urd-gds-volume-io-cache
160: end-volume
161:  
162: volume urd-gds-volume-open-behind
163:     type performance/open-behind
164:     subvolumes urd-gds-volume-quick-read
165: end-volume
166:  
167: volume urd-gds-volume-md-cache
168:     type performance/md-cache
169:     option cache-posix-acl true
170:     subvolumes urd-gds-volume-open-behind
171: end-volume
172:  
173: volume urd-gds-volume-io-threads
174:     type performance/io-threads
175:     subvolumes urd-gds-volume-md-cache
176: end-volume
177:  
178: volume urd-gds-volume-snapd-client
179:     type protocol/client
180:     option remote-host localhost
181:     option ping-timeout 42
182:     option remote-subvolume snapd-urd-gds-volume
183:     option transport-type socket
184:     option transport.address-family inet
185:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
186:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
187:     option transport.tcp-user-timeout 0
188:     option transport.socket.keepalive-time 20
189:     option transport.socket.keepalive-interval 2
190:     option transport.socket.keepalive-count 9
191:     option send-gids true
192: end-volume
193:  
194: volume urd-gds-volume-snapview-client
195:     type features/snapview-client
196:     option snapshot-directory .snaps
197:     option show-snapshot-directory on
198:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
199: end-volume
200:  
201: volume urd-gds-volume
202:     type debug/io-stats
203:     option log-level INFO
204:     option latency-measurement off
205:     option count-fop-hits off
206:     subvolumes urd-gds-volume-snapview-client
207: end-volume
208:  
209: volume posix-acl-autoload
210:     type system/posix-acl
211:     subvolumes urd-gds-volume
212: end-volume
213:  
214: volume gfid-access-autoload
215:     type features/gfid-access
216:     subvolumes posix-acl-autoload
217: end-volume
218:  
219: volume meta-autoload
220:     type meta
221:     subvolumes gfid-access-autoload
222: end-volume
223:  
+------------------------------------------------------------------------------+
[2018-09-01 11:00:57.870437] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.870482] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.870520] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.870561] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:00:57.870578] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
[2018-09-01 11:00:57.871587] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.871657] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fc03570331a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc04325b266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fc043257ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:00:57.871672] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:00:57.871684] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.871703] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
[2018-09-01 11:00:57.871723] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.871749] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.874154] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.874186] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:00:57.874213] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:00:57.877631] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fc03570331a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc04325b266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fc043257ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:00:57.877644] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:00:57.877661] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.877682] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.877791] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fc03570331a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc04325b266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fc043257ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:00:57.877799] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:00:57.877812] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.877846] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.878317] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fc03570331a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc04325b266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fc043257ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:00:57.878326] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:00:57.878339] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.881020] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:00:57.881184] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.881691] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fc03570331a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc04325b266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fc043257ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:00:57.881708] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:00:57.881724] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:00:57.884314] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:00:57.889325] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:00:57.889339] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-4' came back up; going online.
[2018-09-01 11:00:57.890422] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:00:57.893154] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:00:57.893169] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:00:57.896283] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:00:57.896300] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:00:57.897818] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:00:57.898990] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:00:57.900555] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:01:12.500621] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:01:54.087692] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:02:16.126644] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-zTScgg
[2018-09-01 11:02:16.126755] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7fc0420c6e25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x558fc6277d65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x558fc6277b8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:02:16.126776] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-zTScgg'.
[2018-09-01 11:02:16.126784] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-zTScgg'.
[2018-09-01 11:02:27.872002] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-EQhDVc)
[2018-09-01 11:02:27.880884] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:02:27.883262] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:02:27.883264] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:02:27.885402] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:02:27.889726] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:02:27.889779] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.889907] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.889962] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:02:27.894753] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:02:27.896537] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.896563] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.896745] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f15d095831a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f15de4b0266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f15de4acea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:02:27.896756] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:02:27.896777] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.896997] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.897370] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:02:27.904303] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.904888] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f15d095831a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f15de4b0266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f15de4acea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:02:27.904903] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:02:27.904917] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.906191] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:02:27.906211] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:02:27.906781] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:02:27.916253] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:02:27.920891] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.920920] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.920947] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:02:27.920961] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
[2018-09-01 11:02:27.922442] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:02:27.928707] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
[2018-09-01 11:02:27.928713] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.928761] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.928822] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.929033] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.929081] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:d7fd7492-3c37-449e-807b-c6244af7848b-GRAPH_ID:0-PID:1789-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:d7fd7492-3c37-449e-807b-c6244af7848b-GRAPH_ID:0-PID:1789-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option ping-timeout 42
 50:     option remote-host urd-gds-000
 51:     option remote-subvolume /urd-gds/gluster1
 52:     option transport-type socket
 53:     option transport.address-family inet
 54:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 55:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 56:     option transport.tcp-user-timeout 0
 57:     option transport.socket.keepalive-time 20
 58:     option transport.socket.keepalive-interval 2
 59:     option transport.socket.keepalive-count 9
 60:     option send-gids true
 61: end-volume
 62:  
 63: volume urd-gds-volume-replicate-0
 64:     type cluster/replicate
 65:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 66:     option arbiter-count 1
 67:     option use-compound-fops off
 68:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 69: end-volume
 70:  
 71: volume urd-gds-volume-client-3
 72:     type protocol/client
 73:     option ping-timeout 42
 74:     option remote-host urd-gds-003
 75:     option remote-subvolume /urd-gds/gluster
 76:     option transport-type socket
 77:     option transport.address-family inet
 78:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 79:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 80:     option transport.tcp-user-timeout 0
 81:     option transport.socket.keepalive-time 20
 82:     option transport.socket.keepalive-interval 2
 83:     option transport.socket.keepalive-count 9
 84:     option send-gids true
 85: end-volume
 86:  
 87: volume urd-gds-volume-client-4
 88:     type protocol/client
 89:     option ping-timeout 42
 90:     option remote-host urd-gds-004
 91:     option remote-subvolume /urd-gds/gluster
 92:     option transport-type socket
 93:     option transport.address-family inet
 94:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 95:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 96:     option transport.tcp-user-timeout 0
 97:     option transport.socket.keepalive-time 20
 98:     option transport.socket.keepalive-interval 2
 99:     option transport.socket.keepalive-count 9
100:     option send-gids true
101: end-volume
102:  
103: volume urd-gds-volume-client-5
104:     type protocol/client
105:     option ping-timeout 42
106:     option remote-host urd-gds-000
107:     option remote-subvolume /urd-gds/gluster2
108:     option transport-type socket
109:     option transport.address-family inet
110:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
111:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
112:     option transport.tcp-user-timeout 0
113:     option transport.socket.keepalive-time 20
114:     option transport.socket.keepalive-interval 2
115:     option transport.socket.keepalive-count 9
116:     option send-gids true
117: end-volume
118:  
119: volume urd-gds-volume-replicate-1
120:     type cluster/replicate
121:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
122:     option arbiter-count 1
123:     option use-compound-fops off
124:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
125: end-volume
126:  
127: volume urd-gds-volume-dht
128:     type cluster/distribute
129:     option lock-migration off
130:     option force-migration off
131:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
132: end-volume
133:  
134: volume urd-gds-volume-write-behind
135:     type performance/write-behind
136:     subvolumes urd-gds-volume-dht
137: end-volume
138:  
139: volume urd-gds-volume-read-ahead
140:     type performance/read-ahead
141:     subvolumes urd-gds-volume-write-behind
142: end-volume
143:  
144: volume urd-gds-volume-readdir-ahead
145:     type performance/readdir-ahead
146:     option parallel-readdir off
147:     option rda-request-size 131072
148:     option rda-cache-limit 10MB
149:     subvolumes urd-gds-volume-read-ahead
150: end-volume
151:  
152: volume urd-gds-volume-io-cache
153:     type performance/io-cache
154:     subvolumes urd-gds-volume-readdir-ahead
155: end-volume
156:  
157: volume urd-gds-volume-quick-read
158:     type performance/quick-read
159:     subvolumes urd-gds-volume-io-cache
160: end-volume
161:  
162: volume urd-gds-volume-open-behind
163:     type performance/open-behind
164:     subvolumes urd-gds-volume-quick-read
165: end-volume
166:  
167: volume urd-gds-volume-md-cache
168:     type performance/md-cache
169:     option cache-posix-acl true
170:     subvolumes urd-gds-volume-open-behind
171: end-volume
172:  
173: volume urd-gds-volume-io-threads
174:     type performance/io-threads
175:     subvolumes urd-gds-volume-md-cache
176: end-volume
177:  
178: volume urd-gds-volume-snapd-client
179:     type protocol/client
180:     option remote-host localhost
181:     option ping-timeout 42
182:     option remote-subvolume snapd-urd-gds-volume
183:     option transport-type socket
184:     option transport.address-family inet
185:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
186:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
187:     option transport.tcp-user-timeout 0
188:     option transport.socket.keepalive-time 20
189:     option transport.socket.keepalive-interval 2
190:     option transport.socket.keepalive-count 9
191:     option send-gids true
192: end-volume
193:  
194: volume urd-gds-volume-snapview-client
195:     type features/snapview-client
196:     option snapshot-directory .snaps
197:     option show-snapshot-directory on
198:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
199: end-volume
200:  
201: volume urd-gds-volume
202:     type debug/io-stats
203:     option log-level INFO
204:     option latency-measurement off
205:     option count-fop-hits off
206:     subvolumes urd-gds-volume-snapview-client
207: end-volume
208:  
209: volume posix-acl-autoload
210:     type system/posix-acl
211:     subvolumes urd-gds-volume
212: end-volume
213:  
214: volume gfid-access-autoload
215:     type features/gfid-access
216:     subvolumes posix-acl-autoload
217: end-volume
218:  
219: volume meta-autoload
220:     type meta
221:     subvolumes gfid-access-autoload
222: end-volume
223:  
+------------------------------------------------------------------------------+
[2018-09-01 11:02:27.935194] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:02:27.935226] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.935257] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.935288] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.936722] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.936763] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
[2018-09-01 11:02:27.936784] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:02:27.941111] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:02:27.943289] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.945470] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.945500] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.945567] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f15d095831a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f15de4b0266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f15de4acea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:02:27.945579] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:02:27.945594] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.945620] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.945725] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f15d095831a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f15de4b0266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f15de4acea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:02:27.945736] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:02:27.945753] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.945873] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f15d095831a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f15de4b0266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f15de4acea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:02:27.945883] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:02:27.945899] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.945948] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f15d095831a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f15de4b0266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f15de4acea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:02:27.945957] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:02:27.945972] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.948840] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.949319] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f15d095831a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f15de4b0266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f15de4acea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:02:27.949332] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:02:27.949353] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:02:27.954113] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:02:27.959986] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:02:27.960499] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:02:27.960522] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online.
[2018-09-01 11:02:27.961434] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:02:27.964268] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:02:27.966557] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:02:27.968468] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:02:27.968492] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:02:27.970418] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:02:27.971421] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:02:27.972920] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:02:49.659252] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:03:34.987186] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:03:58.547623] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-EQhDVc
[2018-09-01 11:03:58.547717] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f15dd31be25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55da363dad65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55da363dab8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:03:58.547738] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-EQhDVc'.
[2018-09-01 11:03:58.547747] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-EQhDVc'.
[2018-09-01 11:04:10.302046] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-4RS1ta)
[2018-09-01 11:04:10.312730] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:04:10.335112] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:04:10.335133] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:04:10.337648] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:04:10.342789] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:04:10.342850] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.343182] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.343248] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:04:10.349292] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:04:10.351310] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.353705] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.354014] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f1b817db31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f1b8f333266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f1b8f32fea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:04:10.354031] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:04:10.354058] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.354097] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.354395] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:04:10.360112] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:04:10.363978] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.371118] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:04:10.371137] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:04:10.375918] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.380894] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:04:10.380920] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.380962] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.381333] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f1b817db31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f1b8f333266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f1b8f32fea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:04:10.381354] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:04:10.381385] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.381443] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.384441] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:04:10.387193] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:04:10.391719] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:04:10.393906] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.393992] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.398726] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
[2018-09-01 11:04:10.403222] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.403974] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f1b817db31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f1b8f333266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f1b8f32fea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:04:10.404020] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:04:10.404059] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.404151] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:04:10.404177] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
[2018-09-01 11:04:10.410512] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:5463b612-fc83-4113-9ca6-b7f043ca7051-GRAPH_ID:0-PID:31106-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
[2018-09-01 11:04:10.414100] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
[2018-09-01 11:04:10.414194] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
[2018-09-01 11:04:10.414299] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:5463b612-fc83-4113-9ca6-b7f043ca7051-GRAPH_ID:0-PID:31106-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option opversion 40100
 50:     option clnt-lk-version 1
 51:     option volfile-checksum 0
 52:     option volfile-key urd-gds-volume
 53:     option client-version 4.1.3
 54:     option process-uuid CTX_ID:5463b612-fc83-4113-9ca6-b7f043ca7051-GRAPH_ID:0-PID:31106-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-2-RECON_NO:-0
 55:     option fops-version 1298437
 56:     option ping-timeout 42
 57:     option remote-host urd-gds-000
 58:     option remote-subvolume /urd-gds/gluster1
 59:     option transport-type socket
 60:     option transport.address-family inet
 61:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 62:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 63:     option transport.tcp-user-timeout 0
 64:     option transport.socket.keepalive-time 20
 65:     option transport.socket.keepalive-interval 2
 66:     option transport.socket.keepalive-count 9
 67:     option send-gids true
 68: end-volume
 69:  
 70: volume urd-gds-volume-replicate-0
 71:     type cluster/replicate
 72:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 73:     option arbiter-count 1
 74:     option use-compound-fops off
 75:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 76: end-volume
 77:  
 78: volume urd-gds-volume-client-3
 79:     type protocol/client
 80:     option ping-timeout 42
 81:     option remote-host urd-gds-003
 82:     option remote-subvolume /urd-gds/gluster
 83:     option transport-type socket
 84:     option transport.address-family inet
 85:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 86:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 87:     option transport.tcp-user-timeout 0
 88:     option transport.socket.keepalive-time 20
 89:     option transport.socket.keepalive-interval 2
 90:     option transport.socket.keepalive-count 9
 91:     option send-gids true
 92: end-volume
 93:  
 94: volume urd-gds-volume-client-4
 95:     type protocol/client
 96:     option ping-timeout 42
 97:     option remote-host urd-gds-004
 98:     option remote-subvolume /urd-gds/gluster
 99:     option transport-type socket
100:     option transport.address-family inet
101:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
102:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
103:     option transport.tcp-user-timeout 0
104:     option transport.socket.keepalive-time 20
105:     option transport.socket.keepalive-interval 2
106:     option transport.socket.keepalive-count 9
107:     option send-gids true
108: end-volume
109:  
110: volume urd-gds-volume-client-5
111:     type protocol/client
112:     option ping-timeout 42
113:     option remote-host urd-gds-000
114:     option remote-subvolume /urd-gds/gluster2
115:     option transport-type socket
116:     option transport.address-family inet
117:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
118:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
119:     option transport.tcp-user-timeout 0
120:     option transport.socket.keepalive-time 20
121:     option transport.socket.keepalive-interval 2
122:     option transport.socket.keepalive-count 9
123:     option send-gids true
124: end-volume
125:  
126: volume urd-gds-volume-replicate-1
127:     type cluster/replicate
128:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
129:     option arbiter-count 1
130:     option use-compound-fops off
131:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
132: end-volume
133:  
134: volume urd-gds-volume-dht
135:     type cluster/distribute
136:     option lock-migration off
137:     option force-migration off
138:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
139: end-volume
140:  
141: volume urd-gds-volume-write-behind
142:     type performance/write-behind
143:     subvolumes urd-gds-volume-dht
144: end-volume
145:  
146: volume urd-gds-volume-read-ahead
147:     type performance/read-ahead
148:     subvolumes urd-gds-volume-write-behind
149: end-volume
150:  
151: volume urd-gds-volume-readdir-ahead
152:     type performance/readdir-ahead
153:     option parallel-readdir off
154:     option rda-request-size 131072
155:     option rda-cache-limit 10MB
156:     subvolumes urd-gds-volume-read-ahead
157: end-volume
158:  
159: volume urd-gds-volume-io-cache
160:     type performance/io-cache
161:     subvolumes urd-gds-volume-readdir-ahead
162: end-volume
163:  
164: volume urd-gds-volume-quick-read
165:     type performance/quick-read
166:     subvolumes urd-gds-volume-io-cache
167: end-volume
168:  
169: volume urd-gds-volume-open-behind
170:     type performance/open-behind
171:     subvolumes urd-gds-volume-quick-read
172: end-volume
173:  
174: volume urd-gds-volume-md-cache
175:     type performance/md-cache
176:     option cache-posix-acl true
177:     subvolumes urd-gds-volume-open-behind
178: end-volume
179:  
180: volume urd-gds-volume-io-threads
181:     type performance/io-threads
182:     subvolumes urd-gds-volume-md-cache
183: end-volume
184:  
185: volume urd-gds-volume-snapd-client
186:     type protocol/client
187:     option remote-host localhost
188:     option ping-timeout 42
189:     option remote-subvolume snapd-urd-gds-volume
190:     option transport-type socket
191:     option transport.address-family inet
192:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
193:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
194:     option transport.tcp-user-timeout 0
195:     option transport.socket.keepalive-time 20
196:     option transport.socket.keepalive-interval 2
197:     option transport.socket.keepalive-count 9
198:     option send-gids true
199: end-volume
200:  
201: volume urd-gds-volume-snapview-client
202:     type features/snapview-client
203:     option snapshot-directory .snaps
204:     option show-snapshot-directory on
205:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
206: end-volume
207:  
208: volume urd-gds-volume
209:     type debug/io-stats
210:     option log-level INFO
211:     option latency-measurement off
212:     option count-fop-hits off
213:     subvolumes urd-gds-volume-snapview-client
214: end-volume
215:  
216: volume posix-acl-autoload
217:     type system/posix-acl
218:     subvolumes urd-gds-volume
219: end-volume
220:  
221: volume gfid-access-autoload
222:     type features/gfid-access
223:     subvolumes posix-acl-autoload
224: end-volume
225:  
226: volume meta-autoload
227:     type meta
228:     subvolumes gfid-access-autoload
229: end-volume
230:  
+------------------------------------------------------------------------------+
[2018-09-01 11:04:10.423007] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.443109] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:04:10.443125] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f1b817db31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f1b8f333266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f1b8f32fea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:04:10.443187] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:04:10.443253] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.443433] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.443616] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.443671] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:04:10.443696] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:04:10.448415] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f1b817db31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f1b8f333266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f1b8f32fea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:04:10.453199] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.458339] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:04:10.458459] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.458548] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:04:10.458550] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.458603] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online.
[2018-09-01 11:04:10.459154] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f1b817db31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f1b8f333266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f1b8f32fea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:04:10.459187] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:04:10.459224] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.471069] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:04:10.472512] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:04:10.472518] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:04:10.480983] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:04:10.482385] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:04:10.486462] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:04:10.488202] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:04:10.489774] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:04:10.502823] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f1b817db31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f1b8f333266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f1b8f32fea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:04:10.502903] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:04:10.502964] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:04:10.522815] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:04:27.544493] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:04:48.325685] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:05:06.037491] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-4RS1ta
[2018-09-01 11:05:06.037608] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f1b8e19ee25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x56072577cd65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x56072577cb8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:05:06.037622] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-4RS1ta'.
[2018-09-01 11:05:06.037629] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-4RS1ta'.
[2018-09-01 11:05:18.115490] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-jSGiFV)
[2018-09-01 11:05:18.124271] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:05:18.126546] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:05:18.126584] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:05:18.128753] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:05:18.132229] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:05:18.132279] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.132388] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.132449] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:05:18.143566] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:05:18.149436] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.149462] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.149671] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f174559431a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f17530ec266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f17530e8ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:05:18.149682] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:05:18.149703] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.149817] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.150126] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:05:18.151184] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:05:18.156332] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.156548] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:05:18.156567] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:05:18.158125] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:05:18.160194] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.160358] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.160466] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.160870] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f174559431a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f17530ec266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f17530e8ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:05:18.160914] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:05:18.160980] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.161087] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.161168] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:05:18.168650] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:05:18.174438] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:05:18.174465] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.174494] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:05:18.174504] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
[2018-09-01 11:05:18.176120] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
[2018-09-01 11:05:18.181209] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.181242] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.181263] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.181355] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.181570] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
[2018-09-01 11:05:18.181637] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f174559431a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f17530ec266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f17530e8ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:05:18.181646] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:05:18.181671] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:9c98908b-9f87-4984-8c50-35a5feb5c749-GRAPH_ID:0-PID:23793-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:9c98908b-9f87-4984-8c50-35a5feb5c749-GRAPH_ID:0-PID:23793-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option ping-timeout 42
 50:     option remote-host urd-gds-000
 51:     option remote-subvolume /urd-gds/gluster1
 52:     option transport-type socket
 53:     option transport.address-family inet
 54:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 55:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 56:     option transport.tcp-user-timeout 0
 57:     option transport.socket.keepalive-time 20
 58:     option transport.socket.keepalive-interval 2
 59:     option transport.socket.keepalive-count 9
 60:     option send-gids true
 61: end-volume
 62:  
 63: volume urd-gds-volume-replicate-0
 64:     type cluster/replicate
 65:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 66:     option arbiter-count 1
 67:     option use-compound-fops off
 68:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 69: end-volume
 70:  
 71: volume urd-gds-volume-client-3
 72:     type protocol/client
 73:     option opversion 40100
 74:     option clnt-lk-version 1
 75:     option volfile-checksum 0
 76:     option volfile-key urd-gds-volume
 77:     option client-version 4.1.3
 78:     option process-uuid CTX_ID:9c98908b-9f87-4984-8c50-35a5feb5c749-GRAPH_ID:0-PID:23793-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-3-RECON_NO:-0
 79:     option fops-version 1298437
 80:     option ping-timeout 42
 81:     option remote-host urd-gds-003
 82:     option remote-subvolume /urd-gds/gluster
 83:     option transport-type socket
 84:     option transport.address-family inet
 85:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 86:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 87:     option transport.tcp-user-timeout 0
 88:     option transport.socket.keepalive-time 20
 89:     option transport.socket.keepalive-interval 2
 90:     option transport.socket.keepalive-count 9
 91:     option send-gids true
 92: end-volume
 93:  
 94: volume urd-gds-volume-client-4
 95:     type protocol/client
 96:     option ping-timeout 42
 97:     option remote-host urd-gds-004
 98:     option remote-subvolume /urd-gds/gluster
 99:     option transport-type socket
100:     option transport.address-family inet
101:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
102:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
103:     option transport.tcp-user-timeout 0
104:     option transport.socket.keepalive-time 20
105:     option transport.socket.keepalive-interval 2
106:     option transport.socket.keepalive-count 9
107:     option send-gids true
108: end-volume
109:  
110: volume urd-gds-volume-client-5
111:     type protocol/client
112:     option ping-timeout 42
113:     option remote-host urd-gds-000
114:     option remote-subvolume /urd-gds/gluster2
115:     option transport-type socket
116:     option transport.address-family inet
117:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
118:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
119:     option transport.tcp-user-timeout 0
120:     option transport.socket.keepalive-time 20
121:     option transport.socket.keepalive-interval 2
122:     option transport.socket.keepalive-count 9
123:     option send-gids true
124: end-volume
125:  
126: volume urd-gds-volume-replicate-1
127:     type cluster/replicate
128:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
129:     option arbiter-count 1
130:     option use-compound-fops off
131:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
132: end-volume
133:  
134: volume urd-gds-volume-dht
135:     type cluster/distribute
136:     option lock-migration off
137:     option force-migration off
138:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
139: end-volume
140:  
141: volume urd-gds-volume-write-behind
142:     type performance/write-behind
143:     subvolumes urd-gds-volume-dht
144: end-volume
145:  
146: volume urd-gds-volume-read-ahead
147:     type performance/read-ahead
148:     subvolumes urd-gds-volume-write-behind
149: end-volume
150:  
151: volume urd-gds-volume-readdir-ahead
152:     type performance/readdir-ahead
153:     option parallel-readdir off
154:     option rda-request-size 131072
155:     option rda-cache-limit 10MB
156:     subvolumes urd-gds-volume-read-ahead
157: end-volume
158:  
159: volume urd-gds-volume-io-cache
160:     type performance/io-cache
161:     subvolumes urd-gds-volume-readdir-ahead
162: end-volume
163:  
164: volume urd-gds-volume-quick-read
165:     type performance/quick-read
166:     subvolumes urd-gds-volume-io-cache
167: end-volume
168:  
169: volume urd-gds-volume-open-behind
170:     type performance/open-behind
171:     subvolumes urd-gds-volume-quick-read
172: end-volume
173:  
174: volume urd-gds-volume-md-cache
175:     type performance/md-cache
176:     option cache-posix-acl true
177:     subvolumes urd-gds-volume-open-behind
178: end-volume
179:  
180: volume urd-gds-volume-io-threads
181:     type performance/io-threads
182:     subvolumes urd-gds-volume-md-cache
183: end-volume
184:  
185: volume urd-gds-volume-snapd-client
186:     type protocol/client
187:     option remote-host localhost
188:     option ping-timeout 42
189:     option remote-subvolume snapd-urd-gds-volume
190:     option transport-type socket
191:     option transport.address-family inet
192:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
193:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
194:     option transport.tcp-user-timeout 0
195:     option transport.socket.keepalive-time 20
196:     option transport.socket.keepalive-interval 2
197:     option transport.socket.keepalive-count 9
198:     option send-gids true
199: end-volume
200:  
201: volume urd-gds-volume-snapview-client
202:     type features/snapview-client
203:     option snapshot-directory .snaps
204:     option show-snapshot-directory on
205:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
206: end-volume
207:  
208: volume urd-gds-volume
209:     type debug/io-stats
210:     option log-level INFO
211:     option latency-measurement off
212:     option count-fop-hits off
213:     subvolumes urd-gds-volume-snapview-client
214: end-volume
215:  
216: volume posix-acl-autoload
217:     type system/posix-acl
218:     subvolumes urd-gds-volume
219: end-volume
220:  
221: volume gfid-access-autoload
222:     type features/gfid-access
223:     subvolumes posix-acl-autoload
224: end-volume
225:  
226: volume meta-autoload
227:     type meta
228:     subvolumes gfid-access-autoload
229: end-volume
230:  
+------------------------------------------------------------------------------+
[2018-09-01 11:05:18.183542] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.183614] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f174559431a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f17530ec266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f17530e8ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:05:18.183630] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:05:18.183658] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.183687] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.183779] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.183876] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:05:18.183923] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:05:18.187311] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.187474] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f174559431a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f17530ec266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f17530e8ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:05:18.187486] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:05:18.187498] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.187563] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.188032] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f174559431a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f17530ec266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f17530e8ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:05:18.188040] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:05:18.188053] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.191277] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:05:18.191293] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online.
[2018-09-01 11:05:18.193937] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.194434] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f174559431a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f17530ec266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f17530e8ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:05:18.194447] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:05:18.194460] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:05:18.195706] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:05:18.198517] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:05:18.200474] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:05:18.200489] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:05:18.203090] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:05:18.204510] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:05:18.204528] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:05:18.206264] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:05:18.207374] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:05:18.208873] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:05:35.318698] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:06:24.397995] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:06:49.606451] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-jSGiFV
[2018-09-01 11:06:49.606551] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f1751f57e25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x562914810d65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x562914810b8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:06:49.606565] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-jSGiFV'.
[2018-09-01 11:06:49.606571] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-jSGiFV'.
[2018-09-01 11:07:01.351280] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-66x_KB)
[2018-09-01 11:07:01.360525] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:07:01.362792] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:07:01.362795] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:07:01.365021] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:07:01.368743] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:07:01.368783] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.368906] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.369002] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:07:01.373186] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:07:01.375126] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.375157] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.375451] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7facc53cf31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7facd2f27266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7facd2f23ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:07:01.375467] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:07:01.375500] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.375571] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.375819] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:07:01.376957] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:07:01.382191] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.382334] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:07:01.382347] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:07:01.383894] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:07:01.385555] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.385588] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.385606] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.385848] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.385988] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7facc53cf31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7facd2f27266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7facd2f23ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:07:01.386003] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:07:01.386016] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.386038] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:07:01.390009] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:07:01.392637] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:07:01.392662] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.394753] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:5c86dd9f-fc4c-4ee4-81ed-f792f2bec535-GRAPH_ID:0-PID:9456-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:5c86dd9f-fc4c-4ee4-81ed-f792f2bec535-GRAPH_ID:0-PID:9456-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option ping-timeout 42
 50:     option remote-host urd-gds-000
 51:     option remote-subvolume /urd-gds/gluster1
 52:     option transport-type socket
 53:     option transport.address-family inet
 54:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 55:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 56:     option transport.tcp-user-timeout 0
 57:     option transport.socket.keepalive-time 20
 58:     option transport.socket.keepalive-interval 2
 59:     option transport.socket.keepalive-count 9
 60:     option send-gids true
 61: end-volume
 62:  
 63: volume urd-gds-volume-replicate-0
 64:     type cluster/replicate
 65:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 66:     option arbiter-count 1
 67:     option use-compound-fops off
 68:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 69: end-volume
 70:  
 71: volume urd-gds-volume-client-3
 72:     type protocol/client
 73:     option ping-timeout 42
 74:     option remote-host urd-gds-003
 75:     option remote-subvolume /urd-gds/gluster
 76:     option transport-type socket
 77:     option transport.address-family inet
 78:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 79:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 80:     option transport.tcp-user-timeout 0
 81:     option transport.socket.keepalive-time 20
 82:     option transport.socket.keepalive-interval 2
 83:     option transport.socket.keepalive-count 9
 84:     option send-gids true
 85: end-volume
 86:  
 87: volume urd-gds-volume-client-4
 88:     type protocol/client
 89:     option ping-timeout 42
 90:     option remote-host urd-gds-004
 91:     option remote-subvolume /urd-gds/gluster
 92:     option transport-type socket
 93:     option transport.address-family inet
 94:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 95:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 96:     option transport.tcp-user-timeout 0
 97:     option transport.socket.keepalive-time 20
 98:     option transport.socket.keepalive-interval 2
 99:     option transport.socket.keepalive-count 9
100:     option send-gids true
101: end-volume
102:  
103: volume urd-gds-volume-client-5
104:     type protocol/client
105:     option ping-timeout 42
106:     option remote-host urd-gds-000
107:     option remote-subvolume /urd-gds/gluster2
108:     option transport-type socket
109:     option transport.address-family inet
110:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
111:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
112:     option transport.tcp-user-timeout 0
113:     option transport.socket.keepalive-time 20
114:     option transport.socket.keepalive-interval 2
115:     option transport.socket.keepalive-count 9
116:     option send-gids true
117: end-volume
118:  
119: volume urd-gds-volume-replicate-1
120:     type cluster/replicate
121:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
122:     option arbiter-count 1
123:     option use-compound-fops off
124:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
125: end-volume
126:  
127: volume urd-gds-volume-dht
128:     type cluster/distribute
129:     option lock-migration off
130:     option force-migration off
131:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
132: end-volume
133:  
134: volume urd-gds-volume-write-behind
135:     type performance/write-behind
136:     subvolumes urd-gds-volume-dht
137: end-volume
138:  
139: volume urd-gds-volume-read-ahead
140:     type performance/read-ahead
141:     subvolumes urd-gds-volume-write-behind
142: end-volume
143:  
144: volume urd-gds-volume-readdir-ahead
145:     type performance/readdir-ahead
146:     option parallel-readdir off
147:     option rda-request-size 131072
148:     option rda-cache-limit 10MB
149:     subvolumes urd-gds-volume-read-ahead
150: end-volume
151:  
152: volume urd-gds-volume-io-cache
153:     type performance/io-cache
154:     subvolumes urd-gds-volume-readdir-ahead
155: end-volume
156:  
157: volume urd-gds-volume-quick-read
158:     type performance/quick-read
159:     subvolumes urd-gds-volume-io-cache
160: end-volume
161:  
162: volume urd-gds-volume-open-behind
163:     type performance/open-behind
164:     subvolumes urd-gds-volume-quick-read
165: end-volume
166:  
167: volume urd-gds-volume-md-cache
168:     type performance/md-cache
169:     option cache-posix-acl true
170:     subvolumes urd-gds-volume-open-behind
171: end-volume
172:  
173: volume urd-gds-volume-io-threads
174:     type performance/io-threads
175:     subvolumes urd-gds-volume-md-cache
176: end-volume
177:  
178: volume urd-gds-volume-snapd-client
179:     type protocol/client
180:     option remote-host localhost
181:     option ping-timeout 42
182:     option remote-subvolume snapd-urd-gds-volume
183:     option transport-type socket
184:     option transport.address-family inet
185:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
186:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
187:     option transport.tcp-user-timeout 0
188:     option transport.socket.keepalive-time 20
189:     option transport.socket.keepalive-interval 2
190:     option transport.socket.keepalive-count 9
191:     option send-gids true
192: end-volume
193:  
194: volume urd-gds-volume-snapview-client
195:     type features/snapview-client
196:     option snapshot-directory .snaps
197:     option show-snapshot-directory on
198:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
199: end-volume
200:  
201: volume urd-gds-volume
202:     type debug/io-stats
203:     option log-level INFO
204:     option latency-measurement off
205:     option count-fop-hits off
206:     subvolumes urd-gds-volume-snapview-client
207: end-volume
208:  
209: volume posix-acl-autoload
210:     type system/posix-acl
211:     subvolumes urd-gds-volume
212: end-volume
213:  
214: volume gfid-access-autoload
215:     type features/gfid-access
216:     subvolumes posix-acl-autoload
217: end-volume
218:  
219: volume meta-autoload
220:     type meta
221:     subvolumes gfid-access-autoload
222: end-volume
223:  
+------------------------------------------------------------------------------+
[2018-09-01 11:07:01.403095] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.403144] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.403184] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.403227] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:07:01.403246] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
[2018-09-01 11:07:01.404536] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.404617] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7facc53cf31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7facd2f27266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7facd2f23ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:07:01.404632] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:07:01.404647] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.404669] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
[2018-09-01 11:07:01.404694] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.404721] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.410415] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.410452] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:07:01.410487] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:07:01.414516] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7facc53cf31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7facd2f27266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7facd2f23ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:07:01.414533] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:07:01.414554] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.414582] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.414732] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.415206] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7facc53cf31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7facd2f27266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7facd2f23ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:07:01.415217] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:07:01.415234] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.417015] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:07:01.417033] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online.
[2018-09-01 11:07:01.419756] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7facc53cf31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7facd2f27266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7facd2f23ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:07:01.419774] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:07:01.419793] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.419997] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.420557] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7facc53cf31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7facd2f27266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7facd2f23ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:07:01.420576] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:07:01.420601] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:07:01.431260] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:07:01.434029] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:07:01.434061] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:07:01.438323] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:07:01.438514] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:07:01.440080] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:07:01.440120] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:07:01.442579] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:07:01.443746] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:07:01.445203] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:07:22.530107] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:08:13.246630] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:08:30.609168] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-66x_KB
[2018-09-01 11:08:30.609261] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7facd1d92e25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x563a23bdad65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x563a23bdab8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:08:30.609274] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-66x_KB'.
[2018-09-01 11:08:30.609279] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-66x_KB'.
[2018-09-01 11:08:42.320618] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-ToN7eW)
[2018-09-01 11:08:42.330075] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:08:42.332391] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:08:42.332446] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:08:42.334595] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:08:42.338174] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:08:42.338224] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.338332] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.338385] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:08:42.342894] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:08:42.344666] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.344692] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.344881] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fef180d231a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fef25c2a266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fef25c26ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:08:42.344892] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:08:42.344914] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.345081] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.345431] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:08:42.346515] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:08:42.351750] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.351794] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:08:42.351809] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:08:42.353514] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:08:42.354707] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.354749] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.354773] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.355163] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fef180d231a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fef25c2a266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fef25c26ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:08:42.355176] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:08:42.355196] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.355230] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.355250] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:08:42.357958] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:08:42.363850] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:08:42.363880] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.363911] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:08:42.363923] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
[2018-09-01 11:08:42.365908] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:68991020-48a4-4c9e-9c29-e9c1dc77c54a-GRAPH_ID:0-PID:1662-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:68991020-48a4-4c9e-9c29-e9c1dc77c54a-GRAPH_ID:0-PID:1662-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option ping-timeout 42
 50:     option remote-host urd-gds-000
 51:     option remote-subvolume /urd-gds/gluster1
 52:     option transport-type socket
 53:     option transport.address-family inet
 54:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 55:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 56:     option transport.tcp-user-timeout 0
 57:     option transport.socket.keepalive-time 20
 58:     option transport.socket.keepalive-interval 2
 59:     option transport.socket.keepalive-count 9
 60:     option send-gids true
 61: end-volume
 62:  
 63: volume urd-gds-volume-replicate-0
 64:     type cluster/replicate
 65:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 66:     option arbiter-count 1
 67:     option use-compound-fops off
 68:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 69: end-volume
 70:  
 71: volume urd-gds-volume-client-3
 72:     type protocol/client
 73:     option ping-timeout 42
 74:     option remote-host urd-gds-003
 75:     option remote-subvolume /urd-gds/gluster
 76:     option transport-type socket
 77:     option transport.address-family inet
 78:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 79:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 80:     option transport.tcp-user-timeout 0
 81:     option transport.socket.keepalive-time 20
 82:     option transport.socket.keepalive-interval 2
 83:     option transport.socket.keepalive-count 9
 84:     option send-gids true
 85: end-volume
 86:  
 87: volume urd-gds-volume-client-4
 88:     type protocol/client
 89:     option ping-timeout 42
 90:     option remote-host urd-gds-004
 91:     option remote-subvolume /urd-gds/gluster
 92:     option transport-type socket
 93:     option transport.address-family inet
 94:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 95:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 96:     option transport.tcp-user-timeout 0
 97:     option transport.socket.keepalive-time 20
 98:     option transport.socket.keepalive-interval 2
 99:     option transport.socket.keepalive-count 9
100:     option send-gids true
101: end-volume
102:  
103: volume urd-gds-volume-client-5
104:     type protocol/client
105:     option ping-timeout 42
106:     option remote-host urd-gds-000
107:     option remote-subvolume /urd-gds/gluster2
108:     option transport-type socket
109:     option transport.address-family inet
110:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
111:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
112:     option transport.tcp-user-timeout 0
113:     option transport.socket.keepalive-time 20
114:     option transport.socket.keepalive-interval 2
115:     option transport.socket.keepalive-count 9
116:     option send-gids true
117: end-volume
118:  
119: volume urd-gds-volume-replicate-1
120:     type cluster/replicate
121:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
122:     option arbiter-count 1
123:     option use-compound-fops off
124:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
125: end-volume
126:  
127: volume urd-gds-volume-dht
128:     type cluster/distribute
129:     option lock-migration off
130:     option force-migration off
131:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
132: end-volume
133:  
134: volume urd-gds-volume-write-behind
135:     type performance/write-behind
136:     subvolumes urd-gds-volume-dht
137: end-volume
138:  
139: volume urd-gds-volume-read-ahead
140:     type performance/read-ahead
141:     subvolumes urd-gds-volume-write-behind
142: end-volume
143:  
144: volume urd-gds-volume-readdir-ahead
145:     type performance/readdir-ahead
146:     option parallel-readdir off
147:     option rda-request-size 131072
148:     option rda-cache-limit 10MB
149:     subvolumes urd-gds-volume-read-ahead
150: end-volume
151:  
152: volume urd-gds-volume-io-cache
153:     type performance/io-cache
154:     subvolumes urd-gds-volume-readdir-ahead
155: end-volume
156:  
157: volume urd-gds-volume-quick-read
158:     type performance/quick-read
159:     subvolumes urd-gds-volume-io-cache
160: end-volume
161:  
162: volume urd-gds-volume-open-behind
163:     type performance/open-behind
164:     subvolumes urd-gds-volume-quick-read
165: end-volume
166:  
167: volume urd-gds-volume-md-cache
168:     type performance/md-cache
169:     option cache-posix-acl true
170:     subvolumes urd-gds-volume-open-behind
171: end-volume
172:  
173: volume urd-gds-volume-io-threads
174:     type performance/io-threads
175:     subvolumes urd-gds-volume-md-cache
176: end-volume
177:  
178: volume urd-gds-volume-snapd-client
179:     type protocol/client
180:     option remote-host localhost
181:     option ping-timeout 42
182:     option remote-subvolume snapd-urd-gds-volume
183:     option transport-type socket
184:     option transport.address-family inet
185:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
186:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
187:     option transport.tcp-user-timeout 0
188:     option transport.socket.keepalive-time 20
189:     option transport.socket.keepalive-interval 2
190:     option transport.socket.keepalive-count 9
191:     option send-gids true
192: end-volume
193:  
194: volume urd-gds-volume-snapview-client
195:     type features/snapview-client
196:     option snapshot-directory .snaps
197:     option show-snapshot-directory on
198:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
199: end-volume
200:  
201: volume urd-gds-volume
202:     type debug/io-stats
203:     option log-level INFO
204:     option latency-measurement off
205:     option count-fop-hits off
206:     subvolumes urd-gds-volume-snapview-client
207: end-volume
208:  
209: volume posix-acl-autoload
210:     type system/posix-acl
211:     subvolumes urd-gds-volume
212: end-volume
213:  
214: volume gfid-access-autoload
215:     type features/gfid-access
216:     subvolumes posix-acl-autoload
217: end-volume
218:  
219: volume meta-autoload
220:     type meta
221:     subvolumes gfid-access-autoload
222: end-volume
223:  
+------------------------------------------------------------------------------+
[2018-09-01 11:08:42.372991] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.373024] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.373052] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.373073] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.373194] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.373293] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:08:42.374481] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
[2018-09-01 11:08:42.374519] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.374582] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fef180d231a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fef25c2a266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fef25c26ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:08:42.374594] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:08:42.374609] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.376546] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.376580] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:08:42.376601] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.378714] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fef180d231a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fef25c2a266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fef25c26ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:08:42.378731] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:08:42.378751] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.378813] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fef180d231a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fef25c2a266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fef25c26ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:08:42.378823] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:08:42.378840] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.378932] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.379433] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fef180d231a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fef25c2a266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fef25c26ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:08:42.379450] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:08:42.379472] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.381087] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.381660] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fef180d231a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fef25c2a266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fef25c26ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:08:42.381676] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:08:42.381693] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:08:42.386076] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:08:42.392286] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:08:42.393297] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:08:42.393314] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online.
[2018-09-01 11:08:42.393979] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:08:42.397189] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:08:42.398337] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:08:42.400917] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:08:42.400940] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:08:42.402815] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:08:42.403782] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:08:42.405298] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:09:00.963010] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:09:51.040995] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:10:09.595572] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-ToN7eW
[2018-09-01 11:10:09.595683] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7fef24a95e25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55cc547ded65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55cc547deb8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:10:09.595703] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-ToN7eW'.
[2018-09-01 11:10:09.595709] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-ToN7eW'.
[2018-09-01 11:10:21.355748] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-GW9J4k)
[2018-09-01 11:10:21.365211] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:10:21.367537] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:10:21.367555] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:10:21.369723] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:10:21.373369] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:10:21.373442] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.373540] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.373590] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:10:21.378192] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:10:21.379987] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.380014] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.380203] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f2787da731a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f2799b72266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f2799b6eea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:10:21.380214] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:10:21.380236] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.380375] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.380701] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:10:21.381730] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:10:21.387032] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.387073] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:10:21.387085] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:10:21.388733] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:10:21.390018] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.390051] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.390072] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.390409] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f2787da731a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f2799b72266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f2799b6eea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:10:21.390420] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:10:21.390465] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.390490] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.390603] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:10:21.392781] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:10:21.395531] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:10:21.395556] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.397373] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
[2018-09-01 11:10:21.405726] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.405768] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.405795] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.405829] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:10:21.405842] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
[2018-09-01 11:10:21.408330] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.408363] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
[2018-09-01 11:10:21.408464] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f2787da731a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f2799b72266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f2799b6eea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:10:21.408478] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:10:21.408498] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.408528] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:3cbd6e36-d4fe-453b-889a-5f71581606d7-GRAPH_ID:0-PID:23037-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:3cbd6e36-d4fe-453b-889a-5f71581606d7-GRAPH_ID:0-PID:23037-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option opversion 40100
 50:     option clnt-lk-version 1
 51:     option volfile-checksum 0
 52:     option volfile-key urd-gds-volume
 53:     option client-version 4.1.3
 54:     option process-uuid CTX_ID:3cbd6e36-d4fe-453b-889a-5f71581606d7-GRAPH_ID:0-PID:23037-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-2-RECON_NO:-0
 55:     option fops-version 1298437
 56:     option ping-timeout 42
 57:     option remote-host urd-gds-000
 58:     option remote-subvolume /urd-gds/gluster1
 59:     option transport-type socket
 60:     option transport.address-family inet
 61:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 62:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 63:     option transport.tcp-user-timeout 0
 64:     option transport.socket.keepalive-time 20
 65:     option transport.socket.keepalive-interval 2
 66:     option transport.socket.keepalive-count 9
 67:     option send-gids true
 68: end-volume
 69:  
 70: volume urd-gds-volume-replicate-0
 71:     type cluster/replicate
 72:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 73:     option arbiter-count 1
 74:     option use-compound-fops off
 75:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 76: end-volume
 77:  
 78: volume urd-gds-volume-client-3
 79:     type protocol/client
 80:     option ping-timeout 42
 81:     option remote-host urd-gds-003
 82:     option remote-subvolume /urd-gds/gluster
 83:     option transport-type socket
 84:     option transport.address-family inet
 85:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 86:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 87:     option transport.tcp-user-timeout 0
 88:     option transport.socket.keepalive-time 20
 89:     option transport.socket.keepalive-interval 2
 90:     option transport.socket.keepalive-count 9
 91:     option send-gids true
 92: end-volume
 93:  
 94: volume urd-gds-volume-client-4
 95:     type protocol/client
 96:     option ping-timeout 42
 97:     option remote-host urd-gds-004
 98:     option remote-subvolume /urd-gds/gluster
 99:     option transport-type socket
100:     option transport.address-family inet
101:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
102:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
103:     option transport.tcp-user-timeout 0
104:     option transport.socket.keepalive-time 20
105:     option transport.socket.keepalive-interval 2
106:     option transport.socket.keepalive-count 9
107:     option send-gids true
108: end-volume
109:  
110: volume urd-gds-volume-client-5
111:     type protocol/client
112:     option ping-timeout 42
113:     option remote-host urd-gds-000
114:     option remote-subvolume /urd-gds/gluster2
115:     option transport-type socket
116:     option transport.address-family inet
117:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
118:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
119:     option transport.tcp-user-timeout 0
120:     option transport.socket.keepalive-time 20
121:     option transport.socket.keepalive-interval 2
122:     option transport.socket.keepalive-count 9
123:     option send-gids true
124: end-volume
125:  
126: volume urd-gds-volume-replicate-1
127:     type cluster/replicate
128:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
129:     option arbiter-count 1
130:     option use-compound-fops off
131:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
132: end-volume
133:  
134: volume urd-gds-volume-dht
135:     type cluster/distribute
136:     option lock-migration off
137:     option force-migration off
138:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
139: end-volume
140:  
141: volume urd-gds-volume-write-behind
142:     type performance/write-behind
143:     subvolumes urd-gds-volume-dht
144: end-volume
145:  
146: volume urd-gds-volume-read-ahead
147:     type performance/read-ahead
148:     subvolumes urd-gds-volume-write-behind
149: end-volume
150:  
151: volume urd-gds-volume-readdir-ahead
152:     type performance/readdir-ahead
153:     option parallel-readdir off
154:     option rda-request-size 131072
155:     option rda-cache-limit 10MB
156:     subvolumes urd-gds-volume-read-ahead
157: end-volume
158:  
159: volume urd-gds-volume-io-cache
160:     type performance/io-cache
161:     subvolumes urd-gds-volume-readdir-ahead
162: end-volume
163:  
164: volume urd-gds-volume-quick-read
165:     type performance/quick-read
166:     subvolumes urd-gds-volume-io-cache
167: end-volume
168:  
169: volume urd-gds-volume-open-behind
170:     type performance/open-behind
171:     subvolumes urd-gds-volume-quick-read
172: end-volume
173:  
174: volume urd-gds-volume-md-cache
175:     type performance/md-cache
176:     option cache-posix-acl true
177:     subvolumes urd-gds-volume-open-behind
178: end-volume
179:  
180: volume urd-gds-volume-io-threads
181:     type performance/io-threads
182:     subvolumes urd-gds-volume-md-cache
183: end-volume
184:  
185: volume urd-gds-volume-snapd-client
186:     type protocol/client
187:     option remote-host localhost
188:     option ping-timeout 42
189:     option remote-subvolume snapd-urd-gds-volume
190:     option transport-type socket
191:     option transport.address-family inet
192:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
193:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
194:     option transport.tcp-user-timeout 0
195:     option transport.socket.keepalive-time 20
196:     option transport.socket.keepalive-interval 2
197:     option transport.socket.keepalive-count 9
198:     option send-gids true
199: end-volume
200:  
201: volume urd-gds-volume-snapview-client
202:     type features/snapview-client
203:     option snapshot-directory .snaps
204:     option show-snapshot-directory on
205:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
206: end-volume
207:  
208: volume urd-gds-volume
209:     type debug/io-stats
210:     option log-level INFO
211:     option latency-measurement off
212:     option count-fop-hits off
213:     subvolumes urd-gds-volume-snapview-client
214: end-volume
215:  
216: volume posix-acl-autoload
217:     type system/posix-acl
218:     subvolumes urd-gds-volume
219: end-volume
220:  
221: volume gfid-access-autoload
222:     type features/gfid-access
223:     subvolumes posix-acl-autoload
224: end-volume
225:  
226: volume meta-autoload
227:     type meta
228:     subvolumes gfid-access-autoload
229: end-volume
230:  
+------------------------------------------------------------------------------+
[2018-09-01 11:10:21.409955] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f2787da731a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f2799b72266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f2799b6eea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:10:21.409964] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:10:21.409976] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.409996] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:10:21.410013] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.412036] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.412131] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:10:21.413730] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.414047] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.414105] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f2787da731a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f2799b72266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f2799b6eea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:10:21.414119] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:10:21.414133] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.414578] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f2787da731a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f2799b72266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f2799b6eea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:10:21.414592] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:10:21.414609] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.415358] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.415530] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f2787da731a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f2799b72266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f2799b6eea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:10:21.415546] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:10:21.415561] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:10:21.422876] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:10:21.422889] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online.
[2018-09-01 11:10:21.423779] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:10:21.428189] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:10:21.428212] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:10:21.428331] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:10:21.428438] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:10:21.431555] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:10:21.431570] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:10:21.433278] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:10:21.434392] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:10:21.435964] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:10:37.509470] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:11:33.000609] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:11:47.462062] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-GW9J4k
[2018-09-01 11:11:47.462179] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f27989dde25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x56070c891d65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x56070c891b8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:11:47.462199] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-GW9J4k'.
[2018-09-01 11:11:47.462207] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-GW9J4k'.
[2018-09-01 11:11:59.194635] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-2BrZK7)
[2018-09-01 11:11:59.204348] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:11:59.206679] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:11:59.206702] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:11:59.208907] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:11:59.214275] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:11:59.214339] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.214436] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.214482] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:11:59.220095] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:11:59.221899] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.221925] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.222112] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f9083a0a31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f9091562266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f909155eea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:11:59.222122] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:11:59.222145] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.222293] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.222592] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:11:59.223586] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:11:59.229958] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.230213] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.230400] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.230695] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f9083a0a31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f9091562266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f909155eea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:11:59.230709] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:11:59.230729] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.230766] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:11:59.230784] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:11:59.231777] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:11:59.238031] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:11:59.238066] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.239714] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:11:59.247242] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.247278] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.247547] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.247621] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:11:59.249286] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
[2018-09-01 11:11:59.255148] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.255230] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f9083a0a31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f9091562266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f909155eea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:11:59.255244] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:11:59.255261] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.255298] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:11:59.255312] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:f6f16c68-e80b-4fe4-b852-9b5c781f43ba-GRAPH_ID:0-PID:26853-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:f6f16c68-e80b-4fe4-b852-9b5c781f43ba-GRAPH_ID:0-PID:26853-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option opversion 40100
 50:     option clnt-lk-version 1
 51:     option volfile-checksum 0
 52:     option volfile-key urd-gds-volume
 53:     option client-version 4.1.3
 54:     option process-uuid CTX_ID:f6f16c68-e80b-4fe4-b852-9b5c781f43ba-GRAPH_ID:0-PID:26853-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-2-RECON_NO:-0
 55:     option fops-version 1298437
 56:     option ping-timeout 42
 57:     option remote-host urd-gds-000
 58:     option remote-subvolume /urd-gds/gluster1
 59:     option transport-type socket
 60:     option transport.address-family inet
 61:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 62:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 63:     option transport.tcp-user-timeout 0
 64:     option transport.socket.keepalive-time 20
 65:     option transport.socket.keepalive-interval 2
 66:     option transport.socket.keepalive-count 9
 67:     option send-gids true
 68: end-volume
 69:  
 70: volume urd-gds-volume-replicate-0
 71:     type cluster/replicate
 72:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 73:     option arbiter-count 1
 74:     option use-compound-fops off
 75:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 76: end-volume
 77:  
 78: volume urd-gds-volume-client-3
 79:     type protocol/client
 80:     option ping-timeout 42
 81:     option remote-host urd-gds-003
 82:     option remote-subvolume /urd-gds/gluster
 83:     option transport-type socket
 84:     option transport.address-family inet
 85:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 86:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 87:     option transport.tcp-user-timeout 0
 88:     option transport.socket.keepalive-time 20
 89:     option transport.socket.keepalive-interval 2
 90:     option transport.socket.keepalive-count 9
 91:     option send-gids true
 92: end-volume
 93:  
 94: volume urd-gds-volume-client-4
 95:     type protocol/client
 96:     option ping-timeout 42
 97:     option remote-host urd-gds-004
 98:     option remote-subvolume /urd-gds/gluster
 99:     option transport-type socket
100:     option transport.address-family inet
101:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
102:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
103:     option transport.tcp-user-timeout 0
104:     option transport.socket.keepalive-time 20
105:     option transport.socket.keepalive-interval 2
106:     option transport.socket.keepalive-count 9
107:     option send-gids true
108: end-volume
109:  
110: volume urd-gds-volume-client-5
111:     type protocol/client
112:     option ping-timeout 42
113:     option remote-host urd-gds-000
114:     option remote-subvolume /urd-gds/gluster2
115:     option transport-type socket
116:     option transport.address-family inet
117:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
118:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
119:     option transport.tcp-user-timeout 0
120:     option transport.socket.keepalive-time 20
121:     option transport.socket.keepalive-interval 2
122:     option transport.socket.keepalive-count 9
123:     option send-gids true
124: end-volume
125:  
126: volume urd-gds-volume-replicate-1
127:     type cluster/replicate
128:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
129:     option arbiter-count 1
130:     option use-compound-fops off
131:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
132: end-volume
133:  
134: volume urd-gds-volume-dht
135:     type cluster/distribute
136:     option lock-migration off
137:     option force-migration off
138:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
139: end-volume
140:  
141: volume urd-gds-volume-write-behind
142:     type performance/write-behind
143:     subvolumes urd-gds-volume-dht
144: end-volume
145:  
146: volume urd-gds-volume-read-ahead
147:     type performance/read-ahead
148:     subvolumes urd-gds-volume-write-behind
149: end-volume
150:  
151: volume urd-gds-volume-readdir-ahead
152:     type performance/readdir-ahead
153:     option parallel-readdir off
154:     option rda-request-size 131072
155:     option rda-cache-limit 10MB
156:     subvolumes urd-gds-volume-read-ahead
157: end-volume
158:  
159: volume urd-gds-volume-io-cache
160:     type performance/io-cache
161:     subvolumes urd-gds-volume-readdir-ahead
162: end-volume
163:  
164: volume urd-gds-volume-quick-read
165:     type performance/quick-read
166:     subvolumes urd-gds-volume-io-cache
167: end-volume
168:  
169: volume urd-gds-volume-open-behind
170:     type performance/open-behind
171:     subvolumes urd-gds-volume-quick-read
172: end-volume
173:  
174: volume urd-gds-volume-md-cache
175:     type performance/md-cache
176:     option cache-posix-acl true
177:     subvolumes urd-gds-volume-open-behind
178: end-volume
179:  
180: volume urd-gds-volume-io-threads
181:     type performance/io-threads
182:     subvolumes urd-gds-volume-md-cache
[2018-09-01 11:11:59.260879] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
183: end-volume
184:  
185: volume urd-gds-volume-snapd-client
186:     type protocol/client
187:     option remote-host localhost
188:     option ping-timeout 42
189:     option remote-subvolume snapd-urd-gds-volume
190:     option transport-type socket
191:     option transport.address-family inet
192:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
193:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
194:     option transport.tcp-user-timeout 0
195:     option transport.socket.keepalive-time 20
196:     option transport.socket.keepalive-interval 2
[2018-09-01 11:11:59.260925] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
197:     option transport.socket.keepalive-count 9
198:     option send-gids true
199: end-volume
200:  
201: volume urd-gds-volume-snapview-client
202:     type features/snapview-client
203:     option snapshot-directory .snaps
204:     option show-snapshot-directory on
205:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
206: end-volume
207:  
208: volume urd-gds-volume
[2018-09-01 11:11:59.260982] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
209:     type debug/io-stats
210:     option log-level INFO
211:     option latency-measurement off
212:     option count-fop-hits off
213:     subvolumes urd-gds-volume-snapview-client
214: end-volume
215:  
216: volume posix-acl-autoload
[2018-09-01 11:11:59.261030] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
217:     type system/posix-acl
218:     subvolumes urd-gds-volume
219: end-volume
220:  
221: volume gfid-access-autoload
222:     type features/gfid-access
223:     subvolumes posix-acl-autoload
224: end-volume
225:  
226: volume meta-autoload
227:     type meta
228:     subvolumes gfid-access-autoload
229: end-volume
230:  
+------------------------------------------------------------------------------+
[2018-09-01 11:11:59.261188] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.261319] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.261361] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:11:59.261453] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f9083a0a31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f9091562266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f909155eea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:11:59.261471] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:11:59.261499] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.264230] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:11:59.264268] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.268927] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:11:59.269143] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.269700] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f9083a0a31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f9091562266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f909155eea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:11:59.269722] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:11:59.269743] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.272303] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.272930] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f9083a0a31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f9091562266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f909155eea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:11:59.272951] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:11:59.272985] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.281719] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f9083a0a31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f9091562266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f909155eea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:11:59.281744] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:11:59.281779] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:11:59.283936] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:11:59.283969] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-4' came back up; going online.
[2018-09-01 11:11:59.284775] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:11:59.288187] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:11:59.288193] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:11:59.291248] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:11:59.292919] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:11:59.292958] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:11:59.295559] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:11:59.297154] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:11:59.298722] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:12:14.851134] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:12:42.572154] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:13:07.164197] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-2BrZK7
[2018-09-01 11:13:07.164288] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f90903cde25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55d3405c5d65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55d3405c5b8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:13:07.164303] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-2BrZK7'.
[2018-09-01 11:13:07.164308] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-2BrZK7'.
[2018-09-01 11:13:18.878092] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-rmAMtR)
[2018-09-01 11:13:18.887284] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:13:18.889540] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:13:18.889541] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:13:18.891740] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:13:18.895329] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:13:18.895371] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.895496] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.895541] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:13:18.904923] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.905124] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f553e5b331a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f554c10b266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f554c107ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:13:18.905135] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:13:18.905156] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.906512] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:13:18.906773] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.907242] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.907615] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:13:18.912768] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:13:18.916584] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:13:18.916604] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:13:18.918382] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:13:18.919373] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.919401] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.919469] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.919836] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f553e5b331a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f554c10b266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f554c107ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:13:18.919858] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:13:18.919887] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.919923] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.919941] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.920233] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:13:18.920257] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:13:18.922630] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:13:18.927394] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
[2018-09-01 11:13:18.931046] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.931076] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.931094] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.931117] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:13:18.931126] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
[2018-09-01 11:13:18.933555] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.933591] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.933615] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.933663] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f553e5b331a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f554c10b266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f554c107ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:13:18.933671] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:13:18.933683] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.933940] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f553e5b331a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f554c10b266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f554c107ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:13:18.933948] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:13:18.933960] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.933978] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
[2018-09-01 11:13:18.933992] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:80c883e3-e99a-4a93-bc9f-89409808e9b9-GRAPH_ID:0-PID:7295-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:80c883e3-e99a-4a93-bc9f-89409808e9b9-GRAPH_ID:0-PID:7295-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option opversion 40100
 50:     option clnt-lk-version 1
 51:     option volfile-checksum 0
 52:     option volfile-key urd-gds-volume
 53:     option client-version 4.1.3
 54:     option process-uuid CTX_ID:80c883e3-e99a-4a93-bc9f-89409808e9b9-GRAPH_ID:0-PID:7295-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-2-RECON_NO:-0
 55:     option fops-version 1298437
 56:     option ping-timeout 42
 57:     option remote-host urd-gds-000
 58:     option remote-subvolume /urd-gds/gluster1
 59:     option transport-type socket
 60:     option transport.address-family inet
 61:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 62:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 63:     option transport.tcp-user-timeout 0
 64:     option transport.socket.keepalive-time 20
 65:     option transport.socket.keepalive-interval 2
 66:     option transport.socket.keepalive-count 9
 67:     option send-gids true
 68: end-volume
 69:  
 70: volume urd-gds-volume-replicate-0
 71:     type cluster/replicate
 72:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 73:     option arbiter-count 1
 74:     option use-compound-fops off
 75:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 76: end-volume
 77:  
 78: volume urd-gds-volume-client-3
 79:     type protocol/client
 80:     option opversion 40100
 81:     option clnt-lk-version 1
 82:     option volfile-checksum 0
 83:     option volfile-key urd-gds-volume
 84:     option client-version 4.1.3
 85:     option process-uuid CTX_ID:80c883e3-e99a-4a93-bc9f-89409808e9b9-GRAPH_ID:0-PID:7295-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-3-RECON_NO:-0
 86:     option fops-version 1298437
 87:     option ping-timeout 42
 88:     option remote-host urd-gds-003
 89:     option remote-subvolume /urd-gds/gluster
 90:     option transport-type socket
 91:     option transport.address-family inet
 92:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 93:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 94:     option transport.tcp-user-timeout 0
 95:     option transport.socket.keepalive-time 20
 96:     option transport.socket.keepalive-interval 2
 97:     option transport.socket.keepalive-count 9
 98:     option send-gids true
 99: end-volume
100:  
101: volume urd-gds-volume-client-4
102:     type protocol/client
103:     option ping-timeout 42
104:     option remote-host urd-gds-004
105:     option remote-subvolume /urd-gds/gluster
106:     option transport-type socket
107:     option transport.address-family inet
108:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
109:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
110:     option transport.tcp-user-timeout 0
111:     option transport.socket.keepalive-time 20
112:     option transport.socket.keepalive-interval 2
113:     option transport.socket.keepalive-count 9
114:     option send-gids true
115: end-volume
116:  
117: volume urd-gds-volume-client-5
118:     type protocol/client
119:     option ping-timeout 42
120:     option remote-host urd-gds-000
121:     option remote-subvolume /urd-gds/gluster2
122:     option transport-type socket
123:     option transport.address-family inet
124:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
125:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
126:     option transport.tcp-user-timeout 0
127:     option transport.socket.keepalive-time 20
128:     option transport.socket.keepalive-interval 2
129:     option transport.socket.keepalive-count 9
130:     option send-gids true
131: end-volume
132:  
133: volume urd-gds-volume-replicate-1
134:     type cluster/replicate
135:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
136:     option arbiter-count 1
137:     option use-compound-fops off
138:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
139: end-volume
140:  
141: volume urd-gds-volume-dht
142:     type cluster/distribute
143:     option lock-migration off
144:     option force-migration off
145:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
146: end-volume
147:  
148: volume urd-gds-volume-write-behind
149:     type performance/write-behind
150:     subvolumes urd-gds-volume-dht
151: end-volume
152:  
153: volume urd-gds-volume-read-ahead
154:     type performance/read-ahead
155:     subvolumes urd-gds-volume-write-behind
156: end-volume
157:  
158: volume urd-gds-volume-readdir-ahead
159:     type performance/readdir-ahead
160:     option parallel-readdir off
161:     option rda-request-size 131072
162:     option rda-cache-limit 10MB
163:     subvolumes urd-gds-volume-read-ahead
164: end-volume
165:  
166: volume urd-gds-volume-io-cache
167:     type performance/io-cache
168:     subvolumes urd-gds-volume-readdir-ahead
169: end-volume
170:  
171: volume urd-gds-volume-quick-read
172:     type performance/quick-read
173:     subvolumes urd-gds-volume-io-cache
174: end-volume
175:  
176: volume urd-gds-volume-open-behind
177:     type performance/open-behind
178:     subvolumes urd-gds-volume-quick-read
179: end-volume
180:  
181: volume urd-gds-volume-md-cache
182:     type performance/md-cache
183:     option cache-posix-acl true
184:     subvolumes urd-gds-volume-open-behind
185: end-volume
186:  
187: volume urd-gds-volume-io-threads
188:     type performance/io-threads
189:     subvolumes urd-gds-volume-md-cache
190: end-volume
191:  
192: volume urd-gds-volume-snapd-client
193:     type protocol/client
194:     option remote-host localhost
195:     option ping-timeout 42
196:     option remote-subvolume snapd-urd-gds-volume
197:     option transport-type socket
198:     option transport.address-family inet
199:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
200:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
201:     option transport.tcp-user-timeout 0
202:     option transport.socket.keepalive-time 20
203:     option transport.socket.keepalive-interval 2
204:     option transport.socket.keepalive-count 9
205:     option send-gids true
206: end-volume
207:  
208: volume urd-gds-volume-snapview-client
209:     type features/snapview-client
210:     option snapshot-directory .snaps
211:     option show-snapshot-directory on
212:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
213: end-volume
214:  
215: volume urd-gds-volume
216:     type debug/io-stats
217:     option log-level INFO
218:     option latency-measurement off
219:     option count-fop-hits off
220:     subvolumes urd-gds-volume-snapview-client
221: end-volume
222:  
223: volume posix-acl-autoload
224:     type system/posix-acl
225:     subvolumes urd-gds-volume
226: end-volume
227:  
228: volume gfid-access-autoload
229:     type features/gfid-access
230:     subvolumes posix-acl-autoload
231: end-volume
232:  
233: volume meta-autoload
234:     type meta
235:     subvolumes gfid-access-autoload
236: end-volume
237:  
+------------------------------------------------------------------------------+
[2018-09-01 11:13:18.945809] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.946060] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:13:18.946484] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.946654] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.946978] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:13:18.947016] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online.
[2018-09-01 11:13:18.950697] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f553e5b331a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f554c10b266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f554c107ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:13:18.950756] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:13:18.950814] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.950927] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:13:18.952161] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.952733] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f553e5b331a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f554c10b266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f554c107ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:13:18.952745] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:13:18.952762] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.953442] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.953537] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f553e5b331a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f554c10b266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f554c107ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:13:18.953546] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:13:18.953558] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:13:18.959767] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:13:18.960515] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:13:18.960528] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:13:18.962449] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:13:18.964791] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:13:18.964806] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:13:18.966509] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:13:18.967484] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:13:18.969149] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:13:32.030720] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:14:25.291799] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:14:44.791791] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-rmAMtR
[2018-09-01 11:14:44.791870] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f554af76e25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x563242778d65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x563242778b8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:14:44.791884] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-rmAMtR'.
[2018-09-01 11:14:44.791890] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-rmAMtR'.
[2018-09-01 11:14:56.576105] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-FBGDzv)
[2018-09-01 11:14:56.585335] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:14:56.587703] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:14:56.587706] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:14:56.589943] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:14:56.593594] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:14:56.593629] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.593772] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.593862] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:14:56.598546] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:14:56.600422] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.600452] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.600635] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f4a72bcf31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f4a80727266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f4a80723ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:14:56.600646] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:14:56.600670] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.600760] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.600997] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:14:56.604008] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:14:56.610223] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:14:56.614204] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.614246] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:14:56.614258] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:14:56.615907] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:14:56.617155] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.617182] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.617206] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.617227] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.617595] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.617657] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f4a72bcf31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f4a80727266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f4a80723ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:14:56.617666] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:14:56.617678] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.617699] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.617803] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:14:56.619810] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
[2018-09-01 11:14:56.621494] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:14:56.621521] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
[2018-09-01 11:14:56.621539] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:95acc7ca-9d79-42d5-8f77-1b852126938e-GRAPH_ID:0-PID:3945-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:95acc7ca-9d79-42d5-8f77-1b852126938e-GRAPH_ID:0-PID:3945-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option ping-timeout 42
 50:     option remote-host urd-gds-000
 51:     option remote-subvolume /urd-gds/gluster1
 52:     option transport-type socket
 53:     option transport.address-family inet
 54:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 55:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 56:     option transport.tcp-user-timeout 0
 57:     option transport.socket.keepalive-time 20
 58:     option transport.socket.keepalive-interval 2
 59:     option transport.socket.keepalive-count 9
 60:     option send-gids true
 61: end-volume
 62:  
 63: volume urd-gds-volume-replicate-0
 64:     type cluster/replicate
 65:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 66:     option arbiter-count 1
 67:     option use-compound-fops off
 68:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 69: end-volume
 70:  
 71: volume urd-gds-volume-client-3
 72:     type protocol/client
 73:     option ping-timeout 42
 74:     option remote-host urd-gds-003
 75:     option remote-subvolume /urd-gds/gluster
 76:     option transport-type socket
 77:     option transport.address-family inet
 78:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 79:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 80:     option transport.tcp-user-timeout 0
 81:     option transport.socket.keepalive-time 20
 82:     option transport.socket.keepalive-interval 2
 83:     option transport.socket.keepalive-count 9
 84:     option send-gids true
 85: end-volume
 86:  
 87: volume urd-gds-volume-client-4
 88:     type protocol/client
 89:     option ping-timeout 42
 90:     option remote-host urd-gds-004
 91:     option remote-subvolume /urd-gds/gluster
 92:     option transport-type socket
 93:     option transport.address-family inet
 94:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 95:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 96:     option transport.tcp-user-timeout 0
 97:     option transport.socket.keepalive-time 20
 98:     option transport.socket.keepalive-interval 2
 99:     option transport.socket.keepalive-count 9
100:     option send-gids true
101: end-volume
102:  
103: volume urd-gds-volume-client-5
104:     type protocol/client
105:     option ping-timeout 42
106:     option remote-host urd-gds-000
107:     option remote-subvolume /urd-gds/gluster2
108:     option transport-type socket
109:     option transport.address-family inet
110:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
111:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
112:     option transport.tcp-user-timeout 0
113:     option transport.socket.keepalive-time 20
114:     option transport.socket.keepalive-interval 2
115:     option transport.socket.keepalive-count 9
116:     option send-gids true
117: end-volume
118:  
119: volume urd-gds-volume-replicate-1
120:     type cluster/replicate
121:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
122:     option arbiter-count 1
123:     option use-compound-fops off
124:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
125: end-volume
126:  
127: volume urd-gds-volume-dht
128:     type cluster/distribute
129:     option lock-migration off
130:     option force-migration off
131:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
132: end-volume
133:  
134: volume urd-gds-volume-write-behind
135:     type performance/write-behind
136:     subvolumes urd-gds-volume-dht
137: end-volume
138:  
139: volume urd-gds-volume-read-ahead
140:     type performance/read-ahead
141:     subvolumes urd-gds-volume-write-behind
142: end-volume
143:  
144: volume urd-gds-volume-readdir-ahead
145:     type performance/readdir-ahead
146:     option parallel-readdir off
147:     option rda-request-size 131072
148:     option rda-cache-limit 10MB
149:     subvolumes urd-gds-volume-read-ahead
150: end-volume
151:  
152: volume urd-gds-volume-io-cache
153:     type performance/io-cache
154:     subvolumes urd-gds-volume-readdir-ahead
155: end-volume
156:  
157: volume urd-gds-volume-quick-read
158:     type performance/quick-read
159:     subvolumes urd-gds-volume-io-cache
160: end-volume
161:  
162: volume urd-gds-volume-open-behind
163:     type performance/open-behind
164:     subvolumes urd-gds-volume-quick-read
165: end-volume
166:  
167: volume urd-gds-volume-md-cache
168:     type performance/md-cache
169:     option cache-posix-acl true
170:     subvolumes urd-gds-volume-open-behind
171: end-volume
172:  
173: volume urd-gds-volume-io-threads
174:     type performance/io-threads
175:     subvolumes urd-gds-volume-md-cache
176: end-volume
177:  
178: volume urd-gds-volume-snapd-client
179:     type protocol/client
180:     option remote-host localhost
181:     option ping-timeout 42
182:     option remote-subvolume snapd-urd-gds-volume
183:     option transport-type socket
184:     option transport.address-family inet
185:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
186:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
187:     option transport.tcp-user-timeout 0
188:     option transport.socket.keepalive-time 20
189:     option transport.socket.keepalive-interval 2
190:     option transport.socket.keepalive-count 9
191:     option send-gids true
192: end-volume
193:  
194: volume urd-gds-volume-snapview-client
195:     type features/snapview-client
196:     option snapshot-directory .snaps
197:     option show-snapshot-directory on
198:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
199: end-volume
200:  
201: volume urd-gds-volume
202:     type debug/io-stats
203:     option log-level INFO
204:     option latency-measurement off
205:     option count-fop-hits off
206:     subvolumes urd-gds-volume-snapview-client
207: end-volume
208:  
209: volume posix-acl-autoload
210:     type system/posix-acl
211:     subvolumes urd-gds-volume
212: end-volume
213:  
214: volume gfid-access-autoload
215:     type features/gfid-access
216:     subvolumes posix-acl-autoload
217: end-volume
218:  
219: volume meta-autoload
220:     type meta
221:     subvolumes gfid-access-autoload
222: end-volume
223:  
+------------------------------------------------------------------------------+
[2018-09-01 11:14:56.626574] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.626607] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.626628] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.626718] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.626795] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:14:56.626828] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.628595] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f4a72bcf31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f4a80727266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f4a80723ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:14:56.628609] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:14:56.628621] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.628640] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:14:56.628680] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f4a72bcf31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f4a80727266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f4a80723ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:14:56.628687] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:14:56.628701] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.628723] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:14:56.628734] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
[2018-09-01 11:14:56.630286] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.634693] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f4a72bcf31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f4a80727266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f4a80723ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:14:56.634705] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:14:56.634718] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.634744] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.635213] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f4a72bcf31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f4a80727266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f4a80723ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:14:56.635224] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:14:56.635242] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.639862] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.640394] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f4a72bcf31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f4a80727266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f4a80723ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:14:56.640411] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:14:56.640433] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:14:56.645151] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:14:56.645672] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:14:56.645974] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:14:56.645990] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online.
[2018-09-01 11:14:56.650847] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:14:56.652753] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:14:56.653954] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:14:56.658798] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:14:56.658822] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:14:56.660806] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:14:56.661747] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:14:56.663328] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:15:15.738243] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:16:00.664697] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:16:19.420924] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-FBGDzv
[2018-09-01 11:16:19.421017] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f4a7f592e25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x555bfcb20d65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x555bfcb20b8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:16:19.421037] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-FBGDzv'.
[2018-09-01 11:16:19.421046] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-FBGDzv'.
[2018-09-01 11:16:31.283247] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-WxZAZR)
[2018-09-01 11:16:31.292381] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:16:31.294608] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:16:31.294640] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:16:31.296962] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:16:31.301302] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:16:31.301321] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.301447] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.301494] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:16:31.306033] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:16:31.307845] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.307870] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.308050] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7faede0f531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7faeebc4d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7faeebc49ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:16:31.308060] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:16:31.308083] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.308214] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.308595] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:16:31.310548] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:16:31.314229] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.314456] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.314653] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.314956] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7faede0f531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7faeebc4d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7faeebc49ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:16:31.314967] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:16:31.314985] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.315022] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:16:31.316332] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:16:31.318767] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:16:31.318787] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:16:31.320386] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:16:31.324160] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.324189] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.324205] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.324578] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.324641] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7faede0f531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7faeebc4d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7faeebc49ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:16:31.324652] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:16:31.324664] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.324687] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.324850] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:16:31.326319] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
[2018-09-01 11:16:31.329582] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
[2018-09-01 11:16:31.329615] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:16:31.329630] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:1d3a0d49-3649-4644-8ce4-eba2d47d3685-GRAPH_ID:0-PID:24662-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:1d3a0d49-3649-4644-8ce4-eba2d47d3685-GRAPH_ID:0-PID:24662-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option opversion 40100
 50:     option clnt-lk-version 1
 51:     option volfile-checksum 0
 52:     option volfile-key urd-gds-volume
 53:     option client-version 4.1.3
 54:     option process-uuid CTX_ID:1d3a0d49-3649-4644-8ce4-eba2d47d3685-GRAPH_ID:0-PID:24662-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-2-RECON_NO:-0
 55:     option fops-version 1298437
 56:     option ping-timeout 42
 57:     option remote-host urd-gds-000
 58:     option remote-subvolume /urd-gds/gluster1
 59:     option transport-type socket
 60:     option transport.address-family inet
 61:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 62:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 63:     option transport.tcp-user-timeout 0
 64:     option transport.socket.keepalive-time 20
 65:     option transport.socket.keepalive-interval 2
 66:     option transport.socket.keepalive-count 9
 67:     option send-gids true
 68: end-volume
 69:  
 70: volume urd-gds-volume-replicate-0
 71:     type cluster/replicate
 72:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 73:     option arbiter-count 1
 74:     option use-compound-fops off
 75:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 76: end-volume
 77:  
 78: volume urd-gds-volume-client-3
 79:     type protocol/client
 80:     option ping-timeout 42
 81:     option remote-host urd-gds-003
 82:     option remote-subvolume /urd-gds/gluster
 83:     option transport-type socket
 84:     option transport.address-family inet
 85:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 86:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 87:     option transport.tcp-user-timeout 0
 88:     option transport.socket.keepalive-time 20
 89:     option transport.socket.keepalive-interval 2
 90:     option transport.socket.keepalive-count 9
 91:     option send-gids true
 92: end-volume
 93:  
 94: volume urd-gds-volume-client-4
 95:     type protocol/client
 96:     option ping-timeout 42
 97:     option remote-host urd-gds-004
 98:     option remote-subvolume /urd-gds/gluster
 99:     option transport-type socket
100:     option transport.address-family inet
101:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
102:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
103:     option transport.tcp-user-timeout 0
104:     option transport.socket.keepalive-time 20
105:     option transport.socket.keepalive-interval 2
106:     option transport.socket.keepalive-count 9
107:     option send-gids true
108: end-volume
109:  
110: volume urd-gds-volume-client-5
111:     type protocol/client
112:     option ping-timeout 42
113:     option remote-host urd-gds-000
114:     option remote-subvolume /urd-gds/gluster2
115:     option transport-type socket
116:     option transport.address-family inet
117:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
118:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
119:     option transport.tcp-user-timeout 0
120:     option transport.socket.keepalive-time 20
121:     option transport.socket.keepalive-interval 2
122:     option transport.socket.keepalive-count 9
123:     option send-gids true
124: end-volume
125:  
126: volume urd-gds-volume-replicate-1
127:     type cluster/replicate
128:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
129:     option arbiter-count 1
130:     option use-compound-fops off
131:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
132: end-volume
133:  
134: volume urd-gds-volume-dht
135:     type cluster/distribute
136:     option lock-migration off
137:     option force-migration off
138:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
139: end-volume
140:  
141: volume urd-gds-volume-write-behind
142:     type performance/write-behind
143:     subvolumes urd-gds-volume-dht
144: end-volume
145:  
146: volume urd-gds-volume-read-ahead
147:     type performance/read-ahead
148:     subvolumes urd-gds-volume-write-behind
149: end-volume
150:  
151: volume urd-gds-volume-readdir-ahead
152:     type performance/readdir-ahead
153:     option parallel-readdir off
154:     option rda-request-size 131072
155:     option rda-cache-limit 10MB
156:     subvolumes urd-gds-volume-read-ahead
157: end-volume
158:  
159: volume urd-gds-volume-io-cache
160:     type performance/io-cache
161:     subvolumes urd-gds-volume-readdir-ahead
162: end-volume
163:  
164: volume urd-gds-volume-quick-read
165:     type performance/quick-read
166:     subvolumes urd-gds-volume-io-cache
167: end-volume
168:  
169: volume urd-gds-volume-open-behind
170:     type performance/open-behind
171:     subvolumes urd-gds-volume-quick-read
172: end-volume
173:  
174: volume urd-gds-volume-md-cache
175:     type performance/md-cache
176:     option cache-posix-acl true
177:     subvolumes urd-gds-volume-open-behind
178: end-volume
179:  
180: volume urd-gds-volume-io-threads
181:     type performance/io-threads
182:     subvolumes urd-gds-volume-md-cache
183: end-volume
[2018-09-01 11:16:31.333440] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
184:  
185: volume urd-gds-volume-snapd-client
186:     type protocol/client
187:     option remote-host localhost
188:     option ping-timeout 42
189:     option remote-subvolume snapd-urd-gds-volume
190:     option transport-type socket
191:     option transport.address-family inet
192:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
193:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
194:     option transport.tcp-user-timeout 0
195:     option transport.socket.keepalive-time 20
196:     option transport.socket.keepalive-interval 2
197:     option transport.socket.keepalive-count 9
198:     option send-gids true
199: end-volume
200:  
201: volume urd-gds-volume-snapview-client
202:     type features/snapview-client
203:     option snapshot-directory .snaps
204:     option show-snapshot-directory on
205:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
206: end-volume
207:  
208: volume urd-gds-volume
209:     type debug/io-stats
210:     option log-level INFO
211:     option latency-measurement off
212:     option count-fop-hits off
213:     subvolumes urd-gds-volume-snapview-client
214: end-volume
215:  
216: volume posix-acl-autoload
217:     type system/posix-acl
218:     subvolumes urd-gds-volume
219: end-volume
220:  
221: volume gfid-access-autoload
222:     type features/gfid-access
223:     subvolumes posix-acl-autoload
224: end-volume
225:  
226: volume meta-autoload
227:     type meta
228:     subvolumes gfid-access-autoload
229: end-volume
230:  
+------------------------------------------------------------------------------+
[2018-09-01 11:16:31.333620] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.333648] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.333713] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.333756] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:16:31.335789] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.335858] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7faede0f531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7faeebc4d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7faeebc49ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:16:31.335868] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:16:31.335885] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.335906] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.335966] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:16:31.336130] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7faede0f531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7faeebc4d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7faeebc49ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:16:31.336141] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:16:31.336158] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.336192] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:16:31.338135] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.338722] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7faede0f531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7faeebc4d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7faeebc49ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:16:31.338737] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:16:31.338751] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.342490] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:16:31.343385] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.343842] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7faede0f531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7faeebc4d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7faeebc49ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:16:31.343856] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:16:31.343875] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:16:31.345712] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:16:31.345730] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online.
[2018-09-01 11:16:31.349119] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:16:31.350842] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:16:31.352584] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:16:31.355268] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:16:31.355293] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:16:31.357035] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:16:31.358285] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:16:31.359729] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:16:47.427712] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-WxZAZR
[2018-09-01 11:16:47.427781] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7faeeaab8e25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x56367a5bfd65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x56367a5bfb8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:16:47.427803] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-WxZAZR'.
[2018-09-01 11:16:47.427811] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-WxZAZR'.
[2018-09-01 11:17:00.098467] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-VDoXTg)
[2018-09-01 11:17:00.112168] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:17:00.114415] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:17:00.114415] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:17:00.116629] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:17:00.120731] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:17:00.120771] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.120895] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.120992] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:17:00.126764] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:17:00.128530] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.128561] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.128747] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f3fcd85931a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f3fdb3b1266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f3fdb3adea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:17:00.128758] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:17:00.128780] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.128869] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.129144] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:17:00.130890] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:17:00.137078] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.137281] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.137341] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.137540] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:17:00.137607] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f3fcd85931a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f3fdb3b1266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f3fdb3adea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:17:00.137618] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:17:00.137633] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.138775] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:17:00.144344] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:17:00.149039] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
[2018-09-01 11:17:00.150700] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:17:00.150718] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:17:00.152996] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.153024] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.153053] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:17:00.153063] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:f6049c15-7982-49e7-b721-b96d3b683380-GRAPH_ID:0-PID:7363-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:f6049c15-7982-49e7-b721-b96d3b683380-GRAPH_ID:0-PID:7363-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option ping-timeout 42
 50:     option remote-host urd-gds-000
 51:     option remote-subvolume /urd-gds/gluster1
 52:     option transport-type socket
 53:     option transport.address-family inet
 54:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 55:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 56:     option transport.tcp-user-timeout 0
 57:     option transport.socket.keepalive-time 20
 58:     option transport.socket.keepalive-interval 2
 59:     option transport.socket.keepalive-count 9
 60:     option send-gids true
 61: end-volume
 62:  
 63: volume urd-gds-volume-replicate-0
 64:     type cluster/replicate
 65:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 66:     option arbiter-count 1
 67:     option use-compound-fops off
 68:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 69: end-volume
 70:  
 71: volume urd-gds-volume-client-3
 72:     type protocol/client
 73:     option ping-timeout 42
 74:     option remote-host urd-gds-003
 75:     option remote-subvolume /urd-gds/gluster
 76:     option transport-type socket
 77:     option transport.address-family inet
 78:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 79:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 80:     option transport.tcp-user-timeout 0
 81:     option transport.socket.keepalive-time 20
 82:     option transport.socket.keepalive-interval 2
 83:     option transport.socket.keepalive-count 9
 84:     option send-gids true
 85: end-volume
 86:  
 87: volume urd-gds-volume-client-4
 88:     type protocol/client
 89:     option ping-timeout 42
 90:     option remote-host urd-gds-004
 91:     option remote-subvolume /urd-gds/gluster
 92:     option transport-type socket
 93:     option transport.address-family inet
 94:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 95:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 96:     option transport.tcp-user-timeout 0
 97:     option transport.socket.keepalive-time 20
 98:     option transport.socket.keepalive-interval 2
 99:     option transport.socket.keepalive-count 9
100:     option send-gids true
101: end-volume
102:  
103: volume urd-gds-volume-client-5
104:     type protocol/client
105:     option ping-timeout 42
106:     option remote-host urd-gds-000
107:     option remote-subvolume /urd-gds/gluster2
108:     option transport-type socket
109:     option transport.address-family inet
110:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
111:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
112:     option transport.tcp-user-timeout 0
113:     option transport.socket.keepalive-time 20
114:     option transport.socket.keepalive-interval 2
115:     option transport.socket.keepalive-count 9
116:     option send-gids true
117: end-volume
118:  
119: volume urd-gds-volume-replicate-1
120:     type cluster/replicate
121:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
122:     option arbiter-count 1
123:     option use-compound-fops off
124:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
125: end-volume
126:  
127: volume urd-gds-volume-dht
128:     type cluster/distribute
129:     option lock-migration off
130:     option force-migration off
131:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
132: end-volume
133:  
134: volume urd-gds-volume-write-behind
135:     type performance/write-behind
136:     subvolumes urd-gds-volume-dht
137: end-volume
138:  
139: volume urd-gds-volume-read-ahead
140:     type performance/read-ahead
141:     subvolumes urd-gds-volume-write-behind
142: end-volume
143:  
144: volume urd-gds-volume-readdir-ahead
145:     type performance/readdir-ahead
146:     option parallel-readdir off
147:     option rda-request-size 131072
148:     option rda-cache-limit 10MB
149:     subvolumes urd-gds-volume-read-ahead
150: end-volume
151:  
152: volume urd-gds-volume-io-cache
153:     type performance/io-cache
154:     subvolumes urd-gds-volume-readdir-ahead
155: end-volume
156:  
157: volume urd-gds-volume-quick-read
158:     type performance/quick-read
159:     subvolumes urd-gds-volume-io-cache
160: end-volume
161:  
162: volume urd-gds-volume-open-behind
163:     type performance/open-behind
164:     subvolumes urd-gds-volume-quick-read
165: end-volume
166:  
167: volume urd-gds-volume-md-cache
168:     type performance/md-cache
169:     option cache-posix-acl true
170:     subvolumes urd-gds-volume-open-behind
171: end-volume
172:  
173: volume urd-gds-volume-io-threads
174:     type performance/io-threads
175:     subvolumes urd-gds-volume-md-cache
176: end-volume
177:  
178: volume urd-gds-volume-snapd-client
179:     type protocol/client
180:     option remote-host localhost
181:     option ping-timeout 42
182:     option remote-subvolume snapd-urd-gds-volume
183:     option transport-type socket
184:     option transport.address-family inet
185:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
186:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
187:     option transport.tcp-user-timeout 0
188:     option transport.socket.keepalive-time 20
189:     option transport.socket.keepalive-interval 2
190:     option transport.socket.keepalive-count 9
191:     option send-gids true
192: end-volume
193:  
194: volume urd-gds-volume-snapview-client
195:     type features/snapview-client
196:     option snapshot-directory .snaps
197:     option show-snapshot-directory on
198:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
199: end-volume
200:  
201: volume urd-gds-volume
202:     type debug/io-stats
203:     option log-level INFO
204:     option latency-measurement off
205:     option count-fop-hits off
206:     subvolumes urd-gds-volume-snapview-client
207: end-volume
208:  
209: volume posix-acl-autoload
210:     type system/posix-acl
211:     subvolumes urd-gds-volume
212: end-volume
213:  
214: volume gfid-access-autoload
215:     type features/gfid-access
216:     subvolumes posix-acl-autoload
217: end-volume
218:  
219: volume meta-autoload
220:     type meta
221:     subvolumes gfid-access-autoload
222: end-volume
223:  
+------------------------------------------------------------------------------+
[2018-09-01 11:17:00.154600] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.154624] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.154653] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.154679] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.154698] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.154793] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.154848] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:17:00.154868] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
[2018-09-01 11:17:00.156631] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:17:00.156659] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.156712] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f3fcd85931a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f3fdb3b1266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f3fdb3adea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:17:00.156721] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:17:00.156733] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.156750] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.158992] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f3fcd85931a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f3fdb3b1266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f3fdb3adea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:17:00.159005] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:17:00.159019] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.159045] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:17:00.160605] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.160840] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f3fcd85931a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f3fdb3b1266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f3fdb3adea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:17:00.160859] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:17:00.160873] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.160895] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.161305] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f3fcd85931a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f3fdb3b1266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f3fdb3adea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:17:00.161315] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:17:00.161332] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.165586] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:17:00.166526] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.167105] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f3fcd85931a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f3fdb3b1266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f3fdb3adea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:17:00.167116] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:17:00.167131] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:17:00.173245] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:17:00.173260] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-4' came back up; going online.
[2018-09-01 11:17:00.174156] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:17:00.178498] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:17:00.178620] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:17:00.185497] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:17:00.186365] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:17:00.186384] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:17:00.188317] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:17:00.189384] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:17:00.190846] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:17:08.295632] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:17:56.034719] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:18:17.713869] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-VDoXTg
[2018-09-01 11:18:17.713953] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f3fda21ce25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55616c92ad65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55616c92ab8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:18:17.713968] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-VDoXTg'.
[2018-09-01 11:18:17.713974] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-VDoXTg'.
[2018-09-01 11:18:29.509213] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-EU1AxA)
[2018-09-01 11:18:29.518449] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:18:29.520754] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:18:29.520787] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:18:29.523035] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:18:29.526651] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:18:29.526716] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.526822] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.526878] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:18:29.531637] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:18:29.533307] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.533334] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.533557] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f04863be31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f0493f16266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f0493f12ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:18:29.533570] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:18:29.533594] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.533676] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.533992] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:18:29.535400] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:18:29.540260] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.540382] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:18:29.540396] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:18:29.541992] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:18:29.543691] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.543726] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.543745] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.544129] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f04863be31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f0493f16266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f0493f12ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:18:29.544141] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:18:29.544157] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.544181] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.544233] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:18:29.545980] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:18:29.548948] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:18:29.548976] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.550793] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
[2018-09-01 11:18:29.554643] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.554686] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.554713] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.554749] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:18:29.554763] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
[2018-09-01 11:18:29.557203] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.557235] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
[2018-09-01 11:18:29.557305] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f04863be31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f0493f16266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f0493f12ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:18:29.557323] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:18:29.557343] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.557373] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:844113f6-738c-4a97-b219-4cdf66c151a7-GRAPH_ID:0-PID:31654-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:844113f6-738c-4a97-b219-4cdf66c151a7-GRAPH_ID:0-PID:31654-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option opversion 40100
 50:     option clnt-lk-version 1
 51:     option volfile-checksum 0
 52:     option volfile-key urd-gds-volume
 53:     option client-version 4.1.3
 54:     option process-uuid CTX_ID:844113f6-738c-4a97-b219-4cdf66c151a7-GRAPH_ID:0-PID:31654-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-2-RECON_NO:-0
 55:     option fops-version 1298437
 56:     option ping-timeout 42
 57:     option remote-host urd-gds-000
 58:     option remote-subvolume /urd-gds/gluster1
 59:     option transport-type socket
 60:     option transport.address-family inet
 61:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 62:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 63:     option transport.tcp-user-timeout 0
 64:     option transport.socket.keepalive-time 20
 65:     option transport.socket.keepalive-interval 2
 66:     option transport.socket.keepalive-count 9
 67:     option send-gids true
 68: end-volume
 69:  
 70: volume urd-gds-volume-replicate-0
 71:     type cluster/replicate
 72:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 73:     option arbiter-count 1
 74:     option use-compound-fops off
 75:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 76: end-volume
 77:  
 78: volume urd-gds-volume-client-3
 79:     type protocol/client
 80:     option ping-timeout 42
 81:     option remote-host urd-gds-003
 82:     option remote-subvolume /urd-gds/gluster
 83:     option transport-type socket
 84:     option transport.address-family inet
 85:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 86:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 87:     option transport.tcp-user-timeout 0
 88:     option transport.socket.keepalive-time 20
 89:     option transport.socket.keepalive-interval 2
 90:     option transport.socket.keepalive-count 9
 91:     option send-gids true
 92: end-volume
 93:  
 94: volume urd-gds-volume-client-4
 95:     type protocol/client
 96:     option ping-timeout 42
 97:     option remote-host urd-gds-004
 98:     option remote-subvolume /urd-gds/gluster
 99:     option transport-type socket
100:     option transport.address-family inet
101:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
102:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
103:     option transport.tcp-user-timeout 0
104:     option transport.socket.keepalive-time 20
105:     option transport.socket.keepalive-interval 2
106:     option transport.socket.keepalive-count 9
107:     option send-gids true
108: end-volume
109:  
110: volume urd-gds-volume-client-5
111:     type protocol/client
112:     option ping-timeout 42
113:     option remote-host urd-gds-000
114:     option remote-subvolume /urd-gds/gluster2
115:     option transport-type socket
116:     option transport.address-family inet
117:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
118:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
119:     option transport.tcp-user-timeout 0
120:     option transport.socket.keepalive-time 20
121:     option transport.socket.keepalive-interval 2
122:     option transport.socket.keepalive-count 9
123:     option send-gids true
124: end-volume
125:  
126: volume urd-gds-volume-replicate-1
127:     type cluster/replicate
128:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
129:     option arbiter-count 1
130:     option use-compound-fops off
131:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
132: end-volume
133:  
134: volume urd-gds-volume-dht
135:     type cluster/distribute
136:     option lock-migration off
137:     option force-migration off
138:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
139: end-volume
140:  
141: volume urd-gds-volume-write-behind
142:     type performance/write-behind
143:     subvolumes urd-gds-volume-dht
144: end-volume
145:  
146: volume urd-gds-volume-read-ahead
147:     type performance/read-ahead
148:     subvolumes urd-gds-volume-write-behind
149: end-volume
150:  
151: volume urd-gds-volume-readdir-ahead
152:     type performance/readdir-ahead
153:     option parallel-readdir off
154:     option rda-request-size 131072
155:     option rda-cache-limit 10MB
156:     subvolumes urd-gds-volume-read-ahead
157: end-volume
158:  
159: volume urd-gds-volume-io-cache
160:     type performance/io-cache
161:     subvolumes urd-gds-volume-readdir-ahead
162: end-volume
163:  
164: volume urd-gds-volume-quick-read
165:     type performance/quick-read
166:     subvolumes urd-gds-volume-io-cache
167: end-volume
168:  
169: volume urd-gds-volume-open-behind
170:     type performance/open-behind
171:     subvolumes urd-gds-volume-quick-read
172: end-volume
173:  
174: volume urd-gds-volume-md-cache
175:     type performance/md-cache
176:     option cache-posix-acl true
177:     subvolumes urd-gds-volume-open-behind
178: end-volume
179:  
180: volume urd-gds-volume-io-threads
181:     type performance/io-threads
182:     subvolumes urd-gds-volume-md-cache
183: end-volume
184:  
185: volume urd-gds-volume-snapd-client
186:     type protocol/client
187:     option remote-host localhost
188:     option ping-timeout 42
189:     option remote-subvolume snapd-urd-gds-volume
190:     option transport-type socket
191:     option transport.address-family inet
192:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
193:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
194:     option transport.tcp-user-timeout 0
195:     option transport.socket.keepalive-time 20
196:     option transport.socket.keepalive-interval 2
197:     option transport.socket.keepalive-count 9
198:     option send-gids true
199: end-volume
200:  
201: volume urd-gds-volume-snapview-client
202:     type features/snapview-client
203:     option snapshot-directory .snaps
204:     option show-snapshot-directory on
205:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
206: end-volume
207:  
208: volume urd-gds-volume
209:     type debug/io-stats
210:     option log-level INFO
211:     option latency-measurement off
212:     option count-fop-hits off
213:     subvolumes urd-gds-volume-snapview-client
214: end-volume
215:  
216: volume posix-acl-autoload
217:     type system/posix-acl
218:     subvolumes urd-gds-volume
219: end-volume
220:  
221: volume gfid-access-autoload
222:     type features/gfid-access
223:     subvolumes posix-acl-autoload
224: end-volume
225:  
226: volume meta-autoload
227:     type meta
228:     subvolumes gfid-access-autoload
229: end-volume
230:  
+------------------------------------------------------------------------------+
[2018-09-01 11:18:29.559040] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f04863be31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f0493f16266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f0493f12ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:18:29.559056] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:18:29.559079] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.559109] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:18:29.559138] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.562823] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.562914] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:18:29.564704] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.564728] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.564819] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f04863be31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f0493f16266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f0493f12ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:18:29.564829] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:18:29.564844] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.565091] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f04863be31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f0493f16266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f0493f12ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:18:29.565100] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:18:29.565115] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.566780] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.567271] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f04863be31a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f0493f16266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f0493f12ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:18:29.567280] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:18:29.567296] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:18:29.567873] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:18:29.570045] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:18:29.570059] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online.
[2018-09-01 11:18:29.571238] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:18:29.578022] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:18:29.578036] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:18:29.578261] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:18:29.581342] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:18:29.581357] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:18:29.582733] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:18:29.583719] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:18:29.585350] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:18:47.171796] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:19:41.205170] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:20:01.243040] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-EU1AxA
[2018-09-01 11:20:01.243137] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f0492d81e25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x555e62f45d65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x555e62f45b8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:20:01.243153] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-EU1AxA'.
[2018-09-01 11:20:01.243159] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-EU1AxA'.
[2018-09-01 11:20:12.920301] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-zUdMnQ)
[2018-09-01 11:20:12.929358] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:20:13.499488] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:20:13.499497] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:20:13.510330] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:20:13.521370] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:20:13.521492] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.521943] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.522195] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:20:13.530903] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:20:13.535839] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.535912] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.536358] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f7be68e531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f7bf443d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f7bf4439ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:20:13.536383] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:20:13.536446] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.536532] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.536875] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:20:13.539413] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:20:13.543700] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.543778] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:20:13.543801] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:20:13.546397] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.546482] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.546777] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:20:13.546876] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f7be68e531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f7bf443d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f7bf4439ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:20:13.546897] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:20:13.546921] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.546966] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:20:13.547001] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.554218] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.554551] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.554626] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:20:13.559619] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:20:13.562696] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f7be68e531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f7bf443d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f7bf4439ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:20:13.562721] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:20:13.562746] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.562804] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:20:13.562823] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
[2018-09-01 11:20:13.565644] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
[2018-09-01 11:20:13.565943] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.565986] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.566021] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.566441] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f7be68e531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f7bf443d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f7bf4439ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:20:13.566477] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:20:13.566507] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.566557] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.566599] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.566926] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:ff24e3ed-12bc-40ae-bfbf-3921fdc1c142-GRAPH_ID:0-PID:22978-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:ff24e3ed-12bc-40ae-bfbf-3921fdc1c142-GRAPH_ID:0-PID:22978-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option opversion 40100
 50:     option clnt-lk-version 1
 51:     option volfile-checksum 0
 52:     option volfile-key urd-gds-volume
 53:     option client-version 4.1.3
 54:     option process-uuid CTX_ID:ff24e3ed-12bc-40ae-bfbf-3921fdc1c142-GRAPH_ID:0-PID:22978-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-2-RECON_NO:-0
 55:     option fops-version 1298437
 56:     option ping-timeout 42
 57:     option remote-host urd-gds-000
 58:     option remote-subvolume /urd-gds/gluster1
 59:     option transport-type socket
 60:     option transport.address-family inet
 61:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 62:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 63:     option transport.tcp-user-timeout 0
 64:     option transport.socket.keepalive-time 20
 65:     option transport.socket.keepalive-interval 2
 66:     option transport.socket.keepalive-count 9
 67:     option send-gids true
 68: end-volume
 69:  
 70: volume urd-gds-volume-replicate-0
 71:     type cluster/replicate
 72:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 73:     option arbiter-count 1
 74:     option use-compound-fops off
 75:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 76: end-volume
 77:  
 78: volume urd-gds-volume-client-3
 79:     type protocol/client
 80:     option opversion 40100
 81:     option clnt-lk-version 1
 82:     option volfile-checksum 0
 83:     option volfile-key urd-gds-volume
 84:     option client-version 4.1.3
 85:     option process-uuid CTX_ID:ff24e3ed-12bc-40ae-bfbf-3921fdc1c142-GRAPH_ID:0-PID:22978-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-3-RECON_NO:-0
 86:     option fops-version 1298437
 87:     option ping-timeout 42
 88:     option remote-host urd-gds-003
 89:     option remote-subvolume /urd-gds/gluster
 90:     option transport-type socket
 91:     option transport.address-family inet
 92:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 93:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 94:     option transport.tcp-user-timeout 0
 95:     option transport.socket.keepalive-time 20
 96:     option transport.socket.keepalive-interval 2
 97:     option transport.socket.keepalive-count 9
 98:     option send-gids true
 99: end-volume
100:  
101: volume urd-gds-volume-client-4
102:     type protocol/client
103:     option ping-timeout 42
104:     option remote-host urd-gds-004
105:     option remote-subvolume /urd-gds/gluster
106:     option transport-type socket
107:     option transport.address-family inet
108:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
109:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
110:     option transport.tcp-user-timeout 0
111:     option transport.socket.keepalive-time 20
112:     option transport.socket.keepalive-interval 2
113:     option transport.socket.keepalive-count 9
114:     option send-gids true
115: end-volume
116:  
117: volume urd-gds-volume-client-5
118:     type protocol/client
119:     option ping-timeout 42
120:     option remote-host urd-gds-000
121:     option remote-subvolume /urd-gds/gluster2
122:     option transport-type socket
123:     option transport.address-family inet
124:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
125:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
126:     option transport.tcp-user-timeout 0
127:     option transport.socket.keepalive-time 20
128:     option transport.socket.keepalive-interval 2
129:     option transport.socket.keepalive-count 9
130:     option send-gids true
131: end-volume
132:  
133: volume urd-gds-volume-replicate-1
134:     type cluster/replicate
135:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
136:     option arbiter-count 1
137:     option use-compound-fops off
138:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
139: end-volume
140:  
141: volume urd-gds-volume-dht
142:     type cluster/distribute
143:     option lock-migration off
144:     option force-migration off
145:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
146: end-volume
147:  
148: volume urd-gds-volume-write-behind
149:     type performance/write-behind
150:     subvolumes urd-gds-volume-dht
151: end-volume
152:  
153: volume urd-gds-volume-read-ahead
154:     type performance/read-ahead
155:     subvolumes urd-gds-volume-write-behind
156: end-volume
157:  
158: volume urd-gds-volume-readdir-ahead
159:     type performance/readdir-ahead
160:     option parallel-readdir off
161:     option rda-request-size 131072
162:     option rda-cache-limit 10MB
163:     subvolumes urd-gds-volume-read-ahead
164: end-volume
165:  
166: volume urd-gds-volume-io-cache
167:     type performance/io-cache
168:     subvolumes urd-gds-volume-readdir-ahead
169: end-volume
170:  
171: volume urd-gds-volume-quick-read
172:     type performance/quick-read
173:     subvolumes urd-gds-volume-io-cache
174: end-volume
175:  
176: volume urd-gds-volume-open-behind
177:     type performance/open-behind
178:     subvolumes urd-gds-volume-quick-read
179: end-volume
180:  
181: volume urd-gds-volume-md-cache
182:     type performance/md-cache
183:     option cache-posix-acl true
184:     subvolumes urd-gds-volume-open-behind
185: end-volume
186:  
187: volume urd-gds-volume-io-threads
188:     type performance/io-threads
189:     subvolumes urd-gds-volume-md-cache
190: end-volume
191:  
192: volume urd-gds-volume-snapd-client
193:     type protocol/client
194:     option remote-host localhost
195:     option ping-timeout 42
196:     option remote-subvolume snapd-urd-gds-volume
197:     option transport-type socket
198:     option transport.address-family inet
199:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
200:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
201:     option transport.tcp-user-timeout 0
202:     option transport.socket.keepalive-time 20
203:     option transport.socket.keepalive-interval 2
204:     option transport.socket.keepalive-count 9
205:     option send-gids true
206: end-volume
207:  
208: volume urd-gds-volume-snapview-client
209:     type features/snapview-client
210:     option snapshot-directory .snaps
211:     option show-snapshot-directory on
212:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
213: end-volume
214:  
215: volume urd-gds-volume
216:     type debug/io-stats
217:     option log-level INFO
218:     option latency-measurement off
219:     option count-fop-hits off
220:     subvolumes urd-gds-volume-snapview-client
221: end-volume
222:  
223: volume posix-acl-autoload
224:     type system/posix-acl
225:     subvolumes urd-gds-volume
226: end-volume
227:  
228: volume gfid-access-autoload
229:     type features/gfid-access
230:     subvolumes posix-acl-autoload
231: end-volume
232:  
233: volume meta-autoload
234:     type meta
235:     subvolumes gfid-access-autoload
236: end-volume
237:  
+------------------------------------------------------------------------------+
[2018-09-01 11:20:13.574088] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:20:13.574131] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.576346] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.576432] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:20:13.576644] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:20:13.576706] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.576784] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:20:13.576799] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online.
[2018-09-01 11:20:13.580142] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f7be68e531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f7bf443d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f7bf4439ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:20:13.580168] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:20:13.580193] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.580308] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.580855] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f7be68e531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f7bf443d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f7bf4439ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:20:13.580889] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:20:13.580919] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.583176] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.583338] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7f7be68e531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f7bf443d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f7bf4439ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:20:13.583356] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:20:13.583379] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:20:13.589366] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:20:13.589391] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:20:13.590069] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:20:13.593099] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:20:13.594207] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:20:13.594235] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:20:13.596245] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:20:13.597354] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:20:13.599081] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:20:30.664895] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:21:08.781119] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:21:30.127515] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-zUdMnQ
[2018-09-01 11:21:30.127645] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f7bf32a8e25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x5627e4e23d65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x5627e4e23b8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:21:30.127660] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-zUdMnQ'.
[2018-09-01 11:21:30.127666] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-zUdMnQ'.
[2018-09-01 11:21:41.939641] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-DtWvol)
[2018-09-01 11:21:41.949158] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:21:41.951724] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:21:41.951726] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:21:41.953905] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:21:41.957430] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:21:41.957501] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.957679] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.957821] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:21:41.962144] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:21:41.963956] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.963984] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.964172] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fa98f6f031a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fa99d248266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fa99d244ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:21:41.964182] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:21:41.964204] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.964335] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.964616] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:21:41.965829] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:21:41.970641] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.970865] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.971027] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.971124] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:21:41.971135] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:21:41.972375] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:21:41.974502] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fa98f6f031a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fa99d248266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fa99d244ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:21:41.974514] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:21:41.974527] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.974547] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:21:41.974567] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.977106] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:21:41.979622] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.979649] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.979870] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.980010] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:21:41.980051] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.981786] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
[2018-09-01 11:21:41.985409] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
[2018-09-01 11:21:41.985496] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fa98f6f031a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fa99d248266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fa99d244ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:21:41.985506] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:21:41.985520] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.985540] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.985566] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:21:41.985575] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:31109730-3781-43f0-9b50-66753a794c95-GRAPH_ID:0-PID:1237-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:31109730-3781-43f0-9b50-66753a794c95-GRAPH_ID:0-PID:1237-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option opversion 40100
 50:     option clnt-lk-version 1
 51:     option volfile-checksum 0
 52:     option volfile-key urd-gds-volume
 53:     option client-version 4.1.3
 54:     option process-uuid CTX_ID:31109730-3781-43f0-9b50-66753a794c95-GRAPH_ID:0-PID:1237-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-2-RECON_NO:-0
 55:     option fops-version 1298437
 56:     option ping-timeout 42
 57:     option remote-host urd-gds-000
 58:     option remote-subvolume /urd-gds/gluster1
 59:     option transport-type socket
 60:     option transport.address-family inet
 61:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 62:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 63:     option transport.tcp-user-timeout 0
 64:     option transport.socket.keepalive-time 20
 65:     option transport.socket.keepalive-interval 2
 66:     option transport.socket.keepalive-count 9
 67:     option send-gids true
 68: end-volume
 69:  
 70: volume urd-gds-volume-replicate-0
 71:     type cluster/replicate
 72:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 73:     option arbiter-count 1
 74:     option use-compound-fops off
 75:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 76: end-volume
 77:  
 78: volume urd-gds-volume-client-3
 79:     type protocol/client
 80:     option ping-timeout 42
 81:     option remote-host urd-gds-003
 82:     option remote-subvolume /urd-gds/gluster
 83:     option transport-type socket
 84:     option transport.address-family inet
 85:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 86:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 87:     option transport.tcp-user-timeout 0
 88:     option transport.socket.keepalive-time 20
 89:     option transport.socket.keepalive-interval 2
 90:     option transport.socket.keepalive-count 9
 91:     option send-gids true
 92: end-volume
 93:  
 94: volume urd-gds-volume-client-4
 95:     type protocol/client
 96:     option ping-timeout 42
 97:     option remote-host urd-gds-004
 98:     option remote-subvolume /urd-gds/gluster
 99:     option transport-type socket
100:     option transport.address-family inet
101:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
102:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
103:     option transport.tcp-user-timeout 0
104:     option transport.socket.keepalive-time 20
105:     option transport.socket.keepalive-interval 2
106:     option transport.socket.keepalive-count 9
107:     option send-gids true
108: end-volume
109:  
110: volume urd-gds-volume-client-5
111:     type protocol/client
112:     option ping-timeout 42
113:     option remote-host urd-gds-000
114:     option remote-subvolume /urd-gds/gluster2
115:     option transport-type socket
116:     option transport.address-family inet
117:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
118:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
119:     option transport.tcp-user-timeout 0
120:     option transport.socket.keepalive-time 20
121:     option transport.socket.keepalive-interval 2
122:     option transport.socket.keepalive-count 9
123:     option send-gids true
124: end-volume
125:  
126: volume urd-gds-volume-replicate-1
127:     type cluster/replicate
128:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
129:     option arbiter-count 1
130:     option use-compound-fops off
131:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
132: end-volume
133:  
134: volume urd-gds-volume-dht
135:     type cluster/distribute
136:     option lock-migration off
137:     option force-migration off
138:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
139: end-volume
140:  
141: volume urd-gds-volume-write-behind
142:     type performance/write-behind
143:     subvolumes urd-gds-volume-dht
144: end-volume
145:  
146: volume urd-gds-volume-read-ahead
147:     type performance/read-ahead
148:     subvolumes urd-gds-volume-write-behind
149: end-volume
150:  
151: volume urd-gds-volume-readdir-ahead
152:     type performance/readdir-ahead
153:     option parallel-readdir off
154:     option rda-request-size 131072
155:     option rda-cache-limit 10MB
156:     subvolumes urd-gds-volume-read-ahead
157: end-volume
158:  
159: volume urd-gds-volume-io-cache
160:     type performance/io-cache
161:     subvolumes urd-gds-volume-readdir-ahead
162: end-volume
163:  
164: volume urd-gds-volume-quick-read
165:     type performance/quick-read
166:     subvolumes urd-gds-volume-io-cache
167: end-volume
168:  
169: volume urd-gds-volume-open-behind
170:     type performance/open-behind
171:     subvolumes urd-gds-volume-quick-read
172: end-volume
173:  
174: volume urd-gds-volume-md-cache
175:     type performance/md-cache
176:     option cache-posix-acl true
177:     subvolumes urd-gds-volume-open-behind
178: end-volume
179:  
180: volume urd-gds-volume-io-threads
181:     type performance/io-threads
182:     subvolumes urd-gds-volume-md-cache
183: end-volume
184:  
185: volume urd-gds-volume-snapd-client
186:     type protocol/client
187:     option remote-host localhost
188:     option ping-timeout 42
189:     option remote-subvolume snapd-urd-gds-volume
190:     option transport-type socket
191:     option transport.address-family inet
192:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
193:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
194:     option transport.tcp-user-timeout 0
195:     option transport.socket.keepalive-time 20
196:     option transport.socket.keepalive-interval 2
197:     option transport.socket.keepalive-count 9
198:     option send-gids true
199: end-volume
200:  
201: volume urd-gds-volume-snapview-client
202:     type features/snapview-client
203:     option snapshot-directory .snaps
204:     option show-snapshot-directory on
205:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
206: end-volume
207:  
208: volume urd-gds-volume
209:     type debug/io-stats
210:     option log-level INFO
211:     option latency-measurement off
212:     option count-fop-hits off
213:     subvolumes urd-gds-volume-snapview-client
214: end-volume
215:  
216: volume posix-acl-autoload
217:     type system/posix-acl
218:     subvolumes urd-gds-volume
219: end-volume
220:  
221: volume gfid-access-autoload
222:     type features/gfid-access
223:     subvolumes posix-acl-autoload
224: end-volume
225:  
226: volume meta-autoload
227:     type meta
228:     subvolumes gfid-access-autoload
229: end-volume
230:  
+------------------------------------------------------------------------------+
[2018-09-01 11:21:41.989278] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.989307] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.989331] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.989435] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.989508] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:21:41.991283] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fa98f6f031a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fa99d248266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fa99d244ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:21:41.991299] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:21:41.991311] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.991331] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:21:41.991346] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.993740] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fa98f6f031a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fa99d248266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fa99d244ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:21:41.993753] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:21:41.993767] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.993991] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.994470] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fa98f6f031a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fa99d248266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fa99d244ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:21:41.994481] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:21:41.994496] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.994633] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:21:41.998179] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.998684] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fa98f6f031a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fa99d248266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fa99d244ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:21:41.998699] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:21:41.998710] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:21:41.999825] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:21:42.004561] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:21:42.004627] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online.
[2018-09-01 11:21:42.006995] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:21:42.009748] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:21:42.009758] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:21:42.016743] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:21:42.016833] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:21:42.020273] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:21:42.021839] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:21:42.023184] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:21:55.092979] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:22:39.283819] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:23:02.108463] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-DtWvol
[2018-09-01 11:23:02.108544] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7fa99c0b3e25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x562ac0d25d65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x562ac0d25b8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:23:02.108561] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-DtWvol'.
[2018-09-01 11:23:02.108570] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-DtWvol'.
[2018-09-01 11:23:14.963236] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-Dg4SFf)
[2018-09-01 11:23:14.973092] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:23:14.975456] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:23:14.975486] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:23:14.977666] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:23:14.981963] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:23:14.981982] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:14.982109] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:14.982195] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:23:14.986886] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:23:14.988759] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:14.988787] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:14.988963] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fc6b4f0231a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc6c2a5a266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fc6c2a56ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:23:14.988974] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:23:14.988995] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:14.989194] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:14.989488] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:23:14.992082] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:23:14.997199] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:14.997492] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:14.997654] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:14.998019] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fc6b4f0231a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc6c2a5a266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fc6c2a56ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:23:14.998031] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:23:14.998072] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:14.998106] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:23:14.999040] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:23:15.005146] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:23:15.010417] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
[2018-09-01 11:23:15.012553] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:23:15.012576] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:23:15.015026] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.015069] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.015109] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:23:15.015123] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:16b48ae3-815b-47b4-a9b0-1fcbfcd98bc8-GRAPH_ID:0-PID:24182-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:16b48ae3-815b-47b4-a9b0-1fcbfcd98bc8-GRAPH_ID:0-PID:24182-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option ping-timeout 42
 50:     option remote-host urd-gds-000
 51:     option remote-subvolume /urd-gds/gluster1
 52:     option transport-type socket
 53:     option transport.address-family inet
 54:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 55:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 56:     option transport.tcp-user-timeout 0
 57:     option transport.socket.keepalive-time 20
 58:     option transport.socket.keepalive-interval 2
 59:     option transport.socket.keepalive-count 9
 60:     option send-gids true
 61: end-volume
 62:  
 63: volume urd-gds-volume-replicate-0
 64:     type cluster/replicate
 65:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 66:     option arbiter-count 1
 67:     option use-compound-fops off
 68:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 69: end-volume
 70:  
 71: volume urd-gds-volume-client-3
 72:     type protocol/client
 73:     option ping-timeout 42
 74:     option remote-host urd-gds-003
 75:     option remote-subvolume /urd-gds/gluster
 76:     option transport-type socket
 77:     option transport.address-family inet
 78:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 79:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 80:     option transport.tcp-user-timeout 0
 81:     option transport.socket.keepalive-time 20
 82:     option transport.socket.keepalive-interval 2
 83:     option transport.socket.keepalive-count 9
 84:     option send-gids true
 85: end-volume
 86:  
 87: volume urd-gds-volume-client-4
 88:     type protocol/client
 89:     option ping-timeout 42
 90:     option remote-host urd-gds-004
 91:     option remote-subvolume /urd-gds/gluster
 92:     option transport-type socket
 93:     option transport.address-family inet
 94:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 95:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 96:     option transport.tcp-user-timeout 0
 97:     option transport.socket.keepalive-time 20
 98:     option transport.socket.keepalive-interval 2
 99:     option transport.socket.keepalive-count 9
100:     option send-gids true
101: end-volume
102:  
103: volume urd-gds-volume-client-5
104:     type protocol/client
105:     option ping-timeout 42
106:     option remote-host urd-gds-000
107:     option remote-subvolume /urd-gds/gluster2
108:     option transport-type socket
109:     option transport.address-family inet
110:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
111:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
112:     option transport.tcp-user-timeout 0
113:     option transport.socket.keepalive-time 20
114:     option transport.socket.keepalive-interval 2
115:     option transport.socket.keepalive-count 9
116:     option send-gids true
117: end-volume
118:  
119: volume urd-gds-volume-replicate-1
120:     type cluster/replicate
121:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
122:     option arbiter-count 1
123:     option use-compound-fops off
124:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
125: end-volume
126:  
127: volume urd-gds-volume-dht
128:     type cluster/distribute
[2018-09-01 11:23:15.017547] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
129:     option lock-migration off
130:     option force-migration off
131:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
132: end-volume
133:  
134: volume urd-gds-volume-write-behind
135:     type performance/write-behind
136:     subvolumes urd-gds-volume-dht
137: end-volume
138:  
139: volume urd-gds-volume-read-ahead
140:     type performance/read-ahead
141:     subvolumes urd-gds-volume-write-behind
142: end-volume
143:  
144: volume urd-gds-volume-readdir-ahead
145:     type performance/readdir-ahead
146:     option parallel-readdir off
147:     option rda-request-size 131072
148:     option rda-cache-limit 10MB
149:     subvolumes urd-gds-volume-read-ahead
150: end-volume
151:  
152: volume urd-gds-volume-io-cache
153:     type performance/io-cache
154:     subvolumes urd-gds-volume-readdir-ahead
155: end-volume
156:  
157: volume urd-gds-volume-quick-read
158:     type performance/quick-read
159:     subvolumes urd-gds-volume-io-cache
160: end-volume
161:  
162: volume urd-gds-volume-open-behind
163:     type performance/open-behind
164:     subvolumes urd-gds-volume-quick-read
165: end-volume
166:  
167: volume urd-gds-volume-md-cache
168:     type performance/md-cache
169:     option cache-posix-acl true
170:     subvolumes urd-gds-volume-open-behind
171: end-volume
172:  
173: volume urd-gds-volume-io-threads
174:     type performance/io-threads
175:     subvolumes urd-gds-volume-md-cache
176: end-volume
177:  
178: volume urd-gds-volume-snapd-client
179:     type protocol/client
180:     option remote-host localhost
181:     option ping-timeout 42
182:     option remote-subvolume snapd-urd-gds-volume
183:     option transport-type socket
184:     option transport.address-family inet
185:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
186:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
187:     option transport.tcp-user-timeout 0
188:     option transport.socket.keepalive-time 20
189:     option transport.socket.keepalive-interval 2
190:     option transport.socket.keepalive-count 9
191:     option send-gids true
192: end-volume
193:  
194: volume urd-gds-volume-snapview-client
195:     type features/snapview-client
196:     option snapshot-directory .snaps
197:     option show-snapshot-directory on
[2018-09-01 11:23:15.017589] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
198:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
199: end-volume
200:  
201: volume urd-gds-volume
202:     type debug/io-stats
203:     option log-level INFO
204:     option latency-measurement off
205:     option count-fop-hits off
206:     subvolumes urd-gds-volume-snapview-client
207: end-volume
208:  
209: volume posix-acl-autoload
210:     type system/posix-acl
211:     subvolumes urd-gds-volume
212: end-volume
213:  
214: volume gfid-access-autoload
215:     type features/gfid-access
216:     subvolumes posix-acl-autoload
217: end-volume
218:  
219: volume meta-autoload
220:     type meta
221:     subvolumes gfid-access-autoload
222: end-volume
223:  
+------------------------------------------------------------------------------+
[2018-09-01 11:23:15.017893] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.018013] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.018042] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.018096] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.018129] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.018221] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:23:15.018274] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fc6b4f0231a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc6c2a5a266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fc6c2a56ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:23:15.018288] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:23:15.018305] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.018331] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:23:15.018356] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:23:15.018377] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
[2018-09-01 11:23:15.026056] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.026113] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.026224] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fc6b4f0231a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc6c2a5a266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fc6c2a56ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:23:15.026237] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:23:15.026261] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.026323] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.026571] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fc6b4f0231a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc6c2a5a266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fc6c2a56ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:23:15.026589] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:23:15.026619] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.026840] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fc6b4f0231a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc6c2a5a266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fc6c2a56ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:23:15.026852] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:23:15.026874] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.027081] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:23:15.029185] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.029706] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fc6b4f0231a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc6c2a5a266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fc6c2a56ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:23:15.029724] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:23:15.029753] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:23:15.030029] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:23:15.038065] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:23:15.038085] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online.
[2018-09-01 11:23:15.039181] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:23:15.042018] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:23:15.044224] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:23:15.046059] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:23:15.046082] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:23:15.048010] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:23:15.049131] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:23:15.050628] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:23:33.173197] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:24:18.567952] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:24:35.523195] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-Dg4SFf
[2018-09-01 11:24:35.523378] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7fc6c18c5e25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x5559f76c9d65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x5559f76c9b8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:24:35.523393] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-Dg4SFf'.
[2018-09-01 11:24:35.523399] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-Dg4SFf'.
[2018-09-01 11:24:47.324749] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 4.1.3 (args: /usr/local/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/mnt-urd-gds-gluster.log --volfile-server=localhost --volfile-id=urd-gds-volume --client-pid=-1 /tmp/gsyncd-aux-mount-fQRZje)
[2018-09-01 11:24:47.334105] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-09-01 11:24:47.336378] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2018-09-01 11:24:47.336379] I [MSGID: 114007] [client.c:2370:client_check_remote_host] 0-urd-gds-volume-snapd-client: Remote host is not set. Assuming the volfile server as remote host [Invalid argument]
[2018-09-01 11:24:47.338617] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-0: parent translators are ready, attempting connect on transport
[2018-09-01 11:24:47.342269] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-1: parent translators are ready, attempting connect on transport
[2018-09-01 11:24:47.342347] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.342429] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.342503] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-0: changing port to 49152 (from 0)
[2018-09-01 11:24:47.347120] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-2: parent translators are ready, attempting connect on transport
[2018-09-01 11:24:47.348903] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.348938] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.349161] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fe25731531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fe264e6d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fe264e69ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:24:47.349172] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-0: failed to set process-name in handshake msg
[2018-09-01 11:24:47.349194] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-0: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.349287] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.349570] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-1: changing port to 49152 (from 0)
[2018-09-01 11:24:47.350670] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-3: parent translators are ready, attempting connect on transport
[2018-09-01 11:24:47.355561] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.355804] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.355961] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.356042] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-0: Connected to urd-gds-volume-client-0, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:24:47.356054] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-0: Subvolume 'urd-gds-volume-client-0' came back up; going online.
[2018-09-01 11:24:47.357224] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-4: parent translators are ready, attempting connect on transport
[2018-09-01 11:24:47.359457] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fe25731531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fe264e6d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fe264e69ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:24:47.359470] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-1: failed to set process-name in handshake msg
[2018-09-01 11:24:47.359483] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-1: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.359505] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-2: changing port to 49152 (from 0)
[2018-09-01 11:24:47.359522] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.361599] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-client-5: parent translators are ready, attempting connect on transport
[2018-09-01 11:24:47.363223] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.363253] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.363514] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.363553] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-3: changing port to 49152 (from 0)
[2018-09-01 11:24:47.365232] I [MSGID: 114020] [client.c:2328:notify] 0-urd-gds-volume-snapd-client: parent translators are ready, attempting connect on transport
[2018-09-01 11:24:47.369242] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.369303] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fe25731531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fe264e6d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fe264e69ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:24:47.369314] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-2: failed to set process-name in handshake msg
[2018-09-01 11:24:47.369326] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-2: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.369346] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.369483] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.369630] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0)
Final graph:
+------------------------------------------------------------------------------+
  1: volume urd-gds-volume-client-0
  2:     type protocol/client
  3:     option opversion 40100
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key urd-gds-volume
  7:     option client-version 4.1.3
  8:     option process-uuid CTX_ID:11d4e65c-14c9-4f9c-bfef-2b1cd51037a2-GRAPH_ID:0-PID:2782-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-0-RECON_NO:-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host urd-gds-001
 12:     option remote-subvolume /urd-gds/gluster
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 16:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:  
 24: volume urd-gds-volume-client-1
 25:     type protocol/client
 26:     option opversion 40100
 27:     option clnt-lk-version 1
 28:     option volfile-checksum 0
 29:     option volfile-key urd-gds-volume
 30:     option client-version 4.1.3
 31:     option process-uuid CTX_ID:11d4e65c-14c9-4f9c-bfef-2b1cd51037a2-GRAPH_ID:0-PID:2782-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-1-RECON_NO:-0
 32:     option fops-version 1298437
 33:     option ping-timeout 42
 34:     option remote-host urd-gds-002
 35:     option remote-subvolume /urd-gds/gluster
 36:     option transport-type socket
 37:     option transport.address-family inet
 38:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 39:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 40:     option transport.tcp-user-timeout 0
 41:     option transport.socket.keepalive-time 20
 42:     option transport.socket.keepalive-interval 2
 43:     option transport.socket.keepalive-count 9
 44:     option send-gids true
 45: end-volume
 46:  
 47: volume urd-gds-volume-client-2
 48:     type protocol/client
 49:     option opversion 40100
 50:     option clnt-lk-version 1
 51:     option volfile-checksum 0
 52:     option volfile-key urd-gds-volume
 53:     option client-version 4.1.3
 54:     option process-uuid CTX_ID:11d4e65c-14c9-4f9c-bfef-2b1cd51037a2-GRAPH_ID:0-PID:2782-HOST:urd-gds-001-PC_NAME:urd-gds-volume-client-2-RECON_NO:-0
 55:     option fops-version 1298437
 56:     option ping-timeout 42
 57:     option remote-host urd-gds-000
 58:     option remote-subvolume /urd-gds/gluster1
 59:     option transport-type socket
 60:     option transport.address-family inet
 61:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 62:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 63:     option transport.tcp-user-timeout 0
 64:     option transport.socket.keepalive-time 20
 65:     option transport.socket.keepalive-interval 2
 66:     option transport.socket.keepalive-count 9
 67:     option send-gids true
 68: end-volume
 69:  
 70: volume urd-gds-volume-replicate-0
 71:     type cluster/replicate
 72:     option afr-pending-xattr urd-gds-volume-client-0,urd-gds-volume-client-1,urd-gds-volume-client-2
 73:     option arbiter-count 1
 74:     option use-compound-fops off
 75:     subvolumes urd-gds-volume-client-0 urd-gds-volume-client-1 urd-gds-volume-client-2
 76: end-volume
 77:  
 78: volume urd-gds-volume-client-3
 79:     type protocol/client
 80:     option ping-timeout 42
 81:     option remote-host urd-gds-003
 82:     option remote-subvolume /urd-gds/gluster
 83:     option transport-type socket
 84:     option transport.address-family inet
 85:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
 86:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
 87:     option transport.tcp-user-timeout 0
 88:     option transport.socket.keepalive-time 20
 89:     option transport.socket.keepalive-interval 2
 90:     option transport.socket.keepalive-count 9
 91:     option send-gids true
 92: end-volume
 93:  
 94: volume urd-gds-volume-client-4
 95:     type protocol/client
 96:     option ping-timeout 42
 97:     option remote-host urd-gds-004
 98:     option remote-subvolume /urd-gds/gluster
 99:     option transport-type socket
100:     option transport.address-family inet
101:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
102:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
103:     option transport.tcp-user-timeout 0
104:     option transport.socket.keepalive-time 20
105:     option transport.socket.keepalive-interval 2
106:     option transport.socket.keepalive-count 9
107:     option send-gids true
108: end-volume
109:  
110: volume urd-gds-volume-client-5
111:     type protocol/client
112:     option ping-timeout 42
113:     option remote-host urd-gds-000
114:     option remote-subvolume /urd-gds/gluster2
115:     option transport-type socket
116:     option transport.address-family inet
117:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
118:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
119:     option transport.tcp-user-timeout 0
120:     option transport.socket.keepalive-time 20
121:     option transport.socket.keepalive-interval 2
122:     option transport.socket.keepalive-count 9
123:     option send-gids true
124: end-volume
125:  
126: volume urd-gds-volume-replicate-1
127:     type cluster/replicate
128:     option afr-pending-xattr urd-gds-volume-client-3,urd-gds-volume-client-4,urd-gds-volume-client-5
129:     option arbiter-count 1
130:     option use-compound-fops off
131:     subvolumes urd-gds-volume-client-3 urd-gds-volume-client-4 urd-gds-volume-client-5
132: end-volume
133:  
134: volume urd-gds-volume-dht
135:     type cluster/distribute
136:     option lock-migration off
137:     option force-migration off
138:     subvolumes urd-gds-volume-replicate-0 urd-gds-volume-replicate-1
139: end-volume
140:  
141: volume urd-gds-volume-write-behind
142:     type performance/write-behind
143:     subvolumes urd-gds-volume-dht
144: end-volume
145:  
146: volume urd-gds-volume-read-ahead
147:     type performance/read-ahead
148:     subvolumes urd-gds-volume-write-behind
149: end-volume
150:  
151: volume urd-gds-volume-readdir-ahead
152:     type performance/readdir-ahead
153:     option parallel-readdir off
154:     option rda-request-size 131072
155:     option rda-cache-limit 10MB
156:     subvolumes urd-gds-volume-read-ahead
157: end-volume
158:  
159: volume urd-gds-volume-io-cache
160:     type performance/io-cache
161:     subvolumes urd-gds-volume-readdir-ahead
162: end-volume
163:  
164: volume urd-gds-volume-quick-read
165:     type performance/quick-read
166:     subvolumes urd-gds-volume-io-cache
167: end-volume
168:  
169: volume urd-gds-volume-open-behind
170:     type performance/open-behind
171:     subvolumes urd-gds-volume-quick-read
172: end-volume
173:  
174: volume urd-gds-volume-md-cache
175:     type performance/md-cache
176:     option cache-posix-acl true
177:     subvolumes urd-gds-volume-open-behind
178: end-volume
179:  
180: volume urd-gds-volume-io-threads
181:     type performance/io-threads
182:     subvolumes urd-gds-volume-md-cache
183: end-volume
184:  
185: volume urd-gds-volume-snapd-client
186:     type protocol/client
187:     option remote-host localhost
188:     option ping-timeout 42
189:     option remote-subvolume snapd-urd-gds-volume
190:     option transport-type socket
191:     option transport.address-family inet
192:     option username 6208f6a4-6d3d-4328-aa7d-e7ecd1e1e1e9
193:     option password 2f38d6dd-dc27-4fd2-8b3a-fbab9454a6f5
194:     option transport.tcp-user-timeout 0
195:     option transport.socket.keepalive-time 20
196:     option transport.socket.keepalive-interval 2
197:     option transport.socket.keepalive-count 9
198:     option send-gids true
199: end-volume
200:  
201: volume urd-gds-volume-snapview-client
202:     type features/snapview-client
203:     option snapshot-directory .snaps
204:     option show-snapshot-directory on
205:     subvolumes urd-gds-volume-io-threads urd-gds-volume-snapd-client
206: end-volume
207:  
208: volume urd-gds-volume
209:     type debug/io-stats
210:     option log-level INFO
211:     option latency-measurement off
212:     option count-fop-hits off
213:     subvolumes urd-gds-volume-snapview-client
214: end-volume
215:  
216: volume posix-acl-autoload
217:     type system/posix-acl
218:     subvolumes urd-gds-volume
219: end-volume
220:  
221: volume gfid-access-autoload
222:     type features/gfid-access
223:     subvolumes posix-acl-autoload
224: end-volume
225:  
226: volume meta-autoload
227:     type meta
228:     subvolumes gfid-access-autoload
229: end-volume
230:  
+------------------------------------------------------------------------------+
[2018-09-01 11:24:47.373026] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.373054] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:24:47.373064] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is met
[2018-09-01 11:24:47.374205] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fe25731531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fe264e6d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fe264e69ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:24:47.374218] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-3: failed to set process-name in handshake msg
[2018-09-01 11:24:47.374234] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-3: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.374256] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.374287] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0)
[2018-09-01 11:24:47.378997] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.379030] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.379117] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0)
[2018-09-01 11:24:47.380851] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fe25731531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fe264e6d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fe264e69ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:24:47.380864] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-4: failed to set process-name in handshake msg
[2018-09-01 11:24:47.380882] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-4: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.380908] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'.
[2018-09-01 11:24:47.381003] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.381136] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fe25731531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fe264e6d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fe264e69ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:24:47.381144] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-snapd-client: failed to set process-name in handshake msg
[2018-09-01 11:24:47.381159] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-snapd-client: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.382814] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.383294] W [dict.c:923:str_to_data] (-->/usr/lib64/glusterfs/4.1.3/xlator/protocol/client.so(+0x4131a) [0x7fe25731531a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fe264e6d266] -->/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7fe264e69ea1] ) 0-dict: value is NULL [Invalid argument]
[2018-09-01 11:24:47.383304] I [MSGID: 114006] [client-handshake.c:1308:client_setvolume] 0-urd-gds-volume-client-5: failed to set process-name in handshake msg
[2018-09-01 11:24:47.383319] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-urd-gds-volume-client-5: error returned while attempting to connect to host:(null), port:0
[2018-09-01 11:24:47.384027] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:24:47.384043] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online.
[2018-09-01 11:24:47.387989] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-volume'.
[2018-09-01 11:24:47.391967] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'.
[2018-09-01 11:24:47.391980] I [MSGID: 108002] [afr-common.c:5515:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is met
[2018-09-01 11:24:47.393090] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'.
[2018-09-01 11:24:47.395329] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-09-01 11:24:47.395345] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-09-01 11:24:47.396806] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:24:47.399383] I [MSGID: 108031] [afr-common.c:2599:afr_local_discovery_cbk] 0-urd-gds-volume-replicate-0: selecting local read_child urd-gds-volume-client-0
[2018-09-01 11:24:47.400613] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory /
[2018-09-01 11:25:01.982387] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:25:41.918024] E [fuse-bridge.c:3557:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage
[2018-09-01 11:26:01.841261] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /tmp/gsyncd-aux-mount-fQRZje
[2018-09-01 11:26:01.841439] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7fe263cd8e25] -->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x563fe15c2d65] -->/usr/local/sbin/glusterfs(cleanup_and_exit+0x6b) [0x563fe15c2b8b] ) 0-: received signum (15), shutting down
[2018-09-01 11:26:01.841469] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-fQRZje'.
[2018-09-01 11:26:01.841475] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-fQRZje'.
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users