Re: 3.8.3 Shards Healing Glacier Slow

David Gossage <dgossage@xxxxxxxxxxxxxxxxxx> · Mon, 29 Aug 2016 15:52:22 -0500

attached brick and client logs from test machine where same behavior occurred not sure if anything new is there.  its still on 3.8.2

Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.71.10:/gluster2/brick1/1
Brick2: 192.168.71.11:/gluster2/brick2/1
Brick3: 192.168.71.12:/gluster2/brick3/1
Options Reconfigured:
cluster.locking-scheme: granular
performance.strict-o-direct: off
features.shard-block-size: 64MB
features.shard: on
server.allow-insecure: on
storage.owner-uid: 36
storage.owner-gid: 36
cluster.server-quorum-type: server
cluster.quorum-type: auto
network.remote-dio: on
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.quick-read: off
cluster.self-heal-window-size: 1024
cluster.background-self-heal-count: 16
nfs.enable-ino32: off
nfs.addr-namelookup: off
nfs.disable: on
performance.read-ahead: off
performance.readdir-ahead: on
cluster.granular-entry-heal: on

On Mon, Aug 29, 2016 at 2:20 PM, David Gossage <dgossage@xxxxxxxxxxxxxxxxxx> wrote:
On Mon, Aug 29, 2016 at 7:01 AM, Anuradha Talur <atalur@xxxxxxxxxx> wrote:

----- Original Message -----

> From: "David Gossage" <dgossage@xxxxxxxxxxxxxxxxxx>

> To: "Anuradha Talur" <atalur@xxxxxxxxxx>

> Cc: "gluster-users@xxxxxxxxxxx List" <Gluster-users@xxxxxxxxxxx>, "Krutika Dhananjay" <kdhananj@xxxxxxxxxx>

> Sent: Monday, August 29, 2016 5:12:42 PM

> Subject: Re:  3.8.3 Shards Healing Glacier Slow

>

> On Mon, Aug 29, 2016 at 5:39 AM, Anuradha Talur <atalur@xxxxxxxxxx> wrote:

>

> > Response inline.

> >

> > ----- Original Message -----

> > > From: "Krutika Dhananjay" <kdhananj@xxxxxxxxxx>

> > > To: "David Gossage" <dgossage@xxxxxxxxxxxxxxxxxx>

> > > Cc: "gluster-users@xxxxxxxxxxx List" <Gluster-users@xxxxxxxxxxx>

> > > Sent: Monday, August 29, 2016 3:55:04 PM

> > > Subject: Re:  3.8.3 Shards Healing Glacier Slow

> > >

> > > Could you attach both client and brick logs? Meanwhile I will try these

> > steps

> > > out on my machines and see if it is easily recreatable.

> > >

> > > -Krutika

> > >

> > > On Mon, Aug 29, 2016 at 2:31 PM, David Gossage <

> > dgossage@xxxxxxxxxxxxxxxxxx

> > > > wrote:

> > >

> > >

> > >

> > > Centos 7 Gluster 3.8.3

> > >

> > > Brick1: ccgl1.gl.local:/gluster1/BRICK1/1

> > > Brick2: ccgl2.gl.local:/gluster1/BRICK1/1

> > > Brick3: ccgl4.gl.local:/gluster1/BRICK1/1

> > > Options Reconfigured:

> > > cluster.data-self-heal-algorithm: full

> > > cluster.self-heal-daemon: on

> > > cluster.locking-scheme: granular

> > > features.shard-block-size: 64MB

> > > features.shard: on

> > > performance.readdir-ahead: on

> > > storage.owner-uid: 36

> > > storage.owner-gid: 36

> > > performance.quick-read: off

> > > performance.read-ahead: off

> > > performance.io-cache: off

> > > performance.stat-prefetch: on

> > > cluster.eager-lock: enable

> > > network.remote-dio: enable

> > > cluster.quorum-type: auto

> > > cluster.server-quorum-type: server

> > > server.allow-insecure: on

> > > cluster.self-heal-window-size: 1024

> > > cluster.background-self-heal-count: 16

> > > performance.strict-write-ordering: off

> > > nfs.disable: on

> > > nfs.addr-namelookup: off

> > > nfs.enable-ino32: off

> > > cluster.granular-entry-heal: on

> > >

> > > Friday did rolling upgrade from 3.8.3->3.8.3 no issues.

> > > Following steps detailed in previous recommendations began proces of

> > > replacing and healngbricks one node at a time.

> > >

> > > 1) kill pid of brick

> > > 2) reconfigure brick from raid6 to raid10

> > > 3) recreate directory of brick

> > > 4) gluster volume start <> force

> > > 5) gluster volume heal <> full

> > Hi,

> >

> > I'd suggest that full heal is not used. There are a few bugs in full heal.

> > Better safe than sorry ;)

> > Instead I'd suggest the following steps:

> >

> > Currently I brought the node down by systemctl stop glusterd as I was

> getting sporadic io issues and a few VM's paused so hoping that will help.

> I may wait to do this till around 4PM when most work is done in case it

> shoots load up.

>

>

> > 1) kill pid of brick

> > 2) to configuring of brick that you need

> > 3) recreate brick dir

> > 4) while the brick is still down, from the mount point:

> >    a) create a dummy non existent dir under / of mount.

> >

>

> so if noee 2 is down brick, pick node for example 3 and make a test dir

> under its brick directory that doesnt exist on 2 or should I be dong this

> over a gluster mount?

You should be doing this over gluster mount.

>

> >    b) set a non existent extended attribute on / of mount.

> >

>

> Could you give me an example of an attribute to set?   I've read a tad on

> this, and looked up attributes but haven't set any yet myself.

>

Sure. setfattr -n "user.some-name" -v "some-value" <path-to-mount>

> Doing these steps will ensure that heal happens only from updated brick to

> > down brick.

> > 5) gluster v start <> force

> > 6) gluster v heal <>

> >

>

> Will it matter if somewhere in gluster the full heal command was run other

> day?  Not sure if it eventually stops or times out.

>

full heal will stop once the crawl is done. So if you want to trigger heal again,

run gluster v heal <>. Actually even brick up or volume start force should

trigger the heal.

Did this on test bed today.  its one server with 3 bricks on same machine so take that for what its worth.  also it still runs 3.8.2.  Maybe ill update and re-run test.

killed brick
deleted brick dir
recreated brick dir
created fake dir on gluster mount
set suggested fake attribute on it
ran volume start <> force

looked at files it said needed healing and it was just 8 shards that were modified for few minutes I ran through steps

gave it few minutes and it stayed same
ran gluster volume <> heal

it healed all the directories and files you can see over mount including fakedir.  

same issue for shards though.  it adds more shards to heal at glacier pace.  slight jump in speed if I stat every file and dir in VM running but not all shards.

It started with 8 shards to heal and is now only at 33 out of 800 and probably wont finish adding for few days at rate it goes.

> >

> > > 1st node worked as expected took 12 hours to heal 1TB data. Load was

> > little

> > > heavy but nothing shocking.

> > >

> > > About an hour after node 1 finished I began same process on node2. Heal

> > > proces kicked in as before and the files in directories visible from

> > mount

> > > and .glusterfs healed in short time. Then it began crawl of .shard adding

> > > those files to heal count at which point the entire proces ground to a

> > halt

> > > basically. After 48 hours out of 19k shards it has added 5900 to heal

> > list.

> > > Load on all 3 machnes is negligible. It was suggested to change this

> > value

> > > to full cluster.data-self-heal-algorithm and restart volume which I

> > did. No

> > > efffect. Tried relaunching heal no effect, despite any node picked. I

> > > started each VM and performed a stat of all files from within it, or a

> > full

> > > virus scan and that seemed to cause short small spikes in shards added,

> > but

> > > not by much. Logs are showing no real messages indicating anything is

> > going

> > > on. I get hits to brick log on occasion of null lookups making me think

> > its

> > > not really crawling shards directory but waiting for a shard lookup to

> > add

> > > it. I'll get following in brick log but not constant and sometime

> > multiple

> > > for same shard.

> > >

> > > [2016-08-29 08:31:57.478125] W [MSGID: 115009]

> > > [server-resolve.c:569:server_resolve] 0-GLUSTER1-server: no resolution

> > type

> > > for (null) (LOOKUP)

> > > [2016-08-29 08:31:57.478170] E [MSGID: 115050]

> > > [server-rpc-fops.c:156:server_lookup_cbk] 0-GLUSTER1-server: 12591783:

> > > LOOKUP (null) (00000000-0000-0000-00

> > > 00-000000000000/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.221) ==> (Invalid

> > > argument) [Invalid argument]

> > >

> > > This one repeated about 30 times in row then nothing for 10 minutes then

> > one

> > > hit for one different shard by itself.

> > >

> > > How can I determine if Heal is actually running? How can I kill it or

> > force

> > > restart? Does node I start it from determine which directory gets

> > crawled to

> > > determine heals?

> > >

> > > David Gossage

> > > Carousel Checks Inc. | System Administrator

> > > Office 708.613.2284

> > >

> > > _______________________________________________

> > > Gluster-users mailing list

> > > Gluster-users@xxxxxxxxxxx

> > > http://www.gluster.org/mailman/listinfo/gluster-users

> > >

> > >

> > > _______________________________________________

> > > Gluster-users mailing list

> > > Gluster-users@xxxxxxxxxxx

> > > http://www.gluster.org/mailman/listinfo/gluster-users

> >

> > --

> > Thanks,

> > Anuradha.

> >

>

--

Thanks,

Anuradha.

[2016-08-29 18:51:08.454416] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:51:08.813569] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-glustershard-client-0: disconnected from glustershard-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2016-08-29 18:51:09.328730] I [MSGID: 108031] [afr-common.c:1953:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-1
[2016-08-29 18:51:09.328736] I [MSGID: 108031] [afr-common.c:1953:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-2
[2016-08-29 18:51:09.426492] W [MSGID: 114031] [client-rpc-fops.c:1829:client3_3_fxattrop_cbk] 0-glustershard-client-0: remote operation failed
The message "W [MSGID: 114031] [client-rpc-fops.c:1829:client3_3_fxattrop_cbk] 0-glustershard-client-0: remote operation failed" repeated 2 times between [2016-08-29 18:51:09.426492] and [2016-08-29 18:51:09.426677]
[2016-08-29 18:51:09.427353] E [MSGID: 114031] [client-rpc-fops.c:1600:client3_3_finodelk_cbk] 0-glustershard-client-0: remote operation failed [Transport endpoint is not connected]
The message "E [MSGID: 114031] [client-rpc-fops.c:1600:client3_3_finodelk_cbk] 0-glustershard-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2016-08-29 18:51:09.427353] and [2016-08-29 18:51:09.427606]
[2016-08-29 18:51:19.431623] E [MSGID: 114058] [client-handshake.c:1533:client_query_portmap_cbk] 0-glustershard-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2016-08-29 18:51:19.431715] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-glustershard-client-0: disconnected from glustershard-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2016-08-29 18:52:35.295776] I [MSGID: 109036] [dht-common.c:8887:dht_log_new_layout_for_dir_selfheal] 0-glustershard-dht: Setting layout of /fakedir with [Subvol_name: glustershard-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], 
[2016-08-29 18:51:08.454466] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:54:11.617153] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-glustershard-client-0: changing port to 49152 (from 0)
[2016-08-29 18:54:11.621129] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-glustershard-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-08-29 18:54:11.626015] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-glustershard-client-0: Connected to glustershard-client-0, attached to remote volume '/gluster2/brick1/1'.
[2016-08-29 18:54:11.626040] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-glustershard-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-08-29 18:54:11.626050] I [MSGID: 114042] [client-handshake.c:1053:client_post_handshake] 0-glustershard-client-0: 2 fds open - Delaying child_up until they are re-opened
[2016-08-29 18:54:11.626475] W [MSGID: 114060] [client-handshake.c:721:client3_3_reopen_cbk] 0-glustershard-client-0: reopen on <gfid:63705978-6620-445b-b831-7b89c4675c52> failed. [Stale file handle]
[2016-08-29 18:54:11.626525] W [MSGID: 114060] [client-handshake.c:721:client3_3_reopen_cbk] 0-glustershard-client-0: reopen on <gfid:781264e2-27ee-4935-86f3-718b4768812a> failed. [Stale file handle]
[2016-08-29 18:54:11.626538] I [MSGID: 114041] [client-handshake.c:675:client_child_up_reopen_done] 0-glustershard-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2016-08-29 18:54:11.626629] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-glustershard-client-0: Server lk version = 1
[2016-08-29 18:54:12.802796] I [MSGID: 108031] [afr-common.c:1953:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-1
[2016-08-29 18:54:12.802870] I [MSGID: 108031] [afr-common.c:1953:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-2
[2016-08-29 18:54:12.802949] I [MSGID: 108031] [afr-common.c:1953:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-0
[2016-08-29 18:54:12.804899] I [MSGID: 108026] [afr-self-heal-entry.c:840:afr_selfheal_entry_do] 0-glustershard-replicate-0: performing entry selfheal on 00000000-0000-0000-0000-000000000001
[2016-08-29 18:54:12.807534] W [dict.c:418:dict_set] (-->/usr/lib64/glusterfs/3.8.2/xlator/cluster/replicate.so(+0x49890) [0x7f011ccb1890] -->/lib64/libglusterfs.so.0(dict_set_str+0x2c) [0x7f0124a46efc] -->/lib64/libglusterfs.so.0(dict_set+0xe6) [0x7f0124a44db6] ) 0-dict: !this || !value for key=link-count [Invalid argument]
[2016-08-29 18:54:12.810625] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-glustershard-replicate-0: performing metadata selfheal on c8b3c1ad-64f9-4688-89cb-090bf08ba2df
[2016-08-29 18:54:12.814275] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed metadata selfheal on c8b3c1ad-64f9-4688-89cb-090bf08ba2df. sources=[1] 2  sinks=0 
[2016-08-29 18:54:12.814666] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-glustershard-replicate-0: performing metadata selfheal on 15d2f696-2bfa-4eef-86f1-ff709394b189
[2016-08-29 18:54:12.815190] W [MSGID: 114031] [client-rpc-fops.c:2265:client3_3_setattr_cbk] 0-glustershard-client-0: remote operation failed [Operation not permitted]
[2016-08-29 18:54:12.817515] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed metadata selfheal on 15d2f696-2bfa-4eef-86f1-ff709394b189. sources=[1] 2  sinks=
[2016-08-29 18:54:12.834789] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed entry selfheal on 00000000-0000-0000-0000-000000000001. sources=[1] 2  sinks=0 
[2016-08-29 18:54:12.835294] I [MSGID: 108026] [afr-self-heal-entry.c:840:afr_selfheal_entry_do] 0-glustershard-replicate-0: performing entry selfheal on c8b3c1ad-64f9-4688-89cb-090bf08ba2df
[2016-08-29 18:54:12.835953] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-glustershard-replicate-0: performing metadata selfheal on 090f2d08-6a2c-4584-9461-42ff9a9b0158
[2016-08-29 18:54:12.839159] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed metadata selfheal on 090f2d08-6a2c-4584-9461-42ff9a9b0158. sources=[1] 2  sinks=0 
[2016-08-29 18:54:12.859242] I [MSGID: 108026] [afr-self-heal-entry.c:840:afr_selfheal_entry_do] 0-glustershard-replicate-0: performing entry selfheal on 090f2d08-6a2c-4584-9461-42ff9a9b0158
[2016-08-29 18:54:12.859251] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-glustershard-replicate-0: performing metadata selfheal on 4d34c77c-454d-4650-8a11-15d312ca4150
[2016-08-29 18:54:12.862196] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed entry selfheal on c8b3c1ad-64f9-4688-89cb-090bf08ba2df. sources=[1] 2  sinks=0 
[2016-08-29 18:54:12.863476] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed metadata selfheal on 4d34c77c-454d-4650-8a11-15d312ca4150. sources=[1] 2  sinks=0 
[2016-08-29 18:54:12.901484] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed data selfheal on 4d34c77c-454d-4650-8a11-15d312ca4150. sources=[1] 2  sinks=0 
[2016-08-29 18:54:12.907180] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed entry selfheal on 090f2d08-6a2c-4584-9461-42ff9a9b0158. sources=[1] 2  sinks=0 
[2016-08-29 18:54:13.807116] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-glustershard-replicate-0: performing metadata selfheal on cb201568-51de-4b55-8d00-30cd2826ed31
[2016-08-29 18:54:13.810785] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed metadata selfheal on cb201568-51de-4b55-8d00-30cd2826ed31. sources=[1] 2  sinks=0 
[2016-08-29 18:54:13.838467] I [MSGID: 109092] [dht-layout.c:807:dht_layout_dir_mismatch] 0-glustershard-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001
[2016-08-29 18:54:13.838486] I [MSGID: 109018] [dht-common.c:877:dht_revalidate_cbk] 0-glustershard-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001
[2016-08-29 18:54:13.838869] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-glustershard-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
[2016-08-29 18:54:13.840260] I [MSGID: 109036] [dht-common.c:8887:dht_log_new_layout_for_dir_selfheal] 0-glustershard-dht: Setting layout of / with [Subvol_name: glustershard-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], 
[2016-08-29 18:54:13.852316] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-glustershard-replicate-0: performing metadata selfheal on 4463a546-0b68-49a5-a857-a0494d27653c
[2016-08-29 18:54:13.854057] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed data selfheal on cb201568-51de-4b55-8d00-30cd2826ed31. sources=[1] 2  sinks=0 
[2016-08-29 18:54:13.854986] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed metadata selfheal on 4463a546-0b68-49a5-a857-a0494d27653c. sources=[1] 2  sinks=0 
[2016-08-29 18:54:13.864627] I [MSGID: 108026] [afr-self-heal-entry.c:840:afr_selfheal_entry_do] 0-glustershard-replicate-0: performing entry selfheal on 4463a546-0b68-49a5-a857-a0494d27653c
[2016-08-29 18:54:13.865192] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-glustershard-replicate-0: performing metadata selfheal on abd9da9d-5698-44ad-900c-808fbfe55cee
[2016-08-29 18:54:13.868244] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed metadata selfheal on abd9da9d-5698-44ad-900c-808fbfe55cee. sources=[1] 2  sinks=0 
[2016-08-29 18:54:13.876569] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed entry selfheal on 4463a546-0b68-49a5-a857-a0494d27653c. sources=[1] 2  sinks=0 
[2016-08-29 18:54:13.877719] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-glustershard-replicate-0: performing metadata selfheal on 4d83e0de-2fd5-4302-b79a-7292743d0787
[2016-08-29 18:54:13.880741] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed metadata selfheal on 4d83e0de-2fd5-4302-b79a-7292743d0787. sources=[1] 2  sinks=0 
[2016-08-29 18:54:14.201677] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:54:14.210826] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-glustershard-client-0: remote operation failed [No such file or directory]
[2016-08-29 18:54:14.213033] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:54:27.976464] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:54:41.528501] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-glustershard-client-0: remote operation failed [No such file or directory]
[2016-08-29 18:54:41.531121] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-glustershard-client-0: remote operation failed [No such file or directory]
[2016-08-29 18:54:44.296600] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:54:44.299354] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:54:48.517526] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:54:48.517594] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:54:53.466421] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:54:53.466487] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:55:08.465109] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:55:09.059669] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:55:14.391658] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:56:11.831619] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-glustershard-client-0: remote operation failed [No such file or directory]
The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-glustershard-client-0: remote operation failed [No such file or directory]" repeated 2 times between [2016-08-29 18:56:11.831619] and [2016-08-29 18:56:14.607347]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 18:55:14.391658] and [2016-08-29 18:56:14.610730]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 18:55:09.059669] and [2016-08-29 18:56:51.792705]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 18:55:08.465109] and [2016-08-29 18:56:53.470079]
[2016-08-29 18:57:08.472414] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:57:12.323418] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:57:14.550673] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-glustershard-replicate-0: performing metadata selfheal on 00dcd551-ead7-477d-8edc-43824329d0c7
[2016-08-29 18:57:14.554470] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed metadata selfheal on 00dcd551-ead7-477d-8edc-43824329d0c7. sources=[1] 2  sinks=0 
[2016-08-29 18:57:14.595065] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-glustershard-replicate-0: Completed data selfheal on 00dcd551-ead7-477d-8edc-43824329d0c7. sources=[1] 2  sinks=0 
[2016-08-29 18:57:14.771887] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 18:57:14.771887] and [2016-08-29 18:58:14.964945]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 18:57:08.472414] and [2016-08-29 18:58:55.179687]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 18:57:12.323418] and [2016-08-29 18:58:56.746127]
[2016-08-29 18:59:10.179129] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:59:16.864278] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 18:59:17.276783] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 18:59:16.864278] and [2016-08-29 19:00:47.153419]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 13 times between [2016-08-29 18:59:10.179129] and [2016-08-29 19:00:55.175638]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 18:59:17.276783] and [2016-08-29 19:00:59.963705]
[2016-08-29 19:01:10.181222] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:01:17.244899] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:01:20.506805] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:01:47.356943] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-glustershard-client-0: remote operation failed [No such file or directory]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:01:17.244899] and [2016-08-29 19:02:47.534236]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:01:10.181222] and [2016-08-29 19:02:55.179413]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:01:20.506805] and [2016-08-29 19:03:02.219680]
[2016-08-29 19:03:10.178099] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:03:17.625634] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:03:22.762299] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 19:03:22.762299] and [2016-08-29 19:04:44.930651]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 19:03:17.625634] and [2016-08-29 19:04:47.934154]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:03:10.178099] and [2016-08-29 19:04:55.176213]
[2016-08-29 19:05:05.484733] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:05:10.175391] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:05:18.006058] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:05:18.006058] and [2016-08-29 19:06:18.198937]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:05:05.484733] and [2016-08-29 19:06:48.183418]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:05:10.175391] and [2016-08-29 19:06:55.176215]
[2016-08-29 19:07:08.724366] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:07:10.184078] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:07:15.821805] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:08:44.220917] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-glustershard-client-0: remote operation failed [No such file or directory]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:07:08.724366] and [2016-08-29 19:08:51.423863]
The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-glustershard-client-0: remote operation failed [No such file or directory]" repeated 10 times between [2016-08-29 19:08:44.220917] and [2016-08-29 19:08:51.646116]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 13 times between [2016-08-29 19:07:15.821805] and [2016-08-29 19:08:51.648893]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 13 times between [2016-08-29 19:07:10.184078] and [2016-08-29 19:08:55.181484]
[2016-08-29 19:09:10.182052] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:09:11.943086] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:09:18.804413] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:09:18.810059] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-glustershard-client-0: remote operation failed [No such file or directory]
The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-glustershard-client-0: remote operation failed [No such file or directory]" repeated 17 times between [2016-08-29 19:09:18.810059] and [2016-08-29 19:09:41.734328]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 32 times between [2016-08-29 19:09:18.804413] and [2016-08-29 19:10:49.470760]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:09:11.943086] and [2016-08-29 19:10:54.617071]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 15 times between [2016-08-29 19:09:10.182052] and [2016-08-29 19:10:55.182521]
[2016-08-29 19:11:10.182561] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:11:15.248883] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:11:19.216391] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:11:19.216391] and [2016-08-29 19:12:19.449336]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 13 times between [2016-08-29 19:11:10.182561] and [2016-08-29 19:12:55.176349]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:11:15.248883] and [2016-08-29 19:12:57.938922]
[2016-08-29 19:13:10.175676] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:13:18.473968] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:13:19.616793] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 19:13:19.616793] and [2016-08-29 19:14:49.914667]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:13:10.175676] and [2016-08-29 19:14:55.189248]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:13:18.473968] and [2016-08-29 19:15:01.163979]
[2016-08-29 19:15:10.191357] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:15:17.550937] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:15:21.708377] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 19:15:17.550937] and [2016-08-29 19:16:20.206995]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:15:10.191357] and [2016-08-29 19:16:55.199030]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:15:21.708377] and [2016-08-29 19:17:04.415079]
[2016-08-29 19:17:10.198421] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:17:20.397498] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:17:24.971119] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:17:20.397498] and [2016-08-29 19:18:20.593585]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 19:17:24.971119] and [2016-08-29 19:18:47.165303]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:17:10.198421] and [2016-08-29 19:18:55.197480]
[2016-08-29 19:19:07.697658] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:19:10.198071] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:19:20.782390] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:19:07.697658] and [2016-08-29 19:20:50.389853]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 19:19:20.782390] and [2016-08-29 19:20:51.072651]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:19:10.198071] and [2016-08-29 19:20:55.197393]
[2016-08-29 19:21:10.196228] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:21:10.943298] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:21:21.165604] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:21:10.943342] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (781264e2-27ee-4935-86f3-718b4768812a) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:21:21.165604] and [2016-08-29 19:22:51.456194]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:21:10.196228] and [2016-08-29 19:22:55.202388]
[2016-08-29 19:23:10.201713] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:23:21.547752] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 19:23:21.547752] and [2016-08-29 19:24:51.913931]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:23:10.201713] and [2016-08-29 19:24:55.195541]
[2016-08-29 19:25:10.202943] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:25:21.929209] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:25:21.929209] and [2016-08-29 19:26:24.710686]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:25:10.202943] and [2016-08-29 19:26:57.765960]
[2016-08-29 19:27:12.761813] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:27:22.493476] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:27:22.493476] and [2016-08-29 19:28:25.063871]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:27:12.761813] and [2016-08-29 19:28:57.756249]
[2016-08-29 19:29:12.758013] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:29:25.251009] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:29:25.251009] and [2016-08-29 19:30:55.540049]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:29:12.758013] and [2016-08-29 19:30:57.756884]
[2016-08-29 19:31:12.765411] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:31:25.631002] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:31:25.631002] and [2016-08-29 19:32:55.948283]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:31:12.765411] and [2016-08-29 19:32:57.761998]
[2016-08-29 19:33:12.761094] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:33:26.011141] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 19:33:26.011141] and [2016-08-29 19:34:56.310730]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 13 times between [2016-08-29 19:33:12.761094] and [2016-08-29 19:34:57.774516]
[2016-08-29 19:35:12.773728] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:35:26.391579] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 19:35:26.391579] and [2016-08-29 19:36:26.601615]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:35:12.773728] and [2016-08-29 19:36:57.771460]
[2016-08-29 19:37:12.770460] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:37:26.771626] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:37:26.771626] and [2016-08-29 19:38:26.994883]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:37:12.770460] and [2016-08-29 19:38:57.772186]
[2016-08-29 19:39:12.775076] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:39:24.980516] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:39:24.980516] and [2016-08-29 19:40:57.440277]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:39:12.775076] and [2016-08-29 19:40:57.781796]
[2016-08-29 19:41:14.726279] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:41:29.480049] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:41:29.480049] and [2016-08-29 19:42:29.700064]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 19:41:14.726279] and [2016-08-29 19:42:29.725818]
[2016-08-29 19:43:14.743412] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:43:29.860716] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 13 times between [2016-08-29 19:43:14.743412] and [2016-08-29 19:44:59.734874]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 19:43:29.860716] and [2016-08-29 19:45:00.174875]
[2016-08-29 19:45:14.734413] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:45:30.241247] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:45:30.241247] and [2016-08-29 19:46:30.435054]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:45:14.734413] and [2016-08-29 19:46:44.747345]
[2016-08-29 19:47:14.745353] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:47:30.621679] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:47:30.621679] and [2016-08-29 19:48:30.815135]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 19:47:14.745353] and [2016-08-29 19:48:44.748791]
[2016-08-29 19:49:14.746826] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:49:31.001961] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:49:14.746826] and [2016-08-29 19:50:59.763130]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 19:49:31.001961] and [2016-08-29 19:51:01.291039]
[2016-08-29 19:51:14.762193] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:51:31.382298] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 19:51:14.762193] and [2016-08-29 19:52:44.770520]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:51:31.382298] and [2016-08-29 19:53:01.671292]
[2016-08-29 19:53:14.766912] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:53:31.762348] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 19:53:14.766912] and [2016-08-29 19:54:44.768031]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 19:53:31.762348] and [2016-08-29 19:55:02.068331]
[2016-08-29 19:55:14.768718] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:55:32.142707] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:55:32.142707] and [2016-08-29 19:56:32.348206]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 19:55:14.768718] and [2016-08-29 19:56:44.770949]
[2016-08-29 19:57:14.769145] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:57:32.522771] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 19:57:32.522771] and [2016-08-29 19:58:32.728968]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 19:57:14.769145] and [2016-08-29 19:58:44.777160]
[2016-08-29 19:59:14.777783] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 19:59:30.868839] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 19:59:14.777783] and [2016-08-29 20:00:44.792064]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 19:59:30.868839] and [2016-08-29 20:01:03.192190]
[2016-08-29 20:01:16.733601] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:01:35.227118] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 20:01:16.733601] and [2016-08-29 20:02:46.735740]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 20:01:35.227118] and [2016-08-29 20:03:05.533724]
[2016-08-29 20:03:16.729626] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:03:35.607091] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:03:16.729626] and [2016-08-29 20:04:46.734081]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:03:35.607091] and [2016-08-29 20:05:05.916120]
[2016-08-29 20:05:16.732632] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:05:34.029174] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 20:05:34.029174] and [2016-08-29 20:06:36.181268]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 20:05:16.732632] and [2016-08-29 20:06:46.730325]
[2016-08-29 20:07:16.733418] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:07:36.367497] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 20:07:36.367497] and [2016-08-29 20:08:36.561458]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:07:16.733418] and [2016-08-29 20:08:46.734395]
[2016-08-29 20:09:16.734981] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:09:36.747551] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 20:09:16.734981] and [2016-08-29 20:11:01.731524]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:09:36.747551] and [2016-08-29 20:11:07.041470]
[2016-08-29 20:11:16.731110] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:11:37.127795] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 20:11:16.731110] and [2016-08-29 20:12:46.736461]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 20:11:37.127795] and [2016-08-29 20:13:07.416892]
[2016-08-29 20:13:16.734433] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:13:37.507907] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:13:16.734433] and [2016-08-29 20:14:46.734857]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:13:37.507907] and [2016-08-29 20:15:07.824813]
[2016-08-29 20:15:16.735105] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:15:37.887990] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 20:15:37.887990] and [2016-08-29 20:16:38.081907]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 20:15:16.735105] and [2016-08-29 20:16:46.737431]
[2016-08-29 20:17:18.681193] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:17:40.217960] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:17:18.681193] and [2016-08-29 20:18:48.682064]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 20:17:40.217960] and [2016-08-29 20:19:10.507295]
[2016-08-29 20:19:18.679907] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:19:38.777704] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:19:18.679907] and [2016-08-29 20:20:48.706375]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:19:38.777704] and [2016-08-29 20:21:10.912702]
[2016-08-29 20:21:18.699006] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:21:40.979292] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 20:21:18.699006] and [2016-08-29 20:22:48.703838]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 20:21:40.979292] and [2016-08-29 20:23:11.267936]
[2016-08-29 20:23:18.706874] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:23:41.359632] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:23:18.706874] and [2016-08-29 20:24:48.706192]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:23:41.359632] and [2016-08-29 20:25:11.664995]
[2016-08-29 20:25:18.703756] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:25:41.739932] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:25:41.739932] and [2016-08-29 20:26:41.935751]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 20:25:18.703756] and [2016-08-29 20:26:48.708506]
[2016-08-29 20:27:18.715095] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:27:42.120109] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:27:18.715095] and [2016-08-29 20:28:48.713089]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 20:27:42.120109] and [2016-08-29 20:29:12.409430]
[2016-08-29 20:29:18.711051] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:29:42.500352] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:29:18.711051] and [2016-08-29 20:30:48.707663]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 20:29:42.500352] and [2016-08-29 20:31:12.789443]
[2016-08-29 20:31:18.709494] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:31:42.880414] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 20:31:18.709494] and [2016-08-29 20:32:48.717265]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 20:31:42.880414] and [2016-08-29 20:33:13.169313]
[2016-08-29 20:33:18.731800] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:33:41.522647] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:33:18.731800] and [2016-08-29 20:34:48.736653]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:33:41.522647] and [2016-08-29 20:35:13.548747]
[2016-08-29 20:35:18.734720] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:35:43.640814] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 20:35:43.640814] and [2016-08-29 20:36:43.835726]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 20:35:18.734720] and [2016-08-29 20:36:48.735371]
[2016-08-29 20:37:18.733149] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:37:44.020927] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 7 times between [2016-08-29 20:37:18.733149] and [2016-08-29 20:38:50.680002]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 20:37:44.020927] and [2016-08-29 20:39:16.252303]
[2016-08-29 20:39:20.680385] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:39:46.344277] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 5 times between [2016-08-29 20:39:46.344277] and [2016-08-29 20:40:52.918047]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 9 times between [2016-08-29 20:39:20.680385] and [2016-08-29 20:41:05.675195]
[2016-08-29 20:41:16.629391] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:41:20.679494] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
The message "W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 11 times between [2016-08-29 20:41:16.629391] and [2016-08-29 20:42:54.121190]
The message "W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 13 times between [2016-08-29 20:41:20.679494] and [2016-08-29 20:43:05.679933]
[2016-08-29 20:43:17.009479] W [MSGID: 114061] [client-common.c:460:client_pre_fsync] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-29 20:43:20.678466] W [MSGID: 114061] [client-common.c:705:client_pre_fstat] 0-glustershard-client-0:  (63705978-6620-445b-b831-7b89c4675c52) remote_fd is -1. EBADFD [File descriptor in bad state]
[2016-08-28 08:41:09.628667] I [glusterfsd-mgmt.c:1596:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-08-29 18:51:08.809346] W [glusterfsd.c:1286:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f468a50edc5] -->/usr/sbin/glusterfsd(glusterfs_sigwaiter+0xe5) [0x7f468bba2c45] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x7f468bba2abb] ) 0-: received signum (15), shutting down
[2016-08-29 18:54:10.347519] I [MSGID: 100030] [glusterfsd.c:2408:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.8.2 (args: /usr/sbin/glusterfsd -s 192.168.71.10 --volfile-id glustershard.192.168.71.10.gluster2-brick1-1 -p /var/lib/glusterd/vols/glustershard/run/192.168.71.10-gluster2-brick1-1.pid -S /var/run/gluster/78f4acfaf65e49eed806f2fc379f043f.socket --brick-name /gluster2/brick1/1 -l /var/log/glusterfs/bricks/gluster2-brick1-1.log --xlator-option *-posix.glusterd-uuid=6f736bac-fe98-4e68-ae29-bfb4902ddc8c --brick-port 49152 --xlator-option glustershard-server.listen-port=49152)
[2016-08-29 18:54:10.357098] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-08-29 18:54:10.360795] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-glustershard-server: adding option 'listen-port' for volume 'glustershard-server' with value '49152'
[2016-08-29 18:54:10.360831] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-glustershard-posix: adding option 'glusterd-uuid' for volume 'glustershard-posix' with value '6f736bac-fe98-4e68-ae29-bfb4902ddc8c'
[2016-08-29 18:54:10.361053] I [MSGID: 115034] [server.c:398:_check_for_auth_option] 0-glustershard-decompounder: skip format check for non-addr auth option auth.login./gluster2/brick1/1.allow
[2016-08-29 18:54:10.361061] I [MSGID: 115034] [server.c:398:_check_for_auth_option] 0-glustershard-decompounder: skip format check for non-addr auth option auth.login.829a0bdc-b3aa-4711-8e56-155efa464278.password
[2016-08-29 18:54:10.361068] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-08-29 18:54:10.361842] I [rpcsvc.c:2199:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2016-08-29 18:54:10.361930] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-glustershard-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
[2016-08-29 18:54:10.363981] I [MSGID: 121050] [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled.
[2016-08-29 18:54:10.364004] W [MSGID: 101105] [gfdb_sqlite3.h:234:gfdb_set_sql_params] 0-glustershard-changetimerecorder: Failed to retrieve sql-db-pagesize from params.Assigning default value: 4096
[2016-08-29 18:54:10.364013] W [MSGID: 101105] [gfdb_sqlite3.h:234:gfdb_set_sql_params] 0-glustershard-changetimerecorder: Failed to retrieve sql-db-journalmode from params.Assigning default value: wal
[2016-08-29 18:54:10.364021] W [MSGID: 101105] [gfdb_sqlite3.h:234:gfdb_set_sql_params] 0-glustershard-changetimerecorder: Failed to retrieve sql-db-sync from params.Assigning default value: off
[2016-08-29 18:54:10.364028] W [MSGID: 101105] [gfdb_sqlite3.h:234:gfdb_set_sql_params] 0-glustershard-changetimerecorder: Failed to retrieve sql-db-autovacuum from params.Assigning default value: none
[2016-08-29 18:54:10.436472] I [trash.c:2414:init] 0-glustershard-trash: no option specified for 'eliminate', using NULL
[2016-08-29 18:54:10.436701] W [MSGID: 113083] [posix.c:6990:init] 0-glustershard-posix: Posix access control list is not supported. [Operation not supported]
[2016-08-29 18:54:10.437287] W [MSGID: 101174] [graph.c:360:_log_if_unknown_option] 0-glustershard-server: option 'rpc-auth.auth-glusterfs' is not recognized
[2016-08-29 18:54:10.437321] W [MSGID: 101174] [graph.c:360:_log_if_unknown_option] 0-glustershard-server: option 'rpc-auth.auth-unix' is not recognized
[2016-08-29 18:54:10.437343] W [MSGID: 101174] [graph.c:360:_log_if_unknown_option] 0-glustershard-server: option 'rpc-auth.auth-null' is not recognized
[2016-08-29 18:54:10.437388] W [MSGID: 101174] [graph.c:360:_log_if_unknown_option] 0-glustershard-server: option 'auth-path' is not recognized
[2016-08-29 18:54:10.437415] W [MSGID: 101174] [graph.c:360:_log_if_unknown_option] 0-glustershard-quota: option 'timeout' is not recognized
[2016-08-29 18:54:10.437470] W [MSGID: 101174] [graph.c:360:_log_if_unknown_option] 0-glustershard-trash: option 'brick-path' is not recognized
Final graph:
+------------------------------------------------------------------------------+
  1: volume glustershard-posix
  2:     type storage/posix
  3:     option glusterd-uuid 6f736bac-fe98-4e68-ae29-bfb4902ddc8c
  4:     option directory /gluster2/brick1/1
  5:     option volume-id 5889332e-50ba-441e-8fa5-cce3ae6f3a15
  6:     option brick-uid 36
  7:     option brick-gid 36
  8: end-volume
  9:  
 10: volume glustershard-trash
 11:     type features/trash
 12:     option trash-dir .trashcan
 13:     option brick-path /gluster2/brick1/1
 14:     option trash-internal-op off
 15:     subvolumes glustershard-posix
 16: end-volume
 17:  
 18: volume glustershard-changetimerecorder
 19:     type features/changetimerecorder
 20:     option db-type sqlite3
 21:     option hot-brick off
 22:     option db-name 1.db
 23:     option db-path /gluster2/brick1/1/.glusterfs/
 24:     option record-exit off
 25:     option ctr_link_consistency off
 26:     option ctr_lookupheal_link_timeout 300
 27:     option ctr_lookupheal_inode_timeout 300
 28:     option record-entry on
 29:     option ctr-enabled off
 30:     option record-counters off
 31:     option ctr-record-metadata-heat off
 32:     option sql-db-cachesize 1000
 33:     option sql-db-wal-autocheckpoint 1000
 34:     subvolumes glustershard-trash
 35: end-volume
 36:  
 37: volume glustershard-changelog
 38:     type features/changelog
 39:     option changelog-brick /gluster2/brick1/1
 40:     option changelog-dir /gluster2/brick1/1/.glusterfs/changelogs
 41:     option changelog-barrier-timeout 120
 42:     subvolumes glustershard-changetimerecorder
 43: end-volume
 44:  
 45: volume glustershard-bitrot-stub
 46:     type features/bitrot-stub
 47:     option export /gluster2/brick1/1
 48:     subvolumes glustershard-changelog
 49: end-volume
 50:  
 51: volume glustershard-access-control
 52:     type features/access-control
 53:     subvolumes glustershard-bitrot-stub
 54: end-volume
 55:  
 56: volume glustershard-locks
 57:     type features/locks
 58:     subvolumes glustershard-access-control
 59: end-volume
 60:  
 61: volume glustershard-worm
 62:     type features/worm
 63:     option worm off
 64:     option worm-file-level off
 65:     subvolumes glustershard-locks
 66: end-volume
 67:  
 68: volume glustershard-read-only
 69:     type features/read-only
 70:     option read-only off
 71:     subvolumes glustershard-worm
 72: end-volume
 73:  
 74: volume glustershard-leases
 75:     type features/leases
 76:     option leases off
 77:     subvolumes glustershard-read-only
 78: end-volume
 79:  
 80: volume glustershard-upcall
 81:     type features/upcall
 82:     option cache-invalidation off
 83:     subvolumes glustershard-leases
 84: end-volume
 85:  
 86: volume glustershard-io-threads
 87:     type performance/io-threads
 88:     subvolumes glustershard-upcall
 89: end-volume
 90:  
 91: volume glustershard-marker
 92:     type features/marker
 93:     option volume-uuid 5889332e-50ba-441e-8fa5-cce3ae6f3a15
 94:     option timestamp-file /var/lib/glusterd/vols/glustershard/marker.tstamp
 95:     option quota-version 0
 96:     option xtime off
 97:     option gsync-force-xtime off
 98:     option quota off
 99:     option inode-quota off
100:     subvolumes glustershard-io-threads
101: end-volume
102:  
103: volume glustershard-barrier
104:     type features/barrier
105:     option barrier disable
106:     option barrier-timeout 120
107:     subvolumes glustershard-marker
108: end-volume
109:  
110: volume glustershard-index
111:     type features/index
112:     option index-base /gluster2/brick1/1/.glusterfs/indices
113:     option xattrop-dirty-watchlist trusted.afr.dirty
114:     option xattrop-pending-watchlist trusted.afr.glustershard-
115:     subvolumes glustershard-barrier
116: end-volume
117:  
118: volume glustershard-quota
119:     type features/quota
120:     option volume-uuid glustershard
121:     option server-quota off
122:     option timeout 0
123:     option deem-statfs off
124:     subvolumes glustershard-index
125: end-volume
126:  
127: volume /gluster2/brick1/1
128:     type debug/io-stats
129:     option log-level INFO
130:     option latency-measurement off
131:     option count-fop-hits off
132:     subvolumes glustershard-quota
133: end-volume
134:  
135: volume glustershard-decompounder
136:     type performance/decompounder
137:     subvolumes /gluster2/brick1/1
138: end-volume
139:  
140: volume glustershard-server
141:     type protocol/server
142:     option transport.socket.listen-port 49152
143:     option rpc-auth.auth-glusterfs on
144:     option rpc-auth.auth-unix on
145:     option rpc-auth.auth-null on
146:     option transport-type tcp
147:     option auth.login./gluster2/brick1/1.allow 829a0bdc-b3aa-4711-8e56-155efa464278
148:     option auth.login.829a0bdc-b3aa-4711-8e56-155efa464278.password 222c3637-0c6a-4dd8-820f-dc15bda65f4a
149:     option auth-path /gluster2/brick1/1
150:     option auth.addr./gluster2/brick1/1.allow *
151:     option rpc-auth-allow-insecure on
152:     subvolumes glustershard-decompounder
153: end-volume
154:  
+------------------------------------------------------------------------------+
[2016-08-29 18:54:11.621321] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 18:54:11.621351] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-14233-2016/08/15-19:11:50:973716-glustershard-client-0-0-2 (version: 3.8.2)
[2016-08-29 18:54:12.815105] I [MSGID: 115072] [server-rpc-fops.c:1638:server_setattr_cbk] 0-glustershard-server: 13812343: SETATTR /fakedir (15d2f696-2bfa-4eef-86f1-ff709394b189) ==> (Operation not permitted) [Operation not permitted]
[2016-08-29 18:54:15.642783] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 18:54:15.642817] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-27746-2016/08/29-18:54:11:616398-glustershard-client-0-0-0 (version: 3.8.2)
[2016-08-29 18:56:19.885993] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 18:56:19.886019] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-28002-2016/08/29-18:56:19:855501-glustershard-client-0-0-0 (version: 3.8.2)
[2016-08-29 18:56:19.950807] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-28002-2016/08/29-18:56:19:855501-glustershard-client-0-0-0
[2016-08-29 18:56:19.950902] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-28002-2016/08/29-18:56:19:855501-glustershard-client-0-0-0
[2016-08-29 18:56:29.112044] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 18:56:29.112070] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-28027-2016/08/29-18:56:29:82417-glustershard-client-0-0-0 (version: 3.8.2)
[2016-08-29 18:56:29.176891] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-28027-2016/08/29-18:56:29:82417-glustershard-client-0-0-0
[2016-08-29 18:56:29.176980] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-28027-2016/08/29-18:56:29:82417-glustershard-client-0-0-0
[2016-08-29 18:58:51.885784] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 18:58:51.885812] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-28229-2016/08/29-18:58:51:855564-glustershard-client-0-0-0 (version: 3.8.2)
[2016-08-29 18:58:51.948041] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-28229-2016/08/29-18:58:51:855564-glustershard-client-0-0-0
[2016-08-29 18:58:51.948128] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-28229-2016/08/29-18:58:51:855564-glustershard-client-0-0-0
[2016-08-29 19:07:53.017234] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:07:53.017262] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-28913-2016/08/29-19:07:52:985762-glustershard-client-0-0-0 (version: 3.8.2)
[2016-08-29 19:07:53.082387] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-28913-2016/08/29-19:07:52:985762-glustershard-client-0-0-0
[2016-08-29 19:07:53.082476] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-28913-2016/08/29-19:07:52:985762-glustershard-client-0-0-0
[2016-08-29 19:09:01.260558] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:09:01.260585] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-29041-2016/08/29-19:09:01:229983-glustershard-client-0-0-0 (version: 3.8.2)
[2016-08-29 19:09:01.342410] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-29041-2016/08/29-19:09:01:229983-glustershard-client-0-0-0
[2016-08-29 19:09:01.342488] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-29041-2016/08/29-19:09:01:229983-glustershard-client-0-0-0
[2016-08-29 19:09:36.398095] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:09:36.398129] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-29139-2016/08/29-19:09:36:366451-glustershard-client-0-0-0 (version: 3.8.2)
[2016-08-29 19:09:36.500668] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-29139-2016/08/29-19:09:36:366451-glustershard-client-0-0-0
[2016-08-29 19:09:36.500721] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-29139-2016/08/29-19:09:36:366451-glustershard-client-0-0-0
[2016-08-29 19:10:00.421257] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:10:00.421285] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-29197-2016/08/29-19:10:00:392193-glustershard-client-0-0-0 (version: 3.8.2)
[2016-08-29 19:10:00.525481] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-29197-2016/08/29-19:10:00:392193-glustershard-client-0-0-0
[2016-08-29 19:10:00.525555] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-29197-2016/08/29-19:10:00:392193-glustershard-client-0-0-0
[2016-08-29 19:18:25.494886] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:18:25.494936] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-29964-2016/08/29-19:18:25:465005-glustershard-client-0-0-0 (version: 3.8.2)
[2016-08-29 19:18:25.638797] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-29964-2016/08/29-19:18:25:465005-glustershard-client-0-0-0
[2016-08-29 19:18:25.638852] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-29964-2016/08/29-19:18:25:465005-glustershard-client-0-0-0
[2016-08-29 19:19:49.071312] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:19:49.071382] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-30090-2016/08/29-19:19:49:41190-glustershard-client-0-0-0 (version: 3.8.2)
[2016-08-29 19:19:50.684543] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-30090-2016/08/29-19:19:49:41190-glustershard-client-0-0-0
[2016-08-29 19:19:50.684634] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-30090-2016/08/29-19:19:49:41190-glustershard-client-0-0-0
[2016-08-28 08:41:09.628413] I [glusterfsd-mgmt.c:1596:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-08-29 18:54:10.614325] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-16922-2016/08/15-19:40:41:616599-glustershard-client-1-0-0
[2016-08-29 18:54:10.614371] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-16922-2016/08/15-19:40:41:616599-glustershard-client-1-0-0
[2016-08-29 18:54:15.648470] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 18:54:15.648497] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-27746-2016/08/29-18:54:11:616398-glustershard-client-1-0-0 (version: 3.8.2)
[2016-08-29 18:56:19.891386] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 18:56:19.891410] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-28002-2016/08/29-18:56:19:855501-glustershard-client-1-0-0 (version: 3.8.2)
[2016-08-29 18:56:19.950806] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-28002-2016/08/29-18:56:19:855501-glustershard-client-1-0-0
[2016-08-29 18:56:19.950897] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-28002-2016/08/29-18:56:19:855501-glustershard-client-1-0-0
[2016-08-29 18:56:29.117496] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 18:56:29.117522] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-28027-2016/08/29-18:56:29:82417-glustershard-client-1-0-0 (version: 3.8.2)
[2016-08-29 18:56:29.176893] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-28027-2016/08/29-18:56:29:82417-glustershard-client-1-0-0
[2016-08-29 18:56:29.176979] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-28027-2016/08/29-18:56:29:82417-glustershard-client-1-0-0
[2016-08-29 18:58:51.891512] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 18:58:51.891540] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-28229-2016/08/29-18:58:51:855564-glustershard-client-1-0-0 (version: 3.8.2)
[2016-08-29 18:58:51.947982] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-28229-2016/08/29-18:58:51:855564-glustershard-client-1-0-0
[2016-08-29 18:58:51.948025] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-28229-2016/08/29-18:58:51:855564-glustershard-client-1-0-0
[2016-08-29 19:07:53.022636] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:07:53.022658] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-28913-2016/08/29-19:07:52:985762-glustershard-client-1-0-0 (version: 3.8.2)
[2016-08-29 19:07:53.082344] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-28913-2016/08/29-19:07:52:985762-glustershard-client-1-0-0
[2016-08-29 19:07:53.082388] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-28913-2016/08/29-19:07:52:985762-glustershard-client-1-0-0
[2016-08-29 19:09:01.266262] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:09:01.266288] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-29041-2016/08/29-19:09:01:229983-glustershard-client-1-0-0 (version: 3.8.2)
[2016-08-29 19:09:01.342360] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-29041-2016/08/29-19:09:01:229983-glustershard-client-1-0-0
[2016-08-29 19:09:01.342413] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-29041-2016/08/29-19:09:01:229983-glustershard-client-1-0-0
[2016-08-29 19:09:36.403419] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:09:36.403445] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-29139-2016/08/29-19:09:36:366451-glustershard-client-1-0-0 (version: 3.8.2)
[2016-08-29 19:09:36.500654] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-29139-2016/08/29-19:09:36:366451-glustershard-client-1-0-0
[2016-08-29 19:09:36.500706] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-29139-2016/08/29-19:09:36:366451-glustershard-client-1-0-0
[2016-08-29 19:10:00.426657] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:10:00.426683] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-29197-2016/08/29-19:10:00:392193-glustershard-client-1-0-0 (version: 3.8.2)
[2016-08-29 19:10:00.525429] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-29197-2016/08/29-19:10:00:392193-glustershard-client-1-0-0
[2016-08-29 19:10:00.525480] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-29197-2016/08/29-19:10:00:392193-glustershard-client-1-0-0
[2016-08-29 19:18:25.499926] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:18:25.499951] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-29964-2016/08/29-19:18:25:465005-glustershard-client-1-0-0 (version: 3.8.2)
[2016-08-29 19:18:25.638764] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-29964-2016/08/29-19:18:25:465005-glustershard-client-1-0-0
[2016-08-29 19:18:25.638822] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-29964-2016/08/29-19:18:25:465005-glustershard-client-1-0-0
[2016-08-29 19:19:49.076477] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:19:49.076503] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-30090-2016/08/29-19:19:49:41190-glustershard-client-1-0-0 (version: 3.8.2)
[2016-08-29 19:19:50.684542] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-30090-2016/08/29-19:19:49:41190-glustershard-client-1-0-0
[2016-08-29 19:19:50.684632] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-30090-2016/08/29-19:19:49:41190-glustershard-client-1-0-0
[2016-08-28 08:41:09.628390] I [glusterfsd-mgmt.c:1596:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-08-29 18:54:10.614298] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-16922-2016/08/15-19:40:41:616599-glustershard-client-2-0-0
[2016-08-29 18:54:10.614342] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-16922-2016/08/15-19:40:41:616599-glustershard-client-2-0-0
[2016-08-29 18:54:12.833867] E [index.c:665:index_del] 0-glustershard-index: /gluster2/brick3/1/.glusterfs/indices/entry-changes/00000000-0000-0000-0000-000000000001: failed to delete from index (Directory not empty)
[2016-08-29 18:54:15.651229] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 18:54:15.651253] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-27746-2016/08/29-18:54:11:616398-glustershard-client-2-0-0 (version: 3.8.2)
[2016-08-29 18:56:19.894225] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 18:56:19.894252] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-28002-2016/08/29-18:56:19:855501-glustershard-client-2-0-0 (version: 3.8.2)
[2016-08-29 18:56:19.950744] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-28002-2016/08/29-18:56:19:855501-glustershard-client-2-0-0
[2016-08-29 18:56:19.950792] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-28002-2016/08/29-18:56:19:855501-glustershard-client-2-0-0
[2016-08-29 18:56:29.120405] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 18:56:29.120430] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-28027-2016/08/29-18:56:29:82417-glustershard-client-2-0-0 (version: 3.8.2)
[2016-08-29 18:56:29.176830] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-28027-2016/08/29-18:56:29:82417-glustershard-client-2-0-0
[2016-08-29 18:56:29.176880] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-28027-2016/08/29-18:56:29:82417-glustershard-client-2-0-0
[2016-08-29 18:58:51.894350] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 18:58:51.894375] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-28229-2016/08/29-18:58:51:855564-glustershard-client-2-0-0 (version: 3.8.2)
[2016-08-29 18:58:51.948012] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-28229-2016/08/29-18:58:51:855564-glustershard-client-2-0-0
[2016-08-29 18:58:51.948105] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-28229-2016/08/29-18:58:51:855564-glustershard-client-2-0-0
[2016-08-29 19:07:53.025417] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:07:53.025439] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-28913-2016/08/29-19:07:52:985762-glustershard-client-2-0-0 (version: 3.8.2)
[2016-08-29 19:07:53.082363] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-28913-2016/08/29-19:07:52:985762-glustershard-client-2-0-0
[2016-08-29 19:07:53.082473] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-28913-2016/08/29-19:07:52:985762-glustershard-client-2-0-0
[2016-08-29 19:09:01.269260] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:09:01.269286] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-29041-2016/08/29-19:09:01:229983-glustershard-client-2-0-0 (version: 3.8.2)
[2016-08-29 19:09:01.342398] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-29041-2016/08/29-19:09:01:229983-glustershard-client-2-0-0
[2016-08-29 19:09:01.342481] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-29041-2016/08/29-19:09:01:229983-glustershard-client-2-0-0
[2016-08-29 19:09:36.406260] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:09:36.406285] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-29139-2016/08/29-19:09:36:366451-glustershard-client-2-0-0 (version: 3.8.2)
[2016-08-29 19:09:36.500648] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-29139-2016/08/29-19:09:36:366451-glustershard-client-2-0-0
[2016-08-29 19:09:36.500699] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-29139-2016/08/29-19:09:36:366451-glustershard-client-2-0-0
[2016-08-29 19:10:00.429462] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:10:00.429487] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-29197-2016/08/29-19:10:00:392193-glustershard-client-2-0-0 (version: 3.8.2)
[2016-08-29 19:10:00.525422] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-29197-2016/08/29-19:10:00:392193-glustershard-client-2-0-0
[2016-08-29 19:10:00.525480] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-29197-2016/08/29-19:10:00:392193-glustershard-client-2-0-0
[2016-08-29 19:18:25.502736] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:18:25.502761] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-29964-2016/08/29-19:18:25:465005-glustershard-client-2-0-0 (version: 3.8.2)
[2016-08-29 19:18:25.638791] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-29964-2016/08/29-19:18:25:465005-glustershard-client-2-0-0
[2016-08-29 19:18:25.638890] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-29964-2016/08/29-19:18:25:465005-glustershard-client-2-0-0
[2016-08-29 19:19:49.079350] I [login.c:76:gf_auth] 0-auth/login: allowed user names: 829a0bdc-b3aa-4711-8e56-155efa464278
[2016-08-29 19:19:49.079377] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-30090-2016/08/29-19:19:49:41190-glustershard-client-2-0-0 (version: 3.8.2)
[2016-08-29 19:19:50.684470] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-30090-2016/08/29-19:19:49:41190-glustershard-client-2-0-0
[2016-08-29 19:19:50.684521] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-30090-2016/08/29-19:19:49:41190-glustershard-client-2-0-0
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users