Re: [Gluster-users] 3.7.13 & proxmox/qemu

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 






On Fri, Jul 22, 2016 at 9:37 AM, Vijay Bellur <vbellur@xxxxxxxxxx> wrote:
On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen <samppah@xxxxxxxxxxxxx> wrote:
> Here is a quick way how to test this:
> GlusterFS 3.7.13 volume with default settings with brick on ZFS dataset. gluster-test1 is server and gluster-test2 is client mounting with FUSE.
>
> Writing file with oflag=direct is not ok:
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 bs=1024000
> dd: failed to open ‘file’: Invalid argument
>
> Enable network.remote-dio on Gluster Volume:
> [root@gluster-test1 gluster]# gluster volume set gluster network.remote-dio enable
> volume set: success
>
> Writing small file with oflag=direct is ok:
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 bs=1024000
> 1+0 records in
> 1+0 records out
> 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s
>
> Writing bigger file with oflag=direct is ok:
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 bs=1M
> 100+0 records in
> 100+0 records out
> 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s
>
> Enable Sharding on Gluster Volume:
> [root@gluster-test1 gluster]# gluster volume set gluster features.shard enable
> volume set: success
>
> Writing small file  with oflag=direct is ok:
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=1 bs=1M
> 1+0 records in
> 1+0 records out
> 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s
>
> Writing bigger file with oflag=direct is not ok:
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 bs=1M
> dd: error writing ‘file3’: Operation not permitted
> dd: closing output file ‘file3’: Operation not permitted
>


Thank you for these tests! would it be possible to share the brick and
client logs?

Not sure if his tests are same as my setup but here is what I end up with

Volume Name: glustershard
Type: Replicate
Volume ID: 0cc4efb6-3836-4caa-b24a-b3afb6e407c3
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.71.10:/gluster1/shard1/1
Brick2: 192.168.71.11:/gluster1/shard2/1
Brick3: 192.168.71.12:/gluster1/shard3/1
Options Reconfigured:
features.shard-block-size: 64MB
features.shard: on
server.allow-insecure: on
storage.owner-uid: 36
storage.owner-gid: 36
cluster.server-quorum-type: server
cluster.quorum-type: auto
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.quick-read: off
cluster.self-heal-window-size: 1024
cluster.background-self-heal-count: 16
nfs.enable-ino32: off
nfs.addr-namelookup: off
nfs.disable: on
performance.read-ahead: off
performance.readdir-ahead: on



 dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/192.168.71.11\:_glustershard/ oflag=direct count=100 bs=1M
81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/ __DIRECT_IO_TEST__                    .trashcan/                            
[root@ccengine2 ~]# dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/192.168.71.11\:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test oflag=direct count=100 bs=1M
dd: error writing ‘/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test’: Operation not permitted

creates the 64M file in expected location then the shard is 0

# file: gluster1/shard1/1/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x0200000000000000579231f3000e16e7
trusted.gfid=0xec6de302b35f427985639ca3e25d9df0
trusted.glusterfs.shard.block-size=0x0000000004000000
trusted.glusterfs.shard.file-size=0x0000000004000000000000000000000000000000000000010000000000000000


# file: gluster1/shard1/1/.shard/ec6de302-b35f-4279-8563-9ca3e25d9df0.1
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x2bfd3cc8a727489b9a0474241548fe80


Regards,
Vijay
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

[2016-07-22 16:26:44.645166] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.13 (args: /usr/sbin/glusterfs --volfile-server=192.168.71.11 --volfile-server=192.168.71.10 --volfile-server=192.168.71.12 --volfile-id=/glustershard /rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard)
[2016-07-22 16:26:44.655674] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-07-22 16:26:44.661967] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-07-22 16:26:44.662718] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-0: parent translators are ready, attempting connect on transport
[2016-07-22 16:26:44.665996] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-1: parent translators are ready, attempting connect on transport
[2016-07-22 16:26:44.666266] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-0: changing port to 49158 (from 0)
[2016-07-22 16:26:44.669000] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-2: parent translators are ready, attempting connect on transport
[2016-07-22 16:26:44.672423] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-07-22 16:26:44.672556] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-1: changing port to 49159 (from 0)
Final graph:
+------------------------------------------------------------------------------+
  1: volume glustershard-client-0
  2:     type protocol/client
  3:     option clnt-lk-version 1
  4:     option volfile-checksum 0
  5:     option volfile-key /glustershard
  6:     option client-version 3.7.13
  7:     option process-uuid ccengine2.carouselchecks.local-9617-2016/07/22-16:26:44:640508-glustershard-client-0-0-0
  8:     option fops-version 1298437
  9:     option ping-timeout 42
 10:     option remote-host 192.168.71.10
 11:     option remote-subvolume /gluster1/shard1/1
 12:     option transport-type socket
 13:     option username 1d603a2f-0acb-488d-884d-b520297521ff
 14:     option password 1223f473-2316-4e83-988c-c8dc431160ad
 15:     option filter-O_DIRECT enable
 16:     option send-gids true
 17: end-volume
 18:  
 19: volume glustershard-client-1
 20:     type protocol/client
 21:     option ping-timeout 42
 22:     option remote-host 192.168.71.11
 23:     option remote-subvolume /gluster1/shard2/1
 24:     option transport-type socket
 25:     option username 1d603a2f-0acb-488d-884d-b520297521ff
 26:     option password 1223f473-2316-4e83-988c-c8dc431160ad
 27:     option filter-O_DIRECT enable
 28:     option send-gids true
 29: end-volume
 30:  
 31: volume glustershard-client-2
 32:     type protocol/client
 33:     option ping-timeout 42
 34:     option remote-host 192.168.71.12
 35:     option remote-subvolume /gluster1/shard3/1
 36:     option transport-type socket
 37:     option username 1d603a2f-0acb-488d-884d-b520297521ff
 38:     option password 1223f473-2316-4e83-988c-c8dc431160ad
 39:     option filter-O_DIRECT enable
 40:     option send-gids true
 41: end-volume
 42:  
 43: volume glustershard-replicate-0
 44:     type cluster/replicate
 45:     option background-self-heal-count 16
 46:     option data-self-heal-window-size 1024
 47:     option eager-lock enable
 48:     option quorum-type auto
 49:     subvolumes glustershard-client-0 glustershard-client-1 glustershard-client-2
 50: end-volume
 51:  
 52: volume glustershard-dht
 53:     type cluster/distribute
 54:     subvolumes glustershard-replicate-0
 55: end-volume
 56:  
 57: volume glustershard-shard
 58:     type features/shard
 59:     option shard-block-size 64MB
 60:     subvolumes glustershard-dht
 61: end-volume
 62:  
 63: volume glustershard-write-behind
 64:     type performance/write-behind
 65:     subvolumes glustershard-shard
 66: end-volume
 67:  
 68: volume glustershard-readdir-ahead
 69:     type performance/readdir-ahead
 70:     subvolumes glustershard-write-behind
 71: end-volume
 72:  
 73: volume glustershard-open-behind
 74:     type performance/open-behind
 75:     subvolumes glustershard-readdir-ahead
 76: end-volume
 77:  
 78: volume glustershard
 79:     type debug/io-stats
 80:     option log-level INFO
 81:     option latency-measurement off
 82:     option count-fop-hits off
 83:     subvolumes glustershard-open-behind
 84: end-volume
 85:  
 86: volume meta-autoload
 87:     type meta
 88:     subvolumes glustershard
 89: end-volume
 90:  
+------------------------------------------------------------------------------+
[2016-07-22 16:26:44.674788] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-0: Connected to glustershard-client-0, attached to remote volume '/gluster1/shard1/1'.
[2016-07-22 16:26:44.674803] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-07-22 16:26:44.674880] I [MSGID: 108005] [afr-common.c:4142:afr_notify] 0-glustershard-replicate-0: Subvolume 'glustershard-client-0' came back up; going online.
[2016-07-22 16:26:44.675040] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-0: Server lk version = 1
[2016-07-22 16:26:44.675107] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-2: changing port to 49160 (from 0)
[2016-07-22 16:26:44.677423] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-07-22 16:26:44.677770] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-1: Connected to glustershard-client-1, attached to remote volume '/gluster1/shard2/1'.
[2016-07-22 16:26:44.677787] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-07-22 16:26:44.678037] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-1: Server lk version = 1
[2016-07-22 16:26:44.680286] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-07-22 16:26:44.680790] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-2: Connected to glustershard-client-2, attached to remote volume '/gluster1/shard3/1'.
[2016-07-22 16:26:44.680810] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2016-07-22 16:26:44.685439] I [fuse-bridge.c:5172:fuse_graph_setup] 0-fuse: switched to graph 0
[2016-07-22 16:26:44.685570] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-2: Server lk version = 1
[2016-07-22 16:26:44.685672] I [fuse-bridge.c:4083:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22
[2016-07-22 16:26:44.686808] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-1
[2016-07-22 16:26:44.686859] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-2
[2016-07-22 16:26:44.686894] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-0
[2016-07-22 16:27:48.460148] I [fuse-bridge.c:5013:fuse_thread_proc] 0-fuse: unmounting /rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard
[2016-07-22 16:27:48.460688] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fe368831dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fe369eab915] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fe369eab78b] ) 0-: received signum (15), shutting down
[2016-07-22 16:27:48.460712] I [fuse-bridge.c:5720:fini] 0-fuse: Unmounting '/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard'.
[2016-07-22 16:29:29.287998] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.13 (args: /usr/sbin/glusterfs --volfile-server=192.168.71.11 --volfile-server=192.168.71.10 --volfile-server=192.168.71.12 --volfile-id=/glustershard /rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard)
[2016-07-22 16:29:29.299869] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-07-22 16:29:29.306788] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-07-22 16:29:29.307656] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-0: parent translators are ready, attempting connect on transport
[2016-07-22 16:29:29.311044] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-1: parent translators are ready, attempting connect on transport
[2016-07-22 16:29:29.311425] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-0: changing port to 49158 (from 0)
[2016-07-22 16:29:29.314429] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-2: parent translators are ready, attempting connect on transport
[2016-07-22 16:29:29.317449] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-07-22 16:29:29.317584] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-1: changing port to 49159 (from 0)
Final graph:
+------------------------------------------------------------------------------+
  1: volume glustershard-client-0
  2:     type protocol/client
  3:     option clnt-lk-version 1
  4:     option volfile-checksum 0
  5:     option volfile-key /glustershard
  6:     option client-version 3.7.13
  7:     option process-uuid ccengine2.carouselchecks.local-10019-2016/07/22-16:29:29:282874-glustershard-client-0-0-0
  8:     option fops-version 1298437
  9:     option ping-timeout 42
 10:     option remote-host 192.168.71.10
 11:     option remote-subvolume /gluster1/shard1/1
 12:     option transport-type socket
 13:     option username 1d603a2f-0acb-488d-884d-b520297521ff
 14:     option password 1223f473-2316-4e83-988c-c8dc431160ad
 15:     option filter-O_DIRECT enable
 16:     option send-gids true
 17: end-volume
 18:  
 19: volume glustershard-client-1
 20:     type protocol/client
 21:     option ping-timeout 42
 22:     option remote-host 192.168.71.11
 23:     option remote-subvolume /gluster1/shard2/1
 24:     option transport-type socket
 25:     option username 1d603a2f-0acb-488d-884d-b520297521ff
 26:     option password 1223f473-2316-4e83-988c-c8dc431160ad
 27:     option filter-O_DIRECT enable
 28:     option send-gids true
 29: end-volume
 30:  
 31: volume glustershard-client-2
 32:     type protocol/client
 33:     option ping-timeout 42
 34:     option remote-host 192.168.71.12
 35:     option remote-subvolume /gluster1/shard3/1
 36:     option transport-type socket
 37:     option username 1d603a2f-0acb-488d-884d-b520297521ff
 38:     option password 1223f473-2316-4e83-988c-c8dc431160ad
 39:     option filter-O_DIRECT enable
 40:     option send-gids true
 41: end-volume
 42:  
 43: volume glustershard-replicate-0
 44:     type cluster/replicate
 45:     option background-self-heal-count 16
 46:     option data-self-heal-window-size 1024
 47:     option eager-lock enable
 48:     option quorum-type auto
 49:     subvolumes glustershard-client-0 glustershard-client-1 glustershard-client-2
 50: end-volume
 51:  
 52: volume glustershard-dht
 53:     type cluster/distribute
 54:     subvolumes glustershard-replicate-0
 55: end-volume
 56:  
 57: volume glustershard-shard
 58:     type features/shard
 59:     option shard-block-size 64MB
 60:     subvolumes glustershard-dht
 61: end-volume
 62:  
 63: volume glustershard-write-behind
 64:     type performance/write-behind
 65:     subvolumes glustershard-shard
 66: end-volume
 67:  
 68: volume glustershard-readdir-ahead
 69:     type performance/readdir-ahead
 70:     subvolumes glustershard-write-behind
 71: end-volume
 72:  
 73: volume glustershard-open-behind
 74:     type performance/open-behind
 75:     subvolumes glustershard-readdir-ahead
 76: end-volume
 77:  
 78: volume glustershard
 79:     type debug/io-stats
 80:     option log-level INFO
 81:     option latency-measurement off
 82:     option count-fop-hits off
 83:     subvolumes glustershard-open-behind
 84: end-volume
 85:  
 86: volume meta-autoload
 87:     type meta
 88:     subvolumes glustershard
 89: end-volume
 90:  
+------------------------------------------------------------------------------+
[2016-07-22 16:29:29.320203] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-0: Connected to glustershard-client-0, attached to remote volume '/gluster1/shard1/1'.
[2016-07-22 16:29:29.320217] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-07-22 16:29:29.320277] I [MSGID: 108005] [afr-common.c:4142:afr_notify] 0-glustershard-replicate-0: Subvolume 'glustershard-client-0' came back up; going online.
[2016-07-22 16:29:29.320333] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-0: Server lk version = 1
[2016-07-22 16:29:29.320475] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-2: changing port to 49160 (from 0)
[2016-07-22 16:29:29.322932] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-07-22 16:29:29.323349] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-1: Connected to glustershard-client-1, attached to remote volume '/gluster1/shard2/1'.
[2016-07-22 16:29:29.323403] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-07-22 16:29:29.323556] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-1: Server lk version = 1
[2016-07-22 16:29:29.325770] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-07-22 16:29:29.326260] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-2: Connected to glustershard-client-2, attached to remote volume '/gluster1/shard3/1'.
[2016-07-22 16:29:29.326271] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2016-07-22 16:29:29.330885] I [fuse-bridge.c:5172:fuse_graph_setup] 0-fuse: switched to graph 0
[2016-07-22 16:29:29.331017] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-2: Server lk version = 1
[2016-07-22 16:29:29.331105] I [fuse-bridge.c:4083:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22
[2016-07-22 16:29:29.332316] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-1
[2016-07-22 16:29:29.332424] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-0
[2016-07-22 16:29:29.332529] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-2
[2016-07-22 16:29:32.938721] I [MSGID: 109036] [dht-common.c:8888:dht_log_new_layout_for_dir_selfheal] 0-glustershard-dht: Setting layout of /81e19cd3-ae45-449c-b716-ec3e4ad4c2f0 with [Subvol_name: glustershard-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], 
[2016-07-22 16:29:32.944681] I [MSGID: 109036] [dht-common.c:8888:dht_log_new_layout_for_dir_selfheal] 0-glustershard-dht: Setting layout of /81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/dom_md with [Subvol_name: glustershard-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], 
[2016-07-22 16:29:33.014432] I [MSGID: 109066] [dht-rename.c:1568:dht_rename] 0-glustershard-dht: renaming /81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/dom_md/metadata.new (hash=glustershard-replicate-0/cache=glustershard-replicate-0) => /81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/dom_md/metadata (hash=glustershard-replicate-0/cache=<nul>)
[2016-07-22 16:29:33.034967] I [MSGID: 109036] [dht-common.c:8888:dht_log_new_layout_for_dir_selfheal] 0-glustershard-dht: Setting layout of /81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images with [Subvol_name: glustershard-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], 
[2016-07-22 16:29:37.914047] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-glustershard-client-2: remote operation failed [Operation not permitted]
[2016-07-22 16:29:37.915023] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-glustershard-client-0: remote operation failed [Operation not permitted]
[2016-07-22 16:29:37.915644] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-glustershard-client-1: remote operation failed [Operation not permitted]
[2016-07-22 16:29:37.915675] W [fuse-bridge.c:2227:fuse_readv_cbk] 0-glusterfs-fuse: 183: READ => -1 gfid=b00888f5-c098-4915-9b72-c28e9551bd64 fd=0x7f1bf00ba218 (Operation not permitted)
[2016-07-22 17:42:01.782069] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.13 (args: /usr/sbin/glusterfs --volfile-server=192.168.71.11 --volfile-server=192.168.71.10 --volfile-server=192.168.71.12 --volfile-id=/glustershard /rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard)
[2016-07-22 17:42:01.792599] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-07-22 17:42:01.799265] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-07-22 17:42:01.800235] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-0: parent translators are ready, attempting connect on transport
[2016-07-22 17:42:01.804137] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-1: parent translators are ready, attempting connect on transport
[2016-07-22 17:42:01.804403] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-0: changing port to 49158 (from 0)
[2016-07-22 17:42:01.807125] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-2: parent translators are ready, attempting connect on transport
[2016-07-22 17:42:01.810296] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-07-22 17:42:01.810429] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-1: changing port to 49159 (from 0)
Final graph:
+------------------------------------------------------------------------------+
  1: volume glustershard-client-0
  2:     type protocol/client
  3:     option clnt-lk-version 1
  4:     option volfile-checksum 0
  5:     option volfile-key /glustershard
  6:     option client-version 3.7.13
  7:     option process-uuid ccengine2.carouselchecks.local-15473-2016/07/22-17:42:01:777393-glustershard-client-0-0-0
  8:     option fops-version 1298437
  9:     option ping-timeout 42
 10:     option remote-host 192.168.71.10
 11:     option remote-subvolume /gluster1/shard1/1
 12:     option transport-type socket
 13:     option username 1d603a2f-0acb-488d-884d-b520297521ff
 14:     option password 1223f473-2316-4e83-988c-c8dc431160ad
 15:     option filter-O_DIRECT enable
 16:     option send-gids true
 17: end-volume
 18:  
 19: volume glustershard-client-1
 20:     type protocol/client
 21:     option ping-timeout 42
 22:     option remote-host 192.168.71.11
 23:     option remote-subvolume /gluster1/shard2/1
 24:     option transport-type socket
 25:     option username 1d603a2f-0acb-488d-884d-b520297521ff
 26:     option password 1223f473-2316-4e83-988c-c8dc431160ad
 27:     option filter-O_DIRECT enable
 28:     option send-gids true
 29: end-volume
 30:  
 31: volume glustershard-client-2
 32:     type protocol/client
 33:     option ping-timeout 42
 34:     option remote-host 192.168.71.12
 35:     option remote-subvolume /gluster1/shard3/1
 36:     option transport-type socket
 37:     option username 1d603a2f-0acb-488d-884d-b520297521ff
 38:     option password 1223f473-2316-4e83-988c-c8dc431160ad
 39:     option filter-O_DIRECT enable
 40:     option send-gids true
 41: end-volume
 42:  
 43: volume glustershard-replicate-0
 44:     type cluster/replicate
 45:     option background-self-heal-count 16
 46:     option data-self-heal-window-size 1024
 47:     option eager-lock enable
 48:     option quorum-type auto
 49:     subvolumes glustershard-client-0 glustershard-client-1 glustershard-client-2
 50: end-volume
 51:  
 52: volume glustershard-dht
 53:     type cluster/distribute
 54:     subvolumes glustershard-replicate-0
 55: end-volume
 56:  
 57: volume glustershard-shard
 58:     type features/shard
 59:     option shard-block-size 64MB
 60:     subvolumes glustershard-dht
 61: end-volume
 62:  
 63: volume glustershard-write-behind
 64:     type performance/write-behind
 65:     subvolumes glustershard-shard
 66: end-volume
 67:  
 68: volume glustershard-readdir-ahead
 69:     type performance/readdir-ahead
 70:     subvolumes glustershard-write-behind
 71: end-volume
 72:  
 73: volume glustershard-open-behind
 74:     type performance/open-behind
 75:     subvolumes glustershard-readdir-ahead
 76: end-volume
 77:  
 78: volume glustershard
 79:     type debug/io-stats
 80:     option log-level INFO
 81:     option latency-measurement off
 82:     option count-fop-hits off
 83:     subvolumes glustershard-open-behind
 84: end-volume
 85:  
 86: volume meta-autoload
 87:     type meta
 88:     subvolumes glustershard
 89: end-volume
 90:  
+------------------------------------------------------------------------------+
[2016-07-22 17:42:01.813111] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-0: Connected to glustershard-client-0, attached to remote volume '/gluster1/shard1/1'.
[2016-07-22 17:42:01.813123] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-07-22 17:42:01.813198] I [MSGID: 108005] [afr-common.c:4142:afr_notify] 0-glustershard-replicate-0: Subvolume 'glustershard-client-0' came back up; going online.
[2016-07-22 17:42:01.813358] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-0: Server lk version = 1
[2016-07-22 17:42:01.813433] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-2: changing port to 49160 (from 0)
[2016-07-22 17:42:01.815936] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-07-22 17:42:01.816506] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-1: Connected to glustershard-client-1, attached to remote volume '/gluster1/shard2/1'.
[2016-07-22 17:42:01.816558] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-07-22 17:42:01.816749] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-1: Server lk version = 1
[2016-07-22 17:42:01.818836] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-07-22 17:42:01.819263] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-2: Connected to glustershard-client-2, attached to remote volume '/gluster1/shard3/1'.
[2016-07-22 17:42:01.819330] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2016-07-22 17:42:01.823445] I [fuse-bridge.c:5172:fuse_graph_setup] 0-fuse: switched to graph 0
[2016-07-22 17:42:01.823574] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-2: Server lk version = 1
[2016-07-22 17:42:01.823661] I [fuse-bridge.c:4083:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22
[2016-07-22 17:42:01.824814] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-0
[2016-07-22 17:42:01.824848] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-1
[2016-07-22 17:42:01.824874] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-2
[2016-07-22 17:42:01.912806] I [fuse-bridge.c:5013:fuse_thread_proc] 0-fuse: unmounting /rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard
[2016-07-22 17:42:01.913219] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f1bfa492dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f1bfbb0c915] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f1bfbb0c78b] ) 0-: received signum (15), shutting down
[2016-07-22 17:42:01.913248] I [fuse-bridge.c:5720:fini] 0-fuse: Unmounting '/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard'.
[2016-07-22 17:42:05.699668] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.13 (args: /usr/sbin/glusterfs --volfile-server=192.168.71.11 --volfile-server=192.168.71.10 --volfile-server=192.168.71.12 --volfile-id=/glustershard /rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard)
[2016-07-22 17:42:05.710391] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-07-22 17:42:05.717590] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-07-22 17:42:05.718378] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-0: parent translators are ready, attempting connect on transport
[2016-07-22 17:42:05.721685] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-1: parent translators are ready, attempting connect on transport
[2016-07-22 17:42:05.721946] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-0: changing port to 49158 (from 0)
[2016-07-22 17:42:05.724682] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-2: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
  1: volume glustershard-client-0
  2:     type protocol/client
  3:     option ping-timeout 42
  4:     option remote-host 192.168.71.10
  5:     option remote-subvolume /gluster1/shard1/1
  6:     option transport-type socket
  7:     option username 1d603a2f-0acb-488d-884d-b520297521ff
  8:     option password 1223f473-2316-4e83-988c-c8dc431160ad
  9:     option filter-O_DIRECT enable
 10:     option send-gids true
 11: end-volume
 12:  
 13: volume glustershard-client-1
 14:     type protocol/client
 15:     option ping-timeout 42
 16:     option remote-host 192.168.71.11
 17:     option remote-subvolume /gluster1/shard2/1
 18:     option transport-type socket
 19:     option username 1d603a2f-0acb-488d-884d-b520297521ff
 20:     option password 1223f473-2316-4e83-988c-c8dc431160ad
 21:     option filter-O_DIRECT enable
 22:     option send-gids true
 23: end-volume
 24:  
 25: volume glustershard-client-2
 26:     type protocol/client
 27:     option ping-timeout 42
 28:     option remote-host 192.168.71.12
 29:     option remote-subvolume /gluster1/shard3/1
 30:     option transport-type socket
 31:     option username 1d603a2f-0acb-488d-884d-b520297521ff
 32:     option password 1223f473-2316-4e83-988c-c8dc431160ad
 33:     option filter-O_DIRECT enable
 34:     option send-gids true
 35: end-volume
 36:  
 37: volume glustershard-replicate-0
 38:     type cluster/replicate
 39:     option background-self-heal-count 16
 40:     option data-self-heal-window-size 1024
 41:     option eager-lock enable
 42:     option quorum-type auto
 43:     subvolumes glustershard-client-0 glustershard-client-1 glustershard-client-2
 44: end-volume
 45:  
 46: volume glustershard-dht
 47:     type cluster/distribute
 48:     subvolumes glustershard-replicate-0
 49: end-volume
 50:  
 51: volume glustershard-shard
 52:     type features/shard
 53:     option shard-block-size 64MB
 54:     subvolumes glustershard-dht
 55: end-volume
 56:  
 57: volume glustershard-write-behind
 58:     type performance/write-behind
 59:     subvolumes glustershard-shard
 60: end-volume
 61:  
 62: volume glustershard-readdir-ahead
 63:     type performance/readdir-ahead
 64:     subvolumes glustershard-write-behind
 65: end-volume
 66:  
 67: volume glustershard-open-behind
 68:     type performance/open-behind
 69:     subvolumes glustershard-readdir-ahead
 70: end-volume
 71:  
 72: volume glustershard
 73:     type debug/io-stats
 74:     option log-level INFO
 75:     option latency-measurement off
 76:     option count-fop-hits off
 77:     subvolumes glustershard-open-behind
 78: end-volume
 79:  
 80: volume meta-autoload
 81:     type meta
 82:     subvolumes glustershard
 83: end-volume
 84:  
+------------------------------------------------------------------------------+
[2016-07-22 17:42:05.733473] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-07-22 17:42:05.733924] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-1: changing port to 49159 (from 0)
[2016-07-22 17:42:05.733945] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-2: changing port to 49160 (from 0)
[2016-07-22 17:42:05.737402] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-0: Connected to glustershard-client-0, attached to remote volume '/gluster1/shard1/1'.
[2016-07-22 17:42:05.737426] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-07-22 17:42:05.737494] I [MSGID: 108005] [afr-common.c:4142:afr_notify] 0-glustershard-replicate-0: Subvolume 'glustershard-client-0' came back up; going online.
[2016-07-22 17:42:05.737652] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-0: Server lk version = 1
[2016-07-22 17:42:05.737675] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-07-22 17:42:05.738002] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-1: Connected to glustershard-client-1, attached to remote volume '/gluster1/shard2/1'.
[2016-07-22 17:42:05.738029] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-07-22 17:42:05.738115] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-1: Server lk version = 1
[2016-07-22 17:42:05.740723] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-07-22 17:42:05.741047] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-2: Connected to glustershard-client-2, attached to remote volume '/gluster1/shard3/1'.
[2016-07-22 17:42:05.741059] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2016-07-22 17:42:05.746344] I [fuse-bridge.c:5172:fuse_graph_setup] 0-fuse: switched to graph 0
[2016-07-22 17:42:05.746525] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-2: Server lk version = 1
[2016-07-22 17:42:05.746637] I [fuse-bridge.c:4083:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22
[2016-07-22 17:42:05.748082] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-0
[2016-07-22 17:42:05.748120] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-1
[2016-07-22 17:42:05.748150] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-2
[2016-07-22 17:42:08.051249] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-glustershard-client-2: remote operation failed [Operation not permitted]
[2016-07-22 17:42:08.051837] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-glustershard-client-0: remote operation failed [Operation not permitted]
[2016-07-22 17:42:08.052199] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-glustershard-client-1: remote operation failed [Operation not permitted]
[2016-07-22 17:42:08.052310] W [fuse-bridge.c:2227:fuse_readv_cbk] 0-glusterfs-fuse: 50: READ => -1 gfid=b00888f5-c098-4915-9b72-c28e9551bd64 fd=0x7f11440b9e1c (Operation not permitted)
[2016-07-22 17:44:32.374730] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-glustershard-client-0: remote operation failed [Operation not permitted]
[2016-07-22 17:44:32.374789] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-glustershard-client-1: remote operation failed [Operation not permitted]
[2016-07-22 17:44:32.374922] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-glustershard-client-2: remote operation failed [Operation not permitted]
[2016-07-22 17:44:32.388196] W [fuse-bridge.c:2311:fuse_writev_cbk] 0-glusterfs-fuse: 698: WRITE => -1 gfid=ec6de302-b35f-4279-8563-9ca3e25d9df0 fd=0x7f11440b9e1c (Operation not permitted)
[2016-07-22 17:44:32.388325] W [fuse-bridge.c:2311:fuse_writev_cbk] 0-glusterfs-fuse: 699: WRITE => -1 gfid=ec6de302-b35f-4279-8563-9ca3e25d9df0 fd=0x7f11440b9e1c (Operation not permitted)
[2016-07-22 17:44:32.388683] W [fuse-bridge.c:2311:fuse_writev_cbk] 0-glusterfs-fuse: 700: WRITE => -1 gfid=ec6de302-b35f-4279-8563-9ca3e25d9df0 fd=0x7f11440b9e1c (Operation not permitted)
[2016-07-22 17:44:32.392609] W [fuse-bridge.c:2311:fuse_writev_cbk] 0-glusterfs-fuse: 702: WRITE => -1 gfid=ec6de302-b35f-4279-8563-9ca3e25d9df0 fd=0x7f11440b9e1c (Operation not permitted)
[2016-07-22 17:44:32.392772] W [fuse-bridge.c:2311:fuse_writev_cbk] 0-glusterfs-fuse: 703: WRITE => -1 gfid=ec6de302-b35f-4279-8563-9ca3e25d9df0 fd=0x7f11440b9e1c (Operation not permitted)
[2016-07-22 17:44:32.392835] W [fuse-bridge.c:2311:fuse_writev_cbk] 0-glusterfs-fuse: 704: WRITE => -1 gfid=ec6de302-b35f-4279-8563-9ca3e25d9df0 fd=0x7f11440b9e1c (Operation not permitted)
[2016-07-22 17:44:35.722476] W [fuse-bridge.c:2311:fuse_writev_cbk] 0-glusterfs-fuse: 705: WRITE => -1 gfid=ec6de302-b35f-4279-8563-9ca3e25d9df0 fd=0x7f11440b9e1c (Operation not permitted)
[2016-07-22 17:44:35.728337] W [fuse-bridge.c:709:fuse_truncate_cbk] 0-glusterfs-fuse: 706: FTRUNCATE() ERR => -1 (Operation not permitted)
[2016-07-22 17:44:35.729018] E [mem-pool.c:554:mem_put] (-->/usr/lib64/glusterfs/3.7.13/xlator/debug/io-stats.so(io_stats_ftruncate_cbk+0x13b) [0x7f11491efa0b] -->/usr/lib64/glusterfs/3.7.13/xlator/mount/fuse.so(+0xd3c9) [0x7f114ea7b3c9] -->/lib64/libglusterfs.so.0(mem_put+0x156) [0x7f115190cf36] ) 0-mem-pool: mem-pool ptr is NULL
[2016-07-22 17:44:37.723127] W [fuse-bridge.c:1290:fuse_err_cbk] 0-glusterfs-fuse: 707: FLUSH() ERR => -1 (Operation not permitted)
The message "W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-glustershard-client-0: remote operation failed [Operation not permitted]" repeated 37 times between [2016-07-22 17:44:32.374730] and [2016-07-22 17:44:35.730565]
The message "W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-glustershard-client-1: remote operation failed [Operation not permitted]" repeated 37 times between [2016-07-22 17:44:32.374789] and [2016-07-22 17:44:35.730574]
The message "W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-glustershard-client-2: remote operation failed [Operation not permitted]" repeated 37 times between [2016-07-22 17:44:32.374922] and [2016-07-22 17:44:35.730692]
The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-glustershard-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-07-22 17:37:35.005126] and [2016-07-22 17:37:35.008601]
[2016-07-22 17:42:01.810431] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 1d603a2f-0acb-488d-884d-b520297521ff
[2016-07-22 17:42:01.810512] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-15473-2016/07/22-17:42:01:777393-glustershard-client-0-0-0 (version: 3.7.13)
[2016-07-22 17:42:01.915506] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-10019-2016/07/22-16:29:29:282874-glustershard-client-0-0-0
[2016-07-22 17:42:01.915589] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-10019-2016/07/22-16:29:29:282874-glustershard-client-0-0-0
[2016-07-22 17:42:05.733714] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 1d603a2f-0acb-488d-884d-b520297521ff
[2016-07-22 17:42:05.733803] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-15666-2016/07/22-17:42:05:694659-glustershard-client-0-0-0 (version: 3.7.13)
[2016-07-22 17:42:08.051712] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/b0/08/b00888f5-c098-4915-9b72-c28e9551bd64 _fd = -1 [No such file or directory]
[2016-07-22 17:42:08.051747] W [MSGID: 113006] [posix.c:3008:posix_readv] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6b7c [Operation not permitted]
[2016-07-22 17:42:08.051781] E [MSGID: 115068] [server-rpc-fops.c:1486:server_readv_cbk] 0-glustershard-server: 56: READV -2 (b00888f5-c098-4915-9b72-c28e9551bd64) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.374471] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.374606] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.374643] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2872: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.377205] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2889: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.377578] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2890: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
The message "W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]" repeated 3 times between [2016-07-22 17:44:32.374606] and [2016-07-22 17:44:32.377938]
[2016-07-22 17:44:32.377953] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2892: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
The message "E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]" repeated 4 times between [2016-07-22 17:44:32.374471] and [2016-07-22 17:44:32.379628]
[2016-07-22 17:44:32.379631] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.379692] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2902: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.380830] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.380867] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.380925] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2903: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.382454] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.382488] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.382529] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2913: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.382845] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.382868] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.382895] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2915: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.383177] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.383201] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.383225] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2916: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.384118] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.384148] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.384191] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2921: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.384809] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.384870] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.384901] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2923: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.385257] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.385283] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.385307] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2924: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.385954] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.385986] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.386010] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2925: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.386092] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.386112] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.386133] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2926: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.386392] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.386425] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.386469] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.386493] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.386499] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2927: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.386520] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2928: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.386769] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.386798] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.386843] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2930: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.388975] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.389010] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.389036] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2941: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.389228] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.389255] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.389283] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2942: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.389583] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.389604] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.389642] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2943: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.389904] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.389927] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.389953] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2944: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.390305] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.390327] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.390353] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2945: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.390783] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.390819] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.390850] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2946: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.391065] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.391083] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.391110] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2947: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.391683] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.391774] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.391806] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2948: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.392046] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.392075] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.392099] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2949: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:32.392409] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:32.392434] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:32.392459] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2951: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:33.723695] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:33.723845] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:33.723879] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2957: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:35.724660] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:35.724709] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6be8 [Operation not permitted]
[2016-07-22 17:44:35.724753] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2978: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:35.724893] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:35.724942] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6be8 [Operation not permitted]
[2016-07-22 17:44:35.724985] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2979: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:35.725387] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:35.725413] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:35.725436] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2980: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:35.725493] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:35.725519] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:35.725542] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2981: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:35.725897] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:35.725927] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]
[2016-07-22 17:44:35.725955] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2982: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:35.726580] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:35.726605] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6be8 [Operation not permitted]
[2016-07-22 17:44:35.726629] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2985: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:35.727177] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:35.727203] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6be8 [Operation not permitted]
[2016-07-22 17:44:35.727249] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2987: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:35.729521] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:35.729556] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6be8 [Operation not permitted]
[2016-07-22 17:44:35.729581] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2999: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:35.729932] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:35.729974] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6be8 [Operation not permitted]
[2016-07-22 17:44:35.730007] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 3000: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
[2016-07-22 17:44:35.730453] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]
[2016-07-22 17:44:35.730483] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6be8 [Operation not permitted]
[2016-07-22 17:44:35.730511] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 3001: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel

[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux