On Fri, Jul 22, 2016 at 9:37 AM, Vijay Bellur <vbellur@xxxxxxxxxx> wrote:
On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen <samppah@xxxxxxxxxxxxx> wrote:
> Here is a quick way how to test this:
> GlusterFS 3.7.13 volume with default settings with brick on ZFS dataset. gluster-test1 is server and gluster-test2 is client mounting with FUSE.
>
> Writing file with oflag=direct is not ok:
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 bs=1024000
> dd: failed to open ‘file’: Invalid argument
>
> Enable network.remote-dio on Gluster Volume:
> [root@gluster-test1 gluster]# gluster volume set gluster network.remote-dio enable
> volume set: success
>
> Writing small file with oflag=direct is ok:
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 bs=1024000
> 1+0 records in
> 1+0 records out
> 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s
>
> Writing bigger file with oflag=direct is ok:
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 bs=1M
> 100+0 records in
> 100+0 records out
> 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s
>
> Enable Sharding on Gluster Volume:
> [root@gluster-test1 gluster]# gluster volume set gluster features.shard enable
> volume set: success
>
> Writing small file with oflag=direct is ok:
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=1 bs=1M
> 1+0 records in
> 1+0 records out
> 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s
>
> Writing bigger file with oflag=direct is not ok:
> [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 bs=1M
> dd: error writing ‘file3’: Operation not permitted
> dd: closing output file ‘file3’: Operation not permitted
>
Thank you for these tests! would it be possible to share the brick and
client logs?
Not sure if his tests are same as my setup but here is what I end up with
Volume Name: glustershard
Type: Replicate
Volume ID: 0cc4efb6-3836-4caa-b24a-b3afb6e407c3
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.71.10:/gluster1/shard1/1
Brick2: 192.168.71.11:/gluster1/shard2/1
Brick3: 192.168.71.12:/gluster1/shard3/1
Options Reconfigured:
features.shard-block-size: 64MB
features.shard: on
server.allow-insecure: on
storage.owner-uid: 36
storage.owner-gid: 36
cluster.server-quorum-type: server
cluster.quorum-type: auto
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.quick-read: off
cluster.self-heal-window-size: 1024
cluster.background-self-heal-count: 16
nfs.enable-ino32: off
nfs.addr-namelookup: off
nfs.disable: on
performance.read-ahead: off
performance.readdir-ahead: on
dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/192.168.71.11\:_glustershard/ oflag=direct count=100 bs=1M
81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/ __DIRECT_IO_TEST__ .trashcan/
[root@ccengine2 ~]# dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/192.168.71.11\:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test oflag=direct count=100 bs=1M
dd: error writing ‘/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test’: Operation not permitted
creates the 64M file in expected location then the shard is 0
# file: gluster1/shard1/1/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x0200000000000000579231f3000e16e7
trusted.gfid=0xec6de302b35f427985639ca3e25d9df0
trusted.glusterfs.shard.block-size=0x0000000004000000
trusted.glusterfs.shard.file-size=0x0000000004000000000000000000000000000000000000010000000000000000
# file: gluster1/shard1/1/.shard/ec6de302-b35f-4279-8563-9ca3e25d9df0.1
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x2bfd3cc8a727489b9a0474241548fe80
Regards,
Vijay
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
[2016-07-22 16:26:44.645166] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.13 (args: /usr/sbin/glusterfs --volfile-server=192.168.71.11 --volfile-server=192.168.71.10 --volfile-server=192.168.71.12 --volfile-id=/glustershard /rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard) [2016-07-22 16:26:44.655674] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-07-22 16:26:44.661967] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-07-22 16:26:44.662718] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-0: parent translators are ready, attempting connect on transport [2016-07-22 16:26:44.665996] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-1: parent translators are ready, attempting connect on transport [2016-07-22 16:26:44.666266] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-0: changing port to 49158 (from 0) [2016-07-22 16:26:44.669000] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-2: parent translators are ready, attempting connect on transport [2016-07-22 16:26:44.672423] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-07-22 16:26:44.672556] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-1: changing port to 49159 (from 0) Final graph: +------------------------------------------------------------------------------+ 1: volume glustershard-client-0 2: type protocol/client 3: option clnt-lk-version 1 4: option volfile-checksum 0 5: option volfile-key /glustershard 6: option client-version 3.7.13 7: option process-uuid ccengine2.carouselchecks.local-9617-2016/07/22-16:26:44:640508-glustershard-client-0-0-0 8: option fops-version 1298437 9: option ping-timeout 42 10: option remote-host 192.168.71.10 11: option remote-subvolume /gluster1/shard1/1 12: option transport-type socket 13: option username 1d603a2f-0acb-488d-884d-b520297521ff 14: option password 1223f473-2316-4e83-988c-c8dc431160ad 15: option filter-O_DIRECT enable 16: option send-gids true 17: end-volume 18: 19: volume glustershard-client-1 20: type protocol/client 21: option ping-timeout 42 22: option remote-host 192.168.71.11 23: option remote-subvolume /gluster1/shard2/1 24: option transport-type socket 25: option username 1d603a2f-0acb-488d-884d-b520297521ff 26: option password 1223f473-2316-4e83-988c-c8dc431160ad 27: option filter-O_DIRECT enable 28: option send-gids true 29: end-volume 30: 31: volume glustershard-client-2 32: type protocol/client 33: option ping-timeout 42 34: option remote-host 192.168.71.12 35: option remote-subvolume /gluster1/shard3/1 36: option transport-type socket 37: option username 1d603a2f-0acb-488d-884d-b520297521ff 38: option password 1223f473-2316-4e83-988c-c8dc431160ad 39: option filter-O_DIRECT enable 40: option send-gids true 41: end-volume 42: 43: volume glustershard-replicate-0 44: type cluster/replicate 45: option background-self-heal-count 16 46: option data-self-heal-window-size 1024 47: option eager-lock enable 48: option quorum-type auto 49: subvolumes glustershard-client-0 glustershard-client-1 glustershard-client-2 50: end-volume 51: 52: volume glustershard-dht 53: type cluster/distribute 54: subvolumes glustershard-replicate-0 55: end-volume 56: 57: volume glustershard-shard 58: type features/shard 59: option shard-block-size 64MB 60: subvolumes glustershard-dht 61: end-volume 62: 63: volume glustershard-write-behind 64: type performance/write-behind 65: subvolumes glustershard-shard 66: end-volume 67: 68: volume glustershard-readdir-ahead 69: type performance/readdir-ahead 70: subvolumes glustershard-write-behind 71: end-volume 72: 73: volume glustershard-open-behind 74: type performance/open-behind 75: subvolumes glustershard-readdir-ahead 76: end-volume 77: 78: volume glustershard 79: type debug/io-stats 80: option log-level INFO 81: option latency-measurement off 82: option count-fop-hits off 83: subvolumes glustershard-open-behind 84: end-volume 85: 86: volume meta-autoload 87: type meta 88: subvolumes glustershard 89: end-volume 90: +------------------------------------------------------------------------------+ [2016-07-22 16:26:44.674788] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-0: Connected to glustershard-client-0, attached to remote volume '/gluster1/shard1/1'. [2016-07-22 16:26:44.674803] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-07-22 16:26:44.674880] I [MSGID: 108005] [afr-common.c:4142:afr_notify] 0-glustershard-replicate-0: Subvolume 'glustershard-client-0' came back up; going online. [2016-07-22 16:26:44.675040] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-0: Server lk version = 1 [2016-07-22 16:26:44.675107] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-2: changing port to 49160 (from 0) [2016-07-22 16:26:44.677423] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-07-22 16:26:44.677770] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-1: Connected to glustershard-client-1, attached to remote volume '/gluster1/shard2/1'. [2016-07-22 16:26:44.677787] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-07-22 16:26:44.678037] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-1: Server lk version = 1 [2016-07-22 16:26:44.680286] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-07-22 16:26:44.680790] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-2: Connected to glustershard-client-2, attached to remote volume '/gluster1/shard3/1'. [2016-07-22 16:26:44.680810] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-2: Server and Client lk-version numbers are not same, reopening the fds [2016-07-22 16:26:44.685439] I [fuse-bridge.c:5172:fuse_graph_setup] 0-fuse: switched to graph 0 [2016-07-22 16:26:44.685570] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-2: Server lk version = 1 [2016-07-22 16:26:44.685672] I [fuse-bridge.c:4083:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22 [2016-07-22 16:26:44.686808] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-1 [2016-07-22 16:26:44.686859] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-2 [2016-07-22 16:26:44.686894] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-0 [2016-07-22 16:27:48.460148] I [fuse-bridge.c:5013:fuse_thread_proc] 0-fuse: unmounting /rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard [2016-07-22 16:27:48.460688] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fe368831dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fe369eab915] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fe369eab78b] ) 0-: received signum (15), shutting down [2016-07-22 16:27:48.460712] I [fuse-bridge.c:5720:fini] 0-fuse: Unmounting '/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard'. [2016-07-22 16:29:29.287998] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.13 (args: /usr/sbin/glusterfs --volfile-server=192.168.71.11 --volfile-server=192.168.71.10 --volfile-server=192.168.71.12 --volfile-id=/glustershard /rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard) [2016-07-22 16:29:29.299869] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-07-22 16:29:29.306788] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-07-22 16:29:29.307656] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-0: parent translators are ready, attempting connect on transport [2016-07-22 16:29:29.311044] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-1: parent translators are ready, attempting connect on transport [2016-07-22 16:29:29.311425] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-0: changing port to 49158 (from 0) [2016-07-22 16:29:29.314429] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-2: parent translators are ready, attempting connect on transport [2016-07-22 16:29:29.317449] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-07-22 16:29:29.317584] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-1: changing port to 49159 (from 0) Final graph: +------------------------------------------------------------------------------+ 1: volume glustershard-client-0 2: type protocol/client 3: option clnt-lk-version 1 4: option volfile-checksum 0 5: option volfile-key /glustershard 6: option client-version 3.7.13 7: option process-uuid ccengine2.carouselchecks.local-10019-2016/07/22-16:29:29:282874-glustershard-client-0-0-0 8: option fops-version 1298437 9: option ping-timeout 42 10: option remote-host 192.168.71.10 11: option remote-subvolume /gluster1/shard1/1 12: option transport-type socket 13: option username 1d603a2f-0acb-488d-884d-b520297521ff 14: option password 1223f473-2316-4e83-988c-c8dc431160ad 15: option filter-O_DIRECT enable 16: option send-gids true 17: end-volume 18: 19: volume glustershard-client-1 20: type protocol/client 21: option ping-timeout 42 22: option remote-host 192.168.71.11 23: option remote-subvolume /gluster1/shard2/1 24: option transport-type socket 25: option username 1d603a2f-0acb-488d-884d-b520297521ff 26: option password 1223f473-2316-4e83-988c-c8dc431160ad 27: option filter-O_DIRECT enable 28: option send-gids true 29: end-volume 30: 31: volume glustershard-client-2 32: type protocol/client 33: option ping-timeout 42 34: option remote-host 192.168.71.12 35: option remote-subvolume /gluster1/shard3/1 36: option transport-type socket 37: option username 1d603a2f-0acb-488d-884d-b520297521ff 38: option password 1223f473-2316-4e83-988c-c8dc431160ad 39: option filter-O_DIRECT enable 40: option send-gids true 41: end-volume 42: 43: volume glustershard-replicate-0 44: type cluster/replicate 45: option background-self-heal-count 16 46: option data-self-heal-window-size 1024 47: option eager-lock enable 48: option quorum-type auto 49: subvolumes glustershard-client-0 glustershard-client-1 glustershard-client-2 50: end-volume 51: 52: volume glustershard-dht 53: type cluster/distribute 54: subvolumes glustershard-replicate-0 55: end-volume 56: 57: volume glustershard-shard 58: type features/shard 59: option shard-block-size 64MB 60: subvolumes glustershard-dht 61: end-volume 62: 63: volume glustershard-write-behind 64: type performance/write-behind 65: subvolumes glustershard-shard 66: end-volume 67: 68: volume glustershard-readdir-ahead 69: type performance/readdir-ahead 70: subvolumes glustershard-write-behind 71: end-volume 72: 73: volume glustershard-open-behind 74: type performance/open-behind 75: subvolumes glustershard-readdir-ahead 76: end-volume 77: 78: volume glustershard 79: type debug/io-stats 80: option log-level INFO 81: option latency-measurement off 82: option count-fop-hits off 83: subvolumes glustershard-open-behind 84: end-volume 85: 86: volume meta-autoload 87: type meta 88: subvolumes glustershard 89: end-volume 90: +------------------------------------------------------------------------------+ [2016-07-22 16:29:29.320203] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-0: Connected to glustershard-client-0, attached to remote volume '/gluster1/shard1/1'. [2016-07-22 16:29:29.320217] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-07-22 16:29:29.320277] I [MSGID: 108005] [afr-common.c:4142:afr_notify] 0-glustershard-replicate-0: Subvolume 'glustershard-client-0' came back up; going online. [2016-07-22 16:29:29.320333] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-0: Server lk version = 1 [2016-07-22 16:29:29.320475] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-2: changing port to 49160 (from 0) [2016-07-22 16:29:29.322932] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-07-22 16:29:29.323349] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-1: Connected to glustershard-client-1, attached to remote volume '/gluster1/shard2/1'. [2016-07-22 16:29:29.323403] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-07-22 16:29:29.323556] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-1: Server lk version = 1 [2016-07-22 16:29:29.325770] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-07-22 16:29:29.326260] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-2: Connected to glustershard-client-2, attached to remote volume '/gluster1/shard3/1'. [2016-07-22 16:29:29.326271] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-2: Server and Client lk-version numbers are not same, reopening the fds [2016-07-22 16:29:29.330885] I [fuse-bridge.c:5172:fuse_graph_setup] 0-fuse: switched to graph 0 [2016-07-22 16:29:29.331017] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-2: Server lk version = 1 [2016-07-22 16:29:29.331105] I [fuse-bridge.c:4083:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22 [2016-07-22 16:29:29.332316] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-1 [2016-07-22 16:29:29.332424] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-0 [2016-07-22 16:29:29.332529] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-2 [2016-07-22 16:29:32.938721] I [MSGID: 109036] [dht-common.c:8888:dht_log_new_layout_for_dir_selfheal] 0-glustershard-dht: Setting layout of /81e19cd3-ae45-449c-b716-ec3e4ad4c2f0 with [Subvol_name: glustershard-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], [2016-07-22 16:29:32.944681] I [MSGID: 109036] [dht-common.c:8888:dht_log_new_layout_for_dir_selfheal] 0-glustershard-dht: Setting layout of /81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/dom_md with [Subvol_name: glustershard-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], [2016-07-22 16:29:33.014432] I [MSGID: 109066] [dht-rename.c:1568:dht_rename] 0-glustershard-dht: renaming /81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/dom_md/metadata.new (hash=glustershard-replicate-0/cache=glustershard-replicate-0) => /81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/dom_md/metadata (hash=glustershard-replicate-0/cache=<nul>) [2016-07-22 16:29:33.034967] I [MSGID: 109036] [dht-common.c:8888:dht_log_new_layout_for_dir_selfheal] 0-glustershard-dht: Setting layout of /81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images with [Subvol_name: glustershard-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], [2016-07-22 16:29:37.914047] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-glustershard-client-2: remote operation failed [Operation not permitted] [2016-07-22 16:29:37.915023] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-glustershard-client-0: remote operation failed [Operation not permitted] [2016-07-22 16:29:37.915644] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-glustershard-client-1: remote operation failed [Operation not permitted] [2016-07-22 16:29:37.915675] W [fuse-bridge.c:2227:fuse_readv_cbk] 0-glusterfs-fuse: 183: READ => -1 gfid=b00888f5-c098-4915-9b72-c28e9551bd64 fd=0x7f1bf00ba218 (Operation not permitted) [2016-07-22 17:42:01.782069] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.13 (args: /usr/sbin/glusterfs --volfile-server=192.168.71.11 --volfile-server=192.168.71.10 --volfile-server=192.168.71.12 --volfile-id=/glustershard /rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard) [2016-07-22 17:42:01.792599] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-07-22 17:42:01.799265] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-07-22 17:42:01.800235] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-0: parent translators are ready, attempting connect on transport [2016-07-22 17:42:01.804137] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-1: parent translators are ready, attempting connect on transport [2016-07-22 17:42:01.804403] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-0: changing port to 49158 (from 0) [2016-07-22 17:42:01.807125] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-2: parent translators are ready, attempting connect on transport [2016-07-22 17:42:01.810296] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-07-22 17:42:01.810429] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-1: changing port to 49159 (from 0) Final graph: +------------------------------------------------------------------------------+ 1: volume glustershard-client-0 2: type protocol/client 3: option clnt-lk-version 1 4: option volfile-checksum 0 5: option volfile-key /glustershard 6: option client-version 3.7.13 7: option process-uuid ccengine2.carouselchecks.local-15473-2016/07/22-17:42:01:777393-glustershard-client-0-0-0 8: option fops-version 1298437 9: option ping-timeout 42 10: option remote-host 192.168.71.10 11: option remote-subvolume /gluster1/shard1/1 12: option transport-type socket 13: option username 1d603a2f-0acb-488d-884d-b520297521ff 14: option password 1223f473-2316-4e83-988c-c8dc431160ad 15: option filter-O_DIRECT enable 16: option send-gids true 17: end-volume 18: 19: volume glustershard-client-1 20: type protocol/client 21: option ping-timeout 42 22: option remote-host 192.168.71.11 23: option remote-subvolume /gluster1/shard2/1 24: option transport-type socket 25: option username 1d603a2f-0acb-488d-884d-b520297521ff 26: option password 1223f473-2316-4e83-988c-c8dc431160ad 27: option filter-O_DIRECT enable 28: option send-gids true 29: end-volume 30: 31: volume glustershard-client-2 32: type protocol/client 33: option ping-timeout 42 34: option remote-host 192.168.71.12 35: option remote-subvolume /gluster1/shard3/1 36: option transport-type socket 37: option username 1d603a2f-0acb-488d-884d-b520297521ff 38: option password 1223f473-2316-4e83-988c-c8dc431160ad 39: option filter-O_DIRECT enable 40: option send-gids true 41: end-volume 42: 43: volume glustershard-replicate-0 44: type cluster/replicate 45: option background-self-heal-count 16 46: option data-self-heal-window-size 1024 47: option eager-lock enable 48: option quorum-type auto 49: subvolumes glustershard-client-0 glustershard-client-1 glustershard-client-2 50: end-volume 51: 52: volume glustershard-dht 53: type cluster/distribute 54: subvolumes glustershard-replicate-0 55: end-volume 56: 57: volume glustershard-shard 58: type features/shard 59: option shard-block-size 64MB 60: subvolumes glustershard-dht 61: end-volume 62: 63: volume glustershard-write-behind 64: type performance/write-behind 65: subvolumes glustershard-shard 66: end-volume 67: 68: volume glustershard-readdir-ahead 69: type performance/readdir-ahead 70: subvolumes glustershard-write-behind 71: end-volume 72: 73: volume glustershard-open-behind 74: type performance/open-behind 75: subvolumes glustershard-readdir-ahead 76: end-volume 77: 78: volume glustershard 79: type debug/io-stats 80: option log-level INFO 81: option latency-measurement off 82: option count-fop-hits off 83: subvolumes glustershard-open-behind 84: end-volume 85: 86: volume meta-autoload 87: type meta 88: subvolumes glustershard 89: end-volume 90: +------------------------------------------------------------------------------+ [2016-07-22 17:42:01.813111] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-0: Connected to glustershard-client-0, attached to remote volume '/gluster1/shard1/1'. [2016-07-22 17:42:01.813123] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-07-22 17:42:01.813198] I [MSGID: 108005] [afr-common.c:4142:afr_notify] 0-glustershard-replicate-0: Subvolume 'glustershard-client-0' came back up; going online. [2016-07-22 17:42:01.813358] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-0: Server lk version = 1 [2016-07-22 17:42:01.813433] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-2: changing port to 49160 (from 0) [2016-07-22 17:42:01.815936] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-07-22 17:42:01.816506] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-1: Connected to glustershard-client-1, attached to remote volume '/gluster1/shard2/1'. [2016-07-22 17:42:01.816558] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-07-22 17:42:01.816749] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-1: Server lk version = 1 [2016-07-22 17:42:01.818836] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-07-22 17:42:01.819263] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-2: Connected to glustershard-client-2, attached to remote volume '/gluster1/shard3/1'. [2016-07-22 17:42:01.819330] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-2: Server and Client lk-version numbers are not same, reopening the fds [2016-07-22 17:42:01.823445] I [fuse-bridge.c:5172:fuse_graph_setup] 0-fuse: switched to graph 0 [2016-07-22 17:42:01.823574] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-2: Server lk version = 1 [2016-07-22 17:42:01.823661] I [fuse-bridge.c:4083:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22 [2016-07-22 17:42:01.824814] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-0 [2016-07-22 17:42:01.824848] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-1 [2016-07-22 17:42:01.824874] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-2 [2016-07-22 17:42:01.912806] I [fuse-bridge.c:5013:fuse_thread_proc] 0-fuse: unmounting /rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard [2016-07-22 17:42:01.913219] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f1bfa492dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f1bfbb0c915] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f1bfbb0c78b] ) 0-: received signum (15), shutting down [2016-07-22 17:42:01.913248] I [fuse-bridge.c:5720:fini] 0-fuse: Unmounting '/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard'. [2016-07-22 17:42:05.699668] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.13 (args: /usr/sbin/glusterfs --volfile-server=192.168.71.11 --volfile-server=192.168.71.10 --volfile-server=192.168.71.12 --volfile-id=/glustershard /rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard) [2016-07-22 17:42:05.710391] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-07-22 17:42:05.717590] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-07-22 17:42:05.718378] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-0: parent translators are ready, attempting connect on transport [2016-07-22 17:42:05.721685] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-1: parent translators are ready, attempting connect on transport [2016-07-22 17:42:05.721946] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-0: changing port to 49158 (from 0) [2016-07-22 17:42:05.724682] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-2: parent translators are ready, attempting connect on transport Final graph: +------------------------------------------------------------------------------+ 1: volume glustershard-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host 192.168.71.10 5: option remote-subvolume /gluster1/shard1/1 6: option transport-type socket 7: option username 1d603a2f-0acb-488d-884d-b520297521ff 8: option password 1223f473-2316-4e83-988c-c8dc431160ad 9: option filter-O_DIRECT enable 10: option send-gids true 11: end-volume 12: 13: volume glustershard-client-1 14: type protocol/client 15: option ping-timeout 42 16: option remote-host 192.168.71.11 17: option remote-subvolume /gluster1/shard2/1 18: option transport-type socket 19: option username 1d603a2f-0acb-488d-884d-b520297521ff 20: option password 1223f473-2316-4e83-988c-c8dc431160ad 21: option filter-O_DIRECT enable 22: option send-gids true 23: end-volume 24: 25: volume glustershard-client-2 26: type protocol/client 27: option ping-timeout 42 28: option remote-host 192.168.71.12 29: option remote-subvolume /gluster1/shard3/1 30: option transport-type socket 31: option username 1d603a2f-0acb-488d-884d-b520297521ff 32: option password 1223f473-2316-4e83-988c-c8dc431160ad 33: option filter-O_DIRECT enable 34: option send-gids true 35: end-volume 36: 37: volume glustershard-replicate-0 38: type cluster/replicate 39: option background-self-heal-count 16 40: option data-self-heal-window-size 1024 41: option eager-lock enable 42: option quorum-type auto 43: subvolumes glustershard-client-0 glustershard-client-1 glustershard-client-2 44: end-volume 45: 46: volume glustershard-dht 47: type cluster/distribute 48: subvolumes glustershard-replicate-0 49: end-volume 50: 51: volume glustershard-shard 52: type features/shard 53: option shard-block-size 64MB 54: subvolumes glustershard-dht 55: end-volume 56: 57: volume glustershard-write-behind 58: type performance/write-behind 59: subvolumes glustershard-shard 60: end-volume 61: 62: volume glustershard-readdir-ahead 63: type performance/readdir-ahead 64: subvolumes glustershard-write-behind 65: end-volume 66: 67: volume glustershard-open-behind 68: type performance/open-behind 69: subvolumes glustershard-readdir-ahead 70: end-volume 71: 72: volume glustershard 73: type debug/io-stats 74: option log-level INFO 75: option latency-measurement off 76: option count-fop-hits off 77: subvolumes glustershard-open-behind 78: end-volume 79: 80: volume meta-autoload 81: type meta 82: subvolumes glustershard 83: end-volume 84: +------------------------------------------------------------------------------+ [2016-07-22 17:42:05.733473] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-07-22 17:42:05.733924] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-1: changing port to 49159 (from 0) [2016-07-22 17:42:05.733945] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-glustershard-client-2: changing port to 49160 (from 0) [2016-07-22 17:42:05.737402] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-0: Connected to glustershard-client-0, attached to remote volume '/gluster1/shard1/1'. [2016-07-22 17:42:05.737426] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-07-22 17:42:05.737494] I [MSGID: 108005] [afr-common.c:4142:afr_notify] 0-glustershard-replicate-0: Subvolume 'glustershard-client-0' came back up; going online. [2016-07-22 17:42:05.737652] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-0: Server lk version = 1 [2016-07-22 17:42:05.737675] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-07-22 17:42:05.738002] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-1: Connected to glustershard-client-1, attached to remote volume '/gluster1/shard2/1'. [2016-07-22 17:42:05.738029] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-07-22 17:42:05.738115] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-1: Server lk version = 1 [2016-07-22 17:42:05.740723] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustershard-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-07-22 17:42:05.741047] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustershard-client-2: Connected to glustershard-client-2, attached to remote volume '/gluster1/shard3/1'. [2016-07-22 17:42:05.741059] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustershard-client-2: Server and Client lk-version numbers are not same, reopening the fds [2016-07-22 17:42:05.746344] I [fuse-bridge.c:5172:fuse_graph_setup] 0-fuse: switched to graph 0 [2016-07-22 17:42:05.746525] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustershard-client-2: Server lk version = 1 [2016-07-22 17:42:05.746637] I [fuse-bridge.c:4083:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22 [2016-07-22 17:42:05.748082] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-0 [2016-07-22 17:42:05.748120] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-1 [2016-07-22 17:42:05.748150] I [MSGID: 108031] [afr-common.c:1913:afr_local_discovery_cbk] 0-glustershard-replicate-0: selecting local read_child glustershard-client-2 [2016-07-22 17:42:08.051249] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-glustershard-client-2: remote operation failed [Operation not permitted] [2016-07-22 17:42:08.051837] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-glustershard-client-0: remote operation failed [Operation not permitted] [2016-07-22 17:42:08.052199] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-glustershard-client-1: remote operation failed [Operation not permitted] [2016-07-22 17:42:08.052310] W [fuse-bridge.c:2227:fuse_readv_cbk] 0-glusterfs-fuse: 50: READ => -1 gfid=b00888f5-c098-4915-9b72-c28e9551bd64 fd=0x7f11440b9e1c (Operation not permitted) [2016-07-22 17:44:32.374730] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-glustershard-client-0: remote operation failed [Operation not permitted] [2016-07-22 17:44:32.374789] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-glustershard-client-1: remote operation failed [Operation not permitted] [2016-07-22 17:44:32.374922] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-glustershard-client-2: remote operation failed [Operation not permitted] [2016-07-22 17:44:32.388196] W [fuse-bridge.c:2311:fuse_writev_cbk] 0-glusterfs-fuse: 698: WRITE => -1 gfid=ec6de302-b35f-4279-8563-9ca3e25d9df0 fd=0x7f11440b9e1c (Operation not permitted) [2016-07-22 17:44:32.388325] W [fuse-bridge.c:2311:fuse_writev_cbk] 0-glusterfs-fuse: 699: WRITE => -1 gfid=ec6de302-b35f-4279-8563-9ca3e25d9df0 fd=0x7f11440b9e1c (Operation not permitted) [2016-07-22 17:44:32.388683] W [fuse-bridge.c:2311:fuse_writev_cbk] 0-glusterfs-fuse: 700: WRITE => -1 gfid=ec6de302-b35f-4279-8563-9ca3e25d9df0 fd=0x7f11440b9e1c (Operation not permitted) [2016-07-22 17:44:32.392609] W [fuse-bridge.c:2311:fuse_writev_cbk] 0-glusterfs-fuse: 702: WRITE => -1 gfid=ec6de302-b35f-4279-8563-9ca3e25d9df0 fd=0x7f11440b9e1c (Operation not permitted) [2016-07-22 17:44:32.392772] W [fuse-bridge.c:2311:fuse_writev_cbk] 0-glusterfs-fuse: 703: WRITE => -1 gfid=ec6de302-b35f-4279-8563-9ca3e25d9df0 fd=0x7f11440b9e1c (Operation not permitted) [2016-07-22 17:44:32.392835] W [fuse-bridge.c:2311:fuse_writev_cbk] 0-glusterfs-fuse: 704: WRITE => -1 gfid=ec6de302-b35f-4279-8563-9ca3e25d9df0 fd=0x7f11440b9e1c (Operation not permitted) [2016-07-22 17:44:35.722476] W [fuse-bridge.c:2311:fuse_writev_cbk] 0-glusterfs-fuse: 705: WRITE => -1 gfid=ec6de302-b35f-4279-8563-9ca3e25d9df0 fd=0x7f11440b9e1c (Operation not permitted) [2016-07-22 17:44:35.728337] W [fuse-bridge.c:709:fuse_truncate_cbk] 0-glusterfs-fuse: 706: FTRUNCATE() ERR => -1 (Operation not permitted) [2016-07-22 17:44:35.729018] E [mem-pool.c:554:mem_put] (-->/usr/lib64/glusterfs/3.7.13/xlator/debug/io-stats.so(io_stats_ftruncate_cbk+0x13b) [0x7f11491efa0b] -->/usr/lib64/glusterfs/3.7.13/xlator/mount/fuse.so(+0xd3c9) [0x7f114ea7b3c9] -->/lib64/libglusterfs.so.0(mem_put+0x156) [0x7f115190cf36] ) 0-mem-pool: mem-pool ptr is NULL [2016-07-22 17:44:37.723127] W [fuse-bridge.c:1290:fuse_err_cbk] 0-glusterfs-fuse: 707: FLUSH() ERR => -1 (Operation not permitted) The message "W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-glustershard-client-0: remote operation failed [Operation not permitted]" repeated 37 times between [2016-07-22 17:44:32.374730] and [2016-07-22 17:44:35.730565] The message "W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-glustershard-client-1: remote operation failed [Operation not permitted]" repeated 37 times between [2016-07-22 17:44:32.374789] and [2016-07-22 17:44:35.730574] The message "W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-glustershard-client-2: remote operation failed [Operation not permitted]" repeated 37 times between [2016-07-22 17:44:32.374922] and [2016-07-22 17:44:35.730692]
The message "E [MSGID: 113018] [posix.c:196:posix_lookup] 0-glustershard-posix: lstat on null failed [Invalid argument]" repeated 3 times between [2016-07-22 17:37:35.005126] and [2016-07-22 17:37:35.008601] [2016-07-22 17:42:01.810431] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 1d603a2f-0acb-488d-884d-b520297521ff [2016-07-22 17:42:01.810512] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-15473-2016/07/22-17:42:01:777393-glustershard-client-0-0-0 (version: 3.7.13) [2016-07-22 17:42:01.915506] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-glustershard-server: disconnecting connection from ccengine2.carouselchecks.local-10019-2016/07/22-16:29:29:282874-glustershard-client-0-0-0 [2016-07-22 17:42:01.915589] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-glustershard-server: Shutting down connection ccengine2.carouselchecks.local-10019-2016/07/22-16:29:29:282874-glustershard-client-0-0-0 [2016-07-22 17:42:05.733714] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 1d603a2f-0acb-488d-884d-b520297521ff [2016-07-22 17:42:05.733803] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-glustershard-server: accepted client from ccengine2.carouselchecks.local-15666-2016/07/22-17:42:05:694659-glustershard-client-0-0-0 (version: 3.7.13) [2016-07-22 17:42:08.051712] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/b0/08/b00888f5-c098-4915-9b72-c28e9551bd64 _fd = -1 [No such file or directory] [2016-07-22 17:42:08.051747] W [MSGID: 113006] [posix.c:3008:posix_readv] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6b7c [Operation not permitted] [2016-07-22 17:42:08.051781] E [MSGID: 115068] [server-rpc-fops.c:1486:server_readv_cbk] 0-glustershard-server: 56: READV -2 (b00888f5-c098-4915-9b72-c28e9551bd64) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.374471] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.374606] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.374643] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2872: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.377205] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2889: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.377578] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2890: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] The message "W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted]" repeated 3 times between [2016-07-22 17:44:32.374606] and [2016-07-22 17:44:32.377938] [2016-07-22 17:44:32.377953] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2892: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] The message "E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory]" repeated 4 times between [2016-07-22 17:44:32.374471] and [2016-07-22 17:44:32.379628] [2016-07-22 17:44:32.379631] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.379692] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2902: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.380830] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.380867] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.380925] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2903: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.382454] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.382488] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.382529] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2913: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.382845] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.382868] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.382895] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2915: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.383177] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.383201] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.383225] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2916: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.384118] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.384148] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.384191] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2921: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.384809] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.384870] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.384901] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2923: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.385257] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.385283] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.385307] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2924: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.385954] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.385986] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.386010] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2925: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.386092] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.386112] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.386133] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2926: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.386392] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.386425] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.386469] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.386493] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.386499] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2927: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.386520] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2928: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.386769] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.386798] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.386843] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2930: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.388975] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.389010] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.389036] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2941: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.389228] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.389255] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.389283] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2942: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.389583] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.389604] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.389642] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2943: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.389904] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.389927] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.389953] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2944: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.390305] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.390327] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.390353] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2945: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.390783] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.390819] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.390850] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2946: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.391065] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.391083] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.391110] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2947: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.391683] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.391774] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.391806] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2948: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.392046] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.392075] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.392099] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2949: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:32.392409] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:32.392434] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:32.392459] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2951: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:33.723695] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:33.723845] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:33.723879] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2957: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:35.724660] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:35.724709] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6be8 [Operation not permitted] [2016-07-22 17:44:35.724753] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2978: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:35.724893] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:35.724942] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6be8 [Operation not permitted] [2016-07-22 17:44:35.724985] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2979: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:35.725387] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:35.725413] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:35.725436] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2980: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:35.725493] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:35.725519] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:35.725542] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2981: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:35.725897] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:35.725927] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6c54 [Operation not permitted] [2016-07-22 17:44:35.725955] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2982: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:35.726580] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:35.726605] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6be8 [Operation not permitted] [2016-07-22 17:44:35.726629] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2985: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:35.727177] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:35.727203] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6be8 [Operation not permitted] [2016-07-22 17:44:35.727249] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2987: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:35.729521] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:35.729556] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6be8 [Operation not permitted] [2016-07-22 17:44:35.729581] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 2999: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:35.729932] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:35.729974] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6be8 [Operation not permitted] [2016-07-22 17:44:35.730007] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 3000: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted] [2016-07-22 17:44:35.730453] E [MSGID: 113040] [posix-helpers.c:1684:__posix_fd_ctx_get] 0-glustershard-posix: Failed to get anonymous real_path: /gluster1/shard1/1/.glusterfs/2b/fd/2bfd3cc8-a727-489b-9a04-74241548fe80 _fd = -1 [No such file or directory] [2016-07-22 17:44:35.730483] W [MSGID: 113006] [posix.c:4887:posix_fsetxattr] 0-glustershard-posix: pfd is NULL from fd=0x7f37bc0d6be8 [Operation not permitted] [2016-07-22 17:44:35.730511] E [MSGID: 115067] [server-rpc-fops.c:1436:server_writev_cbk] 0-glustershard-server: 3001: WRITEV -2 (2bfd3cc8-a727-489b-9a04-74241548fe80) ==> (Operation not permitted) [Operation not permitted]
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
