On Sun, May 14, 2017 at 12:50 AM, Benjamin Kingston <ben@xxxxxxxxxxxxxxx> wrote:
Hers's some log entries from nfs-ganesha gfapi
[2017-05-13 19:02:54.105936] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on shard 11 failed. Base file gfid = 1494c083-a618-4eba-80a0- 147e656dd9d0 [Input/output error]
[2017-05-13 19:02:54.106176] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on shard 2 failed. Base file gfid = 1494c083-a618-4eba-80a0- 147e656dd9d0 [Input/output error]
[2017-05-13 19:02:54.106288] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on shard 1 failed. Base file gfid = 1494c083-a618-4eba-80a0- 147e656dd9d0 [Input/output error]
[2017-05-13 19:02:54.384922] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-storage2-replicate-2: performing metadata selfheal on fe651475-226e-42a3-be2d- 751d4f58e383
[2017-05-13 19:02:54.385894] W [MSGID: 114031] [client-rpc-fops.c:2258:client3_3_setattr_cbk] 0-storage2-client-8: remote operation failed [Operation not permitted]
[2017-05-13 19:02:54.401187] I [MSGID: 108026] [afr-self-heal-common.c:1255:afr_log_selfheal] 0-storage2-replicate-2: Completed metadata selfheal on fe651475-226e-42a3-be2d- 751d4f58e383. sources=[0] 1 sinks=
[2017-05-13 19:02:57.830019] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. par2.tmp (hash=storage2-readdir-ahead- 2/cache=storage2-readdir- ahead-2) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. par2 (hash=storage2-readdir-ahead- 0/cache=<nul>)
[2017-05-13 19:08:22.014899] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. srr.tmp (hash=storage2-readdir-ahead- 1/cache=storage2-readdir- ahead-1) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.srr (hash=storage2-readdir-ahead- 1/cache=<nul>)
[2017-05-13 19:08:22.463840] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. r04.tmp (hash=storage2-readdir-ahead- 2/cache=storage2-readdir- ahead-2) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r04 (hash=storage2-readdir-ahead- 0/cache=<nul>)
[2017-05-13 19:08:22.769542] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. r01.tmp (hash=storage2-readdir-ahead- 2/cache=storage2-readdir- ahead-2) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r01 (hash=storage2-readdir-ahead- 0/cache=<nul>)
[2017-05-13 19:08:23.141069] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. nfo.tmp (hash=storage2-readdir-ahead- 1/cache=storage2-readdir- ahead-1) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.nfo (hash=storage2-readdir-ahead- 0/cache=<nul>)
[2017-05-13 19:08:23.468554] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. r00.tmp (hash=storage2-readdir-ahead- 0/cache=storage2-readdir- ahead-0) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r00 (hash=storage2-readdir-ahead- 2/cache=<nul>)
[2017-05-13 19:08:23.671753] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. sfv.tmp (hash=storage2-readdir-ahead- 2/cache=storage2-readdir- ahead-2) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.sfv (hash=storage2-readdir-ahead- 2/cache=<nul>)
[2017-05-13 19:08:23.812152] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. r11.tmp (hash=storage2-readdir-ahead- 0/cache=storage2-readdir- ahead-0) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r11 (hash=storage2-readdir-ahead- 1/cache=<nul>)
[2017-05-13 19:08:24.244584] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. r12.tmp (hash=storage2-readdir-ahead- 0/cache=storage2-readdir- ahead-0) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r12 (hash=storage2-readdir-ahead- 1/cache=<nul>)
[2017-05-13 19:08:24.711781] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. r09.tmp (hash=storage2-readdir-ahead- 1/cache=storage2-readdir- ahead-1) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r09 (hash=storage2-readdir-ahead- 1/cache=<nul>)
[2017-05-13 19:08:25.029102] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. r06.tmp (hash=storage2-readdir-ahead- 0/cache=storage2-readdir- ahead-0) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r06 (hash=storage2-readdir-ahead- 0/cache=<nul>)
[2017-05-13 19:08:25.206011] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. r05.tmp (hash=storage2-readdir-ahead- 0/cache=storage2-readdir- ahead-0) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r05 (hash=storage2-readdir-ahead- 0/cache=<nul>)
[2017-05-13 19:08:25.467312] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. r03.tmp (hash=storage2-readdir-ahead- 1/cache=storage2-readdir- ahead-1) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r03 (hash=storage2-readdir-ahead- 1/cache=<nul>)
[2017-05-13 19:08:25.726208] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. r07.tmp (hash=storage2-readdir-ahead- 0/cache=storage2-readdir- ahead-0) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r07 (hash=storage2-readdir-ahead- 0/cache=<nul>)
[2017-05-13 19:08:25.860386] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. rar.tmp (hash=storage2-readdir-ahead- 1/cache=storage2-readdir- ahead-1) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.rar (hash=storage2-readdir-ahead- 1/cache=<nul>)
[2017-05-13 19:09:00.803859] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. r08.tmp (hash=storage2-readdir-ahead- 2/cache=storage2-readdir- ahead-2) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r08 (hash=storage2-readdir-ahead- 0/cache=<nul>)
[2017-05-13 19:09:01.826525] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. r02.tmp (hash=storage2-readdir-ahead- 2/cache=storage2-readdir- ahead-2) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r02 (hash=storage2-readdir-ahead- 2/cache=<nul>)
[2017-05-13 19:09:02.531354] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Videos/Cinema/Bambi (1942)/.Bambi (1942).mkv.hp9oW8 (hash=storage2-readdir-ahead-2/cache=storage2-readdir- ahead-2) => /content/Videos/Cinema/Bambi (1942)/. nfsa223c532d51b3c7700000006 (hash=storage2-readdir-ahead- 2/cache=<nul>)
[2017-05-13 19:09:39.695354] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web. r10.tmp (hash=storage2-readdir-ahead- 2/cache=storage2-readdir- ahead-2) => /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r10 (hash=storage2-readdir-ahead- 2/cache=<nul>)
[2017-05-13 19:10:59.938286] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9- 1732d24f30f7): trying duplicate remote fd set.
[2017-05-13 19:10:59.938510] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9- 1732d24f30f7): trying duplicate remote fd set.
[2017-05-13 19:10:59.939004] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9- 1732d24f30f7): trying duplicate remote fd set.
[2017-05-13 19:10:59.939246] E [shard.c:426:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/3.10. 1/xlator/cluster/distribute. so(+0x65df0) [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10. 1/xlator/features/shard.so(+ 0xba6e) [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10. 1/xlator/features/shard.so(+ 0xb29b) [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get trusted.glusterfs.shard.file- size for 7dcd980b-4f8b-4e0c-8fb9- 1732d24f30f7
[2017-05-13 19:11:01.884482] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25- fb0205f5d99e): trying duplicate remote fd set.
[2017-05-13 19:11:01.884482] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25- fb0205f5d99e): trying duplicate remote fd set.
[2017-05-13 19:11:01.884557] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25- fb0205f5d99e): trying duplicate remote fd set.
[2017-05-13 19:11:01.884918] E [shard.c:426:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/3.10. 1/xlator/cluster/distribute. so(+0x65df0) [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10. 1/xlator/features/shard.so(+ 0xba6e) [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10. 1/xlator/features/shard.so(+ 0xb29b) [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get trusted.glusterfs.shard.file- size for 2b3e3ab2-ba6a-4ad3-af25- fb0205f5d99e
[2017-05-13 19:11:05.136517] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/ usenet/Attack.on.Titan.S02E05. 720p.WEB.x264-ANiURL.#27/ aniurl-aot.s02e05.720p.web.r00 (b2745d17-1972-4738-afa9- 22e9597fa787): trying duplicate remote fd set.
[2017-05-13 19:11:05.136518] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/