Re: Reliability issues with Gluster 3.10 and shard

[Benjamin Kingston <ben@xxxxxxxxxxxxxxx>]

Hers's some log entries from nfs-ganesha gfapi

[2017-05-13 19:02:54.105936] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on shard 11 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error]
[2017-05-13 19:02:54.106176] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on shard 2 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error]
[2017-05-13 19:02:54.106288] E [MSGID: 133010] [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on shard 1 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error]
[2017-05-13 19:02:54.384922] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-storage2-replicate-2: performing metadata selfheal on fe651475-226e-42a3-be2d-751d4f58e383
[2017-05-13 19:02:54.385894] W [MSGID: 114031] [client-rpc-fops.c:2258:client3_3_setattr_cbk] 0-storage2-client-8: remote operation failed [Operation not permitted]
[2017-05-13 19:02:54.401187] I [MSGID: 108026] [afr-self-heal-common.c:1255:afr_log_selfheal] 0-storage2-replicate-2: Completed metadata selfheal on fe651475-226e-42a3-be2d-751d4f58e383. sources=[0] 1  sinks=
[2017-05-13 19:02:57.830019] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2 (hash=storage2-readdir-ahead-0/cache=<nul>)

[2017-05-13 19:08:22.014899] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr (hash=storage2-readdir-ahead-1/cache=<nul>)
[2017-05-13 19:08:22.463840] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04 (hash=storage2-readdir-ahead-0/cache=<nul>)
[2017-05-13 19:08:22.769542] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01 (hash=storage2-readdir-ahead-0/cache=<nul>)
[2017-05-13 19:08:23.141069] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo (hash=storage2-readdir-ahead-0/cache=<nul>)
[2017-05-13 19:08:23.468554] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 (hash=storage2-readdir-ahead-2/cache=<nul>)
[2017-05-13 19:08:23.671753] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv (hash=storage2-readdir-ahead-2/cache=<nul>)
[2017-05-13 19:08:23.812152] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 (hash=storage2-readdir-ahead-1/cache=<nul>)
[2017-05-13 19:08:24.244584] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 (hash=storage2-readdir-ahead-1/cache=<nul>)
[2017-05-13 19:08:24.711781] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r09.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r09 (hash=storage2-readdir-ahead-1/cache=<nul>)
[2017-05-13 19:08:25.029102] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r06.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r06 (hash=storage2-readdir-ahead-0/cache=<nul>)
[2017-05-13 19:08:25.206011] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r05.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r05 (hash=storage2-readdir-ahead-0/cache=<nul>)
[2017-05-13 19:08:25.467312] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r03.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r03 (hash=storage2-readdir-ahead-1/cache=<nul>)
[2017-05-13 19:08:25.726208] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r07.tmp (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r07 (hash=storage2-readdir-ahead-0/cache=<nul>)
[2017-05-13 19:08:25.860386] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.rar.tmp (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.rar (hash=storage2-readdir-ahead-1/cache=<nul>)
[2017-05-13 19:09:00.803859] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r08.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r08 (hash=storage2-readdir-ahead-0/cache=<nul>)
[2017-05-13 19:09:01.826525] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r02.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r02 (hash=storage2-readdir-ahead-2/cache=<nul>)
[2017-05-13 19:09:02.531354] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Videos/Cinema/Bambi (1942)/.Bambi (1942).mkv.hp9oW8 (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Videos/Cinema/Bambi (1942)/.nfsa223c532d51b3c7700000006 (hash=storage2-readdir-ahead-2/cache=<nul>)
[2017-05-13 19:09:39.695354] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r10.tmp (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) => /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r10 (hash=storage2-readdir-ahead-2/cache=<nul>)
[2017-05-13 19:10:59.938286] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set.
[2017-05-13 19:10:59.938510] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set.
[2017-05-13 19:10:59.939004] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set.
[2017-05-13 19:10:59.939246] E [shard.c:426:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get trusted.glusterfs.shard.file-size for 7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7
[2017-05-13 19:11:01.884482] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set.
[2017-05-13 19:11:01.884482] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set.
[2017-05-13 19:11:01.884557] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set.
[2017-05-13 19:11:01.884918] E [shard.c:426:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get trusted.glusterfs.shard.file-size for 2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e
[2017-05-13 19:11:05.136517] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set.
[2017-05-13 19:11:05.136518] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set.
[2017-05-13 19:11:05.136648] I [MSGID: 114024] [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6: /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00 (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set.
[2017-05-13 19:11:05.136871] E [shard.c:426:shard_modify_size_and_block_count] (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0) [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xba6e) [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b) [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get trusted.glusterfs.shard.file-size for b2745d17-1972-4738-afa9-22e9597fa787

-ben

On Fri, May 12, 2017 at 11:46 PM, Benjamin Kingston <ben@xxxxxxxxxxxxxxx> wrote:

Hello all,

I'm trying to take advantage of the shard xlator, however I've found it causes a lot of issues that I hope is easily resolvable

1) large file operations work well (copy file from folder a to folder b

2) seek operations and list operations frequently fail (ls directory, read bytes xyz at offset 235567)

3) Another issue is samba shares through samba-vfs show all files as 4MB, I've also seen this when mounting with fuse, however nfs-ganesha reflects correct file sizes always-

Turning off the shard feature resolves this issue for new files created in the volume. mounted using the gluster fuse mount

here's my volume settings, please let me know if there's some changes I can make.

Volume Name: storage2
Type: Distributed-Replicate
Volume ID: adaabca5-25ed-4e7f-ae86-2f20fc0143a8
Status: Started
Snapshot Count: 0
Number of Bricks: 3 x (2 + 1) = 9
Transport-type: tcp
Bricks:
Brick1: fd00:0:0:3::6:/mnt/gluster/storage/brick0/glusterfs2
Brick2: fd00:0:0:3::8:/mnt/gluster/storage/brick0/glusterfs2
Brick3: fd00:0:0:3::10:/mnt/gluster/storage/brick0/glusterfs (arbiter)
Brick4: fd00:0:0:3::6:/mnt/gluster/storage/brick1/glusterfs2
Brick5: fd00:0:0:3::8:/mnt/gluster/storage/brick1/glusterfs2
Brick6: fd00:0:0:3::10:/mnt/gluster/storage/brick1/glusterfs (arbiter)
Brick7: fd00:0:0:3::6:/mnt/gluster/storage/brick2/glusterfs2
Brick8: fd00:0:0:3::8:/mnt/gluster/storage/brick2/glusterfs2
Brick9: fd00:0:0:3::10:/mnt/gluster/storage/brick2/glusterfs (arbiter)
Options Reconfigured:
features.ctr-enabled: on
features.shard-block-size: 4MB
network.inode-lru-limit: 90000
features.cache-invalidation: on
performance.readdir-ahead: on
client.event-threads: 3
performance.cache-ima-xattrs: on
cluster.data-self-heal-algorithm: diff
network.remote-dio: disable
cluster.use-compound-fops: on
cluster.read-freq-threshold: 2
cluster.write-freq-threshold: 2
features.record-counters: on
disperse.shd-max-threads: 4
performance.parallel-readdir: on
performance.client-io-threads: on
server.event-threads: 3
cluster.lookup-optimize: on
performance.open-behind: on
performance.stat-prefetch: on
performance.quick-read: off
performance.io-cache: on
performance.read-ahead: off
performance.write-behind: on
features.scrub: Active
features.bitrot: on
features.leases: on
features.shard: off
transport.address-family: inet6
nfs.disable: on
server.allow-insecure: on
cluster.shd-max-threads: 8
performance.low-prio-threads: 32
cluster.locking-scheme: granular
cluster.shd-wait-qlength: 10000
user.cifs: off
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.tier-compact: on
storage.linux-aio: on
transport.keepalive: on
performance.write-behind-window-size: 2GB
performance.flush-behind: on
performance.cache-size: 1GB
cluster.choose-local: on
performance.io-thread-count: 64
cluster.brick-multiplex: off
cluster.enable-shared-storage: enable
nfs-ganesha: enable

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users