Hi everybody,
I am running a 2x2 node + arbiter distributed - replicated volume (~60TB, 70% used) since January
I've been experiencing problems with the fuse mounts when copying data into the volume.
Once in a while during copy a newly created directory disappears on the mount where the copy is performed. The mounts on the other nodes are not affected, as they show the "missing" directory. The current workaround is unmounting and mounting the volume.
I am using rsync, the data I'm copying is mostly media, 1MB~20GBfile size , it was 6 terabytes initially.
excerpt from mount log:
[2020-04-21 20:53:19.128073] I [MSGID: 108031] [afr-common.c:2581:afr_local_discovery_cbk] 0-tapeless-replicate-1: selecting local read_child tapeless-client-3
[2020-04-21 21:00:53.680413] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fea041e08ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fe9fb590221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fe9fb591998] (--> /lib64/libpthread.
so.0(+0x7e65)[0x7fea03021e65] (--> /lib64/libc.so.6(clone+0x6d)[0x7fea028e988d] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2020-04-21 21:00:53.680880] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fea041e08ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fe9fb590221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fe9fb591998] (--> /lib64/libpthread.
so.0(+0x7e65)[0x7fea03021e65] (--> /lib64/libc.so.6(clone+0x6d)[0x7fea028e988d] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2020-04-21 21:02:07.062090] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fea041e08ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fe9fb590221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fe9fb591998] (--> /lib64/libpthread.
so.0(+0x7e65)[0x7fea03021e65] (--> /lib64/libc.so.6(clone+0x6d)[0x7fea028e988d] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2020-04-21 21:02:56.695667] W [fuse-bridge.c:949:fuse_entry_cbk] 0-glusterfs-fuse: 62900477: MKDIR() /GR/graficos/OPERATIVA_no_borrar/LA MANANA EN CASA/ANIMADOS HD/2018/SP/ZOCALO_CHOCOTINA_203740_3 => -1 (File exists)
[2020-04-21 21:02:57.029790] W [fuse-bridge.c:949:fuse_entry_cbk] 0-glusterfs-fuse: 62900528: MKDIR() /GR/graficos/OPERATIVA_no_borrar/LA MANANA EN CASA/ANIMADOS HD/2018/SP/ZOCALO_CHOCOTINA_203740_3 => -1 (File exists)
[2020-04-21 21:00:53.680413] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fea041e08ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fe9fb590221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fe9fb591998] (--> /lib64/libpthread.
so.0(+0x7e65)[0x7fea03021e65] (--> /lib64/libc.so.6(clone+0x6d)[0x7fea028e988d] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2020-04-21 21:00:53.680880] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fea041e08ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fe9fb590221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fe9fb591998] (--> /lib64/libpthread.
so.0(+0x7e65)[0x7fea03021e65] (--> /lib64/libc.so.6(clone+0x6d)[0x7fea028e988d] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2020-04-21 21:02:07.062090] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fea041e08ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fe9fb590221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fe9fb591998] (--> /lib64/libpthread.
so.0(+0x7e65)[0x7fea03021e65] (--> /lib64/libc.so.6(clone+0x6d)[0x7fea028e988d] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2020-04-21 21:02:56.695667] W [fuse-bridge.c:949:fuse_entry_cbk] 0-glusterfs-fuse: 62900477: MKDIR() /GR/graficos/OPERATIVA_no_borrar/LA MANANA EN CASA/ANIMADOS HD/2018/SP/ZOCALO_CHOCOTINA_203740_3 => -1 (File exists)
[2020-04-21 21:02:57.029790] W [fuse-bridge.c:949:fuse_entry_cbk] 0-glusterfs-fuse: 62900528: MKDIR() /GR/graficos/OPERATIVA_no_borrar/LA MANANA EN CASA/ANIMADOS HD/2018/SP/ZOCALO_CHOCOTINA_203740_3 => -1 (File exists)
version glusterfs 7.3
Volume Name: tapeless
Type: Distributed-Replicate
Volume ID: 53bfa86d-b390-496b-bbd7-c4bba625c956
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: gluster6.glustersaeta.net:/data/glusterfs/tapeless/brick_6/brick
Brick2: gluster7.glustersaeta.net:/data/glusterfs/tapeless/brick_7/brick
Brick3: kitchen-store.glustersaeta.net:/data/glusterfs/tapeless/brick_1a/brick (arbiter)
Brick4: gluster12.glustersaeta.net:/data/glusterfs/tapeless/brick_12/brick
Brick5: gluster13.glustersaeta.net:/data/glusterfs/tapeless/brick_13/brick
Brick6: kitchen-store.glustersaeta.net:/data/glusterfs/tapeless/brick_2a/brick (arbiter)
Options Reconfigured:
features.quota-deem-statfs: on
cluster.self-heal-daemon: on
cluster.entry-self-heal: on
cluster.metadata-self-heal: on
cluster.data-self-heal: on
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
storage.batch-fsync-delay-usec: 0
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
features.quota: on
features.inode-quota: on
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.cache-samba-metadata: on
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
network.inode-lru-limit: 200000
performance.nl-cache: on
performance.nl-cache-timeout: 600
performance.readdir-ahead: on
performance.parallel-readdir: on
performance.cache-size: 1GB
client.event-threads: 4
server.event-threads: 4
performance.normal-prio-threads: 16
performance.io-thread-count: 32
performance.write-behind-window-size: 4MB
Type: Distributed-Replicate
Volume ID: 53bfa86d-b390-496b-bbd7-c4bba625c956
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: gluster6.glustersaeta.net:/data/glusterfs/tapeless/brick_6/brick
Brick2: gluster7.glustersaeta.net:/data/glusterfs/tapeless/brick_7/brick
Brick3: kitchen-store.glustersaeta.net:/data/glusterfs/tapeless/brick_1a/brick (arbiter)
Brick4: gluster12.glustersaeta.net:/data/glusterfs/tapeless/brick_12/brick
Brick5: gluster13.glustersaeta.net:/data/glusterfs/tapeless/brick_13/brick
Brick6: kitchen-store.glustersaeta.net:/data/glusterfs/tapeless/brick_2a/brick (arbiter)
Options Reconfigured:
features.quota-deem-statfs: on
cluster.self-heal-daemon: on
cluster.entry-self-heal: on
cluster.metadata-self-heal: on
cluster.data-self-heal: on
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
storage.batch-fsync-delay-usec: 0
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
features.quota: on
features.inode-quota: on
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.cache-samba-metadata: on
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
network.inode-lru-limit: 200000
performance.nl-cache: on
performance.nl-cache-timeout: 600
performance.readdir-ahead: on
performance.parallel-readdir: on
performance.cache-size: 1GB
client.event-threads: 4
server.event-threads: 4
performance.normal-prio-threads: 16
performance.io-thread-count: 32
performance.write-behind-window-size: 4MB
Please let me know if you need additional information
Thanks!
Martin Lorenzo
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
