Dear all,we would like to describe the situation that we have and that does not solve since a long time, that means after many minor
and major upgrades of GlusterFSWe use a KVM environment for VMs for glusterfs and host servers are updated regularly. Hosts are disomogeneous hardware,
but configured with same characteristics.The VMs have been also harmonized to use the virtio drivers where available for devices and resources reserved are the same
on each host. Physical switch for hosts has been substituted with a reliable one.Probing peers has been and is quite quick in the heartbeat network and communication between the servers for apparently has no issues on disruptions.
And I say apparently because what we have is:- always pending failed heals that used to resolve by a rotated reboot of the gluster vms (replica 3). Restarting only glusterfs related services (daemon, events etc.) has no effect, only reboot brings results
- very often failed heals are directoriesWe lately removed a brick that was on a vm on a host that has been entirely substituted. Re-added the brick, sync went on and all data was eventually synced and started with 0 pending failed heals. Now it develops failed heals too like its fellow bricks. Please take into account we healed all the failed entries (manually with various methods) before adding the third brick.
After some days of operating, the count of failed heals rises again, not really fast but with new entries for sure (which might solve
with rotated reboots, or not).We have gluster clients also on ctdbs that connect to the gluster and mount via glusterfs client. Windows roaming profiles shared via smb become frequently corrupted,(they are composed of a great number small files and are though of big total dimension). Gluster nodes are formatted with xfs.
Also what we observer is that mounting with the vfs option in smb on the ctdbs has some kind of delay. This means that you can see the shared folder on for example a Windows client machine on a ctdb, but not on another ctdb in the cluster and then after a while it appears there too. And this frequently st
This is an excerpt of entries on our shd logs:
2024-04-08 10:13:26.213596 +0000] I [MSGID: 108026] [afr-self-heal-entry.c:1080:afr_selfheal_entry_do] 0-gv-ho-replicate-0: performing full entry selfheal on 2c621415-6223-4b66-a4ca-3f6f267a448d [2024-04-08 10:14:08.135911 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2457:client4_0_link_cbk] 0-gv-ho-client-5: remote operation failed. [{source=<gfid:91d83f0e-1864-4ff3-9174-b7c956e20596>}, {target=(null)}, {errno=116}, {error=Veraltete Dateizugriffsnummer (file handle)}] [2024-04-08 10:15:59.135908 +0000] W [MSGID: 114061] [client-common.c:2992:client_pre_readdir_v2] 0-gv-ho-client-5: remote_fd is -1. EBADFD [{gfid=6b5e599e-c836-4ebe-b16a-8224425b88c7}, {errno=77}, {error=Die Dateizugriffsnummer ist in schlechter Verfassung}] [2024-04-08 10:30:25.013592 +0000] I [MSGID: 108026] [afr-self-heal-entry.c:1080:afr_selfheal_entry_do] 0-gv-ho-replicate-0: performing full entry selfheal on 24e82e12-5512-4679-9eb3-8bd098367db7 [2024-04-08 10:33:17.613594 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2457:client4_0_link_cbk] 0-gv-ho-client-5: remote operation failed. [{source=<gfid:ef9068fc-a329-4a21-88d2-265ecd3d208c>}, {target=(null)}, {errno=116}, {error=Veraltete Dateizugriffsnummer (file handle)}] [2024-04-08 10:33:21.201359 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2457:client4_0_link_cbk] 0-gv-ho-client-5: remote operation failed. [{source=
How are he clients mapped to real hosts in order to know on which one´s logs to look at?
We would like to go by exclusion to finally eradicate this, possibly in a conservative way (not rebuilding everything) and we
are becoming clueless as to where to look at as we also tried various options settings regarding performance etc.
Here is the set on our main volume:
cluster.lookup-unhashed on (DEFAULT) cluster.lookup-optimize on (DEFAULT) cluster.min-free-disk 10% (DEFAULT) cluster.min-free-inodes 5% (DEFAULT) cluster.rebalance-stats off (DEFAULT) cluster.subvols-per-directory (null) (DEFAULT) cluster.readdir-optimize off (DEFAULT) cluster.rsync-hash-regex (null) (DEFAULT) cluster.extra-hash-regex (null) (DEFAULT) cluster.dht-xattr-name trusted.glusterfs.dht (DEFAULT) cluster.randomize-hash-range-by-gfid off (DEFAULT) cluster.rebal-throttle normal (DEFAULT) cluster.lock-migration off cluster.force-migration off cluster.local-volume-name (null) (DEFAULT) cluster.weighted-rebalance on (DEFAULT) cluster.switch-pattern (null) (DEFAULT) cluster.entry-change-log on (DEFAULT) cluster.read-subvolume (null) (DEFAULT) cluster.read-subvolume-index -1 (DEFAULT) cluster.read-hash-mode 1 (DEFAULT) cluster.background-self-heal-count 8 (DEFAULT) cluster.metadata-self-heal on cluster.data-self-heal on cluster.entry-self-heal on cluster.self-heal-daemon enable cluster.heal-timeout 600 (DEFAULT) cluster.self-heal-window-size 8 (DEFAULT) cluster.data-change-log on (DEFAULT) cluster.metadata-change-log on (DEFAULT) cluster.data-self-heal-algorithm (null) (DEFAULT) cluster.eager-lock on (DEFAULT) disperse.eager-lock on (DEFAULT) disperse.other-eager-lock on (DEFAULT) disperse.eager-lock-timeout 1 (DEFAULT) disperse.other-eager-lock-timeout 1 (DEFAULT) cluster.quorum-type auto cluster.quorum-count 2 cluster.choose-local true (DEFAULT) cluster.self-heal-readdir-size 1KB (DEFAULT) cluster.post-op-delay-secs 1 (DEFAULT) cluster.ensure-durability on (DEFAULT) cluster.consistent-metadata no (DEFAULT) cluster.heal-wait-queue-length 128 (DEFAULT) cluster.favorite-child-policy none cluster.full-lock yes (DEFAULT) cluster.optimistic-change-log on (DEFAULT) diagnostics.latency-measurement off diagnostics.dump-fd-stats off (DEFAULT) diagnostics.count-fop-hits off diagnostics.brick-log-level INFO diagnostics.client-log-level INFO diagnostics.brick-sys-log-level CRITICAL (DEFAULT) diagnostics.client-sys-log-level CRITICAL (DEFAULT) diagnostics.brick-logger (null) (DEFAULT) diagnostics.client-logger (null) (DEFAULT) diagnostics.brick-log-format (null) (DEFAULT) diagnostics.client-log-format (null) (DEFAULT) diagnostics.brick-log-buf-size 5 (DEFAULT) diagnostics.client-log-buf-size 5 (DEFAULT) diagnostics.brick-log-flush-timeout 120 (DEFAULT) diagnostics.client-log-flush-timeout 120 (DEFAULT) diagnostics.stats-dump-interval 0 (DEFAULT) diagnostics.fop-sample-interval 0 (DEFAULT) diagnostics.stats-dump-format json (DEFAULT) diagnostics.fop-sample-buf-size 65535 (DEFAULT) diagnostics.stats-dnscache-ttl-sec 86400 (DEFAULT) performance.cache-max-file-size 10 performance.cache-min-file-size 0 (DEFAULT) performance.cache-refresh-timeout 1 (DEFAULT) performance.cache-priority (DEFAULT) performance.io-cache-size 32MB (DEFAULT) performance.cache-size 32MB (DEFAULT) performance.io-thread-count 16 (DEFAULT) performance.high-prio-threads 16 (DEFAULT) performance.normal-prio-threads 16 (DEFAULT) performance.low-prio-threads 16 (DEFAULT) performance.least-prio-threads 1 (DEFAULT) performance.enable-least-priority on (DEFAULT) performance.iot-watchdog-secs (null) (DEFAULT) performance.iot-cleanup-disconnected-reqs off (DEFAULT) performance.iot-pass-through false (DEFAULT) performance.io-cache-pass-through false (DEFAULT) performance.quick-read-cache-size 128MB (DEFAULT) performance.cache-size 128MB (DEFAULT) performance.quick-read-cache-timeout 1 (DEFAULT) performance.qr-cache-timeout 600 performance.quick-read-cache-invalidation false (DEFAULT) performance.ctime-invalidation false (DEFAULT) performance.flush-behind on (DEFAULT) performance.nfs.flush-behind on (DEFAULT) performance.write-behind-window-size 4MB performance.resync-failed-syncs-after-fsync off (DEFAULT) performance.nfs.write-behind-window-size 1MB (DEFAULT) performance.strict-o-direct off (DEFAULT) performance.nfs.strict-o-direct off (DEFAULT) performance.strict-write-ordering off (DEFAULT) performance.nfs.strict-write-ordering off (DEFAULT) performance.write-behind-trickling-writes on (DEFAULT) performance.aggregate-size 128KB (DEFAULT) performance.nfs.write-behind-trickling-writes on (DEFAULT) performance.lazy-open yes (DEFAULT) performance.read-after-open yes (DEFAULT) performance.open-behind-pass-through false (DEFAULT) performance.read-ahead-page-count 4 (DEFAULT) performance.read-ahead-pass-through false (DEFAULT) performance.readdir-ahead-pass-through false (DEFAULT) performance.md-cache-pass-through false (DEFAULT) performance.write-behind-pass-through false (DEFAULT) performance.md-cache-timeout 600 performance.cache-swift-metadata false (DEFAULT) performance.cache-samba-metadata on performance.cache-capability-xattrs true (DEFAULT) performance.cache-ima-xattrs true (DEFAULT) performance.md-cache-statfs off (DEFAULT) performance.xattr-cache-list (DEFAULT) performance.nl-cache-pass-through false (DEFAULT) network.frame-timeout 1800 (DEFAULT) network.ping-timeout 20 network.tcp-window-size (null) (DEFAULT) client.ssl off network.remote-dio disable (DEFAULT) client.event-threads 4 client.tcp-user-timeout 0 client.keepalive-time 20 client.keepalive-interval 2 client.keepalive-count 9 client.strict-locks off network.tcp-window-size (null) (DEFAULT) network.inode-lru-limit 200000 auth.allow * auth.reject (null) (DEFAULT) transport.keepalive 1 server.allow-insecure on (DEFAULT) server.root-squash off (DEFAULT) server.all-squash off (DEFAULT) server.anonuid 65534 (DEFAULT) server.anongid 65534 (DEFAULT) server.statedump-path /var/run/gluster (DEFAULT) server.outstanding-rpc-limit 64 (DEFAULT) server.ssl off auth.ssl-allow * server.manage-gids off (DEFAULT) server.dynamic-auth on (DEFAULT) client.send-gids on (DEFAULT) server.gid-timeout 300 (DEFAULT) server.own-thread (null) (DEFAULT) server.event-threads 4 server.tcp-user-timeout 42 (DEFAULT) server.keepalive-time 20 server.keepalive-interval 2 server.keepalive-count 9 transport.listen-backlog 1024 ssl.own-cert (null) (DEFAULT) ssl.private-key (null) (DEFAULT) ssl.ca-list (null) (DEFAULT) ssl.crl-path (null) (DEFAULT) ssl.certificate-depth (null) (DEFAULT) ssl.cipher-list (null) (DEFAULT) ssl.dh-param (null) (DEFAULT) ssl.ec-curve (null) (DEFAULT) transport.address-family inet performance.write-behind off performance.read-ahead on performance.readdir-ahead on performance.io-cache off performance.open-behind on performance.quick-read on performance.nl-cache on performance.stat-prefetch on performance.client-io-threads off performance.nfs.write-behind on performance.nfs.read-ahead off performance.nfs.io-cache off performance.nfs.quick-read off performance.nfs.stat-prefetch off performance.nfs.io-threads off performance.force-readdirp true (DEFAULT) performance.cache-invalidation on performance.global-cache-invalidation true (DEFAULT) features.uss off features.snapshot-directory .snaps features.show-snapshot-directory off features.tag-namespaces off network.compression off network.compression.window-size -15 (DEFAULT) network.compression.mem-level 8 (DEFAULT) network.compression.min-size 0 (DEFAULT) network.compression.compression-level -1 (DEFAULT) network.compression.debug false (DEFAULT) features.default-soft-limit 80% (DEFAULT) features.soft-timeout 60 (DEFAULT) features.hard-timeout 5 (DEFAULT) features.alert-time 86400 (DEFAULT) features.quota-deem-statfs off geo-replication.indexing off geo-replication.indexing off geo-replication.ignore-pid-check off geo-replication.ignore-pid-check off features.quota off features.inode-quota off features.bitrot disable debug.trace off debug.log-history no (DEFAULT) debug.log-file no (DEFAULT) debug.exclude-ops (null) (DEFAULT) debug.include-ops (null) (DEFAULT) debug.error-gen off debug.error-failure (null) (DEFAULT) debug.error-number (null) (DEFAULT) debug.random-failure off (DEFAULT) debug.error-fops (null) (DEFAULT) nfs.disable on features.read-only off (DEFAULT) features.worm off features.worm-file-level off features.worm-files-deletable on features.default-retention-period 120 (DEFAULT) features.retention-mode relax (DEFAULT) features.auto-commit-period 180 (DEFAULT) storage.linux-aio off (DEFAULT) storage.linux-io_uring off (DEFAULT) storage.batch-fsync-mode reverse-fsync (DEFAULT) storage.batch-fsync-delay-usec 0 (DEFAULT) storage.owner-uid -1 (DEFAULT) storage.owner-gid -1 (DEFAULT) storage.node-uuid-pathinfo off (DEFAULT) storage.health-check-interval 30 (DEFAULT) storage.build-pgfid off (DEFAULT) storage.gfid2path on (DEFAULT) storage.gfid2path-separator : (DEFAULT) storage.reserve 1 (DEFAULT) storage.health-check-timeout 20 (DEFAULT) storage.fips-mode-rchecksum on storage.force-create-mode 0000 (DEFAULT) storage.force-directory-mode 0000 (DEFAULT) storage.create-mask 0777 (DEFAULT) storage.create-directory-mask 0777 (DEFAULT) storage.max-hardlinks 100 (DEFAULT) features.ctime on (DEFAULT) config.gfproxyd off cluster.server-quorum-type server cluster.server-quorum-ratio 51 changelog.changelog off (DEFAULT)changelog.changelog-dir {{ brick.path }}/.glusterfs/changelogs (DEFAULT)changelog.encoding ascii (DEFAULT) changelog.rollover-time 15 (DEFAULT) changelog.fsync-interval 5 (DEFAULT) changelog.changelog-barrier-timeout 120 changelog.capture-del-path off (DEFAULT) features.barrier disable features.barrier-timeout 120 features.trash off (DEFAULT) features.trash-dir .trashcan (DEFAULT) features.trash-eliminate-path (null) (DEFAULT) features.trash-max-filesize 5MB (DEFAULT) features.trash-internal-op off (DEFAULT) cluster.enable-shared-storage disable locks.trace off (DEFAULT) locks.mandatory-locking off (DEFAULT) cluster.disperse-self-heal-daemon enable (DEFAULT) cluster.quorum-reads no (DEFAULT) client.bind-insecure (null) (DEFAULT) features.timeout 45 (DEFAULT) features.failover-hosts (null) (DEFAULT) features.shard off features.shard-block-size 64MB (DEFAULT) features.shard-lru-limit 16384 (DEFAULT) features.shard-deletion-rate 100 (DEFAULT) features.scrub-throttle lazy features.scrub-freq biweekly features.scrub false (DEFAULT) features.expiry-time 120 features.signer-threads 4 features.cache-invalidation on features.cache-invalidation-timeout 600 ganesha.enable off features.leases off features.lease-lock-recall-timeout 60 (DEFAULT) disperse.background-heals 8 (DEFAULT) disperse.heal-wait-qlength 128 (DEFAULT) cluster.heal-timeout 600 (DEFAULT) dht.force-readdirp on (DEFAULT) disperse.read-policy gfid-hash (DEFAULT) cluster.shd-max-threads 4 cluster.shd-wait-qlength 1024 (DEFAULT) cluster.locking-scheme full (DEFAULT) cluster.granular-entry-heal no (DEFAULT) features.locks-revocation-secs 0 (DEFAULT) features.locks-revocation-clear-all false (DEFAULT) features.locks-revocation-max-blocked 0 (DEFAULT) features.locks-monkey-unlocking false (DEFAULT) features.locks-notify-contention yes (DEFAULT) features.locks-notify-contention-delay 5 (DEFAULT) disperse.shd-max-threads 1 (DEFAULT) disperse.shd-wait-qlength 4096 disperse.cpu-extensions auto (DEFAULT) disperse.self-heal-window-size 32 (DEFAULT) cluster.use-compound-fops off performance.parallel-readdir on performance.rda-request-size 131072 performance.rda-low-wmark 4096 (DEFAULT) performance.rda-high-wmark 128KB (DEFAULT) performance.rda-cache-limit 10MB performance.nl-cache-positive-entry false (DEFAULT) performance.nl-cache-limit 10MB performance.nl-cache-timeout 600 cluster.brick-multiplex disable cluster.brick-graceful-cleanup disable glusterd.vol_count_per_thread 100 cluster.max-bricks-per-process 250 disperse.optimistic-change-log on (DEFAULT) disperse.stripe-cache 4 (DEFAULT) cluster.halo-enabled False (DEFAULT) cluster.halo-shd-max-latency 99999 (DEFAULT) cluster.halo-nfsd-max-latency 5 (DEFAULT) cluster.halo-max-latency 5 (DEFAULT) cluster.halo-max-replicas 99999 (DEFAULT) cluster.halo-min-replicas 2 (DEFAULT) features.selinux on cluster.daemon-log-level INFO debug.delay-gen off delay-gen.delay-percentage 10% (DEFAULT) delay-gen.delay-duration 100000 (DEFAULT) delay-gen.enable (DEFAULT) disperse.parallel-writes on (DEFAULT) disperse.quorum-count 0 (DEFAULT) features.sdfs off features.cloudsync off features.ctime on ctime.noatime on features.cloudsync-storetype (null) (DEFAULT) features.enforce-mandatory-lock off config.global-threading off config.client-threads 16 config.brick-threads 16 features.cloudsync-remote-read off features.cloudsync-store-id (null) (DEFAULT) features.cloudsync-product-id (null) (DEFAULT) features.acl enable cluster.use-anonymous-inode yes rebalance.ensure-durability on (DEFAULT)
Again, sorry for the long post. We would be happy to have this solved as we are excited using glusterfs and we would like to go back to having a stable configuration.
We always appreciate the spirit of collaboration and reciprocal help on this list.
Best Ilias -- forumZFD Entschieden für Frieden | Committed to Peace Ilias Chasapakis Referent IT | IT Consultant Forum Ziviler Friedensdienst e.V. | Forum Civil Peace Service Am Kölner Brett 8 | 50825 Köln | Germany Tel 0221 91273243 | Fax 0221 91273299 | http://www.forumZFD.de Vorstand nach § 26 BGB, einzelvertretungsberechtigt|Executive Board: Alexander Mauz, Sonja Wiekenberg-Mlalandle, Jens von Bargen VR 17651 Amtsgericht Köln Spenden|Donations: IBAN DE90 4306 0967 4103 7264 00 BIC GENODEM1GLS
Attachment:
OpenPGP_signature.asc
Description: OpenPGP digital signature
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
