Glusterfs 10.5-1 healing issues

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


Dear all,

we would like to describe the situation that we have and that does not solve since a long time, that means after many minor
and major upgrades of GlusterFS

We use a KVM environment for VMs for glusterfs and host servers are updated regularly. Hosts are disomogeneous hardware,
but configured with same characteristics.

The VMs have been also harmonized to use the virtio drivers where available for devices and resources reserved are the same
on each host.

Physical switch for hosts has been substituted with a reliable one.

Probing peers has been and is quite quick in the heartbeat network and communication between the servers for apparently has no issues on disruptions.

And I say apparently because what we have is:

- always pending failed heals that used to resolve by a rotated reboot of the gluster vms (replica 3). Restarting only glusterfs related services (daemon, events etc.) has no effect, only reboot brings results
- very often failed heals are directories

We lately removed a brick that was on a vm on a host that has been entirely substituted. Re-added the brick, sync went on and all data was eventually synced and started with 0 pending failed heals. Now it develops failed heals too like its fellow bricks. Please take into account we healed all the failed entries (manually with various methods) before adding the third brick.

After some days of operating, the count of failed heals rises again, not really fast but with new entries for sure (which might solve
with rotated reboots, or not).

We have gluster clients also on ctdbs that connect to the gluster and mount via glusterfs client. Windows roaming profiles shared via smb become frequently corrupted,(they are composed of a great number small files and are though of big total dimension). Gluster nodes are formatted with xfs.

Also what we observer is that mounting with the vfs option in smb on the ctdbs has some kind of delay. This means that you can see the shared folder on for example a Windows client machine on a ctdb, but not on another ctdb in the cluster and then after a while it appears there too. And this frequently st

This is an excerpt of entries on our shd logs:

2024-04-08 10:13:26.213596 +0000] I [MSGID: 108026] [afr-self-heal-entry.c:1080:afr_selfheal_entry_do] 0-gv-ho-replicate-0: performing full entry selfheal on 2c621415-6223-4b66-a4ca-3f6f267a448d [2024-04-08 10:14:08.135911 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2457:client4_0_link_cbk] 0-gv-ho-client-5: remote operation failed. [{source=<gfid:91d83f0e-1864-4ff3-9174-b7c956e20596>}, {target=(null)}, {errno=116}, {error=Veraltete Dateizugriffsnummer (file handle)}] [2024-04-08 10:15:59.135908 +0000] W [MSGID: 114061] [client-common.c:2992:client_pre_readdir_v2] 0-gv-ho-client-5: remote_fd is -1. EBADFD [{gfid=6b5e599e-c836-4ebe-b16a-8224425b88c7}, {errno=77}, {error=Die Dateizugriffsnummer ist in schlechter Verfassung}] [2024-04-08 10:30:25.013592 +0000] I [MSGID: 108026] [afr-self-heal-entry.c:1080:afr_selfheal_entry_do] 0-gv-ho-replicate-0: performing full entry selfheal on 24e82e12-5512-4679-9eb3-8bd098367db7 [2024-04-08 10:33:17.613594 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2457:client4_0_link_cbk] 0-gv-ho-client-5: remote operation failed. [{source=<gfid:ef9068fc-a329-4a21-88d2-265ecd3d208c>}, {target=(null)}, {errno=116}, {error=Veraltete Dateizugriffsnummer (file handle)}] [2024-04-08 10:33:21.201359 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2457:client4_0_link_cbk] 0-gv-ho-client-5: remote operation failed. [{source=

How are he clients mapped to real hosts in order to know on which one´s logs to look at?

We would like to go by exclusion to finally eradicate this, possibly in a conservative way (not rebuilding everything) and we

are becoming clueless as to where to look at as we also tried various options settings regarding performance etc.

Here is the set on our main volume:

cluster.lookup-unhashed                  on (DEFAULT)
cluster.lookup-optimize                  on (DEFAULT)
cluster.min-free-disk                    10% (DEFAULT)
cluster.min-free-inodes                  5% (DEFAULT)
cluster.rebalance-stats                  off (DEFAULT)
cluster.subvols-per-directory            (null) (DEFAULT)
cluster.readdir-optimize                 off (DEFAULT)
cluster.rsync-hash-regex                 (null) (DEFAULT)
cluster.extra-hash-regex                 (null) (DEFAULT)
cluster.dht-xattr-name                   trusted.glusterfs.dht (DEFAULT)
cluster.randomize-hash-range-by-gfid     off (DEFAULT)
cluster.rebal-throttle                   normal (DEFAULT)
cluster.lock-migration off
cluster.force-migration off
cluster.local-volume-name                (null) (DEFAULT)
cluster.weighted-rebalance               on (DEFAULT)
cluster.switch-pattern                   (null) (DEFAULT)
cluster.entry-change-log                 on (DEFAULT)                   (null) (DEFAULT)             -1 (DEFAULT)                   1 (DEFAULT)
cluster.background-self-heal-count       8 (DEFAULT)
cluster.metadata-self-heal on on
cluster.entry-self-heal on
cluster.self-heal-daemon enable
cluster.heal-timeout                     600 (DEFAULT)
cluster.self-heal-window-size            8 (DEFAULT)                  on (DEFAULT)
cluster.metadata-change-log              on (DEFAULT)         (null) (DEFAULT)
cluster.eager-lock                       on (DEFAULT)
disperse.eager-lock                      on (DEFAULT)
disperse.other-eager-lock                on (DEFAULT)
disperse.eager-lock-timeout              1 (DEFAULT)
disperse.other-eager-lock-timeout        1 (DEFAULT)
cluster.quorum-type auto
cluster.quorum-count 2
cluster.choose-local                     true (DEFAULT)
cluster.self-heal-readdir-size           1KB (DEFAULT)               1 (DEFAULT)
cluster.ensure-durability                on (DEFAULT)
cluster.consistent-metadata              no (DEFAULT)
cluster.heal-wait-queue-length           128 (DEFAULT)
cluster.favorite-child-policy none
cluster.full-lock                        yes (DEFAULT)
cluster.optimistic-change-log            on (DEFAULT)
diagnostics.latency-measurement off
diagnostics.dump-fd-stats                off (DEFAULT)
diagnostics.count-fop-hits off
diagnostics.brick-log-level INFO
diagnostics.client-log-level INFO
diagnostics.brick-sys-log-level          CRITICAL (DEFAULT)
diagnostics.client-sys-log-level         CRITICAL (DEFAULT)
diagnostics.brick-logger                 (null) (DEFAULT)
diagnostics.client-logger                (null) (DEFAULT)
diagnostics.brick-log-format             (null) (DEFAULT)
diagnostics.client-log-format            (null) (DEFAULT)
diagnostics.brick-log-buf-size           5 (DEFAULT)
diagnostics.client-log-buf-size          5 (DEFAULT)
diagnostics.brick-log-flush-timeout      120 (DEFAULT)
diagnostics.client-log-flush-timeout     120 (DEFAULT)
diagnostics.stats-dump-interval          0 (DEFAULT)
diagnostics.fop-sample-interval          0 (DEFAULT)
diagnostics.stats-dump-format            json (DEFAULT)
diagnostics.fop-sample-buf-size          65535 (DEFAULT)
diagnostics.stats-dnscache-ttl-sec       86400 (DEFAULT)
performance.cache-max-file-size 10
performance.cache-min-file-size          0 (DEFAULT)
performance.cache-refresh-timeout        1 (DEFAULT)
performance.cache-priority (DEFAULT)                32MB (DEFAULT)
performance.cache-size                   32MB (DEFAULT)              16 (DEFAULT)
performance.high-prio-threads            16 (DEFAULT)
performance.normal-prio-threads          16 (DEFAULT)
performance.low-prio-threads             16 (DEFAULT)
performance.least-prio-threads           1 (DEFAULT)
performance.enable-least-priority        on (DEFAULT)
performance.iot-watchdog-secs            (null) (DEFAULT)
performance.iot-cleanup-disconnected-reqs off (DEFAULT)
performance.iot-pass-through             false (DEFAULT)        false (DEFAULT)
performance.quick-read-cache-size        128MB (DEFAULT)
performance.cache-size                   128MB (DEFAULT)
performance.quick-read-cache-timeout     1 (DEFAULT)
performance.qr-cache-timeout 600
performance.quick-read-cache-invalidation false (DEFAULT)
performance.ctime-invalidation           false (DEFAULT)
performance.flush-behind                 on (DEFAULT)
performance.nfs.flush-behind             on (DEFAULT)
performance.write-behind-window-size 4MB
performance.resync-failed-syncs-after-fsync off (DEFAULT)
performance.nfs.write-behind-window-size 1MB (DEFAULT)
performance.strict-o-direct              off (DEFAULT)
performance.nfs.strict-o-direct          off (DEFAULT)
performance.strict-write-ordering        off (DEFAULT)
performance.nfs.strict-write-ordering    off (DEFAULT)
performance.write-behind-trickling-writes on (DEFAULT)
performance.aggregate-size               128KB (DEFAULT)
performance.nfs.write-behind-trickling-writes on (DEFAULT)
performance.lazy-open                    yes (DEFAULT)              yes (DEFAULT)     false (DEFAULT)        4 (DEFAULT)      false (DEFAULT)
performance.readdir-ahead-pass-through   false (DEFAULT)        false (DEFAULT)
performance.write-behind-pass-through    false (DEFAULT) 600
performance.cache-swift-metadata         false (DEFAULT)
performance.cache-samba-metadata on
performance.cache-capability-xattrs      true (DEFAULT)
performance.cache-ima-xattrs             true (DEFAULT)              off (DEFAULT)
performance.xattr-cache-list (DEFAULT)        false (DEFAULT)
network.frame-timeout                    1800 (DEFAULT) 20
network.tcp-window-size                  (null) (DEFAULT)
client.ssl off
network.remote-dio                       disable (DEFAULT)
client.event-threads 4
client.tcp-user-timeout 0
client.keepalive-time 20
client.keepalive-interval 2
client.keepalive-count 9
client.strict-locks off
network.tcp-window-size                  (null) (DEFAULT)
network.inode-lru-limit 200000
auth.allow *
auth.reject                              (null) (DEFAULT)
transport.keepalive 1
server.allow-insecure                    on (DEFAULT)
server.root-squash                       off (DEFAULT)
server.all-squash                        off (DEFAULT)
server.anonuid                           65534 (DEFAULT)
server.anongid                           65534 (DEFAULT)
server.statedump-path                    /var/run/gluster (DEFAULT)
server.outstanding-rpc-limit             64 (DEFAULT)
server.ssl off
auth.ssl-allow *
server.manage-gids                       off (DEFAULT)
server.dynamic-auth                      on (DEFAULT)
client.send-gids                         on (DEFAULT)
server.gid-timeout                       300 (DEFAULT)
server.own-thread                        (null) (DEFAULT)
server.event-threads 4
server.tcp-user-timeout                  42 (DEFAULT)
server.keepalive-time 20
server.keepalive-interval 2
server.keepalive-count 9
transport.listen-backlog 1024
ssl.own-cert                             (null) (DEFAULT)
ssl.private-key                          (null) (DEFAULT)                              (null) (DEFAULT)
ssl.crl-path                             (null) (DEFAULT)
ssl.certificate-depth                    (null) (DEFAULT)
ssl.cipher-list                          (null) (DEFAULT)
ssl.dh-param                             (null) (DEFAULT)                             (null) (DEFAULT)
transport.address-family inet
performance.write-behind off on
performance.readdir-ahead on off on
performance.quick-read on on
performance.stat-prefetch on
performance.client-io-threads off
performance.nfs.write-behind on off off
performance.nfs.quick-read off
performance.nfs.stat-prefetch off off
performance.force-readdirp               true (DEFAULT)
performance.cache-invalidation on    true (DEFAULT)
features.uss off
features.snapshot-directory .snaps off
features.tag-namespaces off
network.compression off
network.compression.window-size          -15 (DEFAULT)
network.compression.mem-level            8 (DEFAULT)
network.compression.min-size             0 (DEFAULT)
network.compression.compression-level    -1 (DEFAULT)
network.compression.debug                false (DEFAULT)
features.default-soft-limit              80% (DEFAULT)
features.soft-timeout                    60 (DEFAULT)
features.hard-timeout                    5 (DEFAULT)
features.alert-time                      86400 (DEFAULT)
features.quota-deem-statfs off
geo-replication.indexing off
geo-replication.indexing off
geo-replication.ignore-pid-check off
geo-replication.ignore-pid-check off
features.quota off
features.inode-quota off
features.bitrot disable
debug.trace off
debug.log-history                        no (DEFAULT)
debug.log-file                           no (DEFAULT)
debug.exclude-ops                        (null) (DEFAULT)
debug.include-ops                        (null) (DEFAULT)
debug.error-gen off
debug.error-failure                      (null) (DEFAULT)
debug.error-number                       (null) (DEFAULT)
debug.random-failure                     off (DEFAULT)
debug.error-fops                         (null) (DEFAULT)
nfs.disable on                       off (DEFAULT)
features.worm off
features.worm-file-level off
features.worm-files-deletable on
features.default-retention-period        120 (DEFAULT)
features.retention-mode                  relax (DEFAULT)              180 (DEFAULT)
storage.linux-aio                        off (DEFAULT)
storage.linux-io_uring                   off (DEFAULT)
storage.batch-fsync-mode                 reverse-fsync (DEFAULT)
storage.batch-fsync-delay-usec           0 (DEFAULT)
storage.owner-uid                        -1 (DEFAULT)
storage.owner-gid                        -1 (DEFAULT)
storage.node-uuid-pathinfo               off (DEFAULT)            30 (DEFAULT)                      off (DEFAULT)
storage.gfid2path                        on (DEFAULT)
storage.gfid2path-separator              : (DEFAULT)
storage.reserve                          1 (DEFAULT)             20 (DEFAULT)
storage.fips-mode-rchecksum on
storage.force-create-mode                0000 (DEFAULT)
storage.force-directory-mode             0000 (DEFAULT)
storage.create-mask                      0777 (DEFAULT)
storage.create-directory-mask            0777 (DEFAULT)
storage.max-hardlinks                    100 (DEFAULT)
features.ctime                           on (DEFAULT)
config.gfproxyd off
cluster.server-quorum-type server
cluster.server-quorum-ratio 51
changelog.changelog                      off (DEFAULT)
changelog.changelog-dir                  {{ brick.path }}/.glusterfs/changelogs (DEFAULT)
changelog.encoding                       ascii (DEFAULT)
changelog.rollover-time                  15 (DEFAULT)
changelog.fsync-interval                 5 (DEFAULT)
changelog.changelog-barrier-timeout 120
changelog.capture-del-path               off (DEFAULT)
features.barrier disable
features.barrier-timeout 120
features.trash                           off (DEFAULT)
features.trash-dir                       .trashcan (DEFAULT)
features.trash-eliminate-path            (null) (DEFAULT)
features.trash-max-filesize              5MB (DEFAULT)
features.trash-internal-op               off (DEFAULT)
cluster.enable-shared-storage disable
locks.trace                              off (DEFAULT)
locks.mandatory-locking                  off (DEFAULT)
cluster.disperse-self-heal-daemon        enable (DEFAULT)
cluster.quorum-reads                     no (DEFAULT)
client.bind-insecure                     (null) (DEFAULT)
features.timeout                         45 (DEFAULT)
features.failover-hosts                  (null) (DEFAULT)
features.shard off
features.shard-block-size                64MB (DEFAULT)
features.shard-lru-limit                 16384 (DEFAULT)
features.shard-deletion-rate             100 (DEFAULT)
features.scrub-throttle lazy
features.scrub-freq biweekly
features.scrub                           false (DEFAULT)
features.expiry-time 120
features.signer-threads 4
features.cache-invalidation on
features.cache-invalidation-timeout 600
ganesha.enable off
features.leases off       60 (DEFAULT)
disperse.background-heals                8 (DEFAULT)
disperse.heal-wait-qlength               128 (DEFAULT)
cluster.heal-timeout                     600 (DEFAULT)
dht.force-readdirp                       on (DEFAULT)                     gfid-hash (DEFAULT)
cluster.shd-max-threads 4
cluster.shd-wait-qlength                 1024 (DEFAULT)
cluster.locking-scheme                   full (DEFAULT)
cluster.granular-entry-heal              no (DEFAULT)
features.locks-revocation-secs           0 (DEFAULT)
features.locks-revocation-clear-all      false (DEFAULT)
features.locks-revocation-max-blocked    0 (DEFAULT)
features.locks-monkey-unlocking          false (DEFAULT)
features.locks-notify-contention         yes (DEFAULT)
features.locks-notify-contention-delay   5 (DEFAULT)
disperse.shd-max-threads                 1 (DEFAULT)
disperse.shd-wait-qlength 4096
disperse.cpu-extensions                  auto (DEFAULT)
disperse.self-heal-window-size           32 (DEFAULT)
cluster.use-compound-fops off
performance.parallel-readdir on
performance.rda-request-size 131072
performance.rda-low-wmark                4096 (DEFAULT)
performance.rda-high-wmark               128KB (DEFAULT)
performance.rda-cache-limit 10MB      false (DEFAULT) 10MB 600
cluster.brick-multiplex disable
cluster.brick-graceful-cleanup disable
glusterd.vol_count_per_thread 100
cluster.max-bricks-per-process 250
disperse.optimistic-change-log           on (DEFAULT)
disperse.stripe-cache                    4 (DEFAULT)
cluster.halo-enabled                     False (DEFAULT)
cluster.halo-shd-max-latency             99999 (DEFAULT)
cluster.halo-nfsd-max-latency            5 (DEFAULT)
cluster.halo-max-latency                 5 (DEFAULT)
cluster.halo-max-replicas                99999 (DEFAULT)
cluster.halo-min-replicas                2 (DEFAULT)
features.selinux on
cluster.daemon-log-level INFO
debug.delay-gen off
delay-gen.delay-percentage               10% (DEFAULT)
delay-gen.delay-duration                 100000 (DEFAULT)
delay-gen.enable (DEFAULT)
disperse.parallel-writes                 on (DEFAULT)
disperse.quorum-count                    0 (DEFAULT)
features.sdfs off
features.cloudsync off
features.ctime on
ctime.noatime on
features.cloudsync-storetype             (null) (DEFAULT)
features.enforce-mandatory-lock off off
config.client-threads 16
config.brick-threads 16
features.cloudsync-remote-read off
features.cloudsync-store-id              (null) (DEFAULT)
features.cloudsync-product-id            (null) (DEFAULT)
features.acl enable
cluster.use-anonymous-inode yes
rebalance.ensure-durability              on (DEFAULT)

Again, sorry for the long post. We would be happy to have this solved as we are excited using glusterfs and we would like to go back to having a stable configuration.

We always appreciate the spirit of collaboration and reciprocal help on this list.


Entschieden für Frieden | Committed to Peace

Ilias Chasapakis
Referent IT | IT Consultant

Forum Ziviler Friedensdienst e.V. | Forum Civil Peace Service
Am Kölner Brett 8 | 50825 Köln | Germany

Tel 0221 91273243 | Fax 0221 91273299 |

Vorstand nach § 26 BGB, einzelvertretungsberechtigt|Executive Board:
Alexander Mauz, Sonja Wiekenberg-Mlalandle, Jens von Bargen
VR 17651 Amtsgericht Köln

Spenden|Donations: IBAN DE90 4306 0967 4103 7264 00   BIC GENODEM1GLS

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature


Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Gluster-users mailing list

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux