Hi Jiri,
Unfortunately i don't know a solution to fix it, other than what i already mentioned, which doesn't seem to be applicable to your specific setup.
I don't think it's ovirt related (running ovirt my-self as well, but being stuck at 4.3 atm, since centos 7 is not supported for 4.4).
If memory serves me well, i believe i start seeing this issue after upgrading from glusterfs 3.12 to 4.x (I believe this whent together with upgrade from ovirt 4.1 to 4.2), then since is version i've observed this issue. currently running 7.9.
It would be nice to get to the bottom of this. I'm still not 100% sure it might even be a glusterfs issue, or something might be wrong with XFS or somewhere else in the IO stack. But I don't know what the next debugging steps could be.
Just as a side note i've also observed this issue on systems without LVM cache.
Cheers Olaf
Op do 8 jul. 2021 om 16:53 schreef Jiří Sléžka <jiri.slezka@xxxxxx>:
Hi Olaf,
thanks for reply.
On 7/8/21 3:29 PM, Olaf Buitelaar wrote:
> Hi Jiri,
>
> your probleem looks pretty similar to mine, see;
> https://lists.gluster.org/pipermail/gluster-users/2021-February/039134.html
> <https://lists.gluster.org/pipermail/gluster-users/2021-February/039134.html>
> Any chance you also see the xfs errors in de brick logs?
yes, I can see this log lines related to "health-check failed" items
[root@ovirt-hci02 ~]# grep "aio_read" /var/log/glusterfs/bricks/*
/var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07
07:13:37.408010] W [MSGID: 113075]
[posix-helpers.c:2135:posix_fs_health_check] 0-vms-posix:
aio_read_cmp_buf() on /gluster_bricks/vms2/vms2/.glusterfs/health_check
returned ret is -1 error is Structure needs cleaning
/var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07
16:11:14.518844] W [MSGID: 113075]
[posix-helpers.c:2135:posix_fs_health_check] 0-vms-posix:
aio_read_cmp_buf() on /gluster_bricks/vms2/vms2/.glusterfs/health_check
returned ret is -1 error is Structure needs cleaning
[root@ovirt-hci01 ~]# grep "aio_read" /var/log/glusterfs/bricks/*
/var/log/glusterfs/bricks/gluster_bricks-engine-engine.log:[2021-07-05
13:15:51.982938] W [MSGID: 113075]
[posix-helpers.c:2135:posix_fs_health_check] 0-engine-posix:
aio_read_cmp_buf() on
/gluster_bricks/engine/engine/.glusterfs/health_check returned ret is -1
error is Structure needs cleaning
/var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-05
01:53:35.768534] W [MSGID: 113075]
[posix-helpers.c:2135:posix_fs_health_check] 0-vms-posix:
aio_read_cmp_buf() on /gluster_bricks/vms2/vms2/.glusterfs/health_check
returned ret is -1 error is Structure needs cleaning
it looks very similar to your issue but in my case I don't use LVM cache
and brick disks are JBOD (but connected through Broadcom / LSI MegaRAID
SAS-3 3008 [Fury] (rev 02)).
> For me the situation improved once i disabled brick multiplexing, but i
> don't see that in your volume configuration.
probably important is your note...
> When i kill the brick process and start with "gluser v start x force" the
> issue seems much more unlikely to occur, but when started from a fresh
> reboot, or when killing the process and let it being started by glusterd
> (e.g. service glusterd start) the error seems to arise after a couple of
> minutes.
...because in the ovirt list Jayme replied this
https://lists.ovirt.org/archives/list/users@xxxxxxxxx/message/BZRONK53OGWSOPUSGQ76GIXUM7J6HHMJ/
and it looks to me like something you also observes.
Cheers, Jiri
>
> Cheers Olaf
>
> Op do 8 jul. 2021 om 12:28 schreef Jiří Sléžka <jiri.slezka@xxxxxx
> <mailto:jiri.slezka@xxxxxx>>:
>
> Hello gluster community,
>
> I am new to this list but using glusterfs for log time as our SDS
> solution for storing 80+TiB of data. I'm also using glusterfs for small
> 3 node HCI cluster with oVirt 4.4.6 and CentOS 8 (not stream yet).
> Glusterfs version here is 8.5-2.el8.x86_64.
>
> For time to time (I belive) random brick on random host goes down
> because health-check. It looks like
>
> [root@ovirt-hci02 ~]# grep "posix_health_check"
> /var/log/glusterfs/bricks/*
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07
> 07:13:37.408184] M [MSGID: 113075]
> [posix-helpers.c:2214:posix_health_check_thread_proc] 0-vms-posix:
> health-check failed, going down
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07
> 07:13:37.408407] M [MSGID: 113075]
> [posix-helpers.c:2232:posix_health_check_thread_proc] 0-vms-posix:
> still
> alive! -> SIGTERM
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07
> 16:11:14.518971] M [MSGID: 113075]
> [posix-helpers.c:2214:posix_health_check_thread_proc] 0-vms-posix:
> health-check failed, going down
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07
> 16:11:14.519200] M [MSGID: 113075]
> [posix-helpers.c:2232:posix_health_check_thread_proc] 0-vms-posix:
> still
> alive! -> SIGTERM
>
> on other host
>
> [root@ovirt-hci01 ~]# grep "posix_health_check"
> /var/log/glusterfs/bricks/*
> /var/log/glusterfs/bricks/gluster_bricks-engine-engine.log:[2021-07-05
> 13:15:51.983327] M [MSGID: 113075]
> [posix-helpers.c:2214:posix_health_check_thread_proc] 0-engine-posix:
> health-check failed, going down
> /var/log/glusterfs/bricks/gluster_bricks-engine-engine.log:[2021-07-05
> 13:15:51.983728] M [MSGID: 113075]
> [posix-helpers.c:2232:posix_health_check_thread_proc] 0-engine-posix:
> still alive! -> SIGTERM
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-05
> 01:53:35.769129] M [MSGID: 113075]
> [posix-helpers.c:2214:posix_health_check_thread_proc] 0-vms-posix:
> health-check failed, going down
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-05
> 01:53:35.769819] M [MSGID: 113075]
> [posix-helpers.c:2232:posix_health_check_thread_proc] 0-vms-posix:
> still
> alive! -> SIGTERM
>
> I cannot link these errors to any storage/fs issue (in dmesg or
> /var/log/messages), brick devices looks healthy (smartd).
>
> I can force start brick with
>
> gluster volume start vms|engine force
>
> and after some healing all works fine for few days
>
> Did anybody observe this behavior?
>
> vms volume has this structure (two bricks per host, each is separate
> JBOD ssd disk), engine volume has one brick on each host...
>
> gluster volume info vms
>
> Volume Name: vms
> Type: Distributed-Replicate
> Volume ID: 52032ec6-99d4-4210-8fb8-ffbd7a1e0bf7
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 2 x 3 = 6
> Transport-type: tcp
> Bricks:
> Brick1: 10.0.4.11:/gluster_bricks/vms/vms
> Brick2: 10.0.4.13:/gluster_bricks/vms/vms
> Brick3: 10.0.4.12:/gluster_bricks/vms/vms
> Brick4: 10.0.4.11:/gluster_bricks/vms2/vms2
> Brick5: 10.0.4.13:/gluster_bricks/vms2/vms2
> Brick6: 10.0.4.12:/gluster_bricks/vms2/vms2
> Options Reconfigured:
> cluster.granular-entry-heal: enable
> performance.stat-prefetch: off
> cluster.eager-lock: enable
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> user.cifs: off
> network.ping-timeout: 30
> network.remote-dio: off
> performance.strict-o-direct: on
> performance.low-prio-threads: 32
> features.shard: on
> storage.owner-gid: 36
> storage.owner-uid: 36
> transport.address-family: inet
> storage.fips-mode-rchecksum: on
> nfs.disable: on
> performance.client-io-threads: off
>
>
> Cheers,
>
> Jiri
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> <https://meet.google.com/cpu-eiue-hvk>
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx>
> https://lists.gluster.org/mailman/listinfo/gluster-users
> <https://lists.gluster.org/mailman/listinfo/gluster-users>
>
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
