I have no clue. Have you checked for errors in the logs ? Maybe you
might find something useful.
Best Regards,
Strahil Nikolov
On Tue, Mar 21, 2023 at 9:56, Diego Zuccato
<diego.zuccato@xxxxxxxx> wrote:
Killed glfsheal, after a day there were 218 processes, then they got
killed by OOM during the weekend. Now there are no processes active.
Trying to run "heal info" reports lots of files quite quickly but does
not spawn any glfsheal process. And neither does restarting glusterd.
Is there some way to selectively run glfsheal to fix one brick at a
time?
Diego
Il 21/03/2023 01:21, Strahil Nikolov ha scritto:
> Theoretically it might help.
> If possible, try to resolve any pending heals.
>
> Best Regards,
> Strahil Nikolov
>
> On Thu, Mar 16, 2023 at 15:29, Diego Zuccato
> <diego.zuccato@xxxxxxxx <mailto:diego.zuccato@xxxxxxxx>> wrote:
> In Debian stopping glusterd does not stop brick processes: to stop
> everything (and free the memory) I have to
> systemctl stop glusterd
> killall glusterfs{,d}
> killall glfsheal
> systemctl start glusterd
> [this behaviour hangs a simple reboot of a machine running
glusterd...
> not nice]
>
> For now I just restarted glusterd w/o killing the bricks:
>
> root@str957-clustor00:~# ps aux|grep glfsheal|wc -l ;
systemctl restart
> glusterd ; ps aux|grep glfsheal|wc -l
> 618
> 618
>
> No change neither in glfsheal processes nor in free memory :(
> Should I "killall glfsheal" before OOK kicks in?
>
> Diego
>
> Il 16/03/2023 12:37, Strahil Nikolov ha scritto:
> > Can you restart glusterd service (first check that it was not
> modified
> > to kill the bricks)?
> >
> > Best Regards,
> > Strahil Nikolov
> >
> > On Thu, Mar 16, 2023 at 8:26, Diego Zuccato
> > <diego.zuccato@xxxxxxxx <mailto:diego.zuccato@xxxxxxxx>
<mailto:diego.zuccato@xxxxxxxx>> wrote:
> > OOM is just just a matter of time.
> >
> > Today mem use is up to 177G/187 and:
> > # ps aux|grep glfsheal|wc -l
> > 551
> >
> > (well, one is actually the grep process, so "only" 550
glfsheal
> > processes.
> >
> > I'll take the last 5:
> > root 3266352 0.5 0.0 600292 93044 ? Sl
06:55 0:07
> > /usr/libexec/glusterfs/glfsheal cluster_data
info-summary --xml
> > root 3267220 0.7 0.0 600292 91964 ? Sl
07:00 0:07
> > /usr/libexec/glusterfs/glfsheal cluster_data
info-summary --xml
> > root 3268076 1.0 0.0 600160 88216 ? Sl
07:05 0:08
> > /usr/libexec/glusterfs/glfsheal cluster_data
info-summary --xml
> > root 3269492 1.6 0.0 600292 91248 ? Sl
07:10 0:07
> > /usr/libexec/glusterfs/glfsheal cluster_data
info-summary --xml
> > root 3270354 4.4 0.0 600292 93260 ? Sl
07:15 0:07
> > /usr/libexec/glusterfs/glfsheal cluster_data
info-summary --xml
> >
> > -8<--
> > root@str957-clustor00:~# ps -o ppid= 3266352
> > 3266345
> > root@str957-clustor00:~# ps -o ppid= 3267220
> > 3267213
> > root@str957-clustor00:~# ps -o ppid= 3268076
> > 3268069
> > root@str957-clustor00:~# ps -o ppid= 3269492
> > 3269485
> > root@str957-clustor00:~# ps -o ppid= 3270354
> > 3270347
> > root@str957-clustor00:~# ps aux|grep 3266345
> > root 3266345 0.0 0.0 430536 10764 ? Sl
06:55 0:00
> > gluster volume heal cluster_data info summary --xml
> > root 3271532 0.0 0.0 6260 2500 pts/1 S+
07:21 0:00
> grep
> > 3266345
> > root@str957-clustor00:~# ps aux|grep 3267213
> > root 3267213 0.0 0.0 430536 10644 ? Sl
07:00 0:00
> > gluster volume heal cluster_data info summary --xml
> > root 3271599 0.0 0.0 6260 2480 pts/1 S+
07:22 0:00
> grep
> > 3267213
> > root@str957-clustor00:~# ps aux|grep 3268069
> > root 3268069 0.0 0.0 430536 10704 ? Sl
07:05 0:00
> > gluster volume heal cluster_data info summary --xml
> > root 3271626 0.0 0.0 6260 2516 pts/1 S+
07:22 0:00
> grep
> > 3268069
> > root@str957-clustor00:~# ps aux|grep 3269485
> > root 3269485 0.0 0.0 430536 10756 ? Sl
07:10 0:00
> > gluster volume heal cluster_data info summary --xml
> > root 3271647 0.0 0.0 6260 2480 pts/1 S+
07:22 0:00
> grep
> > 3269485
> > root@str957-clustor00:~# ps aux|grep 3270347
> > root 3270347 0.0 0.0 430536 10672 ? Sl
07:15 0:00
> > gluster volume heal cluster_data info summary --xml
> > root 3271666 0.0 0.0 6260 2568 pts/1 S+
07:22 0:00
> grep
> > 3270347
> > -8<--
> >
> > Seems glfsheal is spawning more processes.
> > I can't rule out a metadata corruption (or at least a
desync),
> but it
> > shouldn't happen...
> >
> > Diego
> >
> > Il 15/03/2023 20:11, Strahil Nikolov ha scritto:
> > > If you don't experience any OOM , you can focus on
the heals.
> > >
> > > 284 processes of glfsheal seems odd.
> > >
> > > Can you check the ppid for 2-3 randomly picked ?
> > > ps -o ppid= <pid>
> > >
> > > Best Regards,
> > > Strahil Nikolov
> > >
> > > On Wed, Mar 15, 2023 at 9:54, Diego Zuccato
> > > <diego.zuccato@xxxxxxxx
<mailto:diego.zuccato@xxxxxxxx> <mailto:diego.zuccato@xxxxxxxx>
> <mailto:diego.zuccato@xxxxxxxx>> wrote:
> > > I enabled it yesterday and that greatly reduced
memory
> pressure.
> > > Current volume info:
> > > -8<--
> > > Volume Name: cluster_data
> > > Type: Distributed-Replicate
> > > Volume ID: a8caaa90-d161-45bb-a68c-278263a8531a
> > > Status: Started
> > > Snapshot Count: 0
> > > Number of Bricks: 45 x (2 + 1) = 135
> > > Transport-type: tcp
> > > Bricks:
> > > Brick1: clustor00:/srv/bricks/00/d
> > > Brick2: clustor01:/srv/bricks/00/d
> > > Brick3: clustor02:/srv/bricks/00/q (arbiter)
> > > [...]
> > > Brick133: clustor01:/srv/bricks/29/d
> > > Brick134: clustor02:/srv/bricks/29/d
> > > Brick135: clustor00:/srv/bricks/14/q (arbiter)
> > > Options Reconfigured:
> > > performance.quick-read: off
> > > cluster.entry-self-heal: on
> > > cluster.data-self-heal-algorithm: full
> > > cluster.metadata-self-heal: on
> > > cluster.shd-max-threads: 2
> > > network.inode-lru-limit: 500000
> > > performance.md-cache-timeout: 600
> > > performance.cache-invalidation: on
> > > features.cache-invalidation-timeout: 600
> > > features.cache-invalidation: on
> > > features.quota-deem-statfs: on
> > > performance.readdir-ahead: on
> > > cluster.granular-entry-heal: enable
> > > features.scrub: Active
> > > features.bitrot: on
> > > cluster.lookup-optimize: on
> > > performance.stat-prefetch: on
> > > performance.cache-refresh-timeout: 60
> > > performance.parallel-readdir: on
> > > performance.write-behind-window-size: 128MB
> > > cluster.self-heal-daemon: enable
> > > features.inode-quota: on
> > > features.quota: on
> > > transport.address-family: inet
> > > nfs.disable: on
> > > performance.client-io-threads: off
> > > client.event-threads: 1
> > > features.scrub-throttle: normal
> > > diagnostics.brick-log-level: ERROR
> > > diagnostics.client-log-level: ERROR
> > > config.brick-threads: 0
> > > cluster.lookup-unhashed: on
> > > config.client-threads: 1
> > > cluster.use-anonymous-inode: off
> > > diagnostics.brick-sys-log-level: CRITICAL
> > > features.scrub-freq: monthly
> > > cluster.data-self-heal: on
> > > cluster.brick-multiplex: on
> > > cluster.daemon-log-level: ERROR
> > > -8<--
> > >
> > > htop reports that memory usage is up to 143G,
there are 602
> > tasks and
> > > 5232 threads (~20 running) on clustor00, 117G/49
tasks/1565
> > threads on
> > > clustor01 and 126G/45 tasks/1574 threads on
clustor02.
> > > I see quite a lot (284!) of glfsheal processes
running on
> > clustor00 (a
> > > "gluster v heal cluster_data info summary" is
running
> on clustor02
> > > since
> > > yesterday, still no output). Shouldn't be just
one per
> brick?
> > >
> > > Diego
> > >
> > > Il 15/03/2023 08:30, Strahil Nikolov ha scritto:
> > > > Do you use brick multiplexing ?
> > > >
> > > > Best Regards,
> > > > Strahil Nikolov
> > > >
> > > > On Tue, Mar 14, 2023 at 16:44, Diego Zuccato
> > > > <diego.zuccato@xxxxxxxx
<mailto:diego.zuccato@xxxxxxxx>
> <mailto:diego.zuccato@xxxxxxxx> <mailto:diego.zuccato@xxxxxxxx>
> > <mailto:diego.zuccato@xxxxxxxx>> wrote:
> > > > Hello all.
> > > >
> > > > Our Gluster 9.6 cluster is showing increasing
> problems.
> > > > Currently it's composed of 3 servers (2x
Intel Xeon
> > 4210 [20
> > > cores dual
> > > > thread, total 40 threads], 192GB RAM, 30x
HGST
> > HUH721212AL5200
> > > [12TB]),
> > > > configured in replica 3 arbiter 1. Using
Debian
> > packages from
> > > Gluster
> > > > 9.x latest repository.
> > > >
> > > > Seems 192G RAM are not enough to handle
30 data
> bricks + 15
> > > arbiters
> > > > and
> > > > I often had to reload glusterfsd because
glusterfs
> > processed
> > > got killed
> > > > for OOM.
> > > > On top of that, performance have been
quite bad,
> especially
> > > when we
> > > > reached about 20M files. On top of that,
one of
> the servers
> > > have had
> > > > mobo issues that resulted in memory
errors that
> > corrupted some
> > > > bricks fs
> > > > (XFS, it required "xfs_reparir -L" to fix).
> > > > Now I'm getting lots of "stale file handle"
> errors and
> > other
> > > errors
> > > > (like directories that seem empty from the
> client but still
> > > containing
> > > > files in some bricks) and auto healing seems
> unable to
> > complete.
> > > >
> > > > Since I can't keep up continuing to
manually fix
> all the
> > > issues, I'm
> > > > thinking about backup+destroy+recreate
strategy.
> > > >
> > > > I think that if I reduce the number of
bricks per
> > server to just 5
> > > > (RAID1 of 6x12TB disks) I might resolve RAM
> issues - at the
> > > cost of
> > > > longer heal times in case a disk fails. Am I
> right or it's
> > > useless?
> > > > Other recommendations?
> > > > Servers have space for another 6 disks.
Maybe those
> > could be
> > > used for
> > > > some SSDs to speed up access?
> > > >
> > > > TIA.
> > > >
> > > > --
> > > > Diego Zuccato
> > > > DIFA - Dip. di Fisica e Astronomia
> > > > Servizi Informatici
> > > > Alma Mater Studiorum - Università di Bologna
> > > > V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
> > > > tel.: +39 051 20 95786
> > > > ________
> > > >
> > > >
> > > >
> > > > Community Meeting Calendar:
> > > >
> > > > Schedule -
> > > > Every 2nd and 4th Tuesday at 14:30 IST /
09:00 UTC
> > > > Bridge:
https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>
> <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>>
> > <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>
> <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>>>
> > > <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>
> <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>>
> > <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>
> <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>>>>
> > > > <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>
> <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>>
> > <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>
> <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>>>
> > > <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>
> <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>>
> > <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>
> <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>>>>>
> > > > Gluster-users mailing list
> > > > Gluster-users@xxxxxxxxxxx
<mailto:Gluster-users@xxxxxxxxxxx>
> <mailto:Gluster-users@xxxxxxxxxxx>
> > <mailto:Gluster-users@xxxxxxxxxxx>
> <mailto:Gluster-users@xxxxxxxxxxx>
> > > <mailto:Gluster-users@xxxxxxxxxxx>
> > > >
> https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>>
> >
<https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
> > >
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>>
> >
<https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>
> > > >
> >
<https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>>
> >
<https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
> > >
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>>
> >
<https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>>
>
> >
> > >
> > > >
> > >
> > > --
> > > Diego Zuccato
> > > DIFA - Dip. di Fisica e Astronomia
> > > Servizi Informatici
> > > Alma Mater Studiorum - Università di Bologna
> > > V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
> > > tel.: +39 051 20 95786
> > > ________
> > >
> > >
> > >
> > > Community Meeting Calendar:
> > >
> > > Schedule -
> > > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> > > Bridge: https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>
> <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>>
> > <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>
> <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>>>
> > > <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>
> <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>>
> > <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>
> <https://meet.google.com/cpu-eiue-hvk
<https://meet.google.com/cpu-eiue-hvk>>>>
> > > Gluster-users mailing list
> > > Gluster-users@xxxxxxxxxxx
<mailto:Gluster-users@xxxxxxxxxxx>
> <mailto:Gluster-users@xxxxxxxxxxx>
<mailto:Gluster-users@xxxxxxxxxxx>
> > <mailto:Gluster-users@xxxxxxxxxxx>
> > >
https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>>
> >
<https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
> > >
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>>
> >
<https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>
> <https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>
> > >
> >
> > --
> > Diego Zuccato
> > DIFA - Dip. di Fisica e Astronomia
> > Servizi Informatici
> > Alma Mater Studiorum - Università di Bologna
> > V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
> > tel.: +39 051 20 95786
> >
>
> --
> Diego Zuccato
> DIFA - Dip. di Fisica e Astronomia
> Servizi Informatici
> Alma Mater Studiorum - Università di Bologna
> V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
> tel.: +39 051 20 95786
>
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
