Re: How to configure?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Killed glfsheal, after a day there were 218 processes, then they got killed by OOM during the weekend. Now there are no processes active. Trying to run "heal info" reports lots of files quite quickly but does not spawn any glfsheal process. And neither does restarting glusterd.
Is there some way to selectively run glfsheal to fix one brick at a time?

Diego

Il 21/03/2023 01:21, Strahil Nikolov ha scritto:
Theoretically it might help.
If possible, try to resolve any pending heals.

Best Regards,
Strahil Nikolov

    On Thu, Mar 16, 2023 at 15:29, Diego Zuccato
    <diego.zuccato@xxxxxxxx> wrote:
    In Debian stopping glusterd does not stop brick processes: to stop
    everything (and free the memory) I have to
    systemctl stop glusterd
       killall glusterfs{,d}
       killall glfsheal
       systemctl start glusterd
    [this behaviour hangs a simple reboot of a machine running glusterd...
    not nice]

    For now I just restarted glusterd w/o killing the bricks:

    root@str957-clustor00:~# ps aux|grep glfsheal|wc -l ; systemctl restart
    glusterd ; ps aux|grep glfsheal|wc -l
    618
    618

    No change neither in glfsheal processes nor in free memory :(
    Should I "killall glfsheal" before OOK kicks in?

    Diego

    Il 16/03/2023 12:37, Strahil Nikolov ha scritto:
     > Can you restart glusterd service (first check that it was not
    modified
     > to kill the bricks)?
     >
     > Best Regards,
     > Strahil Nikolov
     >
     >    On Thu, Mar 16, 2023 at 8:26, Diego Zuccato
     >    <diego.zuccato@xxxxxxxx <mailto:diego.zuccato@xxxxxxxx>> wrote:
     >    OOM is just just a matter of time.
     >
     >    Today mem use is up to 177G/187 and:
     >    # ps aux|grep glfsheal|wc -l
     >    551
     >
     >    (well, one is actually the grep process, so "only" 550 glfsheal
     >    processes.
     >
     >    I'll take the last 5:
     >    root    3266352  0.5  0.0 600292 93044 ?        Sl  06:55  0:07
     >    /usr/libexec/glusterfs/glfsheal cluster_data info-summary --xml
     >    root    3267220  0.7  0.0 600292 91964 ?        Sl  07:00  0:07
     >    /usr/libexec/glusterfs/glfsheal cluster_data info-summary --xml
     >    root    3268076  1.0  0.0 600160 88216 ?        Sl  07:05  0:08
     >    /usr/libexec/glusterfs/glfsheal cluster_data info-summary --xml
     >    root    3269492  1.6  0.0 600292 91248 ?        Sl  07:10  0:07
     >    /usr/libexec/glusterfs/glfsheal cluster_data info-summary --xml
     >    root    3270354  4.4  0.0 600292 93260 ?        Sl  07:15  0:07
     >    /usr/libexec/glusterfs/glfsheal cluster_data info-summary --xml
     >
     >    -8<--
     >    root@str957-clustor00:~# ps -o ppid= 3266352
     >    3266345
     >    root@str957-clustor00:~# ps -o ppid= 3267220
     >    3267213
     >    root@str957-clustor00:~# ps -o ppid= 3268076
     >    3268069
     >    root@str957-clustor00:~# ps -o ppid= 3269492
     >    3269485
     >    root@str957-clustor00:~# ps -o ppid= 3270354
     >    3270347
     >    root@str957-clustor00:~# ps aux|grep 3266345
     >    root    3266345  0.0  0.0 430536 10764 ?        Sl  06:55  0:00
     >    gluster volume heal cluster_data info summary --xml
     >    root    3271532  0.0  0.0  6260  2500 pts/1    S+  07:21  0:00
    grep
     >    3266345
     >    root@str957-clustor00:~# ps aux|grep 3267213
     >    root    3267213  0.0  0.0 430536 10644 ?        Sl  07:00  0:00
     >    gluster volume heal cluster_data info summary --xml
     >    root    3271599  0.0  0.0  6260  2480 pts/1    S+  07:22  0:00
    grep
     >    3267213
     >    root@str957-clustor00:~# ps aux|grep 3268069
     >    root    3268069  0.0  0.0 430536 10704 ?        Sl  07:05  0:00
     >    gluster volume heal cluster_data info summary --xml
     >    root    3271626  0.0  0.0  6260  2516 pts/1    S+  07:22  0:00
    grep
     >    3268069
     >    root@str957-clustor00:~# ps aux|grep 3269485
     >    root    3269485  0.0  0.0 430536 10756 ?        Sl  07:10  0:00
     >    gluster volume heal cluster_data info summary --xml
     >    root    3271647  0.0  0.0  6260  2480 pts/1    S+  07:22  0:00
    grep
     >    3269485
     >    root@str957-clustor00:~# ps aux|grep 3270347
     >    root    3270347  0.0  0.0 430536 10672 ?        Sl  07:15  0:00
     >    gluster volume heal cluster_data info summary --xml
     >    root    3271666  0.0  0.0  6260  2568 pts/1    S+  07:22  0:00
    grep
     >    3270347
     >    -8<--
     >
     >    Seems glfsheal is spawning more processes.
     >    I can't rule out a metadata corruption (or at least a desync),
    but it
     >    shouldn't happen...
     >
     >    Diego
     >
     >    Il 15/03/2023 20:11, Strahil Nikolov ha scritto:
     >      > If you don't experience any OOM , you can focus on the heals.
     >      >
     >      > 284 processes of glfsheal seems odd.
     >      >
     >      > Can you check the ppid for 2-3 randomly picked ?
     >      > ps -o ppid= <pid>
     >      >
     >      > Best Regards,
     >      > Strahil Nikolov
     >      >
     >      >    On Wed, Mar 15, 2023 at 9:54, Diego Zuccato
     >      >    <diego.zuccato@xxxxxxxx <mailto:diego.zuccato@xxxxxxxx>
    <mailto:diego.zuccato@xxxxxxxx>> wrote:
     >      >    I enabled it yesterday and that greatly reduced memory
    pressure.
     >      >    Current volume info:
     >      >    -8<--
     >      >    Volume Name: cluster_data
     >      >    Type: Distributed-Replicate
     >      >    Volume ID: a8caaa90-d161-45bb-a68c-278263a8531a
     >      >    Status: Started
     >      >    Snapshot Count: 0
     >      >    Number of Bricks: 45 x (2 + 1) = 135
     >      >    Transport-type: tcp
     >      >    Bricks:
     >      >    Brick1: clustor00:/srv/bricks/00/d
     >      >    Brick2: clustor01:/srv/bricks/00/d
     >      >    Brick3: clustor02:/srv/bricks/00/q (arbiter)
     >      >    [...]
     >      >    Brick133: clustor01:/srv/bricks/29/d
     >      >    Brick134: clustor02:/srv/bricks/29/d
     >      >    Brick135: clustor00:/srv/bricks/14/q (arbiter)
     >      >    Options Reconfigured:
     >      >    performance.quick-read: off
     >      >    cluster.entry-self-heal: on
     >      >    cluster.data-self-heal-algorithm: full
     >      >    cluster.metadata-self-heal: on
     >      >    cluster.shd-max-threads: 2
     >      >    network.inode-lru-limit: 500000
     >      >    performance.md-cache-timeout: 600
     >      >    performance.cache-invalidation: on
     >      >    features.cache-invalidation-timeout: 600
     >      >    features.cache-invalidation: on
     >      >    features.quota-deem-statfs: on
     >      >    performance.readdir-ahead: on
     >      >    cluster.granular-entry-heal: enable
     >      >    features.scrub: Active
     >      >    features.bitrot: on
     >      >    cluster.lookup-optimize: on
     >      >    performance.stat-prefetch: on
     >      >    performance.cache-refresh-timeout: 60
     >      >    performance.parallel-readdir: on
     >      >    performance.write-behind-window-size: 128MB
     >      >    cluster.self-heal-daemon: enable
     >      >    features.inode-quota: on
     >      >    features.quota: on
     >      >    transport.address-family: inet
     >      >    nfs.disable: on
     >      >    performance.client-io-threads: off
     >      >    client.event-threads: 1
     >      >    features.scrub-throttle: normal
     >      >    diagnostics.brick-log-level: ERROR
     >      >    diagnostics.client-log-level: ERROR
     >      >    config.brick-threads: 0
     >      >    cluster.lookup-unhashed: on
     >      >    config.client-threads: 1
     >      >    cluster.use-anonymous-inode: off
     >      >    diagnostics.brick-sys-log-level: CRITICAL
     >      >    features.scrub-freq: monthly
     >      >    cluster.data-self-heal: on
     >      >    cluster.brick-multiplex: on
     >      >    cluster.daemon-log-level: ERROR
     >      >    -8<--
     >      >
     >      >    htop reports that memory usage is up to 143G, there are 602
     >    tasks and
     >      >    5232 threads (~20 running) on clustor00, 117G/49 tasks/1565
     >    threads on
     >      >    clustor01 and 126G/45 tasks/1574 threads on clustor02.
     >      >    I see quite a lot (284!) of glfsheal processes running on
     >    clustor00 (a
     >      >    "gluster v heal cluster_data info summary" is running
    on clustor02
     >      >    since
     >      >    yesterday, still no output). Shouldn't be just one per
    brick?
     >      >
     >      >    Diego
     >      >
     >      >    Il 15/03/2023 08:30, Strahil Nikolov ha scritto:
     >      >      > Do you use brick multiplexing ?
     >      >      >
     >      >      > Best Regards,
     >      >      > Strahil Nikolov
     >      >      >
     >      >      >    On Tue, Mar 14, 2023 at 16:44, Diego Zuccato
     >      >      >    <diego.zuccato@xxxxxxxx
    <mailto:diego.zuccato@xxxxxxxx> <mailto:diego.zuccato@xxxxxxxx>
     >    <mailto:diego.zuccato@xxxxxxxx>> wrote:
     >      >      >    Hello all.
     >      >      >
     >      >      >    Our Gluster 9.6 cluster is showing increasing
    problems.
     >      >      >    Currently it's composed of 3 servers (2x Intel Xeon
     >    4210 [20
     >      >    cores dual
     >      >      >    thread, total 40 threads], 192GB RAM, 30x HGST
     >    HUH721212AL5200
     >      >    [12TB]),
     >      >      >    configured in replica 3 arbiter 1. Using Debian
     >    packages from
     >      >    Gluster
     >      >      >    9.x latest repository.
     >      >      >
     >      >      >    Seems 192G RAM are not enough to handle 30 data
    bricks + 15
     >      >    arbiters
     >      >      >    and
     >      >      >    I often had to reload glusterfsd because glusterfs
     >    processed
     >      >    got killed
     >      >      >    for OOM.
     >      >      >    On top of that, performance have been quite bad,
    especially
     >      >    when we
     >      >      >    reached about 20M files. On top of that, one of
    the servers
     >      >    have had
     >      >      >    mobo issues that resulted in memory errors that
     >    corrupted some
     >      >      >    bricks fs
     >      >      >    (XFS, it required "xfs_reparir -L" to fix).
     >      >      >    Now I'm getting lots of "stale file handle"
    errors and
     >    other
     >      >    errors
     >      >      >    (like directories that seem empty from the
    client but still
     >      >    containing
     >      >      >    files in some bricks) and auto healing seems
    unable to
     >    complete.
     >      >      >
     >      >      >    Since I can't keep up continuing to manually fix
    all the
     >      >    issues, I'm
     >      >      >    thinking about backup+destroy+recreate strategy.
     >      >      >
     >      >      >    I think that if I reduce the number of bricks per
     >    server to just 5
     >      >      >    (RAID1 of 6x12TB disks) I might resolve RAM
    issues - at the
     >      >    cost of
     >      >      >    longer heal times in case a disk fails. Am I
    right or it's
     >      >    useless?
     >      >      >    Other recommendations?
     >      >      >    Servers have space for another 6 disks. Maybe those
     >    could be
     >      >    used for
     >      >      >    some SSDs to speed up access?
     >      >      >
     >      >      >    TIA.
     >      >      >
     >      >      >    --
     >      >      >    Diego Zuccato
     >      >      >    DIFA - Dip. di Fisica e Astronomia
     >      >      >    Servizi Informatici
     >      >      >    Alma Mater Studiorum - Università di Bologna
     >      >      >    V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
     >      >      >    tel.: +39 051 20 95786
     >      >      >    ________
     >      >      >
     >      >      >
     >      >      >
     >      >      >    Community Meeting Calendar:
     >      >      >
     >      >      >    Schedule -
     >      >      >    Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
     >      >      >    Bridge: https://meet.google.com/cpu-eiue-hvk
    <https://meet.google.com/cpu-eiue-hvk>
     >    <https://meet.google.com/cpu-eiue-hvk
    <https://meet.google.com/cpu-eiue-hvk>>
     >      >    <https://meet.google.com/cpu-eiue-hvk
    <https://meet.google.com/cpu-eiue-hvk>
     >    <https://meet.google.com/cpu-eiue-hvk
    <https://meet.google.com/cpu-eiue-hvk>>>
     >      >      >    <https://meet.google.com/cpu-eiue-hvk
    <https://meet.google.com/cpu-eiue-hvk>
     >    <https://meet.google.com/cpu-eiue-hvk
    <https://meet.google.com/cpu-eiue-hvk>>
     >      >    <https://meet.google.com/cpu-eiue-hvk
    <https://meet.google.com/cpu-eiue-hvk>
     >    <https://meet.google.com/cpu-eiue-hvk
    <https://meet.google.com/cpu-eiue-hvk>>>>
     >      >      >    Gluster-users mailing list
     >      >      > Gluster-users@xxxxxxxxxxx
    <mailto:Gluster-users@xxxxxxxxxxx>
     >    <mailto:Gluster-users@xxxxxxxxxxx>
    <mailto:Gluster-users@xxxxxxxxxxx>
     >      >    <mailto:Gluster-users@xxxxxxxxxxx>
     >      >      >
    https://lists.gluster.org/mailman/listinfo/gluster-users
    <https://lists.gluster.org/mailman/listinfo/gluster-users>
     >    <https://lists.gluster.org/mailman/listinfo/gluster-users
    <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      > <https://lists.gluster.org/mailman/listinfo/gluster-users
    <https://lists.gluster.org/mailman/listinfo/gluster-users>
     >    <https://lists.gluster.org/mailman/listinfo/gluster-users
    <https://lists.gluster.org/mailman/listinfo/gluster-users>>>
     >      >      >
     >    <https://lists.gluster.org/mailman/listinfo/gluster-users
    <https://lists.gluster.org/mailman/listinfo/gluster-users>
     >    <https://lists.gluster.org/mailman/listinfo/gluster-users
    <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      > <https://lists.gluster.org/mailman/listinfo/gluster-users
    <https://lists.gluster.org/mailman/listinfo/gluster-users>
     >    <https://lists.gluster.org/mailman/listinfo/gluster-users
    <https://lists.gluster.org/mailman/listinfo/gluster-users>>>>

     >
     >      >
     >      >      >
     >      >
     >      >    --
     >      >    Diego Zuccato
     >      >    DIFA - Dip. di Fisica e Astronomia
     >      >    Servizi Informatici
     >      >    Alma Mater Studiorum - Università di Bologna
     >      >    V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
     >      >    tel.: +39 051 20 95786
     >      >    ________
     >      >
     >      >
     >      >
     >      >    Community Meeting Calendar:
     >      >
     >      >    Schedule -
     >      >    Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
     >      >    Bridge: https://meet.google.com/cpu-eiue-hvk
    <https://meet.google.com/cpu-eiue-hvk>
     >    <https://meet.google.com/cpu-eiue-hvk
    <https://meet.google.com/cpu-eiue-hvk>>
     >      >    <https://meet.google.com/cpu-eiue-hvk
    <https://meet.google.com/cpu-eiue-hvk>
     >    <https://meet.google.com/cpu-eiue-hvk
    <https://meet.google.com/cpu-eiue-hvk>>>
     >      >    Gluster-users mailing list
     >      > Gluster-users@xxxxxxxxxxx
    <mailto:Gluster-users@xxxxxxxxxxx> <mailto:Gluster-users@xxxxxxxxxxx>
     >    <mailto:Gluster-users@xxxxxxxxxxx>
     >      > https://lists.gluster.org/mailman/listinfo/gluster-users
    <https://lists.gluster.org/mailman/listinfo/gluster-users>
     >    <https://lists.gluster.org/mailman/listinfo/gluster-users
    <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      > <https://lists.gluster.org/mailman/listinfo/gluster-users
    <https://lists.gluster.org/mailman/listinfo/gluster-users>
     >    <https://lists.gluster.org/mailman/listinfo/gluster-users
    <https://lists.gluster.org/mailman/listinfo/gluster-users>>>
     >      >
     >
     >    --
     >    Diego Zuccato
     >    DIFA - Dip. di Fisica e Astronomia
     >    Servizi Informatici
     >    Alma Mater Studiorum - Università di Bologna
     >    V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
     >    tel.: +39 051 20 95786
     >

-- Diego Zuccato
    DIFA - Dip. di Fisica e Astronomia
    Servizi Informatici
    Alma Mater Studiorum - Università di Bologna
    V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
    tel.: +39 051 20 95786


--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users




[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux