Re: bitd.log and quotad.log flooding /var

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Have you decreased glusterd log level via:
glusterd --log-level WARNING|ERROR

It seems that bitrot doesn't have it's own log level.

As a workaround, you can configure syslog to send the logs only remotely and thus preventing the overfill of the /var .


Best Regards,
Strahil Nikolov 

On Wed, Aug 10, 2022 at 7:52, Diego Zuccato
<diego.zuccato@xxxxxxxx> wrote:
Hi Strahil.

Sure. Luckily I didn't delete 'em all :)

From bitd.log:
-8<--
[2022-08-09 05:58:12.075999 +0000] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/astro/...omisis.../file.dat},
{gfid=5956af24-5efc-496c-8d7e-ea6656f298de}, {Brick-path=/srv/bricks/10/d}]
[2022-08-09 05:58:12.082264 +0000] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/astro/...omisis.../file.txt},
{gfid=afb75c03-0d29-414e-917a-ff718982c849}, {Brick-path=/srv/bricks/13/d}]
[2022-08-09 05:58:12.082267 +0000] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/astro/...omisis.../file.dat},
{gfid=982bc7a8-d4ba-45d7-9104-044e5d446802}, {Brick-path=/srv/bricks/06/d}]
[2022-08-09 05:58:12.084960 +0000] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/atmos/...omisis.../file},
{gfid=17e4dfb0-1f64-47a3-9aa8-b3fa05b7cd4e}, {Brick-path=/srv/bricks/15/d}]
[2022-08-09 05:58:12.089357 +0000] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/astro/...omisis.../file.txt},
{gfid=e70bf289-5aeb-43c2-aadd-d18979cf62b5}, {Brick-path=/srv/bricks/00/d}]
[2022-08-09 05:58:12.094440 +0000] I [MSGID: 100011]
[glusterfsd.c:1511:reincarnate] 0-glusterfsd: Fetching the volume file
from server... []
[2022-08-09 05:58:12.096299 +0000] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.096653 +0000] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 05:58:12.096853 +0000] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09 05:58:12.096702 +0000] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 05:58:12.102176 +0000] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/astro/...omisis.../file.dat},
{gfid=45f59e3f-eef4-4ccf-baac-bc8bf10c5ced}, {Brick-path=/srv/bricks/09/d}]
[2022-08-09 05:58:12.106120 +0000] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/astro/...omisis.../file.txt},
{gfid=216832dd-0a1c-4593-8a9e-f54d70efc637}, {Brick-path=/srv/bricks/13/d}]
-8<--

And from quotad.log:
-<--
[2022-08-09 05:58:12.291030 +0000] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.291143 +0000] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 05:58:12.291653 +0000] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09 05:58:12.292990 +0000] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.293204 +0000] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.293500 +0000] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09 05:58:12.293663 +0000] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
The message "I [MSGID: 100011] [glusterfsd.c:1511:reincarnate]
0-glusterfsd: Fetching the volume file from server... []" repeated 2
times between [2022-08-09 05:58:12.094470 +0000] and [2022-08-09
05:58:12.291149 +0000]
The message "I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]"
repeated 5 times between [2022-08-09 05:58:12.291143 +0000] and
[2022-08-09 05:58:12.293234 +0000]
[2022-08-09 06:00:23.180856 +0000] I
[glusterfsd-mgmt.c:77:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2022-08-09 06:00:23.324981 +0000] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 06:00:23.325025 +0000] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 06:00:23.325498 +0000] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09 06:00:23.325046 +0000] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 22:00:07.364719 +0000] I [MSGID: 100011]
[glusterfsd.c:1511:reincarnate] 0-glusterfsd: Fetching the volume file
from server... []
[2022-08-09 22:00:07.374040 +0000] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 22:00:07.374099 +0000] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 22:00:07.374569 +0000] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09 22:00:07.385610 +0000] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 22:00:07.386119 +0000] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
-8<--

I've now used
  gluster v set cluster_data diagnostics.brick-sys-log-level CRITICAL
and rate of filling decreased, but I still see many 'I' lines :(

Using Gluster 9.5 packages from
deb [arch=amd64]
https://download.gluster.org/pub/gluster/glusterfs/9/LATEST/Debian/bullseye/amd64/apt
bullseye main

Tks,
  Diego

Il 09/08/2022 22:08, Strahil Nikolov ha scritto:
> Hey Diego,
>
> can you show a sample of such Info entries ?
>
> Best Regards,
> Strahil Nikolov
>
>    On Mon, Aug 8, 2022 at 15:59, Diego Zuccato
>    <diego.zuccato@xxxxxxxx> wrote:
>    Hello all.
>
>    Lately, I noticed some hickups in our Gluster volume. It's a "replica 3
>    arbiter 1" with many bricks (currently 90 data bricks over 3 servers).
>
>    I tried to reduce log level by setting
>    diagnostics.brick-log-level: ERROR
>    diagnostics.client-log-level: ERROR
>    and creating /etc/default/glusterd containing "LOG_LEVEL=ERROR".
>    But I still see a lot of 'I' lines in the logs and have to manually run
>    logrotate way too often or /var gets too full.
>
>    Any hints? What did I forget?
>
>    Tks.
>
>    --
>    Diego Zuccato
>    DIFA - Dip. di Fisica e Astronomia
>    Servizi Informatici
>    Alma Mater Studiorum - Università di Bologna
>    V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
>    tel.: +39 051 20 95786
>    ________
>
>
>
>    Community Meeting Calendar:
>
>    Schedule -
>    Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>    Bridge: https://meet.google.com/cpu-eiue-hvk
>    <https://meet.google.com/cpu-eiue-hvk>
>    Gluster-users mailing list
>    Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx>
>    https://lists.gluster.org/mailman/listinfo/gluster-users
>    <https://lists.gluster.org/mailman/listinfo/gluster-users>

>

--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux