Re: Tons of dht: Found anomalies in (null)

Susant Palai <spalai@xxxxxxxxxx> · Tue, 19 May 2020 12:18:05 +0530

On Tue, May 19, 2020 at 12:15 PM Aravinda VK <aravinda@xxxxxxxxx> wrote:

On 19-May-2020, at 12:05 PM, Susant Palai <spalai@xxxxxxxxxx> wrote:

On Thu, Apr 30, 2020 at 6:31 AM Artem Russakovskii <archon810@xxxxxxxxx> wrote:
Hi,

Every time I ls large dirs in our 1x4 replicate gluster volume, I get a ton of these in the logs.

If I run the same ls right away again, they won't repeat, but inevitably, in a couple of hours or days, they show up again.

What is the meaning of these messages and why am I getting them so frequently?   

[2020-04-30 00:57:39.020757] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 9f879856-36fc-49f9-900f-0e5d742b2e54). Holes=1 overlaps=0
[2020-04-30 00:57:39.021036] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = d6c7f283-3607-4a4d-9099-6f76792d5bdd). Holes=1 overlaps=0
[2020-04-30 00:57:39.021067] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = d8f88e6d-d80e-40fc-a386-9dc35212ad91). Holes=1 overlaps=0
[2020-04-30 00:57:39.021079] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 704b3e6e-9f55-4166-8a0f-4283fb513593). Holes=1 overlaps=0
[2020-04-30 00:57:39.021103] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 2de3da38-c56d-4196-b001-e4fa7d901383). Holes=1 overlaps=0
[2020-04-30 00:57:39.021118] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = c0b2c50e-1abd-4c86-9958-2c6f87d469d1). Holes=1 overlaps=0
[2020-04-30 00:57:39.021400] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 5365eb39-f120-4a1c-9920-1e375c030843). Holes=1 overlaps=0
[2020-04-30 00:57:39.021436] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = f7f46524-28d8-4c57-b244-43f953ec3656). Holes=1 overlaps=0
[2020-04-30 00:57:39.021448] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 6cfd94ad-3b82-4a6f-9f0d-03cc0c59e5f1). Holes=1 overlaps=0
[2020-04-30 00:57:39.021455] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 285061fb-f1b7-49c0-ae10-5d8d8d5cd569). Holes=1 overlaps=0
[2020-04-30 00:57:39.021463] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 38734b38-381f-4825-b65a-4292c5075faa). Holes=1 overlaps=0
[2020-04-30 00:57:39.021470] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 60c15124-18da-40e7-8776-cd78303b79ad). Holes=1 overlaps=0
[2020-04-30 00:57:39.021481] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = e26783bf-bee7-4a59-ae3e-505ada95c5f9). Holes=1 overlaps=0
[2020-04-30 00:57:39.022163] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 6f7da42f-22bd-4e96-828d-02c1c1690afc). Holes=1 overlaps=0
[2020-04-30 00:57:39.022192] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 6d17f907-9fb3-4d13-9a2f-8266b0e3bead). Holes=1 overlaps=0
[2020-04-30 00:57:39.022205] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 64f40fc3-efb3-4359-98df-ba3be29b6326). Holes=1 overlaps=0
[2020-04-30 00:57:39.022214] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = d9081c5d-40e1-4521-a4b5-78fd60d71237). Holes=1 overlaps=0
[2020-04-30 00:57:39.022735] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = b627c175-d6eb-44de-9eed-a13efea52f88). Holes=1 overlaps=0
[2020-04-30 00:57:39.022771] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 9dce8380-ca5d-4947-831c-0db1aae85e09). Holes=1 overlaps=0
[2020-04-30 00:57:39.022788] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = a9c24397-a521-4d1b-8543-42b331fa6794). Holes=1 overlaps=0

Generally, these logs indicate that there were directories missing ("hole = 1") on one of the bricks. These are harmless message and you can ignore them.

Good opportunity to improve the log messages. Can we disable this logging if this can be ignored? Is it when distribute count is 1 or these messages are always harmless?

I will think about it and will update. But my current feeling is that they are helfpful in debugging issues related to directory inconsistency. If I see no value to these messages in INFO level, will send a patch.

Thanks.

Sincerely,
Artem

--
Founder, Android Police, APK Mirror, Illogical Robot LLC
beerpla.net | @ArtemR

________

Community Meeting Calendar:

Schedule -

Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC

Bridge: https://bluejeans.com/441850968

Gluster-users mailing list

Gluster-users@xxxxxxxxxxx

https://lists.gluster.org/mailman/listinfo/gluster-users

________

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

Aravinda Vishwanathapura
https://kadalu.io

________

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users