Re: Tons of dht: Found anomalies in (null)

[Aravinda VK <aravinda@xxxxxxxxx>]

On 19-May-2020, at 12:05 PM, Susant Palai <spalai@xxxxxxxxxx> wrote:

On Thu, Apr 30, 2020 at 6:31 AM Artem Russakovskii <archon810@xxxxxxxxx> wrote:

Hi,

Every time I ls large dirs in our 1x4 replicate gluster volume, I get a ton of these in the logs.

If I run the same ls right away again, they won't repeat, but inevitably, in a couple of hours or days, they show up again.

What is the meaning of these messages and why am I getting them so frequently?   

[2020-04-30 00:57:39.020757] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 9f879856-36fc-49f9-900f-0e5d742b2e54). Holes=1 overlaps=0
[2020-04-30 00:57:39.021036] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = d6c7f283-3607-4a4d-9099-6f76792d5bdd). Holes=1 overlaps=0
[2020-04-30 00:57:39.021067] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = d8f88e6d-d80e-40fc-a386-9dc35212ad91). Holes=1 overlaps=0
[2020-04-30 00:57:39.021079] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 704b3e6e-9f55-4166-8a0f-4283fb513593). Holes=1 overlaps=0
[2020-04-30 00:57:39.021103] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 2de3da38-c56d-4196-b001-e4fa7d901383). Holes=1 overlaps=0
[2020-04-30 00:57:39.021118] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = c0b2c50e-1abd-4c86-9958-2c6f87d469d1). Holes=1 overlaps=0
[2020-04-30 00:57:39.021400] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 5365eb39-f120-4a1c-9920-1e375c030843). Holes=1 overlaps=0
[2020-04-30 00:57:39.021436] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = f7f46524-28d8-4c57-b244-43f953ec3656). Holes=1 overlaps=0
[2020-04-30 00:57:39.021448] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 6cfd94ad-3b82-4a6f-9f0d-03cc0c59e5f1). Holes=1 overlaps=0
[2020-04-30 00:57:39.021455] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 285061fb-f1b7-49c0-ae10-5d8d8d5cd569). Holes=1 overlaps=0
[2020-04-30 00:57:39.021463] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 38734b38-381f-4825-b65a-4292c5075faa). Holes=1 overlaps=0
[2020-04-30 00:57:39.021470] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 60c15124-18da-40e7-8776-cd78303b79ad). Holes=1 overlaps=0
[2020-04-30 00:57:39.021481] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = e26783bf-bee7-4a59-ae3e-505ada95c5f9). Holes=1 overlaps=0
[2020-04-30 00:57:39.022163] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 6f7da42f-22bd-4e96-828d-02c1c1690afc). Holes=1 overlaps=0
[2020-04-30 00:57:39.022192] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 6d17f907-9fb3-4d13-9a2f-8266b0e3bead). Holes=1 overlaps=0
[2020-04-30 00:57:39.022205] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 64f40fc3-efb3-4359-98df-ba3be29b6326). Holes=1 overlaps=0
[2020-04-30 00:57:39.022214] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = d9081c5d-40e1-4521-a4b5-78fd60d71237). Holes=1 overlaps=0
[2020-04-30 00:57:39.022735] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = b627c175-d6eb-44de-9eed-a13efea52f88). Holes=1 overlaps=0
[2020-04-30 00:57:39.022771] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 9dce8380-ca5d-4947-831c-0db1aae85e09). Holes=1 overlaps=0
[2020-04-30 00:57:39.022788] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = a9c24397-a521-4d1b-8543-42b331fa6794). Holes=1 overlaps=0

Generally, these logs indicate that there were directories missing ("hole = 1") on one of the bricks. These are harmless message and you can ignore them.

Good opportunity to improve the log messages. Can we disable this logging if this can be ignored? Is it when distribute count is 1 or these messages are always harmless?

 

 

 

Thanks.

Sincerely,
Artem

--
Founder, Android Police, APK Mirror, Illogical Robot LLC

beerpla.net | @ArtemR

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

Aravinda Vishwanathapura

https://kadalu.io

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users