Re: Wrong directory quota usage

João Baúto <joao.bauto@xxxxxxxxxxxxxxxxxxxxxxx> · Sun, 16 Aug 2020 00:27:56 +0100

Hi Srijan & Strahil,
I ran the quota_fsck script mentioned in Hari's blog post in all bricks and it detected a lot of size mismatch. 

The script was executed as,
python quota_fsck.py --sub-dir projectB --fix-issues /mnt/tank /tank/volume2/brick (in all nodes and bricks)
Here is a snippet from the script,

Size Mismatch    /tank/volume2/brick/projectB {'parents': {'00000000-0000-0000-0000-000000000001': {'contri_file_count': 18446744073035296610L, 'contri_size': 18446645297413872640L, 'contri_dir_count': 18446744073709527653L}}, 'version': '1', 'file_count': 18446744073035296610L, 'dirty': False, 'dir_count': 18446744073709527653L, 'size': 18446645297413872640L} 15204281691754
MARKING DIRTY: /tank/volume2/brick/projectB
stat on /mnt/tank/projectB
Files verified : 683223
Directories verified : 46823
Objects Fixed : 705230

Checking the xattr in the bricks I can see the directory in question marked as dirty,
# getfattr -d -m. -e hex /tank/volume2/brick/projectB
getfattr: Removing leading '/' from absolute path names
# file: tank/volume2/brick/projectB
trusted.gfid=0x3ca2bce0455945efa6662813ce20fc0c
trusted.glusterfs.9582685f-07fa-41fd-b9fc-ebab3a6989cf.xtime=0x5f372478000a7705
trusted.glusterfs.dht=0xe1a4060c000000003ffffffe5ffffffc
trusted.glusterfs.mdata=0x010000000000000000000000005f3724750000000013ddf679000000005ce2aff90000000007fdacb0000000005ce2aff90000000007fdacb0
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x00000ca6ccf7a80000000000000790a1000000000000b6ea
trusted.glusterfs.quota.dirty=0x3100
trusted.glusterfs.quota.limit-set.1=0x0000640000000000ffffffffffffffff
trusted.glusterfs.quota.size.1=0x00000ca6ccf7a80000000000000790a1000000000000b6ea

Now, my question is how do I trigger Gluster to recalculate the quota for this directory? Is it automatic but it takes a while? Because the quota list did change but not to a good "result".

Path                   Hard-limit  Soft-limit           Used            Available         Soft-limit exceeded?   Hard-limit exceeded?
/projectB            100.0TB    80%(80.0TB)   16383.9PB   190.1TB           No                               No

I would like to avoid a disable/enable quota in the volume as it removes the configs.

Thank you for all the help!
João Baúto
---------------
Scientific Computing and Software Platform
Champalimaud Research
Champalimaud Center for the Unknown
Av. Brasília, Doca de Pedrouços
1400-038 Lisbon, Portugal
fchampalimaud.org

Srijan Sivakumar <ssivakum@xxxxxxxxxx> escreveu no dia sábado, 15/08/2020 à(s) 11:57:
Hi João,

The quota accounting error is what we're looking at here. I think you've already looked into the blog post by Hari and are using the script to fix the accounting issue.
That should help you out in fixing this issue. 

Let me know if you face any issues while using it.

Regards,
Srijan Sivakumar

On Fri 14 Aug, 2020, 17:10 João Baúto, <joao.bauto@xxxxxxxxxxxxxxxxxxxxxxx> wrote:
Hi Strahil,
I have tried removing the quota for that specific directory and setting it again but it didn't work (maybe it has to be a quota disable and enable in the volume options). Currently testing a solution 
by Hari with the quota_fsck.py script (https://medium.com/@harigowtham/glusterfs-quota-fix-accounting-840df33fcd3a) and its detecting a lot of size mismatch in files.

Thank you,
João Baúto
---------------
Scientific Computing and Software Platform
Champalimaud Research
Champalimaud Center for the Unknown
Av. Brasília, Doca de Pedrouços
1400-038 Lisbon, Portugal
fchampalimaud.org

Strahil Nikolov <hunter86_bg@xxxxxxxxx> escreveu no dia sexta, 14/08/2020 à(s) 10:16:
Hi João,

Based on your output it seems that the quota size is different on the 2 bricks.

Have you tried to remove the quota and then recreate it ? Maybe it will be the easiest way  to fix it.

Best Regards,

Strahil Nikolov

На 14 август 2020 г. 4:35:14 GMT+03:00, "João Baúto" <joao.bauto@xxxxxxxxxxxxxxxxxxxxxxx> написа:

>Hi all,

>

>We have a 4-node distributed cluster with 2 bricks per node running

>Gluster

>7.7 + ZFS. We use directory quota to limit the space used by our

>members on

>each project. Two days ago we noticed inconsistent space used reported

>by

>Gluster in the quota list.

>

>A small snippet of gluster volume quota vol list,

>

> Path                   Hard-limit  Soft-limit          Used

>Available         Soft-limit exceeded?   Hard-limit exceeded?

>/projectA              5.0TB        80%(4.0TB)    3.1TB           1.9TB

>         No                               No

>*/projectB            100.0TB    80%(80.0TB)  16383.4PB   740.9TB

> No                               No*

>/projectC              70.0TB     80%(56.0TB)   50.0TB         20.0TB

>     No                              No

>

>The total space available in the cluster is 360TB, the quota for

>projectB

>is 100TB and, as you can see, its reporting 16383.4PB used and 740TB

>available (already decreased from 750TB).

>

>There was an issue in Gluster 3.x related to the wrong directory quota

>(

>https://lists.gluster.org/pipermail/gluster-users/2016-February/025305.html

> and

>https://lists.gluster.org/pipermail/gluster-users/2018-November/035374.html)

>but it's marked as solved (not sure if the solution still applies).

>

>*On projectB*

># getfattr -d -m . -e hex projectB

># file: projectB

>trusted.gfid=0x3ca2bce0455945efa6662813ce20fc0c

>trusted.glusterfs.9582685f-07fa-41fd-b9fc-ebab3a6989cf.xtime=0x5f35e69800098ed9

>trusted.glusterfs.dht=0xe1a4060c000000003ffffffe5ffffffc

>trusted.glusterfs.mdata=0x010000000000000000000000005f355c59000000000939079f000000005ce2aff90000000007fdacb0000000005ce2aff90000000007fdacb0

>trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x0000ab0f227a860000000000478e33acffffffffffffc112

>trusted.glusterfs.quota.dirty=0x3000

>trusted.glusterfs.quota.limit-set.1=0x0000640000000000ffffffffffffffff

>trusted.glusterfs.quota.size.1=0x0000ab0f227a860000000000478e33acffffffffffffc112

>

>*On projectA*

># getfattr -d -m . -e hex projectA

># file: projectA

>trusted.gfid=0x05b09ded19354c0eb544d22d4659582e

>trusted.glusterfs.9582685f-07fa-41fd-b9fc-ebab3a6989cf.xtime=0x5f1aeb9f00044c64

>trusted.glusterfs.dht=0xe1a4060c000000001fffffff3ffffffd

>trusted.glusterfs.mdata=0x010000000000000000000000005f1ac6a10000000018f30a4e000000005c338fab0000000017a3135a000000005b0694fb000000001584a21b

>trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x0000067de3bbe20000000000000128610000000000033498

>trusted.glusterfs.quota.dirty=0x3000

>trusted.glusterfs.quota.limit-set.1=0x0000460000000000ffffffffffffffff

>trusted.glusterfs.quota.size.1=0x0000067de3bbe20000000000000128610000000000033498

>

>Any idea on what's happening and how to fix it?

>

>Thanks!

>*João Baúto*

>---------------

>

>*Scientific Computing and Software Platform*

>Champalimaud Research

>Champalimaud Center for the Unknown

>Av. Brasília, Doca de Pedrouços

>1400-038 Lisbon, Portugal

>fchampalimaud.org <https://www.fchampalimaud.org/>

________

Community Meeting Calendar:

Schedule -

Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC

Bridge: https://bluejeans.com/441850968

Gluster-users mailing list

Gluster-users@xxxxxxxxxxx

https://lists.gluster.org/mailman/listinfo/gluster-users

________

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users