Re: Quota list not reflecting disk usage

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Here's a l link to a tarball of one of the gluster hosts logs:
https://dl.dropboxusercontent.com/u/21916057/gluster01.tar.gz

I wanted to include past logs in case they were useful.

Also, the volume I'm trying to get quota's working on is 'storage'
you'll notice I have a brick issue on a different volume 'vm-storage'.

In regards to the 3.7 upgrade. I'm a bit hesitant to move to the
current release, I prefer to stay on a stable release with maintenance
updates if possible.

On Mon, Jan 25, 2016 at 12:09 PM, Manikandan Selvaganesh
<mselvaga@xxxxxxxxxx> wrote:
> Hi Steve,
>
> Also, do you have any plans to upgrade to the latest version. With 3.7,
> we have re factored some approaches used in quota and marker and that have
> fixed quite some issues.
>
> --
> Thanks & Regards,
> Manikandan Selvaganesh.
>
> ----- Original Message -----
> From: "Manikandan Selvaganesh" <mselvaga@xxxxxxxxxx>
> To: "Steve Dainard" <sdainard@xxxxxxxx>
> Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>
> Sent: Tuesday, January 26, 2016 1:31:10 AM
> Subject: Re:  Quota list not reflecting disk usage
>
> Hi Steve,
>
> Could you send us the glusterfs logs, it could help us debug the issue!!
>
> --
> Thanks & Regards,
> Manikandan Selvaganesh.
>
> ----- Original Message -----
> From: "Steve Dainard" <sdainard@xxxxxxxx>
> To: "Manikandan Selvaganesh" <mselvaga@xxxxxxxxxx>
> Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>
> Sent: Tuesday, January 26, 2016 12:56:22 AM
> Subject: Re:  Quota list not reflecting disk usage
>
> Something is seriously wrong with the quota output:
>
> # gluster volume quota storage list
>                   Path                   Hard-limit Soft-limit   Used
> Available  Soft-limit exceeded? Hard-limit exceeded?
> ---------------------------------------------------------------------------------------------------------------------------
> /projects-CanSISE                         10.0TB       80%      27.8TB
>  0Bytes             Yes                  Yes
> /data4/climate                           105.0TB       80%     307.1TB
>  0Bytes             Yes                  Yes
> /data4/forestry                           50.0GB       80%      61.9GB
>  0Bytes             Yes                  Yes
> /data4/projects                          800.0GB       80%       2.0TB
>  0Bytes             Yes                  Yes
> /data4/strays                             85.0GB       80%     230.5GB
>  0Bytes             Yes                  Yes
> /data4/gis                                 2.2TB       80%       6.3TB
>  0Bytes             Yes                  Yes
> /data4/modperl                             1.0TB       80%     953.2GB
>  70.8GB             Yes                   No
> /data4/dem                                 1.0GB       80%      0Bytes
>   1.0GB              No                   No
> /projects-hydrology-archive0               5.0TB       80%      14.4TB
>  0Bytes             Yes                  Yes
> /climate-downscale-idf-ec                  7.5TB       80%       5.1TB
>   2.4TB              No                   No
> /climate-downscale-idf                     5.0TB       80%       6.1TB
>  0Bytes             Yes                  Yes
> /home                                      5.0TB       80%      11.8TB
>  0Bytes             Yes                  Yes
> /projects-hydrology-scratch0               7.0TB       80%     169.1GB
>   6.8TB              No                   No
> /projects-rci-scratch                     10.0TB       80%       1.9TB
>   8.1TB              No                   No
> /projects-dataportal                       1.0TB       80%     775.4GB
> 248.6GB              No                   No
> /modules                                   1.0TB       80%      36.1GB
> 987.9GB              No                   No
> /data4/climate/downscale/CMIP5            65.0TB       80%      56.4TB
>   8.6TB             Yes                   No
>
> Gluster is listing 'Used' space of over 307TB on /data4/climate, but
> the volume capacity is only 146T.
>
> This has happened after disabling quotas on the volume, re-enabling
> quotas, and then setting quotas again. There was a lot of glusterfsd
> CPU usage afterwards, and now 3 days later the quota's I set were all
> missing except
>
> /data4/projects|800.0GB|2.0TB|0Bytes
>
> So I re-set the quotas and the output above is what I have.
>
> Previous to disabling quota's this was the output:
> # gluster volume quota storage list
>                   Path                   Hard-limit Soft-limit   Used
> Available  Soft-limit exceeded? Hard-limit exceeded?
> ---------------------------------------------------------------------------------------------------------------------------
> /data4/climate                           105.0TB       80%     151.6TB
>  0Bytes             Yes                  Yes
> /data4/forestry                           50.0GB       80%      45.4GB
>   4.6GB             Yes                   No
> /data4/projects                          800.0GB       80%     753.1GB
>  46.9GB             Yes                   No
> /data4/strays                             85.0GB       80%      80.8GB
>   4.2GB             Yes                   No
> /data4/gis                                 2.2TB       80%       2.1TB
>  91.8GB             Yes                   No
> /data4/modperl                             1.0TB       80%     948.1GB
>  75.9GB             Yes                   No
> /data4/dem                                 1.0GB       80%      0Bytes
>   1.0GB              No                   No
> /projects-CanSISE                         10.0TB       80%      11.9TB
>  0Bytes             Yes                  Yes
> /projects-hydrology-archive0               5.0TB       80%       4.8TB
> 174.0GB             Yes                   No
> /climate-downscale-idf-ec                  7.5TB       80%       5.0TB
>   2.5TB              No                   No
> /climate-downscale-idf                     5.0TB       80%       3.8TB
>   1.2TB              No                   No
> /home                                      5.0TB       80%       4.7TB
> 283.8GB             Yes                   No
> /projects-hydrology-scratch0               7.0TB       80%      95.9GB
>   6.9TB              No                   No
> /projects-rci-scratch                     10.0TB       80%       1.7TB
>   8.3TB              No                   No
> /projects-dataportal                       1.0TB       80%     775.4GB
> 248.6GB              No                   No
> /modules                                   1.0TB       80%      14.6GB
> 1009.4GB              No                   No
> /data4/climate/downscale/CMIP5            65.0TB       80%      56.4TB
>   8.6TB             Yes                   No
>
> I was so focused on the /projects-CanSISE quota not being accurate
> that I missed that the 'Used' space on /data4/climate is listed higher
> then the total gluster volume capacity.
>
> On Mon, Jan 25, 2016 at 10:52 AM, Steve Dainard <sdainard@xxxxxxxx> wrote:
>> Hi Manikandan
>>
>> I'm using 'du' not df in this case.
>>
>> On Thu, Jan 21, 2016 at 9:20 PM, Manikandan Selvaganesh
>> <mselvaga@xxxxxxxxxx> wrote:
>>> Hi Steve,
>>>
>>> If you would like disk usage using df utility by taking quota limits into
>>> consideration, then you are expected to run the following command.
>>>
>>>    'gluster volume set VOLNAME quota-deem-statfs on'
>>>
>>> with older versions where quota-deem-statfs is OFF by default. However with
>>> the latest versions, quota-deem-statfs is by default ON. In this case, the total
>>> disk space of the directory is taken as the quota hard limit set on the directory
>>> of the volume and disk utility would display accordingly. This answers why there is
>>> a mismatch in disk utility.
>>>
>>> Next, answering to quota mechanism and accuracy: There is something called timeouts
>>> in quota. For performance reasons, quota caches the directory size on client. You can
>>> set timeout indicating the maximum valid duration of directory sizes in cache,
>>> from the time they are populated. By default the hard-timeout is 5s and soft timeout
>>> is 60s. Setting a timeout of zero will do a force fetching of directory sizes from server
>>> for every operation that modifies file data and will effectively disables directory size
>>> caching on client side. If you do not have a timeout of 0(which we do not encourage due to
>>> performance reasons), then till you reach soft-limit, soft timeout will be taken into
>>> consideration, and only for every 60s operations will be synced and that could cause the
>>> usage to exceed more than the hard-limit specified. If you would like quota to
>>> strictly enforce then please run the following commands,
>>>
>>>     'gluster v quota VOLNAME hard-timeout 0s'
>>>     'gluster v quota VOLNAME soft-timeout 0s'
>>>
>>> Appreciate your curiosity in exploring and if you would like to know more about quota
>>> please refer[1]
>>>
>>> [1] http://gluster.readthedocs.org/en/release-3.7.0-1/Administrator%20Guide/Directory%20Quota/
>>>
>>> --
>>> Thanks & Regards,
>>> Manikandan Selvaganesh.
>>>
>>> ----- Original Message -----
>>> From: "Steve Dainard" <sdainard@xxxxxxxx>
>>> To: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>
>>> Sent: Friday, January 22, 2016 1:40:07 AM
>>> Subject: Re:  Quota list not reflecting disk usage
>>>
>>> This is gluster 3.6.6.
>>>
>>> I've attempted to disable and re-enable quota's on the volume, but
>>> when I re-apply the quotas on each directory the same 'Used' value is
>>> present as before.
>>>
>>> Where is quotad getting its information from, and how can I clean
>>> up/regenerate that info?
>>>
>>> On Thu, Jan 21, 2016 at 10:07 AM, Steve Dainard <sdainard@xxxxxxxx> wrote:
>>>> I have a distributed volume with quota's enabled:
>>>>
>>>> Volume Name: storage
>>>> Type: Distribute
>>>> Volume ID: 26d355cb-c486-481f-ac16-e25390e73775
>>>> Status: Started
>>>> Number of Bricks: 4
>>>> Transport-type: tcp
>>>> Bricks:
>>>> Brick1: 10.0.231.50:/mnt/raid6-storage/storage
>>>> Brick2: 10.0.231.51:/mnt/raid6-storage/storage
>>>> Brick3: 10.0.231.52:/mnt/raid6-storage/storage
>>>> Brick4: 10.0.231.53:/mnt/raid6-storage/storage
>>>> Options Reconfigured:
>>>> performance.cache-size: 1GB
>>>> performance.readdir-ahead: on
>>>> features.quota: on
>>>> diagnostics.brick-log-level: WARNING
>>>>
>>>> Here is a partial list of quotas:
>>>> # /usr/sbin/gluster volume quota storage list
>>>>                   Path                   Hard-limit Soft-limit   Used
>>>> Available  Soft-limit exceeded? Hard-limit exceeded?
>>>> ---------------------------------------------------------------------------------------------------------------------------
>>>> ...
>>>> /projects-CanSISE                         10.0TB       80%      11.9TB
>>>>  0Bytes             Yes                  Yes
>>>> ...
>>>>
>>>> If I du on that location I do not get 11.9TB of space used (fuse mount point):
>>>> [root@storage projects-CanSISE]# du -hs
>>>> 9.5T .
>>>>
>>>> Can someone provide an explanation for how the quota mechanism tracks
>>>> disk usage? How often does the quota mechanism check its accuracy? And
>>>> how could it get so far off?
>>>>
>>>> Can I get gluster to rescan that location and update the quota usage?
>>>>
>>>> Thanks,
>>>> Steve
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users@xxxxxxxxxxx
>>> http://www.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users



[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux