Hi All,
Considering we are coming out with major release plan, we would like to revisit the QUOTA feature to decide its path forward.
I have been working on quota feature for a couple of months now and have come across various issues from performance, usability and correctness perspective.
We do have some initial thoughts on how these can be solved. But, to ensure we work on the most important things first, we would like to get a pulse of this from the users of quota feature.
Below is a questionnaire for the same. In order to put the questions in perspective, I have provided rationale, external links to alternative design thoughts.
The focus of this mail thread though is to get a user pulse than being a design review. Please comment on design in the github issue itself.
We can bring the discussion to gluster-devel as they gain traction. We would like the design discussion to be driven by the generated user feedback.
1) On how many directories do u generally have quota limits configured.
How often do we see quota limits added/removed/changed. [numbers would help here than qualitative answers]
Any use case with large number of quota limits (> 100 on a single volume say)?
Is a filesystem Crawl acceptable each time a new quota limit is set?
(crawl may take time equivalent to du command, this would essentially introduce a delay for a limit to take effect after it is set)
Rationale:
Currently, we account the usage of all directories. A performance issue with this approach is we need to recursively account the usage of a file/directory on all its ancestors along the path.
If we consider few directories have limits configured, we could explore alternatives where accounting information is maintained only in directories that have limits set [RFC1].
2) How strict accounting is generally acceptable. Is it acceptable if there is an overshoot by 100MB say?
What are the general value of limits configured. Does anybody set limits in the order of MBs ?
Rationale:
Relaxing the accounting could be another way to gain in performance. We can batch / cache xattr updates. [RFC2]
3) Does directory quota suit your needs? Would you prefer if it works like XFS directory quota?
What are the back-end Filesystem you expect to use quota with?
Is it acceptable to take the route of leveraging backend FS quota with support for limited FS (or just xfs)?
Rationale:
Behavior of directory quota in GlusterFS largely differs from the XFS way. both has its pros and cons.
GlusterFS will allow you to set a limit on /home and /home/user1. So if you write to /home/user1/file1 the limit for both its ancestors are checked and honored.
An admin who has to give storage to 50 users can configure /home to 1 TB and each user home to 25GB say (Hoping that not all users would simultaneously use up their 25 GB). This may not make sense for a cloud storage but it does make sense in a university lab kind of setting.
The same cannot be done with XFS beacuse XFS directory quota relies on project id. Only one project id is associated with a directory so only 1 limit can be honored at any time.
XFS directory quota has its advantages though. Please find details in [RFC3]
4) Do you use quota with large number of bricks in a volume? (How many?)
Do you have quota with large number of volumes hosted on a node? (How many?)
Have you seen any performance issues in such setting?
Rationale:
We have a single quotad process (aggregator of accounting info) on each nodes in the Trusted storage pool.
All the bricks (even from different volumes) hosted on a node share the quotad.
Another issue in this design is large number of bricks within a volume will increase IPC latency during aggregation.
One way of mitigating this is by changing quotad and quota layer to work on a lease based approach [RFC4]
5) If you set directory level quota, do you expect to have hard link across the directories? or want to support rename() across directories?
Rationale:
Supporting these operations consistently does add complexities in design.
XFS itself doesn't support these two operations when quota is set on it.
6) Do u use inode-quota feature?
Any user looking for the user/group quota feature?
7) Are you using current quota feature?
If yes, are you happy?
if yes, and not happy? what are the things you would like to see
improve?
RFC1 - Alternative Accounting method in marker (https://github.com/gluster/glusterfs/issues/182 )
RFC2 - Batched updates (https://github.com/gluster/glusterfs/issues/183 )
RFC3 - XFS based Quota (https://github.com/gluster/glusterfs/issues/184 )
RFC4 - Lease based quotad (https://github.com/gluster/glusterfs/issues/181 )
Note: These RFC are not interdependent.
Considering we are coming out with major release plan, we would like to revisit the QUOTA feature to decide its path forward.
I have been working on quota feature for a couple of months now and have come across various issues from performance, usability and correctness perspective.
We do have some initial thoughts on how these can be solved. But, to ensure we work on the most important things first, we would like to get a pulse of this from the users of quota feature.
Below is a questionnaire for the same. In order to put the questions in perspective, I have provided rationale, external links to alternative design thoughts.
The focus of this mail thread though is to get a user pulse than being a design review. Please comment on design in the github issue itself.
We can bring the discussion to gluster-devel as they gain traction. We would like the design discussion to be driven by the generated user feedback.
1) On how many directories do u generally have quota limits configured.
How often do we see quota limits added/removed/changed. [numbers would help here than qualitative answers]
Any use case with large number of quota limits (> 100 on a single volume say)?
Is a filesystem Crawl acceptable each time a new quota limit is set?
(crawl may take time equivalent to du command, this would essentially introduce a delay for a limit to take effect after it is set)
Rationale:
Currently, we account the usage of all directories. A performance issue with this approach is we need to recursively account the usage of a file/directory on all its ancestors along the path.
If we consider few directories have limits configured, we could explore alternatives where accounting information is maintained only in directories that have limits set [RFC1].
2) How strict accounting is generally acceptable. Is it acceptable if there is an overshoot by 100MB say?
What are the general value of limits configured. Does anybody set limits in the order of MBs ?
Rationale:
Relaxing the accounting could be another way to gain in performance. We can batch / cache xattr updates. [RFC2]
3) Does directory quota suit your needs? Would you prefer if it works like XFS directory quota?
What are the back-end Filesystem you expect to use quota with?
Is it acceptable to take the route of leveraging backend FS quota with support for limited FS (or just xfs)?
Rationale:
Behavior of directory quota in GlusterFS largely differs from the XFS way. both has its pros and cons.
GlusterFS will allow you to set a limit on /home and /home/user1. So if you write to /home/user1/file1 the limit for both its ancestors are checked and honored.
An admin who has to give storage to 50 users can configure /home to 1 TB and each user home to 25GB say (Hoping that not all users would simultaneously use up their 25 GB). This may not make sense for a cloud storage but it does make sense in a university lab kind of setting.
The same cannot be done with XFS beacuse XFS directory quota relies on project id. Only one project id is associated with a directory so only 1 limit can be honored at any time.
XFS directory quota has its advantages though. Please find details in [RFC3]
4) Do you use quota with large number of bricks in a volume? (How many?)
Do you have quota with large number of volumes hosted on a node? (How many?)
Have you seen any performance issues in such setting?
Rationale:
We have a single quotad process (aggregator of accounting info) on each nodes in the Trusted storage pool.
All the bricks (even from different volumes) hosted on a node share the quotad.
Another issue in this design is large number of bricks within a volume will increase IPC latency during aggregation.
One way of mitigating this is by changing quotad and quota layer to work on a lease based approach [RFC4]
5) If you set directory level quota, do you expect to have hard link across the directories? or want to support rename() across directories?
Rationale:
Supporting these operations consistently does add complexities in design.
XFS itself doesn't support these two operations when quota is set on it.
6) Do u use inode-quota feature?
Any user looking for the user/group quota feature?
7) Are you using current quota feature?
If yes, are you happy?
if yes, and not happy? what are the things you would like to see
improve?
RFC1 - Alternative Accounting method in marker (https://github.com/gluster/gl
RFC2 - Batched updates (https://github.com/gluster/gl
RFC3 - XFS based Quota (https://github.com/gluster/gl
RFC4 - Lease based quotad (https://github.com/gluster/gl
Note: These RFC are not interdependent.
Thanks and Regards,
Sanoj_______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-devel
