> Sent: Thursday, March 08, 2018 at 3:23 PM > From: "Amir Goldstein" <amir73il@xxxxxxxxx> > To: "Chengguang Xu" <cgxu519@xxxxxxx> > Cc: overlayfs <linux-unionfs@xxxxxxxxxxxxxxx>, "Miklos Szeredi" <miklos@xxxxxxxxxx> > Subject: Re: "quota-df" feature > > On Thu, Mar 8, 2018 at 4:21 AM, Chengguang Xu <cgxu519@xxxxxxx> wrote: > >> Sent: Wednesday, March 07, 2018 at 9:28 PM > >> From: "Amir Goldstein" <amir73il@xxxxxxxxx> > >> To: "Chengguang Xu" <cgxu519@xxxxxxx> > >> Cc: overlayfs <linux-unionfs@xxxxxxxxxxxxxxx>, "Miklos Szeredi" <miklos@xxxxxxxxxx> > >> Subject: Re: "quota-df" feature > >> > >> On Wed, Mar 7, 2018 at 2:52 PM, Chengguang Xu <cgxu519@xxxxxxx> wrote: > >> > Hello folks, > >> > > >> > Recently I'm trying to implement a feature on overlayfs for our users. > >> > I'd like to call it "quota-df" which means when calling statfs(2), > >> > override/update some fields(e.g., total/used/avail of both block/inode counts） > >> > using proper project quota information if the quota has already set to > >> > underlying directories of overlayfs. > >> > > >> > >> Do you mean project quota setting on underlying fs directories or > >> something else? > > > > Underlying fs directories, more accurately the directory include > > upperdir && workdir. > > > >> > >> > This feature is probably useful for most of container users and there is > >> > no serious side effect to the other parts. We can also provide a new > >> > mount option to enable/disable the function according to different use cases. > >> > > >> > Although I'm still working on it but I'll post the RFC patch here for review > >> > in recent days and hope to get some feedback from you. Any kind of suggestions > >> > are welcomed. > >> > > >> > >> I was asked about quota accounting for the merged content of overalyfs, > >> so quota would display a number resembling du -s on the overlay mount. > >> This means that "covered" lower files are not accounted for in the merge. > >> Is that what you aim to achieve? If not, would that also be an interesting case > >> for you to explore? > >> Basically, the index directory could be used to reconstruct the entire quota > >> information for upper and lower (and copied up) files if full indexing > >> is enabled > >> with nfs_export=on. > > > > To be honest, our use case is standard docker container with overlayfs and > > our requirement of quota-df is accurately counting usage/avail inside container. > > > > My confusion is because I thought this was working correctly today. > When I posted docker support for project quota over year ago I remeber > the tests inside container showed reasonable behavior: > https://github.com/moby/moby/pull/24771 > > So what I am missing is "what is wrong now?". Seems OK except I'm not fully satisfied the adjusting mechanism in specific fs. > > Lowerdirs could be share with different overlayfs, so I'm not sure counting > > the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated > > to one overlayfs then maybe setting same project id with upperdir can resolve > > "covered" problem that you mentioned above. Counting usage information without > > quota might be a hard work when having plenty of files. > > > > Well, its a different use case, but a very well known use case - > When dealing with cloned files (e.g. btrfs, xfs) many files can share the same > blocks, but each clone is fully accounted to the user/group/project. > This is the "thin provisioning" use case - every user gets accounted by files > that the user can reference, but the host does not pay the cost of sum of all > user quotas. > > Without accounting of "covered" files to begin with, is it possible to > get to a state > where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that > can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin > provisioned block, > but a situation that filesystems and administrators try really hard to avoid. Let me make clear about the term of "covered", is it meaning hidden file in lowerdir because of same named file exists in upperdir? or is it meaning the contents in lowerdir but in the merged dir of overlayfs? > All I am saying is that it is "not hard" (TM) to keep track of > "covered" files disk usage > and "not hard" to re-calculate "covered" files disk usage when full > indexing is enabled. > > I don't know exactly what will happen when combining index and nfs_export options, > > I need to read and understand related code later. > > > > nfs_export REQUIRES index and implies indexing of all files on copy up, not > only lower hardlinks. > > >> > >> > > >> > Here are some test examples share with you (based on ext4): > >> > > >> > 1) project quota enabled && without hard-limit > >> > > >> > $ df -h /mnt/test3 /mnt/test3/df/merged > >> > Filesystem Size Used Avail Use% Mounted on > >> > /dev/vdb3 99G 2.5G 91G 3% /mnt/test3 > >> > overlay 92G 201M 91G 1% /mnt/test3/df/merged > >> > > >> > $ df -hi /mnt/test3 /mnt/test3/df/merged > >> > Filesystem Inodes IUsed IFree IUse% Mounted on > >> > /dev/vdb3 6.3M 2.4K 6.3M 1% /mnt/test3 > >> > overlay 6.3M 8 6.3M 1% /mnt/test3/df/merged > >> > > >> > 2) project quota enabled && with hard-limit > >> > > >> > $ df -h /mnt/test3 /mnt/test3/df/merged > >> > Filesystem Size Used Avail Use% Mounted on > >> > /dev/vdb3 99G 2.5G 91G 3% /mnt/test3 > >> > overlay 1.0G 201M 824M 20% /mnt/test3/df/merged > >> > > >> > $ df -hi /mnt/test3 /mnt/test3/df/merged > >> > Filesystem Inodes IUsed IFree IUse% Mounted on > >> > /dev/vdb3 6.3M 2.4K 6.3M 1% /mnt/test3 > >> > overlay 1000 8 992 1% /mnt/test3/df/merged > >> > > >> > > >> > >> I can't follow from the example below what is the expected result and why > >> if you add the quota setup commands that could be useful. > > > > Underlying fs using ext4 and mkfs/mount with project quota option and > > all needed directories(lowerdir, upperdir, workdir, merged) are set same > > project quota. Current quota-df implementation only adjust the couting > > information when having upperdir. > > > > Really? why is lowerdir on the same project id? For containers quota > only upper/work should be on the same project id. lowerdir should belong > to a shared project or no project at all. > In current docker implementation, every overlay2 driver root dir > is assigned a different project id, but lowerdir are symlinks to another > image root dir. For the setting of docker, you are completely right. My description of tesing environment in previous email is only for simple kernel testing and explaining the condition of testing result above, not specific to docker. > > > In case 1: > > There is no hardlimit/softlimit, so the expected result as below. > > Used: The used count in project quota which set to the directory > > include upperdir && workdir. > > Avail: The avail of underlying fs > > Total: Used + Avail > > > > #upperdir used 201M and /mnt/test3 used 2.5G > > > > In case 2: > > Project quota hardlimits are block count = 1G, inode count = 1000. > > So the expected result as below. > > > > Used: The used count in project quota which set to the directory > > include upperdir && workdir. > > Avail: (a)Hardlimit - Used or (b)The avail of underlying fs(when a > b) > > Total: Used + Avail > > > > Are those "expected" results inline with what df shows on project quota > directories without overlayfs? If not, and the new behavior makes sense, > why change overlayfs and not change 'df'? No, those results are based on my change of overlayfs. At the first, I didn't notice that underlying filesystems had already implemented 'quota-df' function. So I plan to do something in overlayfs because it's better than doing same things in every low level filesystems. But now, I'm more willing to persuade xfs/ext4 people to midify the detail mechanism of 'quota-df' in specific filesystems. 'df', hmm... Both solutions could work I think. Thanks, Chengguang. -- To unsubscribe from this list: send the line "unsubscribe linux-unionfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
