> Sent: Monday, March 12, 2018 at 4:55 PM > From: "Amir Goldstein" <amir73il@xxxxxxxxx> > To: "Chengguang Xu" <cgxu519@xxxxxxx> > Cc: "zhangyi (F)" <yi.zhang@xxxxxxxxxx>, overlayfs <linux-unionfs@xxxxxxxxxxxxxxx>, "Miklos Szeredi" <miklos@xxxxxxxxxx> > Subject: Re: "quota-df" feature > > On Mon, Mar 12, 2018 at 9:36 AM, Chengguang Xu <cgxu519@xxxxxxx> wrote: > >> Sent: Monday, March 12, 2018 at 2:50 PM > >> From: "zhangyi (F)" <yi.zhang@xxxxxxxxxx> > >> To: "Chengguang Xu" <cgxu519@xxxxxxx>, "Amir Goldstein" <amir73il@xxxxxxxxx> > >> Cc: overlayfs <linux-unionfs@xxxxxxxxxxxxxxx>, "Miklos Szeredi" <miklos@xxxxxxxxxx> > >> Subject: Re: "quota-df" feature > >> > >> On 2018/3/11 11:16, Chengguang Xu Wrote: > >> >> Sent: Sunday, March 11, 2018 at 4:03 AM > >> >> From: "Amir Goldstein" <amir73il@xxxxxxxxx> > >> >> To: "Chengguang Xu" <cgxu519@xxxxxxx> > >> >> Cc: overlayfs <linux-unionfs@xxxxxxxxxxxxxxx>, "Miklos Szeredi" <miklos@xxxxxxxxxx> > >> >> Subject: Re: "quota-df" feature > >> >> > >> >> On Sat, Mar 10, 2018 at 8:37 AM, Chengguang Xu <cgxu519@xxxxxxx> wrote: > >> >>>> Sent: Thursday, March 08, 2018 at 10:36 PM > >> >>>> From: "Amir Goldstein" <amir73il@xxxxxxxxx> > >> >>>> To: "Chengguang Xu" <cgxu519@xxxxxxx> > >> >>>> Cc: overlayfs <linux-unionfs@xxxxxxxxxxxxxxx>, "Miklos Szeredi" <miklos@xxxxxxxxxx> > >> >>>> Subject: Re: "quota-df" feature > >> >>>> > >> >>>> On Thu, Mar 8, 2018 at 3:10 PM, Chengguang Xu <cgxu519@xxxxxxx> wrote: > >> >>>> [...] > >> >>>>>>>>> Lowerdirs could be share with different overlayfs, so I'm not sure counting > >> >>>>>>>>> the contents of lowerdirs is proper behavior or not. If lowerdirs are dedicated > >> >>>>>>>>> to one overlayfs then maybe setting same project id with upperdir can resolve > >> >>>>>>>>> "covered" problem that you mentioned above. Counting usage information without > >> >>>>>>>>> quota might be a hard work when having plenty of files. > >> >>>>>>>>> > >> >>>>>>>> > >> >>>>>>>> Well, its a different use case, but a very well known use case - > >> >>>>>>>> When dealing with cloned files (e.g. btrfs, xfs) many files can share the same > >> >>>>>>>> blocks, but each clone is fully accounted to the user/group/project. > >> >>>>>>>> This is the "thin provisioning" use case - every user gets accounted by files > >> >>>>>>>> that the user can reference, but the host does not pay the cost of sum of all > >> >>>>>>>> user quotas. > >> >>>>>>>> > >> >>>>>>>> Without accounting of "covered" files to begin with, is it possible to > >> >>>>>>>> get to a state > >> >>>>>>>> where 'touch' on a big file gets ENOSPC/EQUOTA. This is indeed a situation that > >> >>>>>>>> can happen in "thin provisioned" filesystems (e.g. btrfs) or on thin > >> >>>>>>>> provisioned block, > >> >>>>>>>> but a situation that filesystems and administrators try really hard to avoid. > >> >>>>>>> > >> >>>>>>> Let me make clear about the term of "covered", is it meaning hidden file in lowerdir > >> >>>>>>> because of same named file exists in upperdir? or is it meaning the contents in > >> >>>>>>> lowerdir but in the merged dir of overlayfs? > >> >>>>>>> > >> >>>>>>> > >> >>>>>> > >> >>>>>> The former. "covered" file size does not show up in du -s, so the > >> >>>>>> merged disk usage > >> >>>>>> is <upper used> + <lower used> - <covered used> > >> >>>>> > >> >>>>> Thanks, got it. > >> >>>>> But IIUC, "covered" file does not have chance to copy-up, so > >> >>>>> I'm wondering is it the real reason for getting ENOSPC/EQUOTA error? > >> >>>> > >> >>>> Suppose your "image" (i.e. total disk usage of lower) is 1GB > >> >>>> and you want to allow user to touch all the files in the image and > >> >>>> create 1GB of new files. > >> >>>> > >> >>>> If your only tool is project quota on upper then you need to set project > >> >>>> quota hard limit to 2GB, but then user can create 2GB of new files and > >> >>>> later when touching a lower file, will get EQUOTA on copy up. > >> >>>> > >> >>>> If you account lower uncovered files to overlay merged quota then > >> >>>> you set the merged quota to 2GB and start with 50% used. > >> >>>> - Copy up will not change used > >> >>>> - Remove of lower will reduce used > >> >>>> - You can never get EQUOTA from touching a file > >> >>> > >> >>> I get your point, I'd like to think it as "reservation" feature we can implement > >> >>> in overlay for uncovered files, so that we can get rid of EQUOTA error during copy-up. > >> >>> Even might get rid of ENOSPC error during copy-up when underlying filesystem supports > >> >>> block reservation? Are there many people hope to have this kind of function? > >> >>> > >> >> > >> >> Not sure. I heard this requirement once, but I have a feeling people > >> >> still don't know > >> >> what they can get out of overlayfs, so they are used to getting along > >> >> with what they > >> >> have. I think now that docker supports project quotas with overlay2 > >> >> driver, people > >> >> will start using project quota and sooner or later, they will start > >> >> hitting EQUOTA on > >> >> copy up. That's my prophecy anyway ;-) > >> > > >> > I believe in the near future, most container users will experience this. > >> > Indoe will be worse compare to space, not only fail on creating new file > >> > but also on deleting old uncovered files because of plenty whiteout files. > >> > I remembered someone has already posted fix patch to reduce whiteout files, > >> > so after that merged, we can consider block and inode in the same way. > >> > >> And copy up directory with nfs_export=on, which will cost one more inode in > >> work/index dir in upper layer. > >> > >> > > >> > Hi Zhang yi, > >> > > >> > Is that possible to count space and inode of "covered"/"uncovered"/"upper" > >> > via userspace command tool? I think those info will be useful/important reference > >> > to admin to set reservation when mounting overlayfs. Then maybe I can add > >> > some reservation logics in kernel side to avoid EQUOTA error as much as possible. > >> > > >> > > >> Hi Chengguang, > >> > >> I think it is not hard to count thses 3 kinds of space and inode in userspace tool, > >> just figure out the already copied-up target in lower is enough. But I have some > >> questions about the "reservation" feature: > >> > >> 1. If we impliment this tool, does it means that the user should run this tool every > >> time before mount overlayfs? and how to pass this info to overlayfs? > > > > I propose adding a new mount option to pass this info to overlayfs from userspace, > > so it means user/admin can decide how many spaces/inodes they can reserve. Although, > > user can set the value of reservation without any information of overlayfs, but I guess > > they will be happy/grateful if there a tool for checking that even if they don't use > > 'reservation' function. > > > > I think it would be better counting those numbers before mounting, if we do the same > > thing during mounting or inside filesystem, users will think weird by taking much time > > on mounting because mount should be finish in a short period in a common sense. > > > > Before proposing an overlay specific interface for reservation via > mount options, we > should try to use quotatools existing APIs and common quota practices > used by other > file systems. If we do propose to extend the quota APIs for reservation (I'm not > convinced at all that we do) then we should start a thread on fsdevel > to consider other > file systems needs for space reservation (e.g. tune2fs -r). IMO, I don't think reservation should be a part of common quota function or at least not in current stage. Why? I've not seen good benefit by introducing it for common block based filesystems, current quota has already have soft and hard limits for flexibly handling temporary burst. Reservation might be important only for overlayfs in quota environment, so I think I don't have enough reason to persuade people to accept adding new interface which they are never used. For space reservation, if underlying filesystems support then overlayfs can try to utilize the function, though I'm not sure if underlying filesystem allows to do. Thanks, Chengguang. -- To unsubscribe from this list: send the line "unsubscribe linux-unionfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
