On Mon, Jan 25, 2021 at 9:54 PM Sargun Dhillon <sargun@xxxxxxxxx> wrote: > > One of the projects I'm playing with for containers is lazy-loading of layers. > We've found that less than 10% of the files on a layer actually get used, which > is an unfortunate waste. It also means in some cases downloading ~100s of MB, or > ~1s of GB of files before starting a container workload. This is unfortunate. > > It would be nice if there was a way to start a container workload, and have > it so that if it tries to access and unpopulated (not yet downloaded) part > of the filesystem block while trying to be accessed. This is trivial to do > if the "lowest" layer is FUSE, where one can just stall in userspace on > loads. Unfortunately, AFAIK, there's not a good way to swap out the FUSE > filesystem with the "real" filesystem once it's done fully populating, > and you have to pay for the full FUSE cost on each read / write. > > I've tossed around: > 1. Mutable lowerdirs and having something like this: > > layer0 --> Writeable space > layer1 --> Real XFS filesystem > layer2 --> FUSE FS > > and if there is a "miss" on layer 1, it will then look it up on > layer 2 while layer 1 is being populated. Then the FUSE FS can block. > This is neat, but it requires the FUSE FS to always be up, and incurs > a userspace bounce on every miss. > > It also means things like metadata only copies don't work. > > Does anyone have a suggestion of a mechanism to handle this? I've looked into > swapping out layers on the fly, and what it would take to add a mechanism like > userfaultfd to overlayfs, but I was wondering if anything like this was already > built, or if someone has thought it through more than me. > Hi Sargun, I believe that this is the use case that you asked me about in LSFMM, at least the lower part of layer1+layer2. Is that correct? You did not mention three layers in the use case that you described Is that because you decided that layer0 and layer1 can be combined? Technically, you can also setup a nested overlay with the lower overlay layer1+layer2 only doing the lazy loading of the remote read-only layer and the upper overlay is composed of layer0+ovl(layer1+layer2), but this nested overlay configuration has some limitations. Anyway, I have talked with Miklos about the use case that requires detaching the lowermost FUSE layer eventually and the solution that we discussed was to gradually "opaquify" directories whose entire descendant hierarchy is fully copied up at readdir time. I have prepared POC patches for this design: https://github.com/amir73il/linux/commits/ovl-xino-nofollow This was tested using the following patch to unionmount-testsuite: https://github.com/amir73il/unionmount-testsuite/commits/ovl-xino-nofollow commit 026e73c37f3993f56e76128a267e54faedf2322c Author: Amir Goldstein <amir73il@xxxxxxxxx> Date: Mon May 29 17:01:55 2023 +0300 Test detaching lower fs Test that with xino=nofollow, after copying up all files and listing all the directories in DFS order, the lower fs can be detached. Signed-off-by: Amir Goldstein <amir73il@xxxxxxxxx> diff --git a/mount_union.py b/mount_union.py index e905b83..4fad5dd 100644 --- a/mount_union.py +++ b/mount_union.py @@ -54,3 +54,13 @@ def mount_union(ctx): ctx.note_upper_fs(upper_mntroot, testdir, union_mntroot + "/f") ctx.note_lower_layers(lower_mntroot) ctx.note_upper_layer(upperdir) + if cfg.is_xino(): + # Copy up everything, set all dirs opaque and then detach lower fs. + # Instead of iterating in DFS order we iterate 4 times as the depth + # of the dataset tree - on every iteration, level 4-i becomes opaque. + system("chown -R 0.0 " + union_mntroot) + system("find " + union_mntroot + " -inum 0") + system("find " + union_mntroot + " -inum 0") + system("find " + union_mntroot + " -inum 0") + system("find " + union_mntroot + " -inum 0") + system("xfs_io -x -c shutdown " + lower_mntroot) diff --git a/run b/run index 3a6efc3..f8116c1 100755 --- a/run +++ b/run @@ -219,7 +219,7 @@ if redirect_dir is False: # Auto-upgrade xino=auto to xino=on for kernel < v5.7 if xino: - cfg.add_mntopt("xino=on") + cfg.add_mntopt("xino=nofollow") -- It should be pretty self-explanatory - after mounting the overlay, all lower files are copied up using chown -R (no metacopy) and then the overlay is iterated several times, until all the merge directories iterations notice that there is nothing interesting in the lower dirs, so they all become opaque. At this point, the lowest xfs layer is being shutdown and the tests are run. With the 4*find iterations, none of the tests get EIO. This does not mean that the lower xfs can be cleanly unmounted - there may still be references to dentries/inodes from the lower fs, but overlayfs never calls any filesystem methods on the lower dentry/inodes - specifically lookup misses in the upper dir do not end up looking in the lower dir. The reason that I used an opt-in mount option (xino=nofollow) to enable this functionality is because even after all files have been copied up, overlayfs does currently access one bit of information from the lower fs - it calls getattr() to get st_ino from the lower file/directory in order to preserve st_ino across copy up. I used an opt-in mount option to allow st_ino to change across copy up. I hope this change of behavior is acceptable for your use case. Note that after the completion of the migration process (e.g. chown -R + 4*find) all inode numbers are stabilized. Are you interested in testing these patches? If you indicate that they are useful to you, I can post them for review and in that case, I would appreciate if you can write the xfstests for the feature. Thanks, Amir.
