On Mon, Aug 12, 2024 at 09:56:52AM -0400, Brian Foster wrote: > Tweak a few checks to facilitate experimentation with an agcount=1 > filesystem format with a larger agsize than the filesystem data > size. The purpose of this is to POC a filesystem image mode format > for XFS that better supports the typical cloud filesystem image > deployment use case where a very small fs image is created and then > immediately grown orders of magnitude in size once deployed to > container environments. The large grow size delta produces > filesystems with excessive AG counts, which leads to various other > functional problems that eventually derive from this sort of > pathological geometry. > > To experiment with this patch, format a small fs with something like > the following: > > mkfs.xfs -f -lsize=64m -dsize=512m,agcount=1,agsize=8g <imgfile> > > Increase the underlying image file size, mount and grow. The > filesystem will grow according to the format time AG size as if the > AG was a typical runt AG on a traditional multi-AG fs. > > This means that the filesystem remains with an AG count of 1 until > fs size grows beyond AG size. Since the typical deployment workflow > is an immediate very small -> very large, one-time grow, the image > fs can set a reasonable enough default or configurable AG size > (based on user input) that ensures deployed filesystems end up in a > generally supportable geometry (i.e. with multiple AGs for > superblock redundancy) before seeing production workloads. > > Further optional changes are possible on the kernel side to help > provide some simple guardrails against misuse of this mechanism. For > example, the kernel could do anything from warn/fail or restrict > runtime functionality for an insufficient grow. The image mode > itself could set a backwards incompat feature bit that requires a > mount option to enable full functionality (with the exception of > growfs). More discussion is required to determine whether this > provides a usable solution for the common cloud workflows that > exhibit this problem and what the right interface and/or limitations > are to ensure it is used correctly. > > Not-Signed-off-by: Brian Foster <bfoster@xxxxxxxxxx> > --- > > Hi all, > > This is a followup to the idea Darrick brought up in the expansion > discussion here [1]. I poked through the code a bit and found it > somewhat amusing how little was in the way of experimenting with this, > so threw this against an fstests run over the weekend. I see maybe > around ~10 or so test failures, most of which look like simple failures > related to either not expecting agcount == 1 fs' or my generally > hacky/experimental changes. There are a couple or so that require a bit > more investigation to properly characterize before I would consider this > fully sane. > > I'm posting this separately from the expansion discussion to hopefully > avoid further conflating the two. My current sense is that if this turns > out to be a fundamentally workable approach, mkfs would more look > something like 'mkfs --image-size 40g ...' and the kernel side may grow > some optional guardrail logic mentioned above and in the previous > discussion here [2], but others might have different ideas. > > Darrick, you originally raised this idea and then Eric brought up some > legitimate technical concerns in the expansion design thread. I'm > curious if either of you have any further thoughts/ideas on this. Well, it /does/ seem to work as intended -- you can create a filesystem with a 100G agsize even on a 8G filesystem. We've been over the drawbacks of this approach vs. Dave's (i.e. explodefs is really only a one-time event) so I won't rehash that here. But it does solve the much more constrained problem of disk images deployed from a gold master image undergoing a massive onetime expansion during firstboot. The lack of secondary superblocks and xfs_repair warnings are concerning, but to that, I have these things to say: a. If it's a gold master image from a trusted distributor, they should be installing verity information on all the read-mostly files so that one can validate the vendor's signatures. The fs metadata being corrupt is moot if fsverity fails. b. We can turn off the xfs_repair hostility to single-AG filesystems so that people who /do/ want to fsck their goldmaster images don't get annoying warnings. We /could/ add a new compat flag to say it's ok to single-AG, or we could hang that on the fsverity sb flag. c. Over on our end, we know the minimum cloud boot volume size -- it's 47GB. My canned OL9 image seems to come with a 36GB root filesystem. My guess is that most of our users will say yes to the instance creator asking them if they want more space (a princely 67GB minimum if you're stingy like I am!) so I think one could tweak the image creator to spit out a 36GB AG XFS, knowing that only the Ferengi users *won't* end up with a double-AG rootfs. IOWs, this more or less works for the usecase that generates the most internal complaints about xfs sucking. As I said earlier, I've not heard about people wanting to 10000x growfs after firstboot; usually they only grow incrementally as they decide that extra $$$$$ is worth another 10GB. I think this is worth prototyping a bit more. Do you? --D > Brian > > [1] https://lore.kernel.org/linux-xfs/20240721230100.4159699-1-david@xxxxxxxxxxxxx/ > [2] https://lore.kernel.org/linux-xfs/ZqzMay58f0SvdWxV@bfoster/ > > mkfs/xfs_mkfs.c | 11 +++++------ > 1 file changed, 5 insertions(+), 6 deletions(-) > > diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c > index 6d2469c3c..50a874a03 100644 > --- a/mkfs/xfs_mkfs.c > +++ b/mkfs/xfs_mkfs.c > @@ -325,8 +325,7 @@ static struct opt_params dopts = { > }, > .subopt_params = { > { .index = D_AGCOUNT, > - .conflicts = { { &dopts, D_AGSIZE }, > - { &dopts, D_CONCURRENCY }, > + .conflicts = { { &dopts, D_CONCURRENCY }, > { NULL, LAST_CONFLICT } }, > .minval = 1, > .maxval = XFS_MAX_AGNUMBER, > @@ -368,8 +367,7 @@ static struct opt_params dopts = { > .defaultval = SUBOPT_NEEDS_VAL, > }, > { .index = D_AGSIZE, > - .conflicts = { { &dopts, D_AGCOUNT }, > - { &dopts, D_CONCURRENCY }, > + .conflicts = { { &dopts, D_CONCURRENCY }, > { NULL, LAST_CONFLICT } }, > .convert = true, > .minval = XFS_AG_MIN_BYTES, > @@ -1233,7 +1231,7 @@ validate_ag_geometry( > usage(); > } > > - if (agsize > dblocks) { > + if (agsize > dblocks && agcount != 1) { > fprintf(stderr, > _("agsize (%lld blocks) too big, data area is %lld blocks\n"), > (long long)agsize, (long long)dblocks); > @@ -2703,7 +2701,8 @@ validate_supported( > * Filesystems should not have fewer than two AGs, because we need to > * have redundant superblocks. > */ > - if (mp->m_sb.sb_agcount < 2) { > + if (mp->m_sb.sb_agcount < 2 && > + mp->m_sb.sb_agblocks <= mp->m_sb.sb_dblocks) { > fprintf(stderr, > _("Filesystem must have at least 2 superblocks for redundancy!\n")); > usage(); > -- > 2.45.0 >
