On Fri, Aug 27, 2021 at 04:53:47PM +1000, Chris Dunlop wrote: > G'day Dave, > > On Fri, Aug 27, 2021 at 03:49:56PM +1000, Dave Chinner wrote: > > On Fri, Aug 27, 2021 at 12:55:39PM +1000, Chris Dunlop wrote: > > > On Fri, Aug 27, 2021 at 06:56:35AM +1000, Chris Dunlop wrote: > > > > On Thu, Aug 26, 2021 at 10:05:00AM -0500, Eric Sandeen wrote: > > > > > On 8/25/21 9:06 PM, Chris Dunlop wrote: > > > > > > > > > > > > fallocate -l 1GB image.img > > > > > > mkfs.xfs -f image.img > > > > > > mkdir mnt > > > > > > mount -o loop ./image.img mnt > > > > > > fallocate -o 0 -l 700mb mnt/image.img > > > > > > fallocate -o 0 -l 700mb mnt/image.img > > > > > > > > > > > > Why does the second fallocate fail with ENOSPC, and is that considered an XFS bug? > > > > > > > > > > Interesting. Off the top of my head, I assume that xfs is not looking at > > > > > current file space usage when deciding how much is needed to satisfy the > > > > > fallocate request. While filesystems can return ENOSPC at any time for > > > > > any reason, this does seem a bit suboptimal. > > > > > > > > Yes, I would have thought the second fallocate should be a noop. > > > > > > On further reflection, "filesystems can return ENOSPC at any time" is > > > certainly something apps need to be prepared for (and in this case, it's > > > doing the right thing, by logging the error and aborting), but it's not > > > really a "not a bug" excuse for the filesystem in all circumstances (or this > > > one?), is it? E.g. a write(fd, buf, 1) returning ENOSPC on an fresh > > > filesystem would be considered a bug, no? > > > > Sure, but the fallocate case here is different. You're asking to > > preallocate up to 700MB of space on a filesystem that only has 300MB > > of space free. Up front, without knowing anything about the layout > > of the file we might need to allocate 700MB of space into, there's a > > very good chance that we'll get ENOSPC partially through the > > operation. > > But I'm not asking for more space - the space is already there: "Up front, without knowing anything about the layout of the file..." [....] > Sigh. On the other hand that might be a case of "play stupid > games, win stupid prizes". On the gripping hand I can imagine the emails to > the mailing list from people like me asking why their "simple" fallocate is > taking 20 minutes... Yup, we have to chose between behaviours people will complain about. We chose the behaviour that doesn't happen except on really small filesystems because, in practice, we almost never see production workloads asking to fallocate() more than half the entire filesystem capacity at a time..... > > > > > > Background: I'm chasing a mysterious ENOSPC error on an XFS > > > > > > filesystem with way more space than the app should be asking > > > > > > for. There are no quotas on the fs. Unfortunately it's a third > > > > > > party app and I can't tell what sequence is producing the error, > > > > > > but this fallocate issue is a possibility. > > > > More likely speculative preallocation is causing this than > > fallocate. However, we've had a background worker that cleans up > > speculative prealloc before reporting ENOSPC for a while now - what > > kernel version are seeing this on? > > 5.10.60. How long is "a while now"? I vaguely recall something about that > going through. Longer than that. > > Also, it might not even be data allocation that is the issue - if > > the filesystem is full and free space is fragmented, you could be > > getting ENOSPC because inodes cannot be allocated. In which case, > > the output of xfs-info would be useful so we can see if sparse inode > > clusters are enabled or not.... > > $ xfs_info /chroot > meta-data=/dev/mapper/vg00-chroot isize=512 agcount=32, agsize=244184192 blks > = sectsz=4096 attr=2, projid32bit=1 > = crc=1 finobt=1, sparse=1, rmapbt=1 > = reflink=1 bigtime=0 inobtcount=0 > data = bsize=4096 blocks=7813893120, imaxpct=5 > = sunit=128 swidth=512 blks > naming =version 2 bsize=4096 ascii-ci=0, ftype=1 > log =internal log bsize=4096 blocks=521728, version=2 > = sectsz=4096 sunit=1 blks, lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 > > It's currently fuller than I like: > > $ df /chroot > Filesystem 1K-blocks Used Available Use% Mounted on > /dev/mapper/vg00-chroot 31253485568 24541378460 6712107108 79% /chroot > > ...so that's 6.3T free, but this problem was happening with 71% (8.5T) free. > The /maximum/ the app could conceivably be asking for is around 1.1T (to > entirely duplicate an existing file), but it really shouldn't be doing > anywhere near that: I can see it doing write-in-place on the existing file > and should be asking for modest amounts of extention (then again, userland > developers, so who knows, right? ;-}). > > Oh, another reference: this is extensive reflinking happening on this > filesystem. I don't know if that's a factor. You may remember my previous > email relating to that: > > Extreme fragmentation ho! > https://www.spinics.net/lists/linux-xfs/msg47707.html Ah. Details that are likely extremely important. The workload, layout problems and ephemeral ENOSPC symptoms match the description of the problem that was fixed by the series of commits that went into 5.13 that ended in this one: commit fd43cf600cf61c66ae0a1021aca2f636115c7fcb Author: Brian Foster <bfoster@xxxxxxxxxx> Date: Wed Apr 28 15:06:05 2021 -0700 xfs: set aside allocation btree blocks from block reservation The blocks used for allocation btrees (bnobt and countbt) are technically considered free space. This is because as free space is used, allocbt blocks are removed and naturally become available for traditional allocation. However, this means that a significant portion of free space may consist of in-use btree blocks if free space is severely fragmented. On large filesystems with large perag reservations, this can lead to a rare but nasty condition where a significant amount of physical free space is available, but the majority of actual usable blocks consist of in-use allocbt blocks. We have a record of a (~12TB, 32 AG) filesystem with multiple AGs in a state with ~2.5GB or so free blocks tracked across ~300 total allocbt blocks, but effectively at 100% full because the the free space is entirely consumed by refcountbt perag reservation. Such a large perag reservation is by design on large filesystems. The problem is that because the free space is so fragmented, this AG contributes the 300 or so allocbt blocks to the global counters as free space. If this pattern repeats across enough AGs, the filesystem lands in a state where global block reservation can outrun physical block availability. For example, a streaming buffered write on the affected filesystem continues to allow delayed allocation beyond the point where writeback starts to fail due to physical block allocation failures. The expected behavior is for the delalloc block reservation to fail gracefully with -ENOSPC before physical block allocation failure is a possibility. To address this problem, set aside in-use allocbt blocks at reservation time and thus ensure they cannot be reserved until truly available for physical allocation. This allows alloc btree metadata to continue to reside in free space, but dynamically adjusts reservation availability based on internal state. Note that the logic requires that the allocbt counter is fully populated at reservation time before it is fully effective. We currently rely on the mount time AGF scan in the perag reservation initialization code for this dependency on filesystems where it's most important (i.e. with active perag reservations). Signed-off-by: Brian Foster <bfoster@xxxxxxxxxx> Reviewed-by: Chandan Babu R <chandanrlinux@xxxxxxxxx> Reviewed-by: Allison Henderson <allison.henderson@xxxxxxxxxx> Reviewed-by: Darrick J. Wong <djwong@xxxxxxxxxx> Signed-off-by: Darrick J. Wong <djwong@xxxxxxxxxx> Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx
