On 23 Apr 2021 at 18:40, Brian Foster wrote: > The blocks used for allocation btrees (bnobt and countbt) are > technically considered free space. This is because as free space is > used, allocbt blocks are removed and naturally become available for > traditional allocation. However, this means that a significant > portion of free space may consist of in-use btree blocks if free > space is severely fragmented. > > On large filesystems with large perag reservations, this can lead to > a rare but nasty condition where a significant amount of physical > free space is available, but the majority of actual usable blocks > consist of in-use allocbt blocks. We have a record of a (~12TB, 32 > AG) filesystem with multiple AGs in a state with ~2.5GB or so free > blocks tracked across ~300 total allocbt blocks, but effectively at > 100% full because the the free space is entirely consumed by > refcountbt perag reservation. > > Such a large perag reservation is by design on large filesystems. > The problem is that because the free space is so fragmented, this AG > contributes the 300 or so allocbt blocks to the global counters as > free space. If this pattern repeats across enough AGs, the > filesystem lands in a state where global block reservation can > outrun physical block availability. For example, a streaming > buffered write on the affected filesystem continues to allow delayed > allocation beyond the point where writeback starts to fail due to > physical block allocation failures. The expected behavior is for the > delalloc block reservation to fail gracefully with -ENOSPC before > physical block allocation failure is a possibility. > > To address this problem, set aside in-use allocbt blocks at > reservation time and thus ensure they cannot be reserved until truly > available for physical allocation. This allows alloc btree metadata > to continue to reside in free space, but dynamically adjusts > reservation availability based on internal state. Note that the > logic requires that the allocbt counter is fully populated at > reservation time before it is fully effective. We currently rely on > the mount time AGF scan in the perag reservation initialization code > for this dependency on filesystems where it's most important (i.e. > with active perag reservations). > The changes look good to me. Reviewed-by: Chandan Babu R <chandanrlinux@xxxxxxxxx> -- chandan
