On Wed, Dec 02, 2020 at 08:15:57AM +1100, Dave Chinner wrote: > On Tue, Dec 01, 2020 at 09:07:42AM -0500, Brian Foster wrote: > > On Wed, Nov 11, 2020 at 09:59:24AM +1100, Dave Chinner wrote: > > > From: Dave Chinner <dchinner@xxxxxxxxxx> > > > > > > To be able to shrink a filesystem, we need to be able to physically > > > move an inode and all it's data and metadata from it's current > > > location to a new AG. Add a command to spaceman to allow an inode > > > to be moved to a new AG. > > > > > > This new command is not intended to be a perfect solution. I am not > > > trying to handle atomic movement of open files - this is intended to > > > be run as a maintenance operation on idle filesystem. If root > > > filesystems are the target, then this should be run via a rescue > > > environment that is not executing directly on the root fs. With > > > those caveats in place, we can do the entire inode move as a set of > > > non-destructive operations finalised by an atomic inode swap > > > without any needing special kernel support. > > > > > > To ensure we move metadata such as BMBT blocks even if we don't need > > > to move data, we clone the data to a new inode that we've allocated > > > in the destination AG. This will result in new bmbt blocks being > > > allocated in the new location even though the data is not copied. > > > Attributes need to be copied one at a time from the original inode. > > > > > > If data needs to be moved, then we use fallocate(UNSHARE) to create > > > a private copy of the range of data that needs to be moved in the > > > new inode. This will be allocated in the destination AG by normal > > > allocation policy. > > > > > > Once the new inode has been finalised, use RENAME_EXCHANGE to swap > > > it into place and unlink the original inode to free up all the > > > resources it still pins. > > > > > > There are many optimisations still possible to speed this up, but > > > the goal here is "functional" rather than "optimal". Performance can > > > be optimised once all the parts for a "empty the tail of the > > > filesystem before shrink" operation are implemented and solidly > > > tested. > > > > > > > Neat idea. With respect to the shrink use case, what's the reasoning > > behind userspace selecting the target AG? There's no harm in having the > > target AG option in the utility of course, but ISTM that shrink might > > care more about moving some set of inodes from a particular AG as > > opposed to a specific target AG. > > Oh, that's just a mechanism right now to avoid needing kernel > allocator policy support for relocating things. Say for example, we > plan to empty the top six AGs - we don't want the allocator to chose > any of them for relocation, and in the absence of kernel side policy > the only way we can direct that is to select an AG outside that > range manually with a target directory location (as per xfs_fsr). > > IOWs, I'm just trying to implement the move mechanisms without > having to introduce new kernel API dependencies because I kinda want > shrink to be possible with minimal kernel requirements. It's also > not meant to be an optimal implementation at this point, merely a > generic one. Adding policy hooks for controlling AG allocation can > be done once we know exactly what the data movement process needs > for optimal behaviour. > Ok. > > For example, might it make sense to implement a policy where move_inode > > simply moves an inode to the first AG the tempdir lands in that is < the > > AG of the source inode? We'd probably want to be careful to make sure > > that we don't attempt to dump the entire set of moved files into the > > same AG, but I assume the temp dir creation logic would effectively > > rotor across the remaining set of AGs we do want to allow.. Thoughts? > > Yes, we could. But I simply decided that a basic, robust shrink to > the minimum possible size will have to fill the filesystem from AG 0 > up, and not move to AG 1 until AG 0 is full. I also know that the > kernel allocation policies will skip to the next AG if there is lock > contention, space or other allocation setup issues, hence I wanted > to be able to direct movement to the lowest possible AGs first... > > THere's enough complexity in an optimal shrink implementation that > it will keep someone busy full time for a couple of years. I want to > provide the basic functionality userspace needs only spending a > couple of days a week for a couple of months on it. If we want it > fast and deployable on existing systems, compromises will need to be > made... > Yeah, I'm not suggesting we implement the eventual policy here. I do think it would be nice if the userspace command implemented some reasonable default when a target AG is not specified. That could be the "anything less than source AG" logic I described above, a default target of AG 0, or something similarly simple... Brian > Cheers, > > Dave. > -- > Dave Chinner > david@xxxxxxxxxxxxx >
