On Fri, May 19, 2023 at 10:41:31AM -0400, Mike Snitzer wrote: > On Fri, May 19 2023 at 12:09P -0400, > Christoph Hellwig <hch@xxxxxxxxxxxxx> wrote: > > > FYI, I really don't think this primitive is a good idea. In the > > concept of non-overwritable storage (NAND, SMR drives) the entire > > concept of a one-shoot 'provisioning' that will guarantee later writes > > are always possible is simply bogus. > > Valid point for sure, such storage shouldn't advertise support (and > will return -EOPNOTSUPP). > > But the primitive still has utility for other classes of storage. Yet the thing people are wanting to us filesystem developers to use this with is thinly provisioned storage that has snapshot capability. That, by definition, is non-overwritable storage. These are the use cases people are asking filesystes to gracefully handle and report errors when the sparse backing store runs out of space. e.g. journal writes after a snapshot is taken on a busy filesystem are always an overwrite and this requires more space in the storage device for the write to succeed. ENOSPC from the backing device for journal IO is a -fatal error-. Hence if REQ_PROVISION doesn't guarantee space for overwrites after snapshots, then it's not actually useful for solving the real world use cases we actually need device-level provisioning to solve. It is not viable for filesystems to have to reprovision space for in-place metadata overwrites after every snapshot - the filesystem may not even know a snapshot has been taken! And it's not feasible for filesystems to provision on demand before they modify metadata because we don't know what metadata is going to need to be modified before we start modifying metadata in transactions. If we get ENOSPC from provisioning in the middle of a dirty transcation, it's all over just the same as if we get ENOSPC during metadata writeback... Hence what filesystems actually need is device provisioned space to be -always over-writable- without ENOSPC occurring. Ideally, if we provision a range of the block device, the block device *must* guarantee all future writes to that LBA range succeeds. That guarantee needs to stand until we discard or unmap the LBA range, and for however many writes we do to that LBA range. e.g. If the device takes a snapshot, it needs to reprovision the potential COW ranges that overlap with the provisioned LBA range at snapshot time. e.g. by re-reserving the space from the backing pool for the provisioned space so if a COW occurs there is space guaranteed for it to succeed. If there isn't space in the backing pool for the reprovisioning, then whatever operation that triggers the COW behaviour should fail with ENOSPC before doing anything else.... Software devices like dm-thin/snapshot should really only need to keep a persistent map of the provisioned space and refresh space reservations for used space within that map whenever something that triggers COW behaviour occurs. i.e. a snapshot needs to reset the provisioned ranges back to "all ranges are freshly provisioned" before the snapshot is started. If that space is not available in the backing pool, then the snapshot attempt gets ENOSPC.... That means filesystems only need to provision space for journals and fixed metadata at mkfs time, and they only need issue a REQ_PROVISION bio when they first allocate over-write in place metadata. We already have online discard and/or fstrim for releasing provisioned space via discards. This will require some mods to filesystems like ext4 and XFS to issue REQ_PROVISION and fail gracefully during metadata allocation. However, doing so means that we can actually harden filesystems against sparse block device ENOSPC errors by ensuring they will never occur in critical filesystem structures.... -Dave. -- Dave Chinner david@xxxxxxxxxxxxx
