At last year's LSFMM we learned through Ted Ts'o about the interest by
cloud providers in large atomics [0]. It is a good example where cloud
providers innovated in an area perhaps before storage vendors were
providing hardware support for such features. An example use case was
databases. In short, with large atomics databases can disable their own version
of journaling so to increase TPS. Large atomics lets you disabling things like
MySQL innodb_doublewrite. The feature to allow you to disable this and use
large atomcis is known as torn write prevention [1]. At least for MySQL the
default page size for the database (used for columns) is 16k, and so enabling
for example a 16k atomic can allow you to take advantage of this. It was also
mentioned how PostgreSQL only supports buffered-IO and so it would be desirable
for a solution to support buffered-IO with large atomics as well. The way
cloud providers enable torn write protection, is by using direct IO.
John Garry has been working on adding an API for atomic writes, it would
seem some folks refer to this as the no-tears atomic API. It consists of
two parts, one for the block layer [2] and another set of changes for
XFS [3]. It enables Direct IO support with large atomics. It includes
a userspace API which lets you peg a FS_XFLAG_ATOMICWRITES flag onto a
file, and you then create an XFS filesystem using the XFS realtime
subvolume with with an extent alignment. The current users of this API
seems to be SCSI, but obviously this can grow to support others. A neat
feature of this effort is you can have two separate directories with
separate aligment requirements. There is no generic filesystem solution
yet.
Meanwhile we're now at a v2 RFC for LBS support [4]. Although the LBS
effort originally was a completely orthogonal effort to large atomics, it
would seem there is a direct relationship here now worth discussing.
In short LBS enables buffered-IO large atomic support if the hardware
support its. We get both alignment constraints gauranteed and now ensure
we use contigous memory for the IOs for DMA too it is built on using large
folios. We expect NVMe drives which support support large atomics can
easily profit from this without any userspace modification other than
when you create the filesystem.
We reviewed the possible intersection of both efforts at our last LBS cabal
with LBS interested folks and Martin Peterson and John Garry. It is somewhat
unclear exactly how to follow up on some aspects of the no-tear API [5]
but there was agreement about the possible intersection of both efforts,
and that we should discuss this at LSFMM. The goal would be to try to reach
consensus on how no-tear API and how LBS could help with those
interested in leveraging large atomics.
Some things to evaluate or for us to discuss:
* no-tear API:
- allows directories to have separate alignment requirements
- this might be useful for folks who want to use large IOs with
large atomics for some workloads but smaller IOs for another
directory on the same drive. It this a viable option to some
users for large atomics with concerns of being forced to use
only large writes with LBS?
- statx is modified so to display new alignment considerations
- atomics are power of 2
- there seems to be some interest in supporting no-hardware-accel atomic
solution, so a software implemented atomic solution, could someone
clarify if that's accurate? How is the double write avoided? What are
the use cases? Do databases use that today?
- How do we generalize a solution per file? Would extending a min
order per file be desirable? Is that even tenable?
* LBS:
- stat will return the block size set, so userspace applications
using stat / statx will use the larger block size to ensure
alignment
- a drive with support for a large atomic but supporting smaller
logical block sizes will still allow writes to the logical block
size. If a block driver has a "preference" (in NVMe this would
be the NPWG for the IU) to write above the logical block size,
do we want the option to lift the logical block size? In
retrospect I don't think this is needed given Jan Kara's patches
to prevent allowing writes to to mounted devices [4], that should
ensure that if a filesystem takes advantage of a larger physical
block size and creates a filesystem with it as the sector size,
userspace won't be mucking around with lower IOs to the drive
while it is mounted. But, are there any applications which would
get the block device logical block size instead for DIO?
- LBS is transparent to to userspace applications
- We've verified *most* IOs are aligned if you use a 16k block size
but a smaller sector size, the lower IOs were verified to come
from the XFS buffer cache. If your drive supports a large atomic
you can avoid these as you can lift the sector size set as the
physical block size will be larger than the logical block size.
For NVMe today this is possible for drives with a large
NPWG (the IU) and NAWUFP (the large atomic), for example.
Tooling:
- Both efforts stand to gain from a shared verification set of tools
for alignment and atomic use
- We have a block layer eBPF alignent tool written by Daniel Gomez [6]
however there is lack of interested parties to help review a simpler
version of this tool this tool so we merge it [7], we can benefit from more
eyeablls from experienced eBPF / block layer folks.
- More advanced tools are typically not encouraged, and this leaves us
wondering what a better home would be other than side forks
- Other than preventing torn writes, do users of the no-tear API care
about WAF? While we have one for NVMe for WAF [8] would
collaborating on a generic tool be of interest ?
Any other things folks want to get out of this as a session, provided
there is interest?
[0] https://lwn.net/Articles/932900/
[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-twp.html
[2] https://lore.kernel.org/linux-nvme/20240124113841.31824-1-john.g.garry@xxxxxxxxxx/T/#m4ad28b480a8e12eb51467e17208d98ca50041ff2
[3] https://lore.kernel.org/all/20240124142645.9334-1-john.g.garry@xxxxxxxxxx/
[4] https://lore.kernel.org/all/20240213093713.1753368-1-kernel@xxxxxxxxxxxxxxxx/T/#u
[5] https://lkml.kernel.org/r/20231101173542.23597-1-jack@xxxxxxx
[6] https://github.com/dagmcr/bcc/tree/blkalgn-dump
[7] https://github.com/iovisor/bcc/pull/4813
[8] https://github.com/dagmcr/bcc/tree/nvmeiuwaf
Luis
