On 22/05/2024 22:56, Luis Chamberlain wrote:
On Wed, May 15, 2024 at 01:54:39PM -0600, John Garry wrote:
On 27/02/2024 23:12, Theodore Ts'o wrote:
Last year, I talked about an interest to provide database such as
MySQL with the ability to issue writes that would not be torn as they
write 16k database pages[1].
[1] https://urldefense.com/v3/__https://lwn.net/Articles/932900/__;!!ACWV5N9M2RV99hQ!Ij_ZeSZrJ4uPL94Im73udLMjqpkcZwHmuNnznogL68ehu6TDTXqbMsC4xLUqh18hq2Ib77p1D8_4mV5Q$
After discussing this topic earlier this week, I would like to know if there
are still objections or concerns with the untorn-writes userspace API
proposed in https://lore.kernel.org/linux-block/20240326133813.3224593-1-john.g.garry@xxxxxxxxxx/
I feel that the series for supporting direct-IO only, above, is stuck
because of this topic of buffered IO.
I think it was good we had the discussions at LSFMM over it, however
I personally don't percieve it as stuck, however without any consensus
being obviated or written down anywhere it would not be clear to anyone
that we did reach any consensus at all.
Hope is that lwn captures any
consensus if any was indeed reached as you're not making it clear any
was.
That's my point really. There were some positive discussion. I put
across the idea of implementing buffered atomic writes, and now I want
to ensure that everyone is satisfied with that going forward. I think
that a LWN report is now being written.
In case it helps, as we did with the LBS effort it may also be useful to
put together bi-monthly cabals to follow up progress, and divide and
conquer any pending work items.
ok, we can consider that.
So I sent an RFC for buffered untorn-writes last month in https://lore.kernel.org/linux-fsdevel/20240422143923.3927601-1-john.g.garry@xxxxxxxxxx/,
which did leverage the bs > ps effort. Maybe it did not get noticed due to
being an RFC. It works on the following principles:
- A buffered atomic write requires RWF_ATOMIC flag be set, same as
direct IO. The same other atomic writes rules apply.
- For an inode, only a single size of buffered write is allowed. So for
statx, atomic_write_unit_min = atomic_write_unit_max always for
buffered atomic writes.
- A single folio maps to an atomic write in the pagecache. So inode
address_space folio min order = max order = atomic_write_unit_min/max
- A folio is tagged as "atomic" when atomically written and written back
to storage "atomically", same as direct-IO method would do for an
atomic write.
- If userspace wants to guarantee a buffered atomic write is written to
storage atomically after the write syscall returns, it must use
RWF_SYNC or similar (along with RWF_ATOMIC).
From my perspective the above just needs the IOCB atomic support, and
the pending long term work item there is the near-write-through buffered
IO support. We could just wait for buffered-IO support until we have
support for that. I can't think of anying blocking DIO support though,
now that we at least have a mental model of how buffered IO *should*
work.
Yes, these are my thoughts as well.
What about testing? Are you extending fstests, blktests?
Yes, so 3 things to mention here:
- We have been looking at adding full test coverage in xfstests.
Catherine Hoang recently starting working on this. Most tests will
actually cover the forcealign feature. Indeed, just atomic writes
support testing would be quite limited when compared to forcealign
testing. Furthermore we are also looking at forcealign and atomic writes
testing in fsx.c, as finding forcealign corner cases would be quite
limited on the formalized tests
- for blktests, we were going to add some basic atomic writes test
there, like ensuring that misaligned or mis-sized writes are rejected.
This would be the same really for xfstests, above. I don't think that
there are so many tests which we can cover. scsi_debug will support
atomic writes, which can be used for blktests.
- I have done some limited power-fail testing for my NVMe card.
I have 2x challenges here:
- My host does not allow the card port to be manually powered down, so I
need to physically plug out the power cable to test :(
- My NVMe card only supports 4KB power-fail atomic writes, which is
quite small.
The actual power-fail testing involves using fio in verify mode. In
that, each data block has a CRC written per test loop. I just verify
that the CRCs are valid after the power cycle (which they are when block
size is 4KB and lower :)).
Thanks,
John
