Hi Linus,
Can you please pull the XFS updates for 5.19 from the tag listed
below? It merges cleanly with the TOT kernel I just pulled a couple
of minutes ago, though the diffstat I got on merge:
105 files changed, 4862 insertions(+), 2773 deletions(-)
is slightly different to the diffstat the pull request generated.
If I've made any mistakes or done stuff that is considered wrong or
out of date, let me know and I'll fix them up - it's been a while
since I built a tree for upstream merge.
This is a big update with lots of new code. The tag describes them
all, so I'll just touch on teh higlights. The two main new features
are Large Extent Counts and Logged Attribute Replay - these are two
new foundational features that we are building more complex future
features on top of.
For upcoming functionality, we need to be able to store hundreds of
millions of xattrs per inode. The Large Extent Count feature removes
the limits that prevent this scale of xattr storage, and while we
were modifying the on disk extent count format we also increased the
number of data extents we support per inode from 2^32 to 2^47.
We also need to be able to modify xattrs as part of larger atomic
transactions rather than as standalone transactions. The Logged
Attribute Replay feature introduces the infrastructure that allows
us to use intents to record the attribute modifications in the
journal before we start them, hence allowing other atomic
transactions to log attribute modification intents and then defer
the actual modification to later. If we then crash, log recovery
then guarantees that the attribute is replayed in the context of the
atomic transaction that logged the intent.
A significant chunk of the commits in this merge are for the base
attribute replay functionality along with fixes, improvements and
cleanups related to this new functioanlity. Allison deserves a big
round of thanks for her ongoing work to get this functionality into
XFS.
There are also many other smaller changes and improvements, so
overall this is one of the bigger XFS merge requests in some time.
I will be following up next week with another smaller pull request -
we already have another round of fixes and improvements to the
logged attribute replay functionality just about ready to go.
They'll soak and test over the next week, and I'll send a pull
request for them near the end of the merge window.
Thanks!
-Dave.
The following changes since commit 9a5280b312e2e7898b6397b2ca3cfd03f67d7be1:
xfs: reorder iunlink remove operation in xfs_ifree (2022-04-21 08:45:16 +1000)
are available in the Git repository at:
git://git.kernel.org/pub/scm/fs/xfs/xfs-linux.git tags/xfs-5.19-for-linus
for you to fetch changes up to efd409a4329f6927795be5ae080cd3ec8c014f49:
Merge branch 'xfs-5.19-quota-warn-remove' into xfs-5.19-for-next (2022-05-12 15:23:07 +1000)
----------------------------------------------------------------
xfs: Changes for 5.19-rc1
This update includes:
- support for printk message indexing.
- large extent counts to provide support for up to 2^47 data extents and 2^32
attribute extents, allowing us to scale beyond 4 billion data extents
to billions of xattrs per inode.
- conversion of various flags fields to be consistently declared as
unsigned bit fields.
- improvements to realtime extent accounting and converts them to per-cpu
counters to match all the other block and inode accounting.
- reworks core log formatting code to reduce iterations, have a shorter, cleaner
fast path and generally be easier to understand and maintain.
- improvements to rmap btree searches that reduce overhead by up
to 30% resulting in xfs_scrub runtime reductions of 15%.
- improvements to reflink that remove the size limitations in remapping operations
and greatly reduce the size of transaction reservations.
- reworks the minimum log size calculations to allow us to change transaction
reservations without changing the minimum supported log size.
- removal of quota warning support as it has never been used on Linux.
- intent whiteouts to allow us to cancel intents that are completed entirely
in memory rather than having use CPU and disk bandwidth formatting and writing
them into the journal when it is not necessary. This makes rmap, reflink and
extent freeing slightly more efficient, but provides massive improvements
for....
- Logged Attribute Replay feature support. This is a fundamental change to the
way we modify attributes, laying the foundation for future integration of
attribute modifications as part of other atomic transactional operations the
filesystem performs.
- Lots of cleanups and fixes for the logged attribute replay functionality.
----------------------------------------------------------------
Allison Henderson (14):
xfs: Fix double unlock in defer capture code
xfs: Return from xfs_attr_set_iter if there are no more rmtblks to process
xfs: Set up infrastructure for log attribute replay
xfs: Implement attr logging and replay
xfs: Skip flip flags for delayed attrs
xfs: Add xfs_attr_set_deferred and xfs_attr_remove_deferred
xfs: Remove unused xfs_attr_*_args
xfs: Add log attribute error tag
xfs: Add larp debug option
xfs: Merge xfs_delattr_context into xfs_attr_item
xfs: Add helper function xfs_attr_leaf_addname
xfs: Add helper function xfs_init_attr_trans
xfs: add leaf split error tag
xfs: add leaf to node error tag
Brian Foster (1):
xfs: fix soft lockup via spinning in filestream ag selection loop
Catherine Hoang (3):
xfs: remove quota warning limit from struct xfs_quota_limits
xfs: remove warning counters from struct xfs_dquot_res
xfs: don't set quota warning values
Chandan Babu R (19):
xfs: Move extent count limits to xfs_format.h
xfs: Define max extent length based on on-disk format definition
xfs: Introduce xfs_iext_max_nextents() helper
xfs: Use xfs_extnum_t instead of basic data types
xfs: Introduce xfs_dfork_nextents() helper
xfs: Use basic types to define xfs_log_dinode's di_nextents and di_anextents
xfs: Promote xfs_extnum_t and xfs_aextnum_t to 64 and 32-bits respectively
xfs: Introduce XFS_SB_FEAT_INCOMPAT_NREXT64 and associated per-fs feature bit
xfs: Introduce XFS_FSOP_GEOM_FLAGS_NREXT64
xfs: Introduce XFS_DIFLAG2_NREXT64 and associated helpers
xfs: Use uint64_t to count maximum blocks that can be used by BMBT
xfs: Introduce macros to represent new maximum extent counts for data/attr forks
xfs: Replace numbered inode recovery error messages with descriptive ones
xfs: Introduce per-inode 64-bit extent counters
xfs: Directory's data fork extent counter can never overflow
xfs: Conditionally upgrade existing inodes to use large extent counters
xfs: Decouple XFS_IBULK flags from XFS_IWALK flags
xfs: Enable bulkstat ioctl to support 64-bit per-inode extent counters
xfs: Add XFS_SB_FEAT_INCOMPAT_NREXT64 to the list of supported flags
Christoph Hellwig (2):
xfs: change the type of ic_datap
xfs: remove xlog_verify_dest_ptr
Darrick J. Wong (16):
xfs: pass explicit mount pointer to rtalloc query functions
xfs: recalculate free rt extents after log recovery
xfs: use a separate frextents counter for rt extent reservations
xfs: capture buffer ops in the xfs_buf tracepoints
xfs: simplify xfs_rmap_lookup_le call sites
xfs: speed up rmap lookups by using non-overlapped lookups when possible
xfs: speed up write operations by using non-overlapped lookups when possible
xfs: count EFIs when deciding to ask for a continuation of a refcount update
xfs: stop artificially limiting the length of bunmap calls
xfs: remove a __xfs_bunmapi call from reflink
xfs: create shadow transaction reservations for computing minimum log size
xfs: report "max_resp" used for min log size computation
xfs: reduce the absurdly large log operation count
xfs: reduce transaction reservations with reflink
xfs: rewrite xfs_reflink_end_cow to use intents
xfs: rename xfs_*alloc*_log_count to _block_count
Dave Chinner (73):
xfs: factor out the CIL transaction header building
xfs: only CIL pushes require a start record
xfs: embed the xlog_op_header in the unmount record
xfs: embed the xlog_op_header in the commit record
xfs: log tickets don't need log client id
xfs: move log iovec alignment to preparation function
xfs: reserve space and initialise xlog_op_header in item formatting
xfs: log ticket region debug is largely useless
xfs: pass lv chain length into xlog_write()
xfs: introduce xlog_write_full()
xfs: introduce xlog_write_partial()
xfs: xlog_write() no longer needs contwr state
xfs: xlog_write() doesn't need optype anymore
xfs: CIL context doesn't need to count iovecs
xfs: convert attr type flags to unsigned.
xfs: convert scrub type flags to unsigned.
xfs: convert bmap extent type flags to unsigned.
xfs: convert bmapi flags to unsigned.
xfs: convert AGF log flags to unsigned.
xfs: convert AGI log flags to unsigned.
xfs: convert btree buffer log flags to unsigned.
xfs: convert buffer log item flags to unsigned.
xfs: convert da btree operations flags to unsigned.
xfs: convert dquot flags to unsigned.
xfs: convert log item tracepoint flags to unsigned.
xfs: convert inode lock flags to unsigned.
xfs: convert ptag flags to unsigned.
xfs: convert quota options flags to unsigned.
xfs: convert shutdown reasons to unsigned.
xfs: convert log ticket and iclog flags to unsigned.
Merge branch 'guilt/5.19-miscellaneous' into xfs-5.19-for-next
Merge branch 'guilt/xfs-unsigned-flags-5.18' into xfs-5.19-for-next
Merge branch 'guilt/xlog-write-rework' into xfs-5.19-for-next
Merge tag 'large-extent-counters-v9' of https://github.com/chandanr/linux into xfs-5.19-for-next
xfs: zero inode fork buffer at allocation
xfs: fix potential log item leak
xfs: hide log iovec alignment constraints
xfs: don't commit the first deferred transaction without intents
xfs: add log item flags to indicate intents
xfs: tag transactions that contain intent done items
xfs: factor and move some code in xfs_log_cil.c
xfs: add log item method to return related intents
xfs: whiteouts release intents that are not in the AIL
xfs: intent item whiteouts
xfs: detect self referencing btree sibling pointers
xfs: validate inode fork size against fork format
xfs: set XFS_FEAT_NLINK correctly
xfs: validate v5 feature fields
Merge branch 'guilt/xfs-5.19-misc-2' into xfs-5.19-for-next
Merge branch 'guilt/xlog-intent-whiteouts' into xfs-5.19-for-next
Merge tag 'rmap-speedups-5.19_2022-04-28' of git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-5.19-for-next
Merge tag 'reflink-speedups-5.19_2022-04-28' of git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux into xfs-5.19-for-next
Merge branch 'guilt/xfs-5.19-fuzz-fixes' into xfs-5.19-for-next
xfs: avoid empty xattr transaction when attrs are inline
xfs: initialise attrd item to zero
xfs: make xattri_leaf_bp more useful
xfs: rework deferred attribute operation setup
xfs: separate out initial attr_set states
xfs: kill XFS_DAC_LEAF_ADDNAME_INIT
xfs: consolidate leaf/node states in xfs_attr_set_iter
xfs: split remote attr setting out from replace path
xfs: XFS_DAS_LEAF_REPLACE state only needed if !LARP
xfs: remote xattr removal in xfs_attr_set_iter() is conditional
xfs: clean up final attr removal in xfs_attr_set_iter
xfs: xfs_attr_set_iter() does not need to return EAGAIN
xfs: introduce attr remove initial states into xfs_attr_set_iter
xfs: switch attr remove to xfs_attri_set_iter
xfs: remove xfs_attri_remove_iter
xfs: use XFS_DA_OP flags in deferred attr ops
xfs: ATTR_REPLACE algorithm with LARP enabled needs rework
xfs: detect empty attr leaf blocks in xfs_attr3_leaf_verify
xfs: can't use kmem_zalloc() for attribute buffers
Merge branch 'xfs-5.19-quota-warn-remove' into xfs-5.19-for-next
Eric Sandeen (1):
xfs: revert "xfs: actually bump warning counts when we send warnings"
Jonathan Lassoff (2):
xfs: Simplify XFS logging methods.
xfs: Add XFS messages to printk index
Kaixu Xia (1):
xfs: simplify local variable assignment in file write code
Matthew Wilcox (Oracle) (1):
xfs: Use generic_file_open()
Yang Xu (1):
xfs: improve __xfs_set_acl
fs/xfs/Makefile | 1 +
fs/xfs/libxfs/xfs_alloc.c | 12 +-
fs/xfs/libxfs/xfs_alloc.h | 2 +-
fs/xfs/libxfs/xfs_attr.c | 1644 +++++++++++++++++++--------------------
fs/xfs/libxfs/xfs_attr.h | 198 ++++-
fs/xfs/libxfs/xfs_attr_leaf.c | 64 +-
fs/xfs/libxfs/xfs_attr_remote.c | 37 +-
fs/xfs/libxfs/xfs_attr_remote.h | 6 +-
fs/xfs/libxfs/xfs_bmap.c | 167 ++--
fs/xfs/libxfs/xfs_bmap.h | 58 +-
fs/xfs/libxfs/xfs_bmap_btree.c | 9 +-
fs/xfs/libxfs/xfs_btree.c | 150 +++-
fs/xfs/libxfs/xfs_btree.h | 26 +-
fs/xfs/libxfs/xfs_da_btree.c | 4 +
fs/xfs/libxfs/xfs_da_btree.h | 25 +-
fs/xfs/libxfs/xfs_da_format.h | 9 +-
fs/xfs/libxfs/xfs_defer.c | 54 +-
fs/xfs/libxfs/xfs_defer.h | 3 +
fs/xfs/libxfs/xfs_dir2.c | 8 +
fs/xfs/libxfs/xfs_errortag.h | 8 +-
fs/xfs/libxfs/xfs_format.h | 189 +++--
fs/xfs/libxfs/xfs_fs.h | 41 +-
fs/xfs/libxfs/xfs_ialloc.c | 8 +-
fs/xfs/libxfs/xfs_ialloc.h | 2 +-
fs/xfs/libxfs/xfs_inode_buf.c | 118 ++-
fs/xfs/libxfs/xfs_inode_fork.c | 51 +-
fs/xfs/libxfs/xfs_inode_fork.h | 76 +-
fs/xfs/libxfs/xfs_log_format.h | 79 +-
fs/xfs/libxfs/xfs_log_recover.h | 2 +
fs/xfs/libxfs/xfs_log_rlimit.c | 75 +-
fs/xfs/libxfs/xfs_quota_defs.h | 50 +-
fs/xfs/libxfs/xfs_refcount.c | 14 +-
fs/xfs/libxfs/xfs_refcount.h | 13 +-
fs/xfs/libxfs/xfs_rmap.c | 161 ++--
fs/xfs/libxfs/xfs_rmap.h | 7 +-
fs/xfs/libxfs/xfs_rtbitmap.c | 9 +-
fs/xfs/libxfs/xfs_sb.c | 80 +-
fs/xfs/libxfs/xfs_shared.h | 24 +-
fs/xfs/libxfs/xfs_trans_resv.c | 225 ++++--
fs/xfs/libxfs/xfs_trans_resv.h | 16 +-
fs/xfs/libxfs/xfs_types.h | 11 +-
fs/xfs/scrub/bmap.c | 26 +-
fs/xfs/scrub/common.c | 2 +
fs/xfs/scrub/inode.c | 20 +-
fs/xfs/scrub/rtbitmap.c | 9 +-
fs/xfs/xfs_acl.c | 4 +-
fs/xfs/xfs_acl.h | 8 +-
fs/xfs/xfs_attr_item.c | 824 ++++++++++++++++++++
fs/xfs/xfs_attr_item.h | 46 ++
fs/xfs/xfs_attr_list.c | 1 +
fs/xfs/xfs_bmap_item.c | 27 +-
fs/xfs/xfs_bmap_util.c | 27 +-
fs/xfs/xfs_buf_item.h | 24 +-
fs/xfs/xfs_dquot.c | 18 +-
fs/xfs/xfs_dquot.h | 8 -
fs/xfs/xfs_error.c | 9 +
fs/xfs/xfs_error.h | 20 +-
fs/xfs/xfs_extfree_item.c | 23 +-
fs/xfs/xfs_file.c | 24 +-
fs/xfs/xfs_filestream.c | 7 +-
fs/xfs/xfs_fsmap.c | 6 +-
fs/xfs/xfs_fsops.c | 7 +-
fs/xfs/xfs_globals.c | 1 +
fs/xfs/xfs_icache.c | 9 +-
fs/xfs/xfs_icreate_item.c | 1 +
fs/xfs/xfs_inode.c | 80 +-
fs/xfs/xfs_inode.h | 29 +-
fs/xfs/xfs_inode_item.c | 48 +-
fs/xfs/xfs_inode_item_recover.c | 145 +++-
fs/xfs/xfs_ioctl.c | 7 +-
fs/xfs/xfs_ioctl32.c | 2 +
fs/xfs/xfs_iomap.c | 33 +-
fs/xfs/xfs_iops.c | 4 +-
fs/xfs/xfs_itable.c | 15 +-
fs/xfs/xfs_itable.h | 5 +-
fs/xfs/xfs_iwalk.h | 2 +-
fs/xfs/xfs_log.c | 807 +++++++++----------
fs/xfs/xfs_log.h | 90 ++-
fs/xfs/xfs_log_cil.c | 391 ++++++----
fs/xfs/xfs_log_priv.h | 89 +--
fs/xfs/xfs_log_recover.c | 2 +
fs/xfs/xfs_message.c | 58 +-
fs/xfs/xfs_message.h | 55 +-
fs/xfs/xfs_mount.c | 91 ++-
fs/xfs/xfs_mount.h | 32 +-
fs/xfs/xfs_ondisk.h | 2 +
fs/xfs/xfs_qm.c | 9 -
fs/xfs/xfs_qm.h | 5 -
fs/xfs/xfs_qm_syscalls.c | 26 +-
fs/xfs/xfs_quotaops.c | 8 +-
fs/xfs/xfs_refcount_item.c | 25 +-
fs/xfs/xfs_reflink.c | 100 ++-
fs/xfs/xfs_rmap_item.c | 25 +-
fs/xfs/xfs_rtalloc.c | 41 +
fs/xfs/xfs_rtalloc.h | 9 +-
fs/xfs/xfs_super.c | 18 +-
fs/xfs/xfs_symlink.c | 5 -
fs/xfs/xfs_sysctl.h | 1 +
fs/xfs/xfs_sysfs.c | 24 +
fs/xfs/xfs_trace.h | 100 ++-
fs/xfs/xfs_trans.c | 52 +-
fs/xfs/xfs_trans.h | 38 +-
fs/xfs/xfs_trans_dquot.c | 4 +-
fs/xfs/xfs_xattr.c | 2 +-
104 files changed, 4690 insertions(+), 2676 deletions(-)
create mode 100644 fs/xfs/xfs_attr_item.c
create mode 100644 fs/xfs/xfs_attr_item.h
--
Dave Chinner
david@xxxxxxxxxxxxx
