On Mon, Jun 17, 2024 at 03:25:27PM -0700, Krister Johansen wrote: > On Fri, Jun 14, 2024 at 10:45:37AM +1000, Dave Chinner wrote: > > On Thu, Jun 13, 2024 at 01:27:09PM -0700, Krister Johansen wrote: > > > I managed to work out a reproducer for the problem. Debugging that, the > > > steps Gao outlined turned out to be essentially what was necessary to > > > get the problem to happen repeatably. > > > > > > 1. Allocate almost all of the space in an AG > > > 2. Free and reallocate that space to fragement it so the freespace > > > b-trees are just about to split. > > > 3. Allocate blocks in a file such that the next extent allocated for > > > that file will cause its bmbt to get converted from an inline extent to > > > a b-tree. > > > 4. Free space such that the free-space btrees have a contiguous extent > > > with a busy portion on either end > > > 5. Allocate the portion in the middle, splitting the extent and > > > triggering a b-tree split. > > > > Do you have a script that sets up this precondition reliably? > > It sounds like it can be done from a known filesystem config. If you > > do have a script, can you share it? Or maybe even better, turn it > > into an fstest? > > I do have a script that reproduces the problem. At the moment it is in > a pretty embarrasing state. I'm happy to clean it up a bit and share > it, or try to turn it into a fstest, or both. The script currently > creates small loop devices to generate a filesystem layout that's a > little easier to work with. Is it considered acceptable to have a > fstest create a filesystem with a particular geometry? Absolutely. We do that quite a bit in fstests - if you just need one filesystem for this, then _scratch_mkfs_sized() can be used to make a filesystem of a specific size, or you can use _scratch_mkfs and pass the options you want directly to it. An example of this is xfs/227. Otherwise you can create loop devices and use _mkfs_dev() to create the filesystems you need on them. An example of this is xfs/074. > (And would you > consider taking a patch to let mkfs.xfs --unsupported take both size and > agsize arguments so the overall filesystem size and the per-ag size > could be set by a test?) We can already do that, but it's been placed behind a strong environmental guard to restrict such configurations to the fstests environment: # TEST_DIR=foo TEST_DEV=foo QA_CHECK_FS=foo mkfs.xfs -dagcount=2,size=32m,file,name=foo will bypass the "300MB minimum size" checks and allow you to make a filesystem image with 2x16MB AGs (i.e. a 32MB filesystem) ready to have a loop device created over it. If this is run inside a fstest, then you don't need the env variables because they are already set. > > > On older kernels this is all it takes. After the AG-aware allocator > > > changes I also need to start the allocation in the highest numbered AG > > > available while inducing lock contention in the lower numbered AGs. > > > > Ah, so you have to perform a DOS on the lower AGFs so that the > > attempts made by the xfs_alloc_vextent_start_ag() to trylock the > > lower AGFs once it finds it cannot allocate in the highest AG > > anymore also fail. > > > > That was one of the changes made in the perag aware allocator > > rework; it added full-range AG iteration when XFS_ALLOC_FLAG_TRYLOCK > > is set because we can't deadlock on reverse order AGF locking when > > using trylocks. > > > > However, if the trylock iteration fails, it then sets the restart AG > > to the minimum AG be can wait for without deadlocking, removes the > > trylock and restarts the iteration. Hence you've had to create AGF > > lock contention to force the allocator back to being restricted by > > the AGF locking orders. > > The other thing that I really appreciated here is that the patchset > cleaned up a bunch of the different allocation functions and made > everything easier to read and follow. Thanks for that as well. > > > Is this new behaviour sufficient to mitigate the problem being seen > > with this database workload? Has it been tested with kernels that > > have those changes, and if so did it have any impact on the > > frequency of the issue occurring? > > I don't have a good answer for this yet. The team is planning to start > migrating later in the year and this will probably run through to next > year. I'll have that information eventually and will share it when I > do, but don't know yet. OK, that's fine. We still need to fix the issue, I just wanted to get an idea of the urgency of the situation. > Aside from the script, other synethtic > load-tests have not been successful in reproducing the problemr. That > may be the result of the databases that are spun up for load testing not > having filesystems that as full and fragmented as the production ones. Yeah, that's typical when triaging these sorts of issues. We've had issues like this in the past with scyallaDB w.r.t. extent size hint based allocation occasionally failing at ~70% filesystem capacity. Situations like this require a very particular free space pattern to trigger, and it tends to be rare that they are hit in either test or production systems. Having a synthetic reproducer for such an issue is like finding gold. :) -Dave. -- Dave Chinner david@xxxxxxxxxxxxx
