On 21/10/2018 17.28, Dave Chinner wrote:
On Sun, Oct 21, 2018 at 11:55:47AM +0300, Avi Kivity wrote:
On 19/10/2018 10.51, Dave Chinner wrote:
On Thu, Oct 18, 2018 at 04:36:42PM +0300, Avi Kivity wrote:
On 18/10/2018 14.00, Avi Kivity wrote:
Can I get access to the metadump to dig around in the filesystem
directly so I can see how everything has ended up laid out? that
will help me work out what is actually occurring and determine if
mkfs/mount options can address the problem or whether deeper
allocator algorithm changes may be necessary....
I will ask permission to share the dump.
I'll send you a link privately.
Thanks - I've started looking at this - the information here is
just layout stuff - I'm omitted filenames and anything else that
might be identifying from the output.
Looking at a commit log file:
stat.size = 33554432
stat.blocks = 34720
fsxattr.xflags = 0x800 [----------e-----]
fsxattr.projid = 0
fsxattr.extsize = 33554432
fsxattr.cowextsize = 0
fsxattr.nextents = 14
and the layout:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS
0: [0..4079]: 2646677520..2646681599 22 (95606800..95610879) 4080 001010
1: [4080..8159]: 2643130384..2643134463 22 (92059664..92063743) 4080 001010
2: [8160..12239]: 2642124816..2642128895 22 (91054096..91058175) 4080 001010
3: [12240..16319]: 2640666640..2640670719 22 (89595920..89599999) 4080 001010
4: [16320..18367]: 2640523264..2640525311 22 (89452544..89454591) 2048 000000
5: [18368..20415]: 2640119808..2640121855 22 (89049088..89051135) 2048 000000
6: [20416..21287]: 2639874064..2639874935 22 (88803344..88804215) 872 001111
7: [21288..21295]: 2639874936..2639874943 22 (88804216..88804223) 8 011111
8: [21296..24495]: 2639874944..2639878143 22 (88804224..88807423) 3200 001010
9: [24496..26543]: 2639427584..2639429631 22 (88356864..88358911) 2048 000000
10: [26544..28591]: 2638981120..2638983167 22 (87910400..87912447) 2048 000000
11: [28592..30639]: 2638770176..2638772223 22 (87699456..87701503) 2048 000000
12: [30640..31279]: 2638247952..2638248591 22 (87177232..87177871) 640 001111
13: [31280..34719]: 2638248592..2638252031 22 (87177872..87181311) 3440 011010
14: [34720..65535]: hole 30816
The first thing I note is the initial allocations are just short of
2MB and so the extent size hint is, indeed, being truncated here
according to contiguous free space limitations. I had thought that
should occur from reading the code, but it's complex and I wasn't
100% certain what minimum allocation length would be used.
Looking at the system batchlog files, I'm guessing the filesystem
ran out of contiguous 32MB free space extents some time around
September 25. The *Data.db files from 24 Sep and earlier then are
all nice 32MB extents, from 25 sep onwards they never make the full
32MB (30-31MB max). eg, good:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS
0: [0..65535]: 350524552..350590087 3 (2651272..2716807) 65536 001111
1: [65536..131071]: 353378024..353443559 3 (5504744..5570279) 65536 001111
2: [131072..196607]: 355147016..355212551 3 (7273736..7339271) 65536 001111
3: [196608..262143]: 360029416..360094951 3 (12156136..12221671) 65536 001111
4: [262144..327679]: 362244144..362309679 3 (14370864..14436399) 65536 001111
5: [327680..343415]: 365809456..365825191 3 (17936176..17951911) 15736 001111
bad:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS
0: [0..64127]: 512855496..512919623 4 (49024456..49088583) 64128 001111
1: [64128..128247]: 266567048..266631167 2 (34651528..34715647) 64120 001010
2: [128248..142327]: 264401888..264415967 2 (32486368..32500447) 14080 001111
So extent size is a hint but the extent alignment is a hard
requirement.
No, physical alignment is being ignored here, too. THose flags on
the end?
FLAG Values:
0100000 Shared extent
0010000 Unwritten preallocated extent
0001000 Doesn't begin on stripe unit
0000100 Doesn't end on stripe unit
0000010 Doesn't begin on stripe width
0000001 Doesn't end on stripe width
When you have 001111, the allocation was completely unaligned.
When you have 001010, the tail is stripe aligned
When you ahve 000000, the head and tail are stripe aligned
As you can see, there is a mix of aligned, tail aligned and
completely unaligned extents.
So, no, XFS is droping both size hints and alignment hints when
it starts running out of aligned contiguous free space extents.
You are right; I searched for and found some files that where
head-aligned, and jumped to conclusions, but there are many that are
not. Those head-aligned files probably belonged to an era in that
filesystem's life where head-aligned extents less than 1MB were available.
Ok, so the results is not perfect, but there are now huge contiguous
free space extents available again - ~70% of the free space is now
contiguous extents >=32MB in length. There's every chance that the
fs would confinue to help reform large contiguous free spaces as the
database files come and go now, as long as the snapshot problem is
dealt with.
So, what's the problem? Well, it's simply that the workload is
mixing data with vastly different temporal characteristics in the
same physical locality. Every half an hour, a set of ~100 smallish
files are written into a new directory which lands them at the low
endof the largest free space extent in that AG. Each new snapshot
directory ends up in a different AG, so it slowly spreads the
snapshots across all the AGs in the filesystem.
Not exactly - those snapshots are hard links into the live database
files, which eventually get removed. Usually, small files get
removed early, but with the snapshots they get to live forever.
They might be created as hard links, but the effect when the
orginal database file links are removed is the same - the snapshotted
data lives forever, interleaved amongst short term data.
Yes.
Each snapshot effective appends to the current working area in the
AG, chopping it out of the largest contiguous free space. By the
time the next snapshot in that AG comes around, there's other new
short term data between the old snapshot and the new one. The new
snapshot chops up the largest freespace, and on goes the cycle.
Eventually the short term data between the snapshots gets removed,
but this doesn't reform large contiguous free spaces because the
snapshot data is in the way. And so this cycle continues with the
snapshot data chopping up the largest freespace extents in the
filesystem until there's not more large free space extents to be
found.
The solution is to manage the snapshot data better. We need to keep
all the long term data physically isolated from the short term data
so they don't fragment free space. A short term application level
solution would require migrating the snapshot data out of the
filesystem to somewhere else and point to it with symlinks.
Snapshots should not live forever on the disk. The procedure is to
create a snapshot, copy it away, and then delete the snapshot. It's
okay to let snapshots live for a while, but not all of them and not
without a bound on their lifetime.
The filesystem did have a role in this, by requiring alignment of
the extent to the RAID stripe size.
No, in the end it didn't.
Right.
Now, given that this was a RAID
with one member, alignment is pointless, but most of our deployments
are to RAID arrays with >1 members, and alignment does save 12.5% of
IOPS compared to un-aligned extents for compactions and writes (our
scans/writes use 128k buffers, and the alignment is to 1MB). The
database caused the problem by indirectly requiring 1MB alignment
for files that are much smaller than 1MB, and the user contributed
to the problem by causing millions of such small files to be kept.
*nod*
<ding>
Hold on....
<rummage in code>
....we already have an interface so setting those sorts of hints.
fcntl(F_SET_RW_HINT, rw_hint)
/*
* Valid hint values for F_{GET,SET}_RW_HINT. 0 is "not set", or can be
* used to clear any hints previously set.
*/
#define RWF_WRITE_LIFE_NOT_SET 0
#define RWH_WRITE_LIFE_NONE 1
#define RWH_WRITE_LIFE_SHORT 2
#define RWH_WRITE_LIFE_MEDIUM 3
#define RWH_WRITE_LIFE_LONG 4
#define RWH_WRITE_LIFE_EXTREME 5
Avi, does this sound like something that you could use to
classify the different types of data the data base writes out?
So long as the penalty for a mis-classification is not too large, we
can for sure.
OK.
I'll need to have a think about how to apply this to the allocator
policy algorithms before going any further, but I suspect making use
of this hint interface will allow us prevent interleaving of short
and long term data so avoid the freespace fragmentation it is
causing here....
IIUC, the problem (of having ENOSPC on a 10% used disk) is not
fragmentation per se, it's the alignment requirement.
Which, as I've noted above, alignment is a hint, not a requirement.
To take it to
extreme, a 1TB disk can only hold a million files if those files
must be aligned to 1MB, even if everything is perfectly laid out.
For sure fragmentation would have degraded performance sooner or
later, but that's not as bad as that ENOSPC.
What it comes down to is that having looked into it, I don't know
why that ENOSPC error occurred.
Alignment didn't cause it because alignment was being dropped - that
just caused free space fragmentation. Extent size hints didn't
cause it because the size hints were dropped - that just caused
freespace fragmentation. A lack of free space
didn't cause it, because there was heaps of free space in all
allocation groups.
But something tickled a corner case that triggered an allocation
failure that was interpretted as ENOSPC rather than retrying the
allocation. Until I can reproduce the ENOSPC allocation failure
(and I tried!) then it'll be a mystery as to what caused it.
The user reported the error happening multiple times, taking many hours
to reproduce, but on more than one node. So it's an obscure corner case
but not obscure enough to be a one-off event.
I've asked the user to regularly trim their snapshots (they we're not
aware of the snapshots actually - they were performed as a side effect
of a TRUNCATE operation), and we'll remove the default extent hint for
small files. I'll also consider noalign - the 12.5% reduction in IOPS is
perhaps not worth the fragmentation it generates.
entire file. But I think that, given that the extent size is treated
as a hint (or so I infer from the fact that we have <32MB extents),
so should the alignment. Perhaps allocation with a hint should be
performed in two passes, first trying to match size and alignment,
and second relaxing both restrictions.
I think I already mentioned there were 5 separate attmepts to
allocate, each failure reducing restrictions:
1. extent sized and contiguous to adjacent block in file
2. extent sized and aligned, at higher block in AG
3. extent sized, not aligned, at higher block in AG
4. >= minimum length, not aligned, anywhere in AG >= target AG
Surprised at this one. Won't it skew usage in high AGs?
Perhaps it's rare enough not to matter.
Perhaps those higher-block/higher-AG heuristics can be improved for
non-rotational media.
5. minimum length, not aligned, in any AG
Thanks for your patience in helping me understand this issue.
Avi
Cheers,
Dave.
