Hi Dave,
On Thu, Apr 14, 2022 at 03:18:01PM +1000, Dave Chinner wrote:
On Thu, Apr 14, 2022 at 02:00:24PM +1000, Chris Dunlop wrote:
Hi,
I have a nearly full 30T xfs filesystem that I need to grow significantly,
e.g. to, say, 256T, and potentially further in future, e.g. up to, say, 1PB.
That'll be fun. :)
Yeah, good problem to have.
So even if the current fs were to grow to a 1P or more, e.g. 30x - 60x
original, I'm still only going to be ~10% worse off in terms of agcount
than creating a large fs from scratch and copying all the data over.
Yup.
Is that really going to make a significant difference?
No.
But there will be significant differences. e.g. think of the data
layout and free space distribution of a 1PB filesystem that it is
90% full and had it's data evenly distributed throughout it's
capacity. Now consider the free space distribution of a 100TB
filesystem that has been filled to 99% and then grown by 100TB nine
times to a capacity of 90% @ 1PB. Where is all the free space?
That's right - the free space is only in the region that was
appended in the last 100TB grow operation. IOWs, 90% of the AGs are
completley full, and the newly added 10% are compeltely empty.
Yep. But growing from 30T (@ 89% full), to 256T, then possibly to 512T and
then 1P shouldn't suffer from this issue as much - perhaps assisted by
growing when it reaches, say, 60-70% rather than waiting till 90%.
In this case I might be in a decent position: the data is generally
large-ish backup files and the older ones get deleted as they age out, so
that should free up a significant amount of space in the
currently-near-full AGs over time. It sounds like this, in conjunction
with the balancing of the allocation algorithms, should end up
homogenising the usage over all the AGs.
Or, if there's a chance it'll get to 1P, would it be better to grow it out
that far now (on thin storage - ceph rbd in this case)?
On the other hand, I may want newer XFS feature goodies before I need 1P
in this thing.
IOWs, growing by more than 10x really starts to push the limits of
the algorithms regardless of the AG count it results in. It's not a
capacity thing - it's a reflection of the number of AGs with usable
free space in them and the algorithms used to find free space in
those AGs.
I'm not sure if 89% full has already put me behind, but outside of that,
doing a grow at, say, 60-70% full rather than 90+%, in conjunction with a
decent amount of data turnover, should reduce or remove this problem, no?
I.e. would you say, growing by more (possibly a lot more) than 10x would
probably be ok *IF* you're starting with (near) maximally sized AGs, and
growing the fs when it reaches, say, 60-70% full, and with data that has a
reasonable turn over cycle?
Cheers,
Chris
