On 1/22/2013 4:05 PM, Dave Chinner wrote:
On Tue, Jan 22, 2013 at 06:49:46AM -0600, Stan Hoeppner wrote:
On 1/21/2013 10:22 PM, Alvin Ong wrote:
Hi,
We are building a solution with a web front end for our users to store
large files.
Large files starting from the size of 500GB and above and can grow up to
1-2TB's per file.
This is the reason we are trying out XFS to see if we can get a test system
running.
Tell us more about these files. Is this simply bulk file storage?
Start at 500GB and append until 2TB? How often will the files be
appended and at what rate? I.e. will it take 3 days to append from
500GB to 2TB or take 3 months? The answer to this dictates how the
files and filesystem will fragment over time. Constantly expanding with
additional 6 spindle constituent arrays, LVM concatenation, and
xfs_growfs may leave you with an undesirable, possibly disastrous,
fragmentation pattern.
I'd say it's guaranteed, not a possibility.
We plan to use a 6+2 RAID6 to start off with. Then when it gets filled up
to maybe 60-70% we will
expand by adding another 6+2 RAID6 to the array.
The max we can grow this configuration is up to 252TB usable which should
be enough for a year.
Our requirements might grow up to 2PB in 2 years time if all goes well.
I'd not attempt growing a single XFS to the scale you're describing, via
the methods you describe. The odds of catastrophe are too great.
It's a recipe for disaster and not recommended at all.
So I have been testing all of this out on a VM running 3 vmdk's and using
LVM to create a single logical volume of the 3 disks.
I noticed that out of sdb, sdc and sdd, files keep getting written to sdc.
This is probably due to our web app creating a single folder and all files
are written under that folder.
This is the nature of the Allocation Group of XFS? Is there a way to avoid
this?
Yes.
1. Don't put all files in a single directory.
2. Use the inode32 allocator on a filesystem greater than 1TB in size.
This will cause inodes to be located in the first 1TB and files to be
allocated round robin across the AGs via rotor stepping. See page 10:
http://oss.sgi.com/projects/xfs/training/xfs_slides_06_allocators.pdf
3: Use a storage layout that is not affected by hotspots due to
filesystem locality.
That is, build the storage to the scale that you are likely to need
in the future. i.e. use all 112 disks (14x 6+2 RAID = 112 disks) to
begin with and lay the storage and filesystem out optimally
accordingly. I'd build seven 14+2 hardware RAID6 luns (112 disks)
and stripe them in RAID0, setting the XFS stripe unit to be the
width of a hardware RAID6 lun. That way sequential IO to a single
region of the disk still hits every single disk in the array, and
hotspots don't occur
If you do this, it doesn't matter if you use inode64 or inode32 for
a hotspot perspective, only a file fragmentation perspective. This
is the way XFs has been used for exactly this sort of storage for
the last 15 years....
As we will have files keep writing to the same disk thus creating a
hot spot.
Although it might not hurt us that much if we fill up a single RAID6 to
60-70% then adding another RAID6 to the mix. We could go up to a total of
14 RAID6 sets.
Again, you probably don't want to do this. Too many eggs in one basket.
You should investigate using GlusterFS to tie multiple XFS storage
servers together into a single file tree.
Another possible solution. You should talk to RedHat (says the
RedHat employee ;)....
I get the impression the "grow as you go" mindset here is probably due
to budget/cash flow issues, as well as evaluating the system at small
scale before committing to going larger. Thus I'd guess building the
112 drive system up front isn't a real possibility. And this is where
something like Gluster atop XFS would really come in handy, as it would
make "grow as you go" much more feasible, while avoiding the 'game over'
fragmentation issue with simply growing XFS in the manner described by
the OP.
Emmanuel states Gluster is slow, but that's a very relative statement.
For clients streaming single large files over GbE or slower links it
should be plenty fast. Gluster and similar network file systems tend to
be slow with metadata intensive or transactional workloads.
Running an xfs_repair on a single filesystem denies all access, and with
a 252TB XFS this could take some time.
For a filesystem with 1-2TB files, it'll take 30s to run. That's not
an issue.
For some reason I was thinking data size instead of metadata. With only
a few hundred to low thousand files it would be quick indeed, a non issue.
Is LVM a good choice of doing this configuration? Or do you have a better
recommendation?
The reason we thought LVM would be good was so that we could easily grow
XFS.
Why not do the concatenation within the SAN array controller?
Same problem as LVM concatenation. Hot spots.
I was simply suggesting hardware vs software concatenation here
unrelated to his current flawed expansion path idea, as his SAN
controller probably has some nice features and performance here.
Is I was to use the 8-disk RAID6 array with a 256kB stripe size will have a
sunit of 512 and a swidth of (8-2)*512=3072.
So a 256KB strip and a 1.5MB stripe. With RAID6 RMW? I wouldn't
recommend this.
Large files, sequential IO, there will be no RMW cycles in the RAID.
The write cache of RAID controller will do the aggregation of
individual IOs into full stripe writes just fine.
It appears most of your writes will be appends, meaning little
allocation, which means little stripe aligned write out. Here you are
trying to optimize for large IOs which would be fine if you had an all
or mostly allocation workload, but you don't. You have an append heavy
workload.
Using large strips (stripe units, chunks) with parity RAID, especially
RAID6, will simply murder your append performance due to massive
read-modify-write operations on large strips.
No, that's wrong. sequential IO will always fill full stripes in the
cache, so RMW cycles simple will not happen. Remember that RMW
occurs whenteh cache has to be flushed to the back end disks, not
when writes come in to the front end cache....
With RAID6 with a mostly append workload, you should be using a small
strip size. This has been discussed here at length and the consensus is
anything over a 32KB strip size doesn't improve performance, but can
hurt performance, especially with parity RAID. Thus you should create
your 6+2 arrays with a 32KB strip and (6*32)=192KB stripe, and create
your XFS with "-d su=32k,sw=6". This should yield significantly better
append performance.
That's a tuning for an IOPS intensive workload, not a large scale,
large file storage workloads.
While sequential writes are an append workload, it's an append
workload that the RAID controller is optimised to avoid causing RMW
cycles for. As such, the above is bad advice for large files with
sequential IO workloads. Large files, large fielsystem, sequential
IO is ideal for large RAID6 widths....
Yes, of course. WRT XFS you've drilled "allocation=aligned" and "non
allocation=unaligned" so thoroughly into my head that I failed to
actually think for a second about what the hardware does with this type
of large append data stream. I feel a bit silly making this juvenile
oversight. Won't happen again. ;)
External log devices are for systems that modify metadata at rates of
hundreds of IOs per second. So don't specify a log device.
Even at hundreds of thousands of IOs per second, external logs don't
provide much in way of benefit thanks to delayed logging. The only
reason for using an external log these days is a fsync heavy or
synchronous write workload. And in most cases a BBWC means even
those worklaods don't need an external log...
Which bloke provided us with this journal magic code again? Can't
recall his name... ;)
