Hello
So I'm getting weird performance using XFS on a 5TB FusionIO octal (a
solid state device plugged into my pcie bus). It seems to be a newish
problem, but I can't go back to an old version of everything to prove
that, because I've only got one working Octal right now (they are a little
pricy).
At any rate, when doing random 16MB requests to a file with 16 threads,
I get about 4.5GB/s. When writing sequentially with 16 threads doing 16MB
requests, I get about 3.5GB/s -- the first time. Once the file is written
the first time, a second pass results in 4.5GB/s.
The thing is, I'm using preallocate on both types of I/O (that is, I
always preallocate the entire file whether its random or sequential). I
allocate the exact same size file in both cases, its just faster the first
time with random writes rather than sequential writes.
So if you had xdd 7.0 and an octal plugged into slot 6 of an HP DL585 G7
(running CentOS 6.3), you could replicate these test results with the
following commands (note xdd's default block size is 1024, so you'll see
everything is accounting for 1024 byte size blocks):
# Generate a set of random seek offsets within a file
data_size=$((256*8*1024*1024*1024))
rand_range=$((data_size / 16384 / 1024 - 1))
shuf -i 0-$rand_range | awk "{i += 1}; {print i-1, \$0*16384, \"16384
w 0 0\" };" > wseek_file
# Perform the random writes
numactl --cpunodebind=4 xdd -op write -target
/data/xfs-numa4/baseline-rand1 -reqsize 16384 -qd 16 -dio -verbose -bytes
2199023255552 -preallocate 2199023255552 -seek load wseek file
# Perform sequential writes
numactl --cpunodebind=4 xdd -op write -target
/data/xfs-numa4/baseline-seq1 -reqsize 16384 -qd 16 -dio -verbose -bytes
2199023255552 -preallocate 2199023255552
And so for the results I get the following (if you know xdd results):
COMBINED 1 1 16 2199023255552 131072
483.311 4549.910 271.196 0.230 67.644 write 16777216
COMBINED 1 1 16 2199023255552 131072
605.366 3632.550 216.517 0.289 53.971 write 16777216
The bandwidths here are 4549.910 MB/s for the random, and 3632.550 MB/s
for the sequential. If I run it again to the already existing file, I get:
COMBINED 1 1 16 2199023255552 131072 482.851 4554.248 271.454 0.230 66.016
write 16777216
Which is a write bandwidth of 4554.248MB/s.
Weird, right?
Cheers,
Brad
_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs
