Thanks Geoff.
It is always good to get an external opinion on where we stand.
Geoff Kassel wrote:
Hi Shehjar, I feel I should comment on part of your reply to Gordan's
email.
Finally - which translators are deemed stable (no know issues -
memory leaks/bloat, crashes, corruption, etc.)?
We can definitely vouch for a higher degree of stability of the
releases. Otherwise, I dont think there is any performance
translator we can call completely stable/mature because of the
roadmap we have for constantly upgrading algorithms, functionality,
etc.
When will the Gluster team be able to deliver a stable, mature, and
reliable version of GlusterFS?
Continuing from what I said earlier, the fact that GlusterFS releases
work in a stable manner is shown by the deployments among our
customers.
At the same time, are we satisfied with the experience of non-paying
users?
No. I accept there are bottlenecks in our processes. We
acknowledge that and have been working on fixing them. Most visible
aspect of that for users is the move to using bugzilla at
bugs.gluster.com. The earlier setup at Savannah just wasnt scaling.
Personally, in just a few weeks, I am finding handling bugs through
this portal much faster and streamlined than earlier.
I have been using GlusterFS since the v1.3.x days, and I have yet to
see a version since then that doesn't crash at least once a day from
just load on even the simplest configurations.
Then there's the data corruption bug of the early 2.0.0 releases,
which has kept me (and no doubt others) from upgrading to these
releases.
I have read about the Gluster QA team, but quite frankly, I have yet
to see the fruits of this team's work. Letting through a bug of that
magnitude in a major release blew a lot of trust I had in the
Gluster team's QA process.
When will regression tests be used? It's been months now since this
bug, and still I don't see any sign of the use of this simple,
industry-standard technique to minimise the risk of such issues
slipping through again.
I think the QA folks have done some really good work in stabilizing
GlusterFS over the last year or so. The result is there to see in the
2.0.X releases.
Why wasn't this prioritised after such a disasterous bug?
It could've been for any number of reasons ranging from problems with
reproducing it, limited functionality for managing bug reports in
Savannah to even the general constraints of being a commercial
open-source project.
Still, I understand your problems are more important to you than the
problems being faced by other users, I'd so appreciate if you'd give our
bugzilla-based setup a chance at handling this bug. Or, let me
know if you've already filed a report.
When will this even show up on the roadmap?
The QA team is already working on just such a testing and
regression framework.
Thanks
Shehjar
Geoff.
On Sat, 4 Jul 2009, Shehjar Tikoo wrote:
Gordan Bobic wrote:
Just reading through the wiki on this and a few things are
unclear, so I'm hoping someone can clarify.
1) readahead
- Is there any point in using this on systems where the
interconnect <= 1Gb/s? The wiki implies there is no point in
this, but doesn't quite state it explicitly.
I am pretty sure it helps. The question of using read-ahead is more
of a question related to the workload rather than the
interconnect, for eg. it'll be useful for sequential reading,
without any doubts. Of course, there can be cases where excessive
read-ahead chokes the 100 Mib/s link, but then read-ahead can be
configured to reduce its utilization of the network by reducing the
page-count option.
- Is there any point in using this on a server that is also it's
own client when use with replicate/afr? I'm guessing there isn't
since the local fs will be doing it's own read-ahead but I'd
like some confirmation on that.
No. Generally, read-ahead will be most beneficial only on the
client side since it helps avoid the need to go to the network when
an application does need the data already read-ahead. Yes, on the
server side, on-disk file systems read-ahead already does it best.
In your setup above, in case the system has more than a few
CPUs/cores, it might be possible to get a little better performance
while using io-threads on the client. That'll make it possible to
offload the read-ahead to an io-thread without blocking the main
glusterfs thread. Then, the benefit of read-ahead + io-threads
might show up when the data is actually needed, and could be served
without a kernel entry/exit for file system call.
2) io-threads
Is this (usefully) applicable on the client side?
It is. Using io-threads on the client side helps offload the
processing of individual file operations onto a separate thread,
freeing up the main thread to perform other tasks. This is
especially applicable when using io-threads under a write-behind
and/or read-ahead translators where the write-behind and read-ahead
requests, i.e. background or asynchronous requests essentially,
can be offloaded to the threads while freeing up the main glusterfs
thread to handle sync requests, i.e. requests that could make the
application block on a syscall.
Also, using io-threads on client side could help in performing
network IO in a separate thread, again freeing up the main thread
for other in-band tasks.
Then again, if the workload is not concurrent in terms of number of
processes or number of files/dirs, then io-threads might not help
much.
3) io-cache
The wiki page has the same paragraph pasted for both io-threads
and io-cache. Are they the same thing, or is this a documentation
bug?
No, they're not the same. The documentation is still in a flux.
Hope this version will help:
http://www.gluster.org/docs/index.php/Translators_options
What does io-cache do?
io-cache is a translator that caches data from files so that future
references do not lead to network requests. It is generally used
along with read-ahead so that the data that gets read ahead or any
data that gets read, for that matter, will be available from the
local client cache. We're also working on incorporating support for
write buffering in io-cache so that write operations can also
benefit from local buffering until a point in time suitable for
actual transmission to the server.
Finally - which translators are deemed stable (no know issues -
memory leaks/bloat, crashes, corruption, etc.)?
We can definitely vouch for a higher degree of stability of the
releases. Otherwise, I dont think there is any performance
translator we can call completely stable/mature because of the
roadmap we have for constantly upgrading algorithms, functionality,
etc.
Any particular suggestions on which performance translator
combination would be good to apply for a shared root AFR over a
WAN? I already have read-subvolume set to the local mirror, but
any improvement is welcome when latencies soar to 100ms and b/w
gets hammered down to 1-2.5 Mb/s.
WANs are generally characterised as having a large bandwidth-delay
product. That basically means, for good throughput, we should be
pipelining as much data as possible over the link, so that the long
latency overhead can be mitigated or amortised by sending larger
amount of data for the same fixed overhead.
That said, what particular workload is it that gives you a
throughput of 1-2.5 Mb/s?
When you say "latencies soar to 100ms", does that mean, these are
just unusual spikes or is that the normal latency observed?
It'd help to see your volfiles and how the performance translators
are arranged.
Another thing - when a node works standalone in AFR, performance
is pretty good, but as soon as a peer node joins, even though
the original node is the primary, performance degrades on the
primary node quite significantly, even though the interconnect is
direct gigabit, which shouldn't be adding any particular latency
(< 0.1ms) or overheads, especially on the primary node. Is there
any particular reason for this degradation? It's OK in normal
usage, but some operations (e.g. building an big bootstrapping
initrd (50MB compressed, including all the gernel drivers) takes
nearly 10x longer when the peers join than when the node is
standalone. I expected some degradation, but only on the order of
added network latency, and this is way, way more. I tried with
and without direct-io=off, and that didn't make a great amount of
difference. Which performance translators are likely to help
with this use case?
I think Vikas will be able to answer that better.
-Shehjar
Gordan
_______________________________________________ Gluster-devel
mailing list Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel
_______________________________________________ Gluster-devel
mailing list Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel
