Pure transfering speed (from storage to storage, via scp): 80MB/s. As I think, gluster write perfomance in this situation (2 replica bricks) must be about 40MB/s, yes? 2013/3/12 Nikita A Kardashin <differentlocal at gmail.com> > Hello, > > I found other strange thing. > > On the dd-test (dd if=/dev/zero of=2testbin bs=1M count=1024 oflag=direct) > my volume shows only 18-19MB/s. > Full network speed is 90-110MB/s, storage speed - ~200MB/s. > > Volume type - replicated-distributed, 2 replicas, 4 nodes. Volumes mounted > via fuse with direct-io=enable option. > > Its sooo slooow, right? > > > 2013/3/5 harry mangalam <harry.mangalam at uci.edu> > >> This kind of info is surprisingly hard to obtain. The gluster docs do >> contain >> some of it, ie: >> >> <http://community.gluster.org/a/linux-kernel-tuning-for-glusterfs/> >> >> I also found well-described kernel tuning parameters in the FHGFS wiki (as >> another distibuted fs, they share some characteristics) >> >> http://www.fhgfs.com/wiki/wikka.php?wakka=StorageServerTuning >> >> and more XFS tuning filesystem params here: >> >> <http://www.mythtv.org/wiki/Optimizing_Performance#Further_Information> >> >> and here: >> < >> http://www.mysqlperformanceblog.com/2011/12/16/setting-up-xfs-the-simple- >> edition> >> >> But of course, YMMV and a number of these parameters conflict and/or have >> serious tradeoffs, as you discovered. >> >> LSI recently loaned me a Nytro SAS controller (on-card SSD-cached) which >> seems >> pretty phenomenal on a single brick (and is predicted to perform well >> based on >> their profiling), but am waiting for another node to arrive before I can >> test >> it under true gluster conditions. Anyone else tried this hardware? >> >> hjm >> >> On Tuesday, March 05, 2013 12:34:41 PM Nikita A Kardashin wrote: >> > Hello all! >> > >> > This problem is solved by me today. >> > Root of all in the incompatibility of gluster cache and kvm cache. >> > >> > Bug reproduces if KVM virtual machine created with cache=writethrough >> > (default for OpenStack) option and hosted on GlusterFS volume. If any >> other >> > (cache=writeback or cache=none with direct-io) cacher used, performance >> of >> > writing to existing file inside VM is equal to bare storage (from host >> > machine) write performance. >> > >> > I think, it must be documented in Gluster and maybe filled a bug. >> > >> > Other question. Where I can read something about gluster tuning (optimal >> > cache size, write-behind, flush-behind use cases and other)? I found >> only >> > options list, without any how-to or tested cases. >> > >> > >> > 2013/3/5 Toby Corkindale <toby.corkindale at strategicdata.com.au> >> > >> > > On 01/03/13 21:12, Brian Candler wrote: >> > >> On Fri, Mar 01, 2013 at 03:30:07PM +0600, Nikita A Kardashin wrote: >> > >>> If I try to execute above command inside virtual machine (KVM), >> > >>> first >> > >>> time all going right - about 900MB/s (cache effect, I think), >> but if >> > >>> >> > >>> I >> > >>> >> > >>> run this test again on existing file - task (dd) hungs up and >> can be >> > >>> stopped only by Ctrl+C. >> > >>> Overall virtual system latency is poor too. For example, apt-get >> > >>> upgrade upgrading system very, very slow, freezing on "Unpacking >> > >>> replacement" and other io-related steps. >> > >>> Does glusterfs have any tuning options, that can help me? >> > >> >> > >> If you are finding that processes hang or freeze indefinitely, this >> is >> > >> not >> > >> a question of "tuning", this is simply "broken". >> > >> >> > >> Anyway, you're asking the wrong person - I'm currently in the >> process of >> > >> stripping out glusterfs, although I remain interested in the project. >> > >> >> > >> I did find that KVM performed very poorly, but KVM was not my main >> > >> application and that's not why I'm abandoning it. I'm stripping out >> > >> glusterfs primarily because it's not supportable in my environment, >> > >> because >> > >> there is no documentation on how to analyse and recover from failure >> > >> scenarios which can and do happen. This point in more detail: >> > >> http://www.gluster.org/**pipermail/gluster-users/2013-** >> > >> January/035118.html< >> http://www.gluster.org/pipermail/gluster-users/2013-J >> > >> anuary/035118.html> >> > >> >> > >> The other downside of gluster was its lack of flexibility, in >> particular >> > >> the >> > >> fact that there is no usage scaling factor on bricks, so that even >> with a >> > >> simple distributed setup all your bricks have to be the same size. >> Also, >> > >> the object store feature which I wanted to use, has clearly had >> hardly >> > >> any >> > >> testing (even the RPM packages don't install properly). >> > >> >> > >> I *really* wanted to deploy gluster, because in principle I like the >> idea >> > >> of >> > >> a virtual distribution/replication system which sits on top of >> existing >> > >> local filesystems. But for storage, I need something where >> operational >> > >> supportability is at the top of the pile. >> > > >> > > I have to agree; GlusterFS has been in use here in production for a >> while, >> > > and while it mostly works, it's been fragile and documentation has >> been >> > > disappointing. Despite 3.3 being in beta for a year, it still seems to >> > > have >> > > been poorly tested. For eg, I can't believe almost no-one else noticed >> > > that >> > > the log files were busted.. nor that the bug report has been around >> for >> > > quarter of a year without being responded to or fixed. >> > > >> > > I have to ask -- what are you moving to now, Brian? >> > > >> > > -Toby >> > > >> > > >> > > ______________________________**_________________ >> > > Gluster-users mailing list >> > > Gluster-users at gluster.org >> > > http://supercolony.gluster.**org/mailman/listinfo/gluster-**users< >> http://s >> > > upercolony.gluster.org/mailman/listinfo/gluster-users> >> >> --- >> Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine >> [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 >> 415 South Circle View Dr, Irvine, CA, 92697 [shipping] >> MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) >> --- >> "Something must be done. [X] is something. Therefore, we must do it." >> Bruce Schneier, on American response to just about anything. >> > > > > -- > With best regards, > differentlocal (www.differentlocal.ru | differentlocal at gmail.com), > System administrator. > -- With best regards, differentlocal (www.differentlocal.ru | differentlocal at gmail.com), System administrator. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130312/10b849dc/attachment.html>
