Am 21.01.2010 17:27, schrieb Jiann-Ming Su: > On Wed, Jan 20, 2010 at 5:49 PM, pkoelle<pkoelle at gmail.com> wrote: >> >> This looks suspicious. Is this a gfs2 mount? what kind of performance do you >> get wen running bonnie++ directly against /mnt/gluster/gfs2? >> > > Bonnie++ command: ./bonnie++ -u 99:99 -d /mnt/gluster/gfs2 -s 2000 > > Version 1.96 ------Sequential Output------ --Sequential Input- --Random- > Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP > localhost.loc 2000M 74 99 39432 12 27308 10 836 99 65104 9 351.5 11 > Latency 179ms 1207ms 429ms 109ms 109ms 215ms > Version 1.96 ------Sequential Create------ --------Random Create-------- > localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > 16 23991 81 +++++ +++ +++++ +++ 17007 57 +++++ +++ +++++ +++ > Latency 1276us 1189us 787us 1183us 438us 2813us > > Hi Jiann-Ming, IO for your local disk doesn't look very good either. You are CPU bound when using putchar() and you get only 74K/sec with 99% CPU. For reference, I get 68808K/sec with 86% CPU on a XEN vdev on top of cheap SATA. With a two-node mirror and glusterFS I still get 31886K/sec with 41% CPU (these are the -Per Chr- values from bonnie++). >> >> Can you leave out some of the performance/* translators and also try a >> "real" network interface with lower MTU values to see of that helps? I had >> terrible results with bigger MTU (6500) and bigger values for >> net.core.(r/w)mem_max in /etc/sysctl.conf. IOPS where in the range of 5/sec. >> > > GlusterFS server and client are running on the same system, so there's > no network latency involved at all. I don't see how introducing the > network layer into this would help. As a matter of fact, with cluster filesystems things get much more complicated for a single write() or read(). I haven't investigated this throughoutly but my reasoning is: Data goes through network buffers, when the buffer is full an interrupt is triggered and data is copied over. With large buffers this kills small reads and IOPS. Blame it on gluster or fuse, but > the reality is glusterfs read performance, in a simple base config, is > 1/10th of the native file system. In order to have this be worthwhile > at all, I'd have to stripe the data across at least 10 servers to get > the same read performance. This may be a legitimate setup, but it's > not what I need out of gluster. I need the replication capabilities > out of gluster and the ability for the node to read the local copy > reasonably fast. I can't reproduce. I get reasonable performance given the hardware,network and all. Do you know any better cluster filesystems? Do you have experience with gfs2 or ocfs? cheers Paul > >
