Re: [ovirt-users] ovirt glusterfs performance

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Ravi and colleagues

(apologies for hijacking this thread but I’m not sure where else to report this (and it is related).)

With gluster 3.7.10, running
#gluster volume set <volname> group virt
fails with:
volume set: failed: option : eager-lock does not exist
Did you mean eager-lock?

I had to remove the eager-lock setting from /var/lib/glusterd/groups/virt to get this to work. It seems like setting eager-lock has been removed from latest gluster. Is this correct? Either way, is there anything else I should do?

Cheers,

Roderick

On 12 Feb 2016, at 6:18 AM, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:

Hi Bill,
Can you enable virt-profile setting for your volume and see if that helps? You need to enable this optimization when you create the volume using ovrit, or use the following command for an existing volume:

#gluster volume set <volname> group virt


-Ravi


On 02/12/2016 05:22 AM, Bill James wrote:
My apologies, I'm showing how much of a noob I am.
Ignore last direct to gluster numbers, as that wasn't really glusterfs.


[root@ovirt2 test ~]# mount -t glusterfs ovirt2-ks.test.j2noc.com:/gv1 /mnt/tmp/
[root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/tmp/testfile2 bs=1M count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s

That's more how I expected, it is pointing to glusterfs performance.



On 02/11/2016 03:27 PM, Bill James wrote:
don't know if it helps, but I ran a few more tests, all from the same hardware node.

The VM:
[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s

Writing directly to gluster volume:
[root@ovirt2 test ~]# time dd if=/dev/zero of=/gluster-store/brick1/gv1/testfile bs=1M count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s


Writing to NFS volume:
[root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/storage/qa/testfile bs=1M count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s

NFS & Gluster are using the same interface. Tests were not run at same time.

This would suggest my problem isn't glusterfs, but the VM performance.



On 02/11/2016 03:13 PM, Bill James wrote:
xml attached.


On 02/11/2016 12:28 PM, Nir Soffer wrote:
On Thu, Feb 11, 2016 at 8:27 PM, Bill James <bill.james@xxxxxx> wrote:
thank you for the reply.

We setup gluster using the names associated with  NIC 2 IP.
  Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
  Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
  Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1

That's NIC 2's IP.
Using 'iftop -i eno2 -L 5 -t' :

dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s
Can you share the xml of this vm? You can find it in vdsm log,
at the time you start the vm.

Or you can do (on the host):

# virsh
virsh # list
(username: vdsm@ovirt password: shibboleth)
virsh # dumpxml vm-id

Peak rate (sent/received/total):                      281Mb 5.36Mb
282Mb
Cumulative (sent/received/total):                    1.96GB 14.6MB
1.97GB

gluster volume info gv1:
  Options Reconfigured:
  performance.write-behind-window-size: 4MB
  performance.readdir-ahead: on
  performance.cache-size: 1GB
  performance.write-behind: off

performance.write-behind: off didn't help.
Neither did any other changes I've tried.


There is no VM traffic on this VM right now except my test.



On 02/10/2016 11:55 PM, Nir Soffer wrote:
On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N <ravishankar@xxxxxxxxxx>
wrote:
+gluster-users

Does disabling 'performance.write-behind' give a better throughput?



On 02/10/2016 11:06 PM, Bill James wrote:
I'm setting up a ovirt cluster using glusterfs and noticing not stellar
performance.
Maybe my setup could use some adjustments?

3 hardware nodes running centos7.2, glusterfs 3.7.6.1, ovirt 3.6.2.6-1.
Each node has 8 spindles configured in 1 array which is split using LVM
with one logical volume for system and one for gluster.
They each have 4 NICs,
   NIC1 = ovirtmgmt
   NIC2 = gluster  (1GbE)
How do you ensure that gluster trafic is using this nic?

   NIC3 = VM traffic
How do you ensure that vm trafic is using this nic?

I tried with default glusterfs settings
And did you find any difference?

and also with:
performance.cache-size: 1GB
performance.readdir-ahead: on
performance.write-behind-window-size: 4MB

[root@ovirt3 test scripts]# gluster volume info gv1

Volume Name: gv1
Type: Replicate
Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
Options Reconfigured:
performance.cache-size: 1GB
performance.readdir-ahead: on
performance.write-behind-window-size: 4MB


Using simple dd test on VM in ovirt:
    dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
block size of 1G?!

Try 1M (our default for storage operations)

        1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3 MB/s

Another VM not in ovirt using nfs:
     dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
        1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8 MB/s


Is that expected or is there a better way to set it up to get better
performance?
Adding Niels for advice.

This email, its contents and ....
Please avoid this, this is a public mailing list, everything you write
here is public.

Nir
I'll have to look into how to remove this sig for this mailing list....

Cloud Services for Business www.j2.com
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox


This email, its contents and attachments contain information from j2 Global,
Inc. and/or its affiliates which may be privileged, confidential or
otherwise protected from disclosure. The information is intended to be for
the addressee(s) only. If you are not an addressee, any disclosure, copy,
distribution, or use of the contents of this message is prohibited. If you
have received this email in error please notify the sender by reply e-mail
and delete the original message and any copies. (c) 2015 j2 Global, Inc. All
rights reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox
are registered trademarks of j2 Global, Inc. and its affiliates.



_______________________________________________
Users mailing list
Users@xxxxxxxxx
http://lists.ovirt.org/mailman/listinfo/users

www.j2.com

This email, its contents and attachments contain information from j2 Global, Inc. and/or its affiliates which may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use of the contents of this message is prohibited. If you have received this email in error please notify the sender by reply e-mail and delete the original message and any copies. © 2015 j2 Global, Inc. All rights reserved. eFax ®, eVoice ®, Campaigner ®, FuseMail ®, KeepItSafe ® and Onebox ® are r egistered trademarks of j2 Global, Inc. and its affiliates.



_______________________________________________
Users mailing list
Users@xxxxxxxxx
http://lists.ovirt.org/mailman/listinfo/users

www.j2.com

This email, its contents and attachments contain information from j2 Global, Inc. and/or its affiliates which may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use of the contents of this message is prohibited. If you have received this email in error please notify the sender by reply e-mail and delete the original message and any copies. © 2015 j2 Global, Inc. All rights reserved. eFax ®, eVoice ®, Campaigner ®, FuseMail ®, KeepItSafe ® and Onebox ® are r egistered trademarks of j2 Global, Inc. and its affiliates.



_______________________________________________
Users mailing list
Users@xxxxxxxxx
http://lists.ovirt.org/mailman/listinfo/users



--
This message is subject to the CSIR's copyright terms and conditions, e-mail legal notice, and implemented Open Document Format (ODF) standard.
The full disclaimer details can be found at http://www.csir.co.za/disclaimer.html.


This message has been scanned for viruses and dangerous content by MailScanner,
and is believed to be clean.


Please consider the environment before printing this email.

_______________________________________________
Users mailing list
Users@xxxxxxxxx
http://lists.ovirt.org/mailman/listinfo/users


--
This message is subject to the CSIR's copyright terms and conditions, e-mail legal notice, and implemented Open Document Format (ODF) standard.
The full disclaimer details can be found at http://www.csir.co.za/disclaimer.html.


This message has been scanned for viruses and dangerous content by MailScanner,
and is believed to be clean.


Please consider the environment before printing this email.

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux