Re: [ovirt-users] slow performance with export storage on glusterfs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

> 
> If you use Gluster as FUSE mount it's always slower than you expect it
> to be.
> If you want to get better performance out of your oVirt/Gluster storage,
> try the following: 
> 
> - create a Linux VM in your oVirt environment, assign 4/8/12 virtual
> disks (Virtual disks are located on your Gluster storage volume).
> - Boot/configure the VM, then use LVM to create VG/LV with 4 stripes
> (lvcreate -i 4) and use all 4/8/12 virtual disks as PVS.
> - then install NFS server and export LV you created in previous step,
> use the NFS export as export domain in oVirt/RHEV.
> 
> You should get wire speed when you use multiple stripes on Gluster
> storage, FUSE mount on oVirt host will fan out requests to all 4 servers.
> Gluster is very good at distributed/parallel workloads, but when you use
> direct Gluster FUSE mount for Export domain you only have one data
> stream, which is fragmented even more my multiple writes/reads that
> Gluster needs to do to save your data on all member servers.

Thanks for explanation, it is an interesting solution.

Cheers,

Jiri

> 
> 
> 
> On Mon, Nov 27, 2017 at 8:41 PM, Donny Davis <donny@xxxxxxxxxxxxxx
> <mailto:donny@xxxxxxxxxxxxxx>> wrote:
> 
>     What about mounting over nfs instead of the fuse client. Or maybe
>     libgfapi. Is that available for export domains
> 
>     On Fri, Nov 24, 2017 at 3:48 AM Jiří Sléžka <jiri.slezka@xxxxxx
>     <mailto:jiri.slezka@xxxxxx>> wrote:
> 
>         On 11/24/2017 06:41 AM, Sahina Bose wrote:
>         >
>         >
>         > On Thu, Nov 23, 2017 at 4:56 PM, Jiří Sléžka
>         <jiri.slezka@xxxxxx <mailto:jiri.slezka@xxxxxx>
>         > <mailto:jiri.slezka@xxxxxx <mailto:jiri.slezka@xxxxxx>>> wrote:
>         >
>         >     Hi,
>         >
>         >     On 11/22/2017 07:30 PM, Nir Soffer wrote:
>         >     > On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka
>         <jiri.slezka@xxxxxx <mailto:jiri.slezka@xxxxxx>
>         <mailto:jiri.slezka@xxxxxx <mailto:jiri.slezka@xxxxxx>>
>         >     > <mailto:jiri.slezka@xxxxxx <mailto:jiri.slezka@xxxxxx>
>         <mailto:jiri.slezka@xxxxxx <mailto:jiri.slezka@xxxxxx>>>> wrote:
>         >     >
>         >     >     Hi,
>         >     >
>         >     >     I am trying realize why is exporting of vm to export
>         storage on
>         >     >     glusterfs such slow.
>         >     >
>         >     >     I am using oVirt and RHV, both instalations on
>         version 4.1.7.
>         >     >
>         >     >     Hosts have dedicated nics for rhevm network - 1gbps,
>         data
>         >     storage itself
>         >     >     is on FC.
>         >     >
>         >     >     GlusterFS cluster lives separate on 4 dedicated
>         hosts. It has
>         >     slow disks
>         >     >     but I can achieve about 200-400mbit throughput in other
>         >     applications (we
>         >     >     are using it for "cold" data, backups mostly).
>         >     >
>         >     >     I am using this glusterfs cluster as backend for export
>         >     storage. When I
>         >     >     am exporting vm I can see only about 60-80mbit
>         throughput.
>         >     >
>         >     >     What could be the bottleneck here?
>         >     >
>         >     >     Could it be qemu-img utility?
>         >     >
>         >     >     vdsm      97739  0.3  0.0 354212 29148 ?        S<l 
>         15:43   0:06
>         >     >     /usr/bin/qemu-img convert -p -t none -T none -f raw
>         >     >   
>         >   
>           /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>         >     >     -O raw
>         >     >   
>         >   
>           /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>         >     >
>         >     >     Any idea how to make it work faster or what
>         throughput should I
>         >     >     expected?
>         >     >
>         >     >
>         >     > gluster storage operations are using fuse mount - so
>         every write:
>         >     > - travel to the kernel
>         >     > - travel back to the gluster fuse helper process
>         >     > - travel to all 3 replicas - replication is done on
>         client side
>         >     > - return to kernel when all writes succeeded
>         >     > - return to caller
>         >     >
>         >     > So gluster will never set any speed record.
>         >     >
>         >     > Additionally, you are copying from raw lv on FC -
>         qemu-img cannot do
>         >     > anything
>         >     > smart and avoid copying unused clusters. Instead if copies
>         >     gigabytes of
>         >     > zeros
>         >     > from FC.
>         >
>         >     ok, it does make sense
>         >
>         >     > However 7.5-10 MiB/s sounds too slow.
>         >     >
>         >     > I would try to test with dd - how much time it takes to copy
>         >     > the same image from FC to your gluster storage?
>         >     >
>         >     > dd
>         >     >
>         if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>         >     >
>         of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
>         >     > bs=8M oflag=direct status=progress
>         >
>         >     unfrotunately dd performs the same
>         >
>         >     1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
>         >
>         >
>         >     > If dd can do this faster, please ask on qemu-discuss
>         mailing list:
>         >     > https://lists.nongnu.org/mailman/listinfo/qemu-discuss
>         <https://lists.nongnu.org/mailman/listinfo/qemu-discuss>
>         >     <https://lists.nongnu.org/mailman/listinfo/qemu-discuss
>         <https://lists.nongnu.org/mailman/listinfo/qemu-discuss>>
>         >     >
>         >     > If both give similar results, I think asking in gluster
>         mailing list
>         >     > about this can help. Maybe your gluster setup can be
>         optimized.
>         >
>         >     ok, this is definitly on the gluster side. Thanks for your
>         guidance.
>         >
>         >     I will investigate the gluster side and also will try
>         Export on NFS
>         >     share.
>         >
>         >
>         > [Adding gluster users ml]
>         >
>         > Please provide "gluster volume info" output for the rhv_export
>         gluster
>         > volume and also volume profile details (refer to earlier mail
>         from Shani
>         > on how to run this) while performing the dd operation above.
> 
>         you can find all this output on https://pastebin.com/sBK01VS8
> 
>         as mentioned in other posts. Gluster cluster uses really slow
>         (green)
>         disks but without direct io it can achieve throughput around
>         400mbit/s.
> 
>         This storage is used mostly for backup purposes. It is not used
>         as a vm
>         storage.
> 
>         In my case it would be nice not to use direct io in export case
>         but I
>         understand why it might not be wise.
> 
>         Cheers,
> 
>         Jiri
> 
>         >
>         >  
>         >
>         >
>         >     Cheers,
>         >
>         >     Jiri
>         >
>         >
>         >     >
>         >     > Nir
>         >     >  
>         >     >
>         >     >
>         >     >     Cheers,
>         >     >
>         >     >     Jiri
>         >     >
>         >     >
>         >     >     _______________________________________________
>         >     >     Users mailing list
>         >     >     Users@xxxxxxxxx <mailto:Users@xxxxxxxxx>
>         <mailto:Users@xxxxxxxxx <mailto:Users@xxxxxxxxx>>
>         >     <mailto:Users@xxxxxxxxx <mailto:Users@xxxxxxxxx>
>         <mailto:Users@xxxxxxxxx <mailto:Users@xxxxxxxxx>>>
>         >     >     http://lists.ovirt.org/mailman/listinfo/users
>         <http://lists.ovirt.org/mailman/listinfo/users>
>         >     <http://lists.ovirt.org/mailman/listinfo/users
>         <http://lists.ovirt.org/mailman/listinfo/users>>
>         >     >
>         >
>         >
>         >
>         >     _______________________________________________
>         >     Users mailing list
>         >     Users@xxxxxxxxx <mailto:Users@xxxxxxxxx>
>         <mailto:Users@xxxxxxxxx <mailto:Users@xxxxxxxxx>>
>         >     http://lists.ovirt.org/mailman/listinfo/users
>         <http://lists.ovirt.org/mailman/listinfo/users>
>         >     <http://lists.ovirt.org/mailman/listinfo/users
>         <http://lists.ovirt.org/mailman/listinfo/users>>
>         >
>         >
> 
> 
>         _______________________________________________
>         Users mailing list
>         Users@xxxxxxxxx <mailto:Users@xxxxxxxxx>
>         http://lists.ovirt.org/mailman/listinfo/users
>         <http://lists.ovirt.org/mailman/listinfo/users>
> 
> 
>     _______________________________________________
>     Gluster-users mailing list
>     Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx>
>     http://lists.gluster.org/mailman/listinfo/gluster-users
>     <http://lists.gluster.org/mailman/listinfo/gluster-users>
> 
> 


Attachment: smime.p7s
Description: S/MIME Cryptographic Signature

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux