Hello
If you use Gluster as FUSE mount it's always slower than you expect it to be.
If you want to get better performance out of your oVirt/Gluster storage, try the following:
- create a Linux VM in your oVirt environment, assign 4/8/12 virtual disks (Virtual disks are located on your Gluster storage volume).
- Boot/configure the VM, then use LVM to create VG/LV with 4 stripes (lvcreate -i 4) and use all 4/8/12 virtual disks as PVS.
- then install NFS server and export LV you created in previous step, use the NFS export as export domain in oVirt/RHEV.
You should get wire speed when you use multiple stripes on Gluster storage, FUSE mount on oVirt host will fan out requests to all 4 servers.
Gluster is very good at distributed/parallel workloads, but when you use direct Gluster FUSE mount for Export domain you only have one data stream, which is fragmented even more my multiple writes/reads that Gluster needs to do to save your data on all member servers.
On Mon, Nov 27, 2017 at 8:41 PM, Donny Davis <donny@xxxxxxxxxxxxxx> wrote:
What about mounting over nfs instead of the fuse client. Or maybe libgfapi. Is that available for export domainsOn Fri, Nov 24, 2017 at 3:48 AM Jiří Sléžka <jiri.slezka@xxxxxx> wrote:http://lists.ovirt.org/On 11/24/2017 06:41 AM, Sahina Bose wrote:
>
>
> On Thu, Nov 23, 2017 at 4:56 PM, Jiří Sléžka <jiri.slezka@xxxxxx
> <mailto:jiri.slezka@xxxxxx>> wrote:
>
> Hi,
>
> On 11/22/2017 07:30 PM, Nir Soffer wrote:
> > On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka@xxxxxx <mailto:jiri.slezka@xxxxxx>
> > <mailto:jiri.slezka@xxxxxx <mailto:jiri.slezka@xxxxxx>>> wrote:
> >
> > Hi,
> >
> > I am trying realize why is exporting of vm to export storage on
> > glusterfs such slow.
> >
> > I am using oVirt and RHV, both instalations on version 4.1.7.
> >
> > Hosts have dedicated nics for rhevm network - 1gbps, data
> storage itself
> > is on FC.
> >
> > GlusterFS cluster lives separate on 4 dedicated hosts. It has
> slow disks
> > but I can achieve about 200-400mbit throughput in other
> applications (we
> > are using it for "cold" data, backups mostly).
> >
> > I am using this glusterfs cluster as backend for export
> storage. When I
> > am exporting vm I can see only about 60-80mbit throughput.
> >
> > What could be the bottleneck here?
> >
> > Could it be qemu-img utility?
> >
> > vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06
> > /usr/bin/qemu-img convert -p -t none -T none -f raw
> >
> /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ ff3cd56a-1005-4426-8137- 8f422c0b47c1/images/ba42cbcc- c068-4df8-af3d-00f2077b1e27/ c57acd5f-d6cf-48cc-ad0c- 4a7d979c0c1e
> > -O raw
> >
> /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__ export/81094499-a392-4ea2- b081-7c6288fbb636/images/ ba42cbcc-c068-4df8-af3d- 00f2077b1e27/c57acd5f-d6cf- 48cc-ad0c-4a7d979c0c1e
> >
> > Any idea how to make it work faster or what throughput should I
> > expected?
> >
> >
> > gluster storage operations are using fuse mount - so every write:
> > - travel to the kernel
> > - travel back to the gluster fuse helper process
> > - travel to all 3 replicas - replication is done on client side
> > - return to kernel when all writes succeeded
> > - return to caller
> >
> > So gluster will never set any speed record.
> >
> > Additionally, you are copying from raw lv on FC - qemu-img cannot do
> > anything
> > smart and avoid copying unused clusters. Instead if copies
> gigabytes of
> > zeros
> > from FC.
>
> ok, it does make sense
>
> > However 7.5-10 MiB/s sounds too slow.
> >
> > I would try to test with dd - how much time it takes to copy
> > the same image from FC to your gluster storage?
> >
> > dd
> > if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ ff3cd56a-1005-4426-8137- 8f422c0b47c1/images/ba42cbcc- c068-4df8-af3d-00f2077b1e27/ c57acd5f-d6cf-48cc-ad0c- 4a7d979c0c1e
> > of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__ export/81094499-a392-4ea2- b081-7c6288fbb636/__test__
> > bs=8M oflag=direct status=progress
>
> unfrotunately dd performs the same
>
> 1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
>
>
> > If dd can do this faster, please ask on qemu-discuss mailing list:
> > https://lists.nongnu.org/mailman/listinfo/qemu-discuss
> <https://lists.nongnu.org/mailman/listinfo/qemu-discuss >
> >
> > If both give similar results, I think asking in gluster mailing list
> > about this can help. Maybe your gluster setup can be optimized.
>
> ok, this is definitly on the gluster side. Thanks for your guidance.
>
> I will investigate the gluster side and also will try Export on NFS
> share.
>
>
> [Adding gluster users ml]
>
> Please provide "gluster volume info" output for the rhv_export gluster
> volume and also volume profile details (refer to earlier mail from Shani
> on how to run this) while performing the dd operation above.
you can find all this output on https://pastebin.com/sBK01VS8
as mentioned in other posts. Gluster cluster uses really slow (green)
disks but without direct io it can achieve throughput around 400mbit/s.
This storage is used mostly for backup purposes. It is not used as a vm
storage.
In my case it would be nice not to use direct io in export case but I
understand why it might not be wise.
Cheers,
Jiri
>
>
>
>
> Cheers,
>
> Jiri
>
>
> >
> > Nir
> >
> >
> >
> > Cheers,
> >
> > Jiri
> >
> >
> > _______________________________________________
> > Users mailing list
> > Users@xxxxxxxxx <mailto:Users@xxxxxxxxx>
> <mailto:Users@xxxxxxxxx <mailto:Users@xxxxxxxxx>>
> > http://lists.ovirt.org/mailman/listinfo/users
> <http://lists.ovirt.org/mailman/listinfo/users >
> >
>
>
>
> _______________________________________________
> Users mailing list
> Users@xxxxxxxxx <mailto:Users@xxxxxxxxx>
> http://lists.ovirt.org/mailman/listinfo/users
> <http://lists.ovirt.org/mailman/listinfo/users >
>
>
_______________________________________________
Users mailing list
Users@xxxxxxxxx
mailman/listinfo/users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
