Re: [ovirt-users] slow performance with export storage on glusterfs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





On Thu, Nov 23, 2017 at 4:56 PM, Jiří Sléžka <jiri.slezka@xxxxxx> wrote:
Hi,

On 11/22/2017 07:30 PM, Nir Soffer wrote:
> On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka@xxxxxx
> <mailto:jiri.slezka@xxxxxx>> wrote:
>
>     Hi,
>
>     I am trying realize why is exporting of vm to export storage on
>     glusterfs such slow.
>
>     I am using oVirt and RHV, both instalations on version 4.1.7.
>
>     Hosts have dedicated nics for rhevm network - 1gbps, data storage itself
>     is on FC.
>
>     GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks
>     but I can achieve about 200-400mbit throughput in other applications (we
>     are using it for "cold" data, backups mostly).
>
>     I am using this glusterfs cluster as backend for export storage. When I
>     am exporting vm I can see only about 60-80mbit throughput.
>
>     What could be the bottleneck here?
>
>     Could it be qemu-img utility?
>
>     vdsm      97739  0.3  0.0 354212 29148 ?        S<l  15:43   0:06
>     /usr/bin/qemu-img convert -p -t none -T none -f raw
>     /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>     -O raw
>     /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>
>     Any idea how to make it work faster or what throughput should I
>     expected?
>
>
> gluster storage operations are using fuse mount - so every write:
> - travel to the kernel
> - travel back to the gluster fuse helper process
> - travel to all 3 replicas - replication is done on client side
> - return to kernel when all writes succeeded
> - return to caller
>
> So gluster will never set any speed record.
>
> Additionally, you are copying from raw lv on FC - qemu-img cannot do
> anything
> smart and avoid copying unused clusters. Instead if copies gigabytes of
> zeros
> from FC.

ok, it does make sense

> However 7.5-10 MiB/s sounds too slow.
>
> I would try to test with dd - how much time it takes to copy
> the same image from FC to your gluster storage?
>
> dd
> if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
> bs=8M oflag=direct status=progress

unfrotunately dd performs the same

1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s


> If dd can do this faster, please ask on qemu-discuss mailing list:
> https://lists.nongnu.org/mailman/listinfo/qemu-discuss
>
> If both give similar results, I think asking in gluster mailing list
> about this can help. Maybe your gluster setup can be optimized.

ok, this is definitly on the gluster side. Thanks for your guidance.

I will investigate the gluster side and also will try Export on NFS share.

[Adding gluster users ml]

Please provide "gluster volume info" output for the rhv_export gluster volume and also volume profile details (refer to earlier mail from Shani on how to run this) while performing the dd operation above.

 

Cheers,

Jiri


>
> Nir
>  
>
>
>     Cheers,
>
>     Jiri
>
>
>     _______________________________________________
>     Users mailing list
>     Users@xxxxxxxxx <mailto:Users@xxxxxxxxx>
>     http://lists.ovirt.org/mailman/listinfo/users
>



_______________________________________________
Users mailing list
Users@xxxxxxxxx
http://lists.ovirt.org/mailman/listinfo/users


_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux