That's more a limitation in the gluster protocol as it currently sits, not so much with whether you're using FUSE or libgfapi. There are probably other things you can do, though.
One might be to use dispersion instead of replication. Another might be to disable atime on the server. Per this thread ( http://lists.gnu.org/archive/html/gluster-devel/2008-03/msg00149.html ), if the server fs does not have atime enabled, gluster won't update it. That may well save you a few network round trips.
Another thing you might try is taking steps to ensure your ARP caches on your clients and bricks remain hot. (Random guess, here).
I've seen feature request tickets about bundling IOPs on the wire to opportunistically eliminate round trips, but I don't know what the status is of those.
On Thu, Jul 30, 2015 at 12:07 PM David Robinson <david.robinson@xxxxxxxxxxxxx> wrote:
Copy that. I am very aware of the small file performance issues and I
have been tracking 3.7. I am waiting for 3.7 to stabilize before I roll
it out to my production system.
My thought was that it looks like Bareos now has built in use of
libgfapi, which from my understanding should improve performance for
gluster. My issues with rsync aren't with the actual transfer speed of
the files. My issue is with the time it takes to search through the
filesystem to figure out which files to transfer. This takes an
extremely long time on 400TB of data, especially when it is going
through the directories with large numbers of small files.
I was curious if Bareos with built-in support for libgfapi would be
faster than rsync backups between gluster machines.
Anyone on the email list care to comment? Thanks in advance for any
info that can be provided.
David
------ Original Message ------
From: "André Bauer" <abauer@xxxxxxxxx>
To: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx>
Sent: 7/30/2015 11:35:21 AM
Subject: Re: Bareos backup from Gluster mount
>Hi David,
>
>i never used Bareos until now. We like to switch from Bacula in the
>future but i think this will not happen before next Ubuntu LTS release
>(16.04).
>
>I also never directly compared with rsync but i think rsync is faster
>in
>transfering because it does not have to do any compression and so on...
>
>What i can say about Bacula on Glusterfs volumes is, that copying big
>files works at reasonable speed while small files (especialy if there
>are a lot) are a bit slow, whats in Glusterfs nature until versions
>prior 3.6(?).
>
>With Glusterfs 3.6 / 3.7 this should be a bit faster in the meantime
>but
>i have no experience with the performance gains because i'm still on
>Glusterfs 3.5.5.
>
>In conclusion i still prefer Bacula over Rsync even if its slower.
>
>Some more info about Glusterfs small file performance can be found
>here:
>
>https://gluster.readthedocs.org/en/latest/Feature%20Planning/GlusterFS%203.7/Small%20File%20Performance/
>
>Regards
>André
>
>
>
>Am 30.07.2015 um 15:23 schrieb David F. Robinson:
>> Andre,
>>
>> I am looking at a backup alternative to rsnc for gluster. My storage
>>system is growing and rsync takes too long on my system (300TB). Do
>>you have any idea of the relative performance of bareos as compared to
>>that of rsync? Can it be run in a multi-threaded mode? Rsync takes an
>>extremely long time just searching the directory tree to figure out
>>what to copy. Before digging into bareos, I was wondering if you had
>>any thoughts on performance for gluster.
>>
>> David (Sent from mobile)
>>
>> ===============================
>> David F. Robinson, Ph.D.
>> President - Corvid Technologies
>> 704.799.6944 x101 [office]
>> 704.252.1310 [cell]
>> 704.799.7974 [fax]
>> David.Robinson@xxxxxxxxxxxxx
>> http://www.corvidtechnologies.com
>>
>>> On Jul 29, 2015, at 1:36 PM, André Bauer <abauer@xxxxxxxxx> wrote:
>>>
>>> We're using Bacula (Bareos is a fork of it) for backups.
>>> Never had any problems doing backups of Gluster volumes.
>>>
>>>> Am 27.07.2015 um 23:02 schrieb Ryan Clough:
>>>> Hello,
>>>>
>>>> I have cross-posted this question in the bareos-users mailing list.
>>>>
>>>> Wondering if anyone has tried this because I am unable to backup
>>>>data
>>>> that is mounted via Gluster Fuse or Gluster NFS. Basically, I have
>>>>the
>>>> Gluster volume mounted on the Bareos Director which also has the
>>>>tape
>>>> changer attached.
>>>>
>>>> Here is some information about versions:
>>>> Bareos version 14.2.2
>>>> Gluster version 3.7.2
>>>> Scientific Linux version 6.6
>>>>
>>>> Our Gluster volume consists of two nodes in distribute only. Here
>>>>is the
>>>> configuration of our volume:
>>>> [root@hgluster02 ~]# gluster volume info
>>>>
>>>> Volume Name: export_volume
>>>> Type: Distribute
>>>> Volume ID: c74cc970-31e2-4924-a244-4c70d958dadb
>>>> Status: Started
>>>> Number of Bricks: 2
>>>> Transport-type: tcp
>>>> Bricks:
>>>> Brick1: hgluster01:/gluster_data
>>>> Brick2: hgluster02:/gluster_data
>>>> Options Reconfigured:
>>>> performance.io-thread-count: 24
>>>> server.event-threads: 20
>>>> client.event-threads: 4
>>>> performance.readdir-ahead: on
>>>> features.inode-quota: on
>>>> features.quota: on
>>>> nfs.disable: off
>>>> auth.allow: 192.168.10.*,10.0.10.*,10.8.0.*,10.2.0.*,10.0.60.*
>>>> server.allow-insecure: on
>>>> server.root-squash: on
>>>> performance.read-ahead: on
>>>> features.quota-deem-statfs: on
>>>> diagnostics.brick-log-level: WARNING
>>>>
>>>> When I try to backup a directory from Gluster Fuse or Gluster NFS
>>>>mount
>>>> and I monitor the network communication I only see data being
>>>>pulled
>>>> from the hgluster01 brick. When the job finishes Bareos thinks that
>>>>it
>>>> completed without error but included in the messages for the job
>>>>are
>>>> lots and lots of permission denied errors like this:
>>>> 15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open
>>>> "/export/rclough/psdv-2014-archives-2/scan_111.tar.bak":
>>>>ERR=Permission
>>>> denied.
>>>> 15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open
>>>> "/export/rclough/psdv-2014-archives-2/run_219.tar.bak":
>>>>ERR=Permission
>>>> denied.
>>>> 15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open
>>>> "/export/rclough/psdv-2014-archives-2/scan_112.tar.bak":
>>>>ERR=Permission
>>>> denied.
>>>> 15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open
>>>> "/export/rclough/psdv-2014-archives-2/run_220.tar.bak":
>>>>ERR=Permission
>>>> denied.
>>>> 15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open
>>>> "/export/rclough/psdv-2014-archives-2/scan_114.tar.bak":
>>>>ERR=Permission
>>>> denied.
>>>>
>>>> At first I thought this might be a root-squash problem but, if I
>>>>try to
>>>> read/copy a file using the root user from the Bareos server that is
>>>> trying to do the backup, I can read files just fine.
>>>>
>>>> When the job finishes is reports that it finished "OK -- with
>>>>warnings"
>>>> but, again the log for the job is filled with "ERR=Permission
>>>>denied"
>>>> messages. In my opinion, this job did not finish OK and should be
>>>> Failed. Some of the files from the HGluster02 brick are backed up
>>>>but
>>>> all of the ones with permission errors do not. When I restore the
>>>>job,
>>>> all of the files with permission errors are empty.
>>>>
>>>> Has anyone successfully used Bareos to backup data from Gluster
>>>>mounts?
>>>> This is an important use case for us because this is the largest
>>>>single
>>>> volume that we have to prepare large amounts of data to be
>>>>archived.
>>>>
>>>> Thank you for your time,
>>>> ___________________________________________
>>>> ¯\_(ツ)_/¯
>>>> Ryan Clough
>>>> Information Systems
>>>> Decision Sciences International Corporation
>>>>
>>>><http://www.decisionsciencescorp.com/><http://www.decisionsciencescorp.com/>
>>>>
>>>> This email and its contents are confidential. If you are not the
>>>> intended recipient, please do not disclose or use the information
>>>>within
>>>> this email or its attachments. If you have received this email in
>>>>error,
>>>> please report the error to the sender by return email and delete
>>>>this
>>>> communication from your records.
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users@xxxxxxxxxxx
>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>
>>>
>>> --
>>> Mit freundlichen Grüßen
>>> André Bauer
>>>
>>> MAGIX Software GmbH
>>> André Bauer
>>> Administrator
>>> August-Bebel-Straße 48
>>> 01219 Dresden
>>> GERMANY
>>>
>>> tel.: 0351 41884875
>>> e-mail: abauer@xxxxxxxxx
>>> abauer@xxxxxxxxx <mailto:Email>
>>> www.magix.com <http://www.magix.com/>
>>>
>>>
>>> Geschäftsführer | Managing Directors: Dr. Arnd Schröder, Michael
>>>Keith
>>> Amtsgericht | Commercial Register: Berlin Charlottenburg, HRB 127205
>>>
>>> Find us on:
>>>
>>> <http://www.facebook.com/MAGIX> <http://www.twitter.com/magix_de>
>>> <http://www.youtube.com/wwwmagixcom> <http://www.magixmagazin.de>
>>>
>>>----------------------------------------------------------------------
>>> The information in this email is intended only for the addressee
>>>named
>>> above. Access to this email by anyone else is unauthorized. If you
>>>are
>>> not the intended recipient of this message any disclosure, copying,
>>> distribution or any action taken in reliance on it is prohibited and
>>> may be unlawful. MAGIX does not warrant that any attachments are
>>>free
>>> from viruses or other defects and accepts no liability for any
>>>losses
>>> resulting from infected email transmissions. Please note that any
>>> views expressed in this email may be those of the originator and do
>>> not necessarily represent the agenda of the company.
>>>
>>>----------------------------------------------------------------------
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users@xxxxxxxxxxx
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>--
>Mit freundlichen Grüßen
>André Bauer
>
>MAGIX Software GmbH
>André Bauer
>Administrator
>August-Bebel-Straße 48
>01219 Dresden
>GERMANY
>
>tel.: 0351 41884875
>e-mail: abauer@xxxxxxxxx
>abauer@xxxxxxxxx <mailto:Email>
>www.magix.com <http://www.magix.com/>
>
>
>Geschäftsführer | Managing Directors: Dr. Arnd Schröder, Michael Keith
>Amtsgericht | Commercial Register: Berlin Charlottenburg, HRB 127205
>
>Find us on:
>
><http://www.facebook.com/MAGIX> <http://www.twitter.com/magix_de>
><http://www.youtube.com/wwwmagixcom> <http://www.magixmagazin.de>
>----------------------------------------------------------------------
>The information in this email is intended only for the addressee named
>above. Access to this email by anyone else is unauthorized. If you are
>not the intended recipient of this message any disclosure, copying,
>distribution or any action taken in reliance on it is prohibited and
>may be unlawful. MAGIX does not warrant that any attachments are free
>from viruses or other defects and accepts no liability for any losses
>resulting from infected email transmissions. Please note that any
>views expressed in this email may be those of the originator and do
>not necessarily represent the agenda of the company.
>----------------------------------------------------------------------
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
