Re: Sudden, dramatic performance drops with Glusterfs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




Hi,

I had complete crash with gluster and ZFS. Installed ext4 instead and it worked much more better.

Regards,
Birgit

On 06/11/19 11:50, Michael Rightmire wrote:
Hello list!

I'm new to Glusterfs in general. We have chosen to use it as our distributed file system on a new set of HA file servers.

The setup is:
2 SUPERMICRO SuperStorage Server 6049PE1CR36L with 24-4TB spinning disks and NVMe for cache and slog.
HBA not RAID card
Ubuntu 18.04 server (on both systems)
ZFS filestorage
Glusterfs 5.10

Step one was to install Ubuntu, ZFS, and gluster. This all went without issue.
We have 3 ZFS raidz2 identical on both servers
We have three glusterfs mirrored volumes - 1 attached to each raidz on each server. I.e.

And mounted the gluster volumes as (for example) "/glusterfs/homes -> /zpool/homes". I.e. gluster volume create homes replica 2 transport tcp server1:/zpool-homes/homes server2:/zpool-homes/homes force (on server1) server1:/homes     44729413504 16032705152 28696708352  36% /glusterfs/homes

The problem is, the performance has deteriorated terribly.
We needed to copy all of our data from the old server to the new glusterfs volumes (appx. 60TB). We decided to do this with multiple rsync commands (like 400 simultanous rsyncs) The copy went well for the first 4 days, with an average across all rsyncs of 150-200 MBytes per second.
Then, suddenly, on the fourth day, it dropped to about 50 MBytes/s.
Then, by the end of the day, down to ~5MBytes/s (five).
I've stopped the rsyncs, and Ican still copy an individual file across to the glusterfs shared directory at 100MB/s.
But actions such as "ls -la" or "find" take forever!

Are there obvious flaws in my setup to correct?
How can I better troubleshoot this?

Thanks!
--

Mike


________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users




[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux