Hi David,
Did you try setting "direct-io-mode=disable" on the client mounts? As it is mostly static content it would help to use the kernel caching and read-ahead mechanisms.
I think the default is enabled.
Regards,
Jorick Astrego
On 12/19/19 1:28 AM, David Cunningham
wrote:
Hi Raghavendra and Strahil,
We are using GFS version 5.6-1.el7 from the CentOS repository. Unfortunately we can't modify the application and it expects to read and write from a normal filesystem.
There's around 25GB of data being written during a business day, so over 10 hours that's around 0.7 MBps, which has me mystified as to how it can generate 114MBps of network traffic. Granted we have read traffic as well, but still. The chart shows much more inbound traffic to the GFS server than outbound, suggesting the problem is with data writes.
Is it possible with GFS to not check with the other nodes when reading? Our data is mostly static and we don't require 100% guarantee that the data is up-to-date when reading.
Thanks for any assistance.
On Wed, 18 Dec 2019 at 16:39, Raghavendra Gowdappa <rgowdapp@xxxxxxxxxx> wrote:
What version of Glusterfs are you using? Though, not sure what's the root cause of your problem, just wanted to point out a bug with read-ahead which would cause read-amplification over network [1][2], which should be fixed in recent versions.
On Wed, Dec 18, 2019 at 2:50 AM David Cunningham <dcunningham@xxxxxxxxxxxxx> wrote:
________Hello,
We switched a production system to using GFS instead of NFS at the weekend, however it didn't go well on Monday when full load hit. The application started crashing regularly and we had to revert to NFS. It seems that the problem was high network traffic used by GFS.
We've two GFS nodes plus one arbiter node, each about 1.3ms latency from each other. Attached is a chart of network traffic on one of the GFS nodes. We see that it saturated the 1Gbps link before we reverted to NFS at 15:10.
The question is, why does GFS use so much network traffic and is there anything we can do about it? NFS traffic doesn't exceed 4MBps, so 120MBps for GFS seems awfully high.
It would also be good to have faster read performance from GFS, but that's another issue.
Thanks in advance for any assistance.
--
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
--
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
Met vriendelijke groet, With kind regards,
Jorick Astrego
Netbulae Virtualization Experts
| Tel: 053 20 30 270 | info@xxxxxxxxxxx | Staalsteden 4-3A | KvK 08198180 |
| Fax: 053 20 30 271 | www.netbulae.eu | 7547 TA Enschede | BTW NL821234584B01 |
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
