Hi Strahil, thanks for your response.
I have compiled gluster 7.6 from sources on both servers.
There is a 7.7 version which is fixing somw stuff. Why do you have to compile it from source ?
Because I have often found with other stuff in the past compiling from
source makes a bunch of problems go away. software generally works the
way the developers expect it to if you use the sources, so they are
better able to help if required. so now I generally compile most of my
center-piece softwares and use packages for all the supporting stuff.
Servers are 6core/3.4Ghz with 32 GB RAM, no swap, and SSD and gigabit
network connections. They are running debian, and are being used as
redundant web servers. There is some 3Million files on the Gluster
Storage averaging 130KB/file.
This type of workload is called 'metadata-intensive'.
does this mean the metadata-cache group file would be a good one to
enable? will try.
waited 10 minutes, no change that I can see.
There are some recommendations for this type of workload:
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/small_file_performance_enhancements
Keep an eye on the section that mentions dirty-ratio = 5 &dirty-background-ration = 2.
I have actually read that whole manual, and specifically that page
several times. And also this one:
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/small_file_performance_enhancements
Perhaps I am not understanding it correctly. I tried these suggestions
before and it got worse, not better. so I have been operating under the
assumption that maybe these guidelines are not appropriate for newer
versions.
But will try again. adjusting the dirty ratios.
Load average went from around 15 to 35 in about 2-3 minutes, but 20
minutes later, it is back down to 20. It may be having a minimal
positive impact on cpu, though, I haven't see the main glusterfs go over
200% since I changed this, an the brick processes are hovering just
below 50% where they were consistently above 50% before. Might just be
time of day with the system not as busy.
after watching for 30 minutes, load average is fluctuating between 10
and 30, but cpu idle appears marginally better on average than it was.
Interestingly, mostly because it is not something I have ever
experienced before, software interrupts sit between 1 and 5 on each
core, but the last core is usually sitting around 20. Have never
encountered a high load average where the si number was ever
significant. I have googled the crap out of that (as well as gluster
performance in general), there are nearly limitless posts about what it
is, but have yet to see one thing to explain what to do about it.
There is an explanation about that in the link I provided above:
Configuring a higher event threads value than the available processing units could again cause context switches on these threads. As a result reducing the number deduced from the previous step to a number that is less that the available processing units is recommended.
Okay, again, have played with these numbers before and it did not pan
out as expected. if I understand it correctly, I have 3 brick processes
(glusterfsd), so the "deduced" number should be 3, and I should set it
lower than that, so 2. but it also says "If a specific thread consumes
more number of CPU cycles than needed, increasing the event thread count
would enhance the performance of the Red Hat Storage Server." which is
why I had it at 8.
but will set it to 2 now. load average is at 17 to start, waiting a
while to see what happens.
so 15 minutes later, load average is currently 12, but is fluctuating
between 10 and 20, have seen no significant change in cpu usage or
anything else in top.
now try also changing server.outstanding-rpc-limit to 256 and wait.
15 minutes later; load has been above 30 but is currently back down to
12. no significant change in cpu. try increasing to 512 and wait.
15 minutes later, load average is 50. no signficant difference in cpu.
Software interrupts remain around where they were. wa from top remains
about where it was. not sure why load average is climbing so high.
changing rpc-limit to 128.
ugh. 10 minutes later, load average just popped over 100. resetting
rpc-limit.
now trying cluster.lookup-optimize on, lazy rebalancing (probably a bad
idea on the live system, but how much worse can it get?) Ya, bad idea,
80 hours estimated to complete, load is over 50 and server is crawling.
disabling rebalance and turning lookup-optimize off, for now.
right now the only suggested parameter I haven't played with is the
performance.io-thread-count, which I currently have at 64.
sigh. an hour later load average is 80 and climbing. apache processes
are numbering in the hundreds and I am constantly having to restart it.
this brings load average down to 5, but as apache processes climb and
are held open load average gets up to over 100 again with 3-4 minutes,
and system starts going non-responsive. rinse and repeat.
so followed all the recommendations, maybe the dirty settings had a
small positive impact, but overall system is most definitely worse for
having made the changes.
I have returned the configs back to how they were except the dirty
settings and the metadata-cache group. increased performance.cache-size
to 16GB for now, because that is the one thing that seems to help when I
"tune" (aka make worse) the system. have had to restart apache a couple
dozen times or more, but after another 30 minutes or so system has
pretty much settled back to how it was before I started. cpu is like I
originally stated, all 6 cores maxed out most of the time, software
interrupts still have all cpus running around 5 with the last one
consistently sitting around 20-25. Disk is busy but not usually maxed
out. RAM is about half used. network load peaks at about 1/3 capacity.
load average is between 10 and 20. sites are responding, but sluggish.
so am I not reading these recommendations and following the instructions
correctly? am I not waiting long enough after each implementation,
should I be making 1 change per day instead of thinking 15 minutes
should be enough for the system to catch up? I have read the full red
hat documentation and the significant majority of the gluster docs,
maybe I am missing something else there? should these settings have had
a different effect than they did?
For what it's worth, I am running ext4 as my underlying fs and I have
read a few times that XFS might have been a better choice. But that is
not a trivial experiment to make at this time with the system in
production. It's one thing (and still a bad thing to be sure) to
semi-bork the system for an hour or two while I play with
configurations, but would take a day or so offline to reformat and
restore the data.
As 'storage.fips-mode-rchecksum' is using sha256, you can try to disable it - which should use the less cpu intensive md5. Yet, I have never played with that option ...
Done. no signficant difference than I can see.
Check the RH page about the tunings and try different values for the event threads.
in the past I have tried 2, 4, 8, 16, and 32. Playing with just those I
never noticed that any of them made any difference. Though I might have
some different options now than I did then, so might try these again
throughout the day...
Thanks again for your time Strahil, if you have any more thoughts would
love to hear them.
Best Regards,
Strahil Nikolov
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
