Hi Artem and others,
Happy to report the system has been relatively stable for the remainder
of the week. I have one wordpress site that seems to get hung processes
when someone logs in with an incorrect password. Since it is only one,
and reliably reproduceable, I am not sure if the issue is to do with
Gluster or Wordpress itself, but afaik it was not doing it some months
back before the system was using Gluster so I am guessing some combo of
both.
Regardless, that is the one and only time apache processes stacked up to
over 150, and that still only brought the load average up to just under
25; the system did go a bit sluggish, but remained fairly responsive
throughout until I restarted apache. Otherwise 15 minute load average
consistently runs between 8 and 11 during peak hours and between 4 and 7
during off hours, and other than the one time I have not seen the
one-minute load average go over 15. all resources still spike to full
capacity from time to time, but it never remains that way for long like
it did before.
For site responsiveness, first visit to any given site is quite slow,
like 3-5 seconds on straight html pages, 10-15 seconds for some of the
more bloated WP themes, but clicking links within the site after the
first page is loaded is relatively quick, like 1 second on straight html
pages, and ~5-6 seconds on the bloated themes. Again, not sure if that
is a Gluster related thing or something else.
So, still holding my breath a bit, but seems this solution is working,
at least for me. I haven't played with any of the other settings yet to
see if I can improve it further, probably will next week. thinking to
increase the write behind window size further to see what happens, as
well as play with the settings suggested by Strahil.
On 2020-08-05 5:28 p.m., Artem Russakovskii wrote:
I'm very curious whether these improvements hold up over the next few
days. Please report back.
Sincerely,
Artem
--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net <http://beerpla.net/> | @ArtemR <http://twitter.com/ArtemR>
On Wed, Aug 5, 2020 at 9:44 AM Computerisms Corporation
<bob@xxxxxxxxxxxxxxx <mailto:bob@xxxxxxxxxxxxxxx>> wrote:
Hi List,
> So, we just moved into a quieter time of the day, but maybe I just
> stumbled onto something. I was trying to figure out if/how I could
> throw more RAM at the problem. gluster docs says write behind is
not a
> cache unless flush-behind is on. So seems that is a way to throw
ram to
> it? I put performance.write-behind-window-size: 512MB and
> performance.flush-behind: on and the whole system calmed down pretty
> much immediately. could be just timing, though, will have to see
> tomorrow during business hours whether the system stays at a
reasonable
> load.
so reporting back that this seems to have definitely had a significant
positive effect.
So far today I have not seen the load average climb over 13 with the
15minute average hovering around 7. cpus are still spiking from
time to
time, but they are not staying maxed out all the time, and frequently I
am seeing brief periods of up to 80% idle. glusterfs process still
spiking up to 180% or so, but consistently running around 70%, and the
brick processes still spiking up to 70-80%, but consistently running
around 20%. Disk has only been above 50% in atop once so far today
when
it spiked up to 92%, and still lots of RAM left over. So far nload
even
seems indicates I could get away with a 100Mbit network connection.
Websites are snappy relative to what they were, still a bit sluggish on
the first page of any given site, but tolerable or close to. Apache
processes are opening and closing right away, instead of stacking up.
Overall, system is performing pretty much like I would expect it to
without gluster. I haven't played with any of the other settings yet,
just going to leave it like this for a day.
I have to admit I am a little bit suspicious. I have been arguing with
Gluster for a very long time, and I have never known it to play this
nice. kind feels like when your girl tells you she is "fine";
conversation has stopped, but you aren't really sure if it's done...
>
> I will still test the other options you suggested tonight,
though, this
> is probably too good to be true.
>
> Can't thank you enough for your input, Strahil, your help is truly
> appreciated!
>
>
>
>
>
>
>>
>>>>
>>>>
>>>> Best Regards,
>>>> Strahil Nikolov
>>>>
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx>
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx>
> https://lists.gluster.org/mailman/listinfo/gluster-users
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx>
https://lists.gluster.org/mailman/listinfo/gluster-users
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
