On Fri, Oct 11, 2019 at 09:22:11AM -0400, Stephen John Smoogen wrote: > On Fri, 11 Oct 2019 at 09:19, Peter Robinson <pbrobinson@xxxxxxxxx> wrote: > > > > On Fri, Oct 11, 2019 at 10:13 AM Dominik 'Rathann' Mierzejewski > > <dominik@xxxxxxxxxxxxxx> wrote: > > > > > > On Friday, 11 October 2019 at 04:36, Orion Poplawski wrote: > > > > On 10/10/19 4:57 PM, Orion Poplawski wrote: > > > > > On 10/10/19 9:23 AM, Kevin Fenzi wrote: > > > [...] > > > > > > Pretty please can you all indicate approximately when (UTC) you were > > > > > > seeing the slowness? I suspect it may be the nightly database dump job, > > > > > > but I can't tell without knowing when you saw the slowness... > > > [...] > > > > > Thanks, it is much better now. I feel like I've hit periods of slowness > > > > > over several days recently so I will definitely note times if I run into > > > > > it again. > > > > > > > > Definitely slower again: > > > > Fri Oct 11 02:20:53 UTC 2019 > > > > > > Fri 11 Oct 09:11:01 UTC 2019 > > > It takes 30s to show a build status page, e.g. > > > https://koji.fedoraproject.org/koji/taskinfo?taskID=38215205 > > > > Builds/cli is slow on Fri 11th, 13:17 UTC. > > The fixes we tried seems to have put the load average of koji up to > 150 while several mbs builds are going. I am going to put our status > as yellow and see if we can tune this down. So to update this thread some: * Load is fine/responsiveness is fine as long as we aren't doing a database dump. * When we do, things slow down and load goes way up. We did find one very odd/bad thing on the virthost the server is on (one drive in the raid is showing active in multipath and possibly getting copies of all i/o sent to it? unclear exactly, but it's wrong and needs fixing). So, we stopped the backup and we are going to wait until 21UTC (after 5pm EDT). At that point we will try and clean up that multipath issue and see if that helps things. If not we will try and tune the vm/database more until we have it responsive all the time. We hope we can do these changes with minimal to no downtime, but there may be some. I've filed https://pagure.io/fedora-infrastructure/issue/8292 on this issue, folks are welcome to follow along there. Sorry about all this, I was just hoping to make things faster and sadly, did not. kevin
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
