On Fri, Apr 10, 2015 at 11:04:55AM -0600, Kevin Fenzi wrote: > I was going to wait until after freeze for this, but with us slipping a > week I think it might be worth doing now. > > For the last few weeks we have been having issues with db-koji01. > The problem started when I moved it's backend storage from one iscsi/pv > to another iscsi/pv. The load has been high since then and it's not as > performant as it was. > > Effects: > > * koji alerts in nagios make us need to restart httpd on koji01 (which > we can do without outage, but means a human has to wake up and go do > it). > > * If koji01 httpd isn't restarted, kojira sometimes will timeout and > not launch newrepos. (We worked around this by increasing the > timeout, but it's only a matter of time before it hits this again). > > * Pages on koji that need lots of db access are slower than they > were/need to be. > > Cause: > > Not entirely sure what the base cause is. lvdisplay shows the guest is > on the right iscsi volume, there's no iscsi errors or the like. The > host did have stale lvm data due to lvmetad running, but that shouldn't > have affected the running guest(s). I can only think there's something > still trying to hit the old no longer used iscsi volume and causing > extra load. > > What I would like to do: > > * Stop postgres on db-koji01. This will cause the hub to show db down > to anyone looking. > > * rsync /var/lib/pgsql off to backup03. This should take less than > 10min. > > * shutdown db-koji01 and dhcp01. > > * Reboot bvirthost09 > > * See if the issue clears up. If something happens and db-koji01 > doesn't come back up right, we can make a new one and > sync /var/lib/pgsql back to it and be back up pretty quickly. > Hopefully it won't come to that. > > I'd like to schedule this possibly over the weekend off hours when koji > isn't all that busy. > > Thoughts? +1s? +1 for me and fingers crossed :) Pierre _______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
