On 06/09/2016 02:38 PM, Kaushal M wrote: > On Thu, Jun 9, 2016 at 12:13 PM, Kaushal M <kshlmster@xxxxxxxxx> wrote: >> A heads up to all maintainers and developers. >> >> As all of you probably already know, reviews in Gerrit are getting >> random votes for jobs that ran for other patchsets. >> >> We've had people noticing these votes only when they've been negative. >> But these votes can be positive as well (I've got an example in the >> forwarded mail below). >> >> Maintainers need to be make sure that any positive vote given to a >> review is correct and for a job that ran for the particular review, >> before merging it. >> >> To make sure that changes that have been given such a bogus vote don't >> get merged, any developer finding such a vote, can give a Verified-1 >> to the review to block it from merging. I've changed the Verified flag >> so that a Verified-1 blocks a review from being merged. I'll remove >> this change after we figure out what's happening. >> >> I'll be posting updates to the infra-list to the mail-thread I've >> forwarded below. > > This (and the random build failures) should be fixed now. > > There should no longer be any random votes/comments. Anyone who's had > incorrect votes, please re-trigger the jobs. That's a great news Kaushal. Many thanks for taking it to the root cause and fixing it. > > I'll leave the Verified-1 configuration around, as a way to block > changes being merged in the future, if similar situations occur. > > ~kaushal > > PS: For anyone curious about what happened (tl;dr: zombie-jenkins), > please refer to the thread > 'Investigating random votes in Gerrit' in the gluster-infra list. > >> >> ~kaushal >> >> >> ---------- Forwarded message ---------- >> From: Kaushal M <kshlmster@xxxxxxxxx> >> Date: Thu, Jun 9, 2016 at 11:52 AM >> Subject: Investigating random votes in Gerrit >> To: gluster-infra <gluster-infra@xxxxxxxxxxx> >> >> >> In addition to the builder issues we're having, we are also facing >> problems with jenkins voting/commenting randomly. >> >> The comments generally link to older jobs for older patchsets, which >> were run about 2 months back (beginning of April). For example, >> https://review.gluster.org/14665 has a netbsd regression +1 vote, from >> a job run in April for review 13873, and which actually failed. >> >> Another observation that I've made is that these fake votes sometime >> provide a -1 Verified. Jenkins shouldn't be using this flag anymore. >> >> These 2 observations, make me wonder if another jenkins instance is >> running somewhere, from our old backups possibly? Michael, could this >> be possible? >> >> To check from where these votes/comments were coming from, I tried >> checking the Gerrit sshd logs. This wasn't helpful, because all logins >> apparently happen from 127.0.0.1. This is probably some firewall rule >> that has been setup, post migration, because I see older logs giving >> proper IPs. I'll require Michael's help with fixing this, if possible. >> >> I'll continue to investigate, and update this thread with anything I find. >> >> ~kaushal > _______________________________________________ > maintainers mailing list > maintainers@xxxxxxxxxxx > http://www.gluster.org/mailman/listinfo/maintainers > _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel