Hi, so people might have seen that last week, a rather severe vuln was found: https://dirtycow.ninja/ I was at Openstack summit when it was found, and the updated kernel package wasn't on the CDN until I was out for holiday[1]. The main reason is that RH test kernel patchs a bit more than others, especially for something as critical. And Centos wait on RH to push update So while this was not uber urgent as shellshock or heartbleed, it was still rather critical to fix as I have a rather minimal trust in Jenkins and Gerrit to be secure. So once I was back on friday, and after dealing with others fires and infra, I did reboot stuff that wouldn't impact too much production (like rsyslog, freeipa servers, the salt server, the virt hosts with builders) and decided to push for a reboot of jenkins and gerrit for the weekend. In retrospect, I tought I did discuss on irc, but I forgot, sorry about that. Of course, because I like to live dangerously, I did that in the saturday morning, on a travel day. It should have been fast[2]. However, things never go as expected and we did face a few issues: - myrmicinae.rht.gluster.org, the host running our VM decided to take 1h to boot. At the firmware/BIOS level. That's slightly inacceptable, but I have also a limited capacity to fix, since this would requires 1) to test reboot (so lose 1h) 2) to fiddle in the Bios (and so reboot). So that's why jenkins/gerrit were down around 10h CET until 11h. - jenkins didn't (as usual) restart. I found the root cause, this was due to NetworkManager and network init script kinda doing the same stuff, but in different way. This is now fixed, and jenkins VM should reboot without a human to fix stuff around. - gerrit for some reason do not start at boot. I am not sure what was the way it was done before, but I suspect something related to /etc/init.d that got wiped after a upgrade or something, because gerrit initscript is not a real initscript. So I did some hack in /etc/rc.local, since the upgrade to EL7 is around the corner, and I had better things to do in the weekend that fixing some bash stuff (like fixing python stuff). - gerrit VM DNS was incorrect, and no one told me until 6h after the reboot (why no one told on irc and or on the list and or bugzilla is a issue that I will surely have to investigate). Why did the DNS got changed (or if it didn't changed, how did it worked before ?) is the part that I still cannot explain. But it got for some reason reverted to the old setting, using the libvirt gateway as dns, which wasn't working with the current setup. So this was fixed after Nigel pinged me on my phone, and I managed to connect from the train to fix it. So I suspect that's all for today, I will try to schedule my next vacation outside of the unexpected release of a critical kernel patch. [1] yes, it was nice, thanks for asking. [2] famous last word -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS
Attachment:
signature.asc
Description: This is a digitally signed message part
_______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
