On Thu, Oct 12, 2017 at 9:15 AM, David Galloway <dgallowa@xxxxxxxxxx> wrote: > > On 10/12/2017 07:24 AM, Alfredo Deza wrote: >> We are currently going through a severe outage. All critical >> infrastructure services are currently unavailable. >> >> This includes the following: >> >> * Both Jenkins masters (one is for Ceph the other one for ceph-ansible) >> * Anything in Shaman >> * All build nodes >> * All repositories (anything *chacra.ceph.com) >> >> Some services are up, like docs.ceph.com, tracker, and download.ceph.com >> >> We are currently trying to determine the reason why these are >> completely unreachable with the service provider. >> >> We will keep updating once we have anything to report. >> > > All services have been restored. The outage only affected the CI > infrastructure (Jenkins, Shaman, Chacra, etc.). Some of you may have experienced some builds being "stale" and not completed to get repos. This happened because the service that polls to check repos to build was not started on one of the servers. That server is 2.chacra.ceph.com So if you were impacted, I apologize, this was very hard to catch. The good news is that the binaries are there, and the server will pick up where it needs to, building the repos as it finds them. Because there is a backlog on that server, please be patient, at some point it will up to date. Again, apologies, this was totally something that we didn't anticipate. Created an issue to make sure this is tracked next time http://tracker.ceph.com/issues/21779 > > OVH temporarily suspended our CI public cloud account due to a past due > bill. > > I'm currently monitoring a build to verify packages are being created > and pushed to the chacra nodes but I think we're in the clear. -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
