Send CentOS-announce mailing list submissions to
centos-announce@xxxxxxxxxx
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.centos.org/mailman/listinfo/centos-announce
or, via email, send a message with subject or body 'help' to
centos-announce-request@xxxxxxxxxx
You can reach the person managing the list at
centos-announce-owner@xxxxxxxxxx
When replying, please edit your Subject line so it is more specific
than "Re: Contents of CentOS-announce digest..."
Today's Topics:
1. Notice of Service Outage and followup LON1/UK Facility
(Karanbir Singh)
2. Announcing release for Developer Toolset 4 on CentOS Linux 6
x86_64 SCL (Honza Horak)
3. Announcing release for Developer Toolset 4 on CentOS Linux 7
x86_64 SCL (Honza Horak)
----------------------------------------------------------------------
Message: 1
Date: Wed, 30 Mar 2016 11:25:19 +0100
From: Karanbir Singh <kbsingh@xxxxxxxxxx>
To: CentOS Announcements List <centos-announce@xxxxxxxxxx>
Subject: [CentOS-announce] Notice of Service Outage and followup
LON1/UK Facility
Message-ID: <56FBA98F.6000102@xxxxxxxxxx>
Content-Type: text/plain; charset=utf-8
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
== What happened ==
On Wednesday February 24th, at 6pm UTC time, the DC hosting some of
the CentOS equipments used for various roles had suffered from
multiple electricity power outages. The facility was completely dark
for just under 2 hrs, and we were able to start recovering services by
8pm UTC. By midnight we had most services restored, by 2:00AM UTC Feb
25th we had all services restored.
That meant that the machines in those racks were running on batteries
(ups in the racks) but finally went down in an uncontrolled way due to
lack ot communication with that UPS.
Subsequent on Monday March 14th, we suffered another power outage in
the racks, this time due to a overload on the rack power circuits.
== Services that were impacted ==
- severity critical : mirrorlist.centos.org node (IPv6) went down
(while multiple mirrorlist.centos.org nodes for IPv4 nodes were still
online). That means that machines with only IPV6 connectivity couldn't
get yum to work to retrieve the list of nearest mirrors.
- severity medium : Our main buildservices queue management services
were down; note: this did not impact our ability to build, test and
deliver updates.
- severity medium : www.centos.org and www.centos.org/forums weren't
reachable through IPv6 : at the moment, those services are natively
reachable through IPv4, but proxied through nodes in that DC for IPv6
users. Most tested browsers were falling back to IPv4 during that period
- severity medium : CentOS DevCloud
(https://wiki.centos.org/DevCloud) : that means that CentOS Developers
weren't able to instantiate new CentOS test VMs for their work, but
also weren't able to reach the existing ones.
- severity low : several publicly facing small services like
http://planet.centos.org , http://seven.centos.org (not critical and
could be restored quickly to other VMs elsewhere)
- severity low : the server leading the armv7hl builds for the Plague
build farm was also offline, meaning no armhfp build during that
timeframe (but not updates were to be built, so mitigated issue)
= Followup actions and notes
Over the years, the baseline recovery model we've used and tried to
enforce is one of 'restore in place', take a downtime hit if needed -
and ensure we have service continuity for the user facing components (
the mirrorlist service, the centos update and content distribution
services). For other resources, like the main website etc, we ensure
there are good backups available in multiple places, usable to restore
services should there be a need. This model has worked well for us
over the years, and we've had very little, if any, service outages
that had a user impact. The restore in place/restore outside HA also
meant we were able to better utilise the exclusively sponsored
machines we rely on.
However, as the project grows, with a lot more infrastructure being
consolidated into a few locations for non CDN services, our exposure
to service downtime has dramatically increased. Its clear that we need
to expand the scope of where we backup to, how we backup, how we
anticipate failure and our ability to restore services in a timely
manner should there be facilities outages. In the coming weeks, we are
going to undertake a deep dive into our Infrastructure design and
delivery and try to first come up with a consolidated set of risks we
need to manage against, and then work towards reducing the risk,
spreading the availability as needed.
Our backend storage platform for the DevCloud and persistent
storage for other nodes in the facility is run from a distributed,
replicated Gluster setup. Inspite of the sudden loss of power, in a
production environment with hundreds of running VMs and dozens of
running data jobs, we were able to trivially recover our entire data
set with minimum data loss. Some of the running VMs inside the
DevCloud did see local filesystem issues, but we dont think that was a
backing storage issue. This event has dramatically increased out
confidence in the gluster technology stack and we will certainly be
looking at extending deployments for it internally.
== Comments about hosting facility ==
Their Status post about this
http://status.uk2.net/2016/02/24/london-power-outage/
We have multiple racks at this facility, and have a long standing
relationship with them going back to late Summer 2012. Over this
period we have had a near perfect uptime record for our equipment
there. And above all we have been consistently impressed with the
speed of and the knowledgeable support we've recieved at the DC. In
many cases, how the facility reacts to outage defines the real service
value - and in this case, we can only commend the fantastic support we
had through the outage hours. We do however feel there could be better
monitoring and reporting of some of the facilities information and
will be working with them to improve in those regards.
Fabian Arrotin and Karanbir Singh
The CentOS Project
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQEcBAEBAgAGBQJW+6mPAAoJEI3Oi2Mx7xbtHo8IAI+RVIDjGwJOzgJ5Ry7mHwLe
Zc+aBUQklDk5oRaDk7QZHsaGp1lclNsutBk3YujNlXwMC4hUKdPwkTVuX50usQ7s
kd7qF1BlElNyfMPfFJGwchIQBDOZqZxkZP4uOrvQUnIZUYfyx6NnPnGS0uatBdnw
hBJ6TbgP6i50h7U0fNWjHU2I8xe0zsx1jVrvNngDMlQcIHC0d1KMtpOgSMR5f9Bn
bLwghfD4/yPyqJP1sc+021ANk1+a7uXs4KKG3MXpMlFyvYmv2ict0Q/sDtz0jzCx
kbRgDGm/GF1TUUENciESkHPKy3kLWA1oCicOkiEhzNz2YwFQNdNpi9PqWEK/F5Q=
=bDIN
-----END PGP SIGNATURE-----
------------------------------
Message: 2
Date: Thu, 31 Mar 2016 17:38:07 +0200
From: Honza Horak <hhorak@xxxxxxxxxx>
To: centos-announce@xxxxxxxxxx
Subject: [CentOS-announce] Announcing release for Developer Toolset 4
on CentOS Linux 6 x86_64 SCL
Message-ID: <56FD445F.8060304@xxxxxxxxxx>
Content-Type: text/plain; charset=utf-8; format=flowed
I am pleased to announce the immediate availability of Developer Toolset
4 on CentOS Linux 6 x86_64, delivered via a Software Collection (SCL)
built by the SCLo Special Interest Group
(https://wiki.centos.org/SpecialInterestGroup/SCLo).
QuickStart
----------
You can get started in three easy steps:
$ sudo yum install centos-release-scl
$ sudo yum install devtoolset-4-toolchain
$ scl enable devtoolset-4 bash
At this point you should be able to use gcc and other tools just as a
normal application. Examples of commands run might be:
$ gcc hello.c
$ sudo yum install devtoolset-4-valgrind
$ valgrind ./a.out
$ gdb ./a.out
In order to view the individual components included in this collection,
including additional development tools, you can run:
$ sudo yum list devtoolset-4\*
About Software Collections
--------------------------
Software Collections give you the power to build, install, and use
multiple versions of software on the same system, without affecting
system-wide installed packages. Each collection is delivered as a group
of RPMs, with the grouping being done using the name of the collection
as a prefix of all packages that are part of the software collection.
The collection devtoolset-4 delivers version 5.2.1 of the GNU Compiler
Collection, GNU Debugger, and other development, debugging, and
performance monitoring tools as RPMs.
The SCLo SIG in CentOS
----------------------
The Software Collections SIG group is an open community group
co-ordinating the development of the SCL technology, and helping curate
a reference set of collections. In addition to the Developer Toolset
collection being released here, we also build and deliver databases, web
servers, and language stacks including multiple versions of PostgreSQL,
MariaDB, Apache HTTP Server, NodeJS, Ruby, Python and others.
Software Collections SIG release was announced at
https://lists.centos.org/pipermail/centos-announce/2015-October/021446.html
You can learn more about Software Collections concepts at:
http://softwarecollections.org
You can find information on the SIG at
https://wiki.centos.org/SpecialInterestGroup/SCLo ; this includes howto
get involved and help with the effort.
We meet every second Wednesday at 16:00 UTC in #centos-devel (ref:
https://www.centos.org/community/calendar), for an informal open forum
open to anyone who might have comments, concerns or wants to get started
with SCL's in CentOS.
Enjoy!
Honza
SCLo SIG member
------------------------------
Message: 3
Date: Thu, 31 Mar 2016 17:38:11 +0200
From: Honza Horak <hhorak@xxxxxxxxxx>
To: centos-announce@xxxxxxxxxx
Subject: [CentOS-announce] Announcing release for Developer Toolset 4
on CentOS Linux 7 x86_64 SCL
Message-ID: <56FD4463.3040800@xxxxxxxxxx>
Content-Type: text/plain; charset=utf-8; format=flowed
I am pleased to announce the immediate availability of Developer Toolset
4 on CentOS Linux 7 x86_64, delivered via a Software Collection (SCL)
built by the SCLo Special Interest Group
(https://wiki.centos.org/SpecialInterestGroup/SCLo).
QuickStart
----------
You can get started in three easy steps:
$ sudo yum install centos-release-scl
$ sudo yum install devtoolset-4-toolchain
$ scl enable devtoolset-4 bash
At this point you should be able to use gcc and other tools just as a
normal application. Examples of commands run might be:
$ gcc hello.c
$ sudo yum install devtoolset-4-valgrind
$ valgrind ./a.out
$ gdb ./a.out
In order to view the individual components included in this collection,
including additional development tools, you can run:
$ sudo yum list devtoolset-4\*
About Software Collections
--------------------------
Software Collections give you the power to build, install, and use
multiple versions of software on the same system, without affecting
system-wide installed packages. Each collection is delivered as a group
of RPMs, with the grouping being done using the name of the collection
as a prefix of all packages that are part of the software collection.
The collection devtoolset-4 delivers version 5.2.1 of the GNU Compiler
Collection, GNU Debugger, Eclipse development platform, and other
development, debugging, and performance monitoring tools as RPMs.
The SCLo SIG in CentOS
----------------------
The Software Collections SIG group is an open community group
co-ordinating the development of the SCL technology, and helping curate
a reference set of collections. In addition to the Developer Toolset
collection being released here, we also build and deliver databases, web
servers, and language stacks including multiple versions of PostgreSQL,
MariaDB, Apache HTTP Server, NodeJS, Ruby, Python and others.
Software Collections SIG release was announced at
https://lists.centos.org/pipermail/centos-announce/2015-October/021446.html
You can learn more about Software Collections concepts at:
http://softwarecollections.org
You can find information on the SIG at
https://wiki.centos.org/SpecialInterestGroup/SCLo ; this includes howto
get involved and help with the effort.
We meet every second Wednesday at 16:00 UTC in #centos-devel (ref:
https://www.centos.org/community/calendar), for an informal open forum
open to anyone who might have comments, concerns or wants to get started
with SCL's in CentOS.
Enjoy!
Honza
SCLo SIG member
------------------------------
_______________________________________________
CentOS-announce mailing list
CentOS-announce@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos-announce
End of CentOS-announce Digest, Vol 134, Issue 2
***********************************************
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos
