Greetings.
I thought I would share this in hopes we can come up with ways to
handle this sort of thing better. I do not blame anyone and I'd like to
focus on positive steps we can take rather than pointing fingers.
Just before the Fedora 25 development cycle started, I went and
upgraded our koji builders from Fedora 23 to Fedora 24. I've done this
every cycle since we moved them to Fedora instances and its gone
usually pretty smoothly.
Not this time. ;) The kernel maintainers started getting builds that
failed and got stuck in odd ways. A few other larger packages (glibc)
maintainers also hit this issue. The mock logs simply showed a bunch of
"Fork failed" messages and a bunch of processes stuck in D
At first we thought it might be a kernel issue, so we tried various
kernels on the builders without any luck. Then the kernel maintainers
added some patches to see if they could get debugging from the deadlock
it was hitting. That fixed the processes stuck in D, but builds still
failed.
I started looking around for anything else that might be related and
saw there was a systemd update that mentioned task limits, which in
turn pointed us to looking at systemd and seeing this (from systemd 228
NEWS):
" * There's a new system.conf setting DefaultTasksMax= to
control the default TasksMax= setting for services and
scopes running on the system. (TasksMax= is the primary
setting that exposes the "pids" cgroup controller on systemd
and was introduced in the previous systemd release.) The
setting now defaults to 512, which means services that are
not explicitly configured otherwise will only be able to
create 512 processes or threads at maximum, from this
version on. Note that this means that thread- or
process-heavy services might need to be reconfigured to set
TasksMax= to a higher value. It is sufficient to set
TasksMax= in these specific unit files to a higher value, or
even "infinity". Similar, there's now a logind.conf setting
UserTasksMax= that defaults to 4096 and limits the total
number of processes or tasks each user may own
concurrently. nspawn containers also have the TasksMax=
value set by default now, to 8192. Note that all of this
only has an effect if the "pids" cgroup controller is
enabled in the kernel. The general benefit of these changes
should be a more robust and safer system, that provides a
certain amount of per-service fork() bomb protection."
We had systemd 229, and this was the root cause of the issue.
In systemd 231 they changed this to:
" * In similar fashion TasksMax= takes percentage values now,
too. The value is taken relative to the configured maximum number of
processes on the system. The per-service task maximum has been changed
to 15% using this functionality. (Effectively this is an increase of
512 → 4915 for service units, given the kernel's default pid_max
setting.)"
So, how can we do better here?
* IMHO the initial upstream default didn't make sense for Fedora
* The feature doesn't seem to work right (causing processes to go into
D and be deadlocked). Likely this is a rcu/kernel bug(s), but systemd
shouldn't enable this unless it's robust or have some kind of check
for a recent enough kernel with fixes?
* There isn't any logging showing that this was hit. Perhaps there's a
technical limitation with cgroups? This would be very handy in
tracking down things like this.
* This might have been a nice thing for release notes? I try and keep
up, but I don't read every systemd NEWS file as they happen or most
packages NEWS files. Can we make adding release notes any easier?
* Perhaps after beta but before final we ping maintainers of
"important" packages asking what big changes have happened? Or
someone just goes thru the release notes for them all and proposes a
list of them?
* Your brilliant idea here.
kevin
Attachment:
pgpFmzbIyOxaJ.pgp
Description: OpenPGP digital signature
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
