Re: counting file descriptors with a cgroup controller

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi

On 03/06/2017 07:58 PM, Tejun Heo wrote:
Hello,

On Fri, Feb 17, 2017 at 12:37:11PM +0100, Krzysztof Opasiak wrote:
We need to limit and monitor the number of file descriptors processes
keep open. If a process exceeds certain limit we'd like to terminate it
and restart it or reboot the whole system. Currently the RLIMIT API
allows limiting the number of file descriptors but to achieve our goals
we'd need to make sure all programmes we run handle EMFILE errno
properly. That is why we consider developing a cgroup controller that
limits the number of open file descriptors of its members (similar to
 memory controler).

Any comments? Is there any alternative that:

+ does not require modifications of user-land code,
+ enables other process (e.g. init) to be notified and apply policy.

Hmm... I'm not quite sure fds qualify as an independent system-wide
resource.  We did that for pids because pids are globally limited and
can run out way earlier than memory backing it.  I don't think we have
similar restructions for fds, do we?

Well I'm not aware of such restrictions...

So maybe let me clarify our use case so we can have some more discussion about this. We are dealing with task of monitoring system services on an IoT system. So this system needs to run as long as possible without reboot just like server. In server world almost whole system state is being monitored by services like nagios. They measure each parameter (like cpu, memory etc) with some interval. Unfortunately we cannot use this it in an embedded system due to power consumption.

So generally now we consider two approaches:

1) Use rlimits when possible to limit resources for each process.

The problem here is that this creates an implicit requirement that all system services are well written and able to detect that they for example run out of fd and they will just exit with a suitable error code instead of hanging forever and responding to clients that they are unable to handle their request due to lack of fd. This is hard specially when service use a lot of libraries under the hood because they also need to return this error code from each functions which opens some files. This is especially hard when using some proprietary services or libraries for we don't have access to source code.

2) Use cgroups to limit and monitor resources usage

Generally systemd creates a cgroup for each service. cgroups like memory cgroup has an ability to notify userspace when memory usage reaches some level. So for example systemd could get notification that one of cgroups is using more memory than it should but as long as it's not a hard limit of the cgroup this service is not going to even notice this. So instead of returning error from for example malloc() in service, systemd could just send signal to that service and ask it to exit gracefully and the restart it. The disadvantage of this solution is the need of having cgroup for each resource we would like to monitor. For now we have suitable cgroups for everything we need apart from file descriptors.

What do you think about this? Maybe you have some other ideas how we could achieve this?

Best regards,
--
Krzysztof Opasiak
Samsung R&D Institute Poland
Samsung Electronics
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html



[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]     [Monitors]

  Powered by Linux