Re: [RFC PATCH] thermal: add generic cpu hotplug cooling device

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Pinging again... Does anyone have any opinion on this feature?
Thanks,
Zoran

On 4 October 2013 15:52, Zoran Markovic <zoran.markovic@xxxxxxxxxx> wrote:
> Any comments on this proposed feature and implementation? Apparently
> it's also useful for server systems.
> Thanks,
> Zoran
>
> On 20 September 2013 15:15, Zoran Markovic <zoran.markovic@xxxxxxxxxx> wrote:
>> This patch implements a generic CPU hotplug cooling device. The
>> implementation scales down the number of running CPUs when temperature
>> increases through a thermal trip point and prevents booting CPUs
>> until thermal conditions are restored. Upon restoration, the action
>> of starting up a CPU is left to another entity (e.g. CPU offline
>> governor, for which a patch is in the works).
>>
>> In the past two years, ARM considerably reduced the time required for
>> CPUs to boot and shutdown; this time is now measured in microseconds.
>> This patch is predominantly intended for ARM big.LITTLE architectures
>> where big cores are expected to have a much bigger impact on thermal
>> budget than little cores, resulting in fast temperature ramps to a trip
>> point, i.e. thermal runaways. Switching off the big core(s) may be one
>> of the recovery mechanisms to restore system temperature, but the actual
>> strategy is left to the thermal governor.
>>
>> The assumption is that CPU shutdown/startup is a rare event, so no
>> attempt was made to make the code atomic, i.e. the code evidently races
>> with CPU hotplug driver. The set_cur_state() function offlines CPUs
>> iteratively one at a time, checking the cooling state before each CPU
>> shutdown. A hotplug notifier callback validates any CPU boot requests
>> against current cooling state and approves/denies accordingly. This
>> mechanism guarantees that the desired cooling state could be reached in a
>> maximum of d-c iterations, where d and c are the "desired" and "current"
>> cooling states expressed in the number of offline CPUs.
>>
>> Credits to Amit Daniel Kachhap for initial attempt to upstream this feature.
>>
>> Cc: Zhang Rui <rui.zhang@xxxxxxxxx>
>> Cc: Eduardo Valentin <eduardo.valentin@xxxxxx>
>> Cc: Rob Landley <rob@xxxxxxxxxxx>
>> Cc: Amit Daniel Kachhap <amit.daniel@xxxxxxxxxxx>
>> Cc: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
>> Cc: Durgadoss R <durgadoss.r@xxxxxxxxx>
>> Cc: Christian Daudt <bcm@xxxxxxxxxxxxx>
>> Cc: James King <james.king@xxxxxxxxxx>
>> Signed-off-by: Zoran Markovic <zoran.markovic@xxxxxxxxxx>
>> ---
>>  Documentation/thermal/cpu-cooling-api.txt |   17 ++
>>  drivers/thermal/Kconfig                   |   10 +
>>  drivers/thermal/Makefile                  |    1 +
>>  drivers/thermal/cpu_hotplug.c             |  362 +++++++++++++++++++++++++++++
>>  include/linux/cpuhp_cooling.h             |   57 +++++
>>  5 files changed, 447 insertions(+)
>>  create mode 100644 drivers/thermal/cpu_hotplug.c
>>  create mode 100644 include/linux/cpuhp_cooling.h
>>
>> diff --git a/Documentation/thermal/cpu-cooling-api.txt b/Documentation/thermal/cpu-cooling-api.txt
>> index fca24c9..2f94f68 100644
>> --- a/Documentation/thermal/cpu-cooling-api.txt
>> +++ b/Documentation/thermal/cpu-cooling-api.txt
>> @@ -30,3 +30,20 @@ the user. The registration APIs returns the cooling device pointer.
>>      This interface function unregisters the "thermal-cpufreq-%x" cooling device.
>>
>>      cdev: Cooling device pointer which has to be unregistered.
>> +
>> +1.2 cpu hotplug registration/unregistration APIs
>> +1.2.1 struct thermal_cooling_device *cpuhp_cooling_register(
>> +       struct cpumask *cpus, const char *ext)
>> +
>> +    This function creates and registers a cpu hotplug cooling device with
>> +    the name "cpu-hotplug-%s".
>> +
>> +    cpus: cpumask of cpu cores participating in cooling.
>> +    ext: instance-specific name of device
>> +
>> +1.2.2 void cpuhotplug_cooling_unregister(struct thermal_cooling_device *cdev)
>> +
>> +    This function unregisters and frees the cpu hotplug cooling device cdev.
>> +
>> +    cdev: Pointer to cooling device to unregister.
>> +
>> diff --git a/drivers/thermal/Kconfig b/drivers/thermal/Kconfig
>> index 52b6ed7..3509100 100644
>> --- a/drivers/thermal/Kconfig
>> +++ b/drivers/thermal/Kconfig
>> @@ -79,6 +79,16 @@ config CPU_THERMAL
>>
>>           If you want this support, you should say Y here.
>>
>> +config CPU_THERMAL_HOTPLUG
>> +       bool "Generic CPU hotplug cooling"
>> +       depends on HOTPLUG_CPU
>> +       help
>> +         Shutdown CPUs to prevent the device from overheating. This feature
>> +         uses generic CPU hot-unplug capabilities to control device
>> +         temperature. When the temperature increases over a trip point, a
>> +         random subset of CPUs is shut down to reach the desired cooling
>> +         state.
>> +
>>  config THERMAL_EMULATION
>>         bool "Thermal emulation mode support"
>>         help
>> diff --git a/drivers/thermal/Makefile b/drivers/thermal/Makefile
>> index 5ee0db0..0bd08be 100644
>> --- a/drivers/thermal/Makefile
>> +++ b/drivers/thermal/Makefile
>> @@ -12,6 +12,7 @@ thermal_sys-$(CONFIG_THERMAL_GOV_USER_SPACE)  += user_space.o
>>
>>  # cpufreq cooling
>>  thermal_sys-$(CONFIG_CPU_THERMAL)      += cpu_cooling.o
>> +thermal_sys-$(CONFIG_CPU_THERMAL_HOTPLUG)      += cpu_hotplug.o
>>
>>  # platform thermal drivers
>>  obj-$(CONFIG_SPEAR_THERMAL)    += spear_thermal.o
>> diff --git a/drivers/thermal/cpu_hotplug.c b/drivers/thermal/cpu_hotplug.c
>> new file mode 100644
>> index 0000000..8c3021e
>> --- /dev/null
>> +++ b/drivers/thermal/cpu_hotplug.c
>> @@ -0,0 +1,362 @@
>> +/*
>> + *  drivers/thermal/cpu_hotplug.c
>> + *
>> + *  Copyright (C) 2013  Broadcom Corporation Ltd.
>> + *  Copyright (C) 2013  Zoran Markovic <zoran.markovic@xxxxxxxxxx>
>> + *
>> + * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> + *  This program is free software; you can redistribute it and/or modify
>> + *  it under the terms of the GNU General Public License as published by
>> + *  the Free Software Foundation; version 2 of the License.
>> + *
>> + *  This program is distributed in the hope that it will be useful, but
>> + *  WITHOUT ANY WARRANTY; without even the implied warranty of
>> + *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> + *  General Public License for more details.
>> + *
>> + *  You should have received a copy of the GNU General Public License along
>> + *  with this program; if not, write to the Free Software Foundation, Inc.,
>> + *  59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.
>> + *
>> + * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> + */
>> +#include <linux/module.h>
>> +#include <linux/thermal.h>
>> +#include <linux/workqueue.h>
>> +#include <linux/cpu.h>
>> +#include <linux/err.h>
>> +#include <linux/slab.h>
>> +#include <linux/cpuhp_cooling.h>
>> +
>> +/**
>> + * struct cpuhotplug_cooling_device - cpu hotplug cooling device data
>> + * @cpus: cpu mask representing cpus that can be hot-unplugged for cooling
>> + * @cdev: pointer to generic cooling device
>> + */
>> +struct cpuhotplug_cooling_device {
>> +       unsigned int target;
>> +       struct cpumask cpus;
>> +       struct thermal_cooling_device *cdev;
>> +       struct list_head list;
>> +};
>> +
>> +/**
>> + * cpuhotplug_list - list of all cpu hotplug cooling devices. Traversed
>> + * by cpu hotplug notifier to check constraints on booting cpus. Locked
>> + * by cpuhotplug_cooling_lock mutex.
>> + */
>> +static LIST_HEAD(cpuhotplug_list);
>> +static DEFINE_MUTEX(cpuhotplug_cooling_lock);
>> +
>> +/**
>> + * boot_cpu - return index of boot CPU; same criteria as in
>> + * disable_nonboot_cpus()
>> + */
>> +static inline int boot_cpu(void)
>> +{
>> +       int cpu;
>> +       get_online_cpus();
>> +       cpu = cpumask_first(cpu_online_mask);
>> +       put_online_cpus();
>> +       return cpu;
>> +}
>> +
>> +/**
>> + * random_online_cpu - pick any online hot-unpluggable cpu
>> + * @d: pointer to cpuhotplug_cooling_device containing hot-pluggable cpu mask
>> + */
>> +static inline int random_online_cpu(struct cpuhotplug_cooling_device *d)
>> +{
>> +       int cpu;
>> +
>> +       get_online_cpus();
>> +       cpu = any_online_cpu(d->cpus);
>> +       put_online_cpus();
>> +
>> +       return cpu;
>> +}
>> +
>> +/**
>> + * _num_offline_cpus - number of hot-pluggable cpus currently offline
>> + * @d: pointer to cpuhotplug_cooling_device containing hot-pluggable cpu mask
>> + */
>> +static inline int _num_offline_cpus(struct cpuhotplug_cooling_device *d)
>> +{
>> +       struct cpumask offline;
>> +
>> +       cpumask_andnot(&offline, &(d->cpus), cpu_online_mask);
>> +       return cpumask_weight(&offline);
>> +}
>> +
>> +/**
>> + * num_offline_cpus - same as _num_offline_cpus, but safe from background
>> + * hotplug events.
>> + * @d: pointer to cpuhotplug_cooling_device containing hot-pluggable cpu mask
>> + */
>> +static inline int num_offline_cpus(struct cpuhotplug_cooling_device *d)
>> +{
>> +       int num;
>> +
>> +       get_online_cpus();
>> +       num = _num_offline_cpus(d);
>> +       put_online_cpus();
>> +
>> +       return num;
>> +}
>> +
>> +/**
>> + * cpuhotplug_get_max_state - get maximum cooling state of device
>> + * @cdev: pointer to generic cooling device
>> + * @state: returned maximum cooling state
>> + *
>> + * Thermal framework callback to get the maximum cooling state of cpu
>> + * hotplug cooling device.
>> + *
>> + * Return: always 0.
>> + */
>> +static int cpuhotplug_get_max_state(struct thermal_cooling_device *cdev,
>> +                                   unsigned long *state)
>> +{
>> +       struct cpuhotplug_cooling_device *d = cdev->devdata;
>> +
>> +       /* defined as number of CPUs in hot-pluggable mask: this is invariant */
>> +       *state = cpumask_weight(&(d->cpus));
>> +
>> +       return 0;
>> +}
>> +
>> +/**
>> + * cpuhotplug_get_cur_state - get current cooling state of device
>> + * @cdev: pointer to generic cooling device
>> + * @state: current cooling state
>> + *
>> + * Thermal framework callback to get the current cooling state of cpu
>> + * hotplug cooling device.
>> + *
>> + * Return: always 0.
>> + */
>> +static int cpuhotplug_get_cur_state(struct thermal_cooling_device *cdev,
>> +                                   unsigned long *state)
>> +{
>> +       struct cpuhotplug_cooling_device *d = cdev->devdata;
>> +
>> +       *state = d->target;
>> +
>> +       return 0;
>> +}
>> +
>> +/**
>> + * cpuhotplug_get_cur_state - set cooling state of device
>> + * @cdev: pointer to generic cooling device
>> + * @state: cooling state
>> + *
>> + * Thermal framework callback to set/change cooling state of cpu hotplug
>> + * cooling device.
>> + *
>> + * Return: 0 on success, or error code otherwise
>> + */
>> +static int cpuhotplug_set_cur_state(struct thermal_cooling_device *cdev,
>> +                                   unsigned long state)
>> +{
>> +       struct cpuhotplug_cooling_device *d = cdev->devdata;
>> +       unsigned long cstate;
>> +       unsigned int cpu;
>> +       int err = 0;
>> +
>> +       if (state > cpumask_weight(&(d->cpus)))
>> +               return -EINVAL; /* out of allowed range */
>> +
>> +       /*
>> +        * Set target state here; hot-unplug CPUs if we are too hot, but
>> +        * don't attempt to hot-plug CPUs if we're cold. Starting CPUs
>> +        * should be left to CPUOffline governor.
>> +        *
>> +        * There is a chance that CPU hotplug driver is racing with this
>> +        * code. Rather than trying to make the procedure atomic, iterate
>> +        * until we reach the desired state, or signal error if the state
>> +        * cannot be reached.
>> +        *
>> +        * Neither CPU hotplug nor this code is expected to run too often.
>> +        */
>> +       d->target = state;
>> +
>> +       /* compare desired cooling state to current cooling state */
>> +       while ((cstate = num_offline_cpus(d)) < state && !err) {
>> +               /* cstate < cstate: we're too hot, unplug any cpu */
>> +               cpu = random_online_cpu(d);
>> +               if (cpu < nr_cpu_ids)
>> +                       err = work_on_cpu(boot_cpu(),
>> +                                         (long(*)(void *))cpu_down,
>> +                                         (void *)cpu);
>> +                       /* on error, message would come from cpu_down() */
>> +               else {
>> +                       pr_warn("cpuhotplug: CPUs already down\n");
>> +                       err = -EAGAIN;
>> +               }
>> +       }
>> +
>> +       return err;
>> +}
>> +
>> +/* cpu hotplug cooling device ops */
>> +static struct thermal_cooling_device_ops const cpuhotplug_cooling_ops = {
>> +       .get_max_state = cpuhotplug_get_max_state,
>> +       .get_cur_state = cpuhotplug_get_cur_state,
>> +       .set_cur_state = cpuhotplug_set_cur_state,
>> +};
>> +
>> +/**
>> + * _cpu_startup_allowed - traverse list of hotplug cooling devices to
>> + * check if startup of cpu violates thermal constraints
>> + */
>> +static inline int _cpu_startup_allowed(int cpu)
>> +{
>> +       struct cpuhotplug_cooling_device *d;
>> +       int ret = 1;
>> +
>> +       /*
>> +        * Prevent starting CPU if it violates any cooling
>> +        * device's constraint. Called from hotplug notifier, so
>> +        * cpu_up()/cpu_down() already holds a lock on hotplug
>> +        * events.
>> +        */
>> +       mutex_lock(&cpuhotplug_cooling_lock);
>> +       list_for_each_entry(d, &cpuhotplug_list, list) {
>> +               if (cpumask_test_cpu(cpu, &(d->cpus)) &&
>> +                   d->target >= _num_offline_cpus(d)) {
>> +                       pr_warn("%s: CPU%d startup prevented\n",
>> +                               dev_name(&(d->cdev->device)), cpu);
>> +                       ret = 0;
>> +                       break;
>> +               }
>> +       }
>> +       mutex_unlock(&cpuhotplug_cooling_lock);
>> +       return ret;
>> +}
>> +
>> +/**
>> + * cpuhotplug_thermal_notifier - notifier callback for CPU hotplug events.
>> + * @nb: struct notifier_block
>> + * @event: cpu hotplug event for which callback is invoked.
>> + * @data: context data, in this particular case CPU index.
>> + *
>> + * Callback intercepting CPU hotplug events. Compares CPU hotplug action
>> + * with current thermal state and allows/denies accordingly.
>> + *
>> + * Return: 0 (allow) or error (deny).
>> + */
>> +static int cpuhotplug_thermal_notifier(struct notifier_block *nb,
>> +                                      unsigned long event, void *data)
>> +{
>> +       int cpu = (int)data;
>> +
>> +       switch (event) {
>> +       case CPU_UP_PREPARE:
>> +       case CPU_UP_PREPARE_FROZEN:
>> +               /* _cpu_up() only cares about result from CPU_UP_PREPAREs */
>> +               if (!_cpu_startup_allowed(cpu))
>> +                       return notifier_from_errno(-EAGAIN);
>> +               break;
>> +       default:
>> +               /* allow all other actions */
>> +               break;
>> +       }
>> +       return 0;
>> +}
>> +
>> +static struct notifier_block cpuhotplug_thermal_notifier_block = {
>> +       .notifier_call = cpuhotplug_thermal_notifier,
>> +};
>> +
>> +/**
>> + * cpuhotplug_cooling_register - create cpu hotplug cooling device
>> + * @cpus: cpumask of cpu cores participating in cooling
>> + * @ext: instance-specific name of device
>> + *
>> + * Creates and registers a cpu hotplug cooling device with the name
>> + * "cpu-hotplug-<ext>".
>> + *
>> + * Return: valid pointer to cpuhotplug_cooling_device struct on success,
>> + * corresponding ERR_PTR() on failure.
>> + */
>> +struct thermal_cooling_device *
>> +cpuhotplug_cooling_register(const struct cpumask *cpus, const char *ext)
>> +{
>> +       struct thermal_cooling_device *cdev;
>> +       struct cpuhotplug_cooling_device *cpuhotplug_cdev;
>> +       struct cpumask test;
>> +       char name[THERMAL_NAME_LENGTH];
>> +       int err;
>> +
>> +       /* test if we passed in a good cpumask */
>> +       cpu_maps_update_begin();
>> +       cpumask_and(&test, cpus, cpu_possible_mask);
>> +       cpu_maps_update_done();
>> +
>> +       if (cpumask_test_cpu(boot_cpu(), &test)) {
>> +               pr_warn("cannot hot-plug boot CPU%d\n", boot_cpu());
>> +               cpumask_clear_cpu(boot_cpu(), &test);
>> +       }
>> +       if (cpumask_empty(&test)) {
>> +               pr_err("CPUs unavailable for hot-plug cooling\n");
>> +               err = -EINVAL;
>> +               goto out;
>> +       }
>> +
>> +       cpuhotplug_cdev = kzalloc(sizeof(struct cpuhotplug_cooling_device),
>> +                            GFP_KERNEL);
>> +       if (!cpuhotplug_cdev) {
>> +               err = -ENOMEM;
>> +               goto out;
>> +       }
>> +
>> +       cpumask_copy(&cpuhotplug_cdev->cpus, &test);
>> +
>> +       snprintf(name, sizeof(name), "cpu-hotplug-%s", ext);
>> +
>> +       cpuhotplug_cdev->target = 0;
>> +       cdev = thermal_cooling_device_register(name, cpuhotplug_cdev,
>> +                                              &cpuhotplug_cooling_ops);
>> +       if (!cdev) {
>> +               pr_err("%s: cooling device registration failed.\n", name);
>> +               err = -EINVAL;
>> +               goto out_free;
>> +       }
>> +       cpuhotplug_cdev->cdev = cdev;
>> +
>> +       mutex_lock(&cpuhotplug_cooling_lock);
>> +       if (list_empty(&cpuhotplug_list))
>> +               register_cpu_notifier(&cpuhotplug_thermal_notifier_block);
>> +       list_add(&(cpuhotplug_cdev->list), &cpuhotplug_list);
>> +       mutex_unlock(&cpuhotplug_cooling_lock);
>> +
>> +       return cdev;
>> +
>> +out_free:
>> +       kfree(cpuhotplug_cdev);
>> +out:
>> +       return ERR_PTR(err);
>> +}
>> +EXPORT_SYMBOL_GPL(cpuhotplug_cooling_register);
>> +
>> +/**
>> + * cpuhotplug_cooling_unregister - remove cpu hotplug cooling device
>> + * @cdev: cooling device to remove
>> + *
>> + * Unregisters and frees the cpu hotplug cooling device.
>> + */
>> +void cpuhotplug_cooling_unregister(struct thermal_cooling_device *cdev)
>> +{
>> +       struct cpuhotplug_cooling_device *cpuhotplug_cdev = cdev->devdata;
>> +
>> +       mutex_lock(&cpuhotplug_cooling_lock);
>> +       list_del(&(cpuhotplug_cdev->list));
>> +       if (list_empty(&cpuhotplug_list))
>> +               unregister_cpu_notifier(&cpuhotplug_thermal_notifier_block);
>> +       mutex_unlock(&cpuhotplug_cooling_lock);
>> +
>> +       thermal_cooling_device_unregister(cpuhotplug_cdev->cdev);
>> +       kfree(cpuhotplug_cdev);
>> +}
>> +EXPORT_SYMBOL_GPL(cpuhotplug_cooling_unregister);
>> +
>> diff --git a/include/linux/cpuhp_cooling.h b/include/linux/cpuhp_cooling.h
>> new file mode 100644
>> index 0000000..ace1d5b
>> --- /dev/null
>> +++ b/include/linux/cpuhp_cooling.h
>> @@ -0,0 +1,57 @@
>> +/*
>> + *  linux/include/linux/cpuhp_cooling.h
>> + *
>> + *  Copyright (C) 2013 Broadcom Corporation Ltd.
>> + *  Copyright (C) 2013 Zoran Markovic <zoran.markovic@xxxxxxxxxx>
>> + *
>> + * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> + *  This program is free software; you can redistribute it and/or modify
>> + *  it under the terms of the GNU General Public License as published by
>> + *  the Free Software Foundation; version 2 of the License.
>> + *
>> + *  This program is distributed in the hope that it will be useful, but
>> + *  WITHOUT ANY WARRANTY; without even the implied warranty of
>> + *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> + *  General Public License for more details.
>> + *
>> + *  You should have received a copy of the GNU General Public License along
>> + *  with this program; if not, write to the Free Software Foundation, Inc.,
>> + *  59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.
>> + *
>> + * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> + */
>> +
>> +#ifndef __CPUHP_COOLING_H__
>> +#define __CPUHP_COOLING_H__
>> +
>> +#include <linux/thermal.h>
>> +#include <linux/cpumask.h>
>> +
>> +#ifdef CONFIG_CPU_THERMAL_HOTPLUG
>> +/**
>> + * cpuhotplug_cooling_register - create cpu hotplug cooling device.
>> + * @cpus: cpumask of hot-pluggable cpus
>> + * @ext: instance-specific device name
>> + */
>> +struct thermal_cooling_device *
>> +cpuhotplug_cooling_register(const struct cpumask *cpus, const char *ext);
>> +
>> +/**
>> + * cpuhotplug_cooling_unregister - remove cpu hoptlug cooling device.
>> + * @cdev: thermal cooling device pointer.
>> + */
>> +void cpuhotplug_cooling_unregister(struct thermal_cooling_device *cdev);
>> +#else /* !CONFIG_CPU_THERMAL_HOTPLUG */
>> +static inline struct thermal_cooling_device *
>> +cpuhotplug_cooling_register(const struct cpumask *cpus, const char *ext)
>> +{
>> +       return NULL;
>> +}
>> +static inline
>> +void cpuhotplug_cooling_unregister(struct thermal_cooling_device *cdev)
>> +{
>> +       return;
>> +}
>> +#endif /* CONFIG_CPU_THERMAL_HOTPLUG */
>> +
>> +#endif /* __CPU_COOLING_H__ */
>> --
>> 1.7.9.5
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Linux FS]     [Yosemite Forum]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Video 4 Linux]     [Device Mapper]     [Linux Resources]

  Powered by Linux