On Fri, 2012-12-07 at 13:57 +0800, Jiang Liu wrote: > On 2012-12-7 10:57, Toshi Kani wrote: > > On Fri, 2012-12-07 at 00:40 +0800, Jiang Liu wrote: > >> On 12/04/2012 08:10 AM, Toshi Kani wrote: > >>> On Mon, 2012-12-03 at 12:25 +0800, Hanjun Guo wrote: > >>>> On 2012/11/30 6:27, Toshi Kani wrote: > >>>>> On Thu, 2012-11-29 at 12:48 +0800, Hanjun Guo wrote: : > >>> > >>> If I read the code right, the framework calls ACPI drivers differently > >>> at boot-time and hot-add as follows. That is, the new entry points are > >>> called at hot-add only, but .add() is called at both cases. This > >>> requires .add() to work differently. > >>> > >>> Boot : .add() > >>> Hot-Add : .add(), .pre_configure(), configure(), etc. > >>> > >>> I think the boot-time and hot-add initialization should be done > >>> consistently. While there is difficulty with the current boot sequence, > >>> the framework should be designed to allow them consistent, not make them > >>> diverged. > >> Hi Toshi, > >> We have separated hotplug operations from driver binding/unbinding interface > >> due to following considerations. > >> 1) Physical CPU and memory devices are initialized/used before the ACPI subsystem > >> is initialized. So under normal case, .add() of processor and acpi_memhotplug only > >> figures out information about device already in working state instead of starting > >> the device. > > > > I agree that the current boot sequence is not very hot-plug friendly... > > > >> 2) It's impossible to rmmod the processor and acpi_memhotplug driver at runtime > >> if .remove() of CPU and memory drivers do really remove the CPU/memory device > >> from the system. And the ACPI processor driver also implements CPU PM funcitonality > >> other than hotplug. > > > > Agreed. > > > >> And recently Rafael has mentioned that he has a long term view to get rid of the > >> concept of "ACPI device". If that happens, we could easily move the hotplug > >> logic from ACPI device drivers into the hotplug framework if the hotplug logic > >> is separated from the .add()/.remove() callbacks. Actually we could even move all > >> hotplug only logic into the hotplug framework and don't rely on any ACPI device > >> driver any more. So we could get rid of all these messy things. We could achieve > >> that by: > >> 1) moving code shared by ACPI device drivers and the hotplug framework into the core. > >> 2) moving hotplug only code to the framework. > > > > Yes, the framework should allow such future work. I also think that the > > framework itself should be independent from such ACPI issue. Ideally, > > it should be able to support non-ACPI platforms. > The same point here. The ACPI based hotplug framework is designed as: > 1) an ACPI based hotplug slot driver to handle platform specific logic. > Platform may provide platform specific slot drivers to discover, manage > hotplug slots. We have provided a default implementation of slot driver > according to the ACPI spec. The ACPI spec does not define that _EJ0 is required to receive a hot-add request, i.e. bus/device check. This is a major issue. Since Windows only supports hot-add, I think there are platforms that only support hot-add today. > 2) an ACPI based hotplug manager driver, which is a platform independent > driver and manages all hotplug slot created by the slot driver. It is surely impressive work, but I think is is a bit overdoing. I expect hot-pluggable servers come with management console and/or GUI where a user can manage hardware units and initiate hot-plug operations. I do not think the kernel needs to step into such area since it tends to be platform-specific. > We haven't gone further enough to provide an ACPI independent hotplug framework > because we only have experience with x86 and Itanium, both are ACPI based. > We may try to implement an ACPI independent hotplug framework by pushing all > ACPI specific logic into the slot driver, I think it's doable. But we need > suggestions from experts of other architectures, such as SPARC and Power. > But seems Power already have some sorts of hotplug framework, right? I do not know about the Linux hot-plug support on other architectures. PA-RISC SuperDome also supports Node hot-plug, but it is not supported by Linux. Since ARM is getting used by servers, I would not surprise if there will be an ARM based server with hot-plug support in future. > >> Hi Rafael, what's your thoughts here? > >> > >>> > >>>>>>> 1. Validate phase - Verify if the request is a supported operation. All > >>>>>>> known restrictions are verified at this phase. For instance, if a > >>>>>>> hot-remove request involves kernel memory, it is failed in this phase. > >>>>>>> Since this phase makes no change, no rollback is necessary to fail. > >>>>>> > >>>>>> Yes, we have done this in acpihp_drv_pre_execute, and check following things: > >>>>>> > >>>>>> 1) Hot-plugble or not. the instance kernel memory you mentioned is also checked > >>>>>> when memory device remove; > >>>>> > >>>>> Agreed. > >>>>> > >>>>>> 2) Dependency check involved. For instance, if hot-add a memory device, > >>>>>> processor should be added first, otherwise it's not valid to this operation. > >>>>> > >>>>> I think FW should be the one that assures such dependency. That is, > >>>>> when a memory device object is marked as present/enabled/functioning, it > >>>>> should be ready for the OS to use. > >>>> > >>>> Yes, BIOS should do something for the dependency, because BIOS knows the > >>>> actual hardware topology. > >>> > >>> Right. > >>> > >>>> The ACPI specification provides _EDL method to > >>>> tell OS the eject device list, but still has no method to tell OS the add device > >>>> list now. > >>> > >>> Yes, but I do not think the OS needs special handling for add... > >> We have a plan to support triggering hot-adding events from OS provided interfaces, > >> so we also need to solve dependency issues when handling requests from those interfaces. > >> For need to power on the physical processor before powering on a memory device if > >> the memory device is attached to a physical processor. > > > > I am afraid that this issue is platform-specific, and I am not sure if > > there is a common way to handle such things in general. I'd recommend > > to work with FW folks to implement such platform-specific validation > > code in FW. > You are right, we may rely on firmware to validate the dependency. Great! > >>>> For some cases, OS should analyze the dependency in the validate phase. For example, > >>>> when hot remove a node (container device), OS should analyze the dependency to get > >>>> the remove order as following: > >>>> 1) Host bridge; > >>>> 2) Memory devices; > >>>> 3) Processor devices; > >>>> 4) Container device itself; > >>> > >>> This may be off-topic, but how do you plan to delete I/O devices under a > >>> node? Are you planning to delete all I/O devices along with the node? > >>> > >>> On other OS, we made a separate step called I/O chassis delete, which > >>> off-lines all I/O devices under the node, and is required before a node > >>> hot-remove. It basically triggers PCIe hot-remove to detach drivers > >>> from all devices. It does not eject the devices so that they do not > >>> have to be on hot-plug slots. This step runs user-space scripts to > >>> verify if the devices can be off-lined without disrupting user's > >>> applications, and provides comprehensive reports if any of them are in > >>> use. Not sure if Linux's PCI hot-remove has such check, but I thought > >>> I'd mention it. :) > >> Yinghai is working on PCI host bridge hotplug, which just stops all PCI devices > >> under the host bridge. That's really a little dangerous and we do need help > >> from userspace to check whether the hot-removal operaitons is fatal, > >> e.g. removing PCI device hosting the rootfs. > > > > Agreed. > > > >> So in our framework, we have an option to relay hotplug event from firmware > >> to userspace, so the userspace has a chance to reject the hotplug operations > >> if it may cause unacceptable disturbance to userspace services. > > > > I think validation from user-space is necessary for deleting I/O > > devices. For CPU and memory, the kernel check works fine. > Agreed. But we may need help from userspace to handle cgroup/cpuset/cpuisol > etc for cpu and memory hot-removal. Especially for telecom applications, they > have strong dependency on cgroup/cpuisol to guarantee latency. I have not looked at the code, but isn't these cpu attributes managed in the kernel? Thanks, -Toshi -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>