Re: Difference between logical and physical cpu hotplug

[Srivatsa Bhat <bhat.srivatsa@xxxxxxxxx>]

Hi Vaibhav,

On Tue, Aug 23, 2011 at 11:40 AM, Vaibhav Jain <vjoss197@xxxxxxxxx> wrote:

On Sun, Aug 21, 2011 at 1:09 PM, Srivatsa Bhat <bhat.srivatsa@xxxxxxxxx> wrote:

On Sat, Aug 20, 2011 at 4:05 AM, Vaibhav Jain <vjoss197@xxxxxxxxx> wrote:

Awesome explanation Srivatsa!! Thanks a lot!!

Just had one more doubt. I am a little unclear about how the APs get initialized in the beginning. In the case of Boot Processor

its just like a uniprocessor system. But how do the APs start executing code ?

Could you please explain a little ?

Sure. But please note that I will stick to Intel architecture while explaining the details.

The Boot CPU or the Boot-Strap Processor (BSP) is the one which boots the Operating System. Then it wakes up the APs (Application Processors) when it is the right time.

Let us now explore some background details to understand how all this works.
On uniprocessor systems we use PIC (Programmable Interrupt Controller) like the 8259A Interrupt Controller chip to deliver interrupts to the processor.
On Multi-Processor (MP) systems, we use something known as APICs (Advanced Programmable Interrupt Controllers). Every processor has a local APIC.
And there are one or more I/O APICs in the system that are shared by all the processors. As the name suggests, I/O APICs are used to deliver interrupts from I/O devices to the processors, via the local APICs.

All local APICs have unique IDs that are assigned either by the hardware or the BIOS during the initialization phase. Using the local APIC ID we can identify the processors in the system.

Using these local APICs, we can send something known as "Inter-Processor Interrupts" or IPIs. As the name suggests, this is a mechanism for one processor to interrupt another processor in the system. Note that this mechanism can be used by any processor to talk to any other processor in
the system (no distinction between BSP and APs here).

To kick-start the APs, the BSP sends INIT IPI to each AP in turn, waits for some time for the IPI to be delivered to the AP and then checks if that AP booted up. Based on the version of the APIC used, the BSP might have to send 2 STARTUP IPIs to the APs with some time delay after each of the IPIs.
[ If you have discrete APICs (i.e., 82489DX APIC) then INIT IPI will do. If you have integrated APIC, you need to send two STARTUP IPIs. ]
All this is in accordance with the "Universal Start-up Algorithm" to start APs, as specified by Intel architecture. These IPIs cause an INIT at the AP to which it was delivered.

Now the question is, how do you make the APs to execute a particular piece of code (i.e., jump to a specified location) on start-up?
We know that whenever a processor starts after a RESET or INIT, it starts executing code from the reset vector (a predefined location).
However if you want a processor to immediately jump to an address that you have specified, you must use the INIT IPI as part of a "warm-reset".
Warm-reset allows you to send INIT signal to a processor without causing the processor to go through the entire BIOS initialization (POST -- see below for details) and then start the processor's execution at the warm-reset-vector.

By putting the appropriate pointer (i.e., pointer to the AP start-up code) in the warm-reset-vector (system RAM location 40:67h), setting the BIOS shutdown code to 0Ah (which tells the BIOS that this INIT is part of a warm-reset) and then causing an INIT at the processor (via the IPIs), the Operating System can cause the processor to jump immediately to any location and start executing that code. This is how the BSP can boot the APs and make them execute some particular piece of code (in this case, the AP start-up code as designed in the OS).

It would be worthwhile to understand what would be the state of the system (and the APs) before the Operating System gets control from the BIOS after switching ON the machine. The BIOS, upon system start, performs a procedure known as POST (Power-On Self Test). This is to check the status of all the components/circuitry of the system, including the processors, to ensure that they are all functioning properly. During this phase the BIOS initializes all the circuitry (including all the APICs and the processors) to some known configuration and then puts all the APs to the HALT state with interrupts disabled. This is to ensure that the APs don't execute Operating System code (we want only the BSP to execute the OS code initially). Then the BSP starts executing OS code.

To boot APs, the BSP sends IPIs to them. But IPIs are non-maskable (note that the APs were in HALT state, with interrupts disabled). Hence the BSP will be able to kick-start AP execution and by using the warm-reset mechanism, it can direct the APs to execute some particular piece of code at startup. The BSP would have put a pointer to that AP start-up code in the warm-reset-vector address before sending the INIT or STARTUP IPIs to the APs.

You might be wondering how does the BSP specify to its local APIC as to which AP it must send an IPI to..
The answer is simple. During BIOS POST, an MP (Multi-Processor) Configuration Table will be set up (in conjunction with BSP and APs) in a well-known region of memory, which will be read by the OS during boot up. This table contains the local APIC IDs of all the processors.
So, while sending the targeted IPIs using its local APIC, the BSP specifies the local APIC ID of the target AP which it wants to interrupt (and boot in this case). This ensures the delivery of the IPI to the correct AP.

In short, this is how a Multi-Processor system gets rolling ... :-)

For more details you can refer:

1. Intel Multi-Processor Specification, especially Appendix A and B.
    http://www.intel.com/design/pentium/datashts/242016.htm

2. linux/arch/x86/kernel/smpboot.c :

    do_boot_cpu(), wakeup_secondary_cpu_via_init(), native_cpu_up(), start_secondary()

3. linux/arch/x86/kernel/head_32.S:
    startup_32_smp()

4. linux/arch/x86/kernel/trampoline_32.S

5. http://tldp.org/HOWTO/Linux-i386-Boot-Code-HOWTO/smpboot.html

Regards,
Srivatsa S. Bhat

 

Srivatsa, you are awsome!  Thanks a lot!!

Ha ha :) Thank you!
You are welcome :)
 

I am just wondering what all is required to gain this depth of knowledge :)

I feel humbled by your comment!
By the way, you can get information about almost anything and everything about Linux nowadays,
given the amount of interest it has generated and the uncountable resources available on the Internet,
most important among them being the open source code itself..
You just need to look in the right place I guess.. :)

Regards,
Srivatsa S. Bhat

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@xxxxxxxxxxxxxxxxx
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies