Re: [PATCH 01/10] irqchip: irq-mips-gic: export gic_send_ipi

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 08/26/2015 10:40 PM, Thomas Gleixner wrote:
On Wed, 26 Aug 2015, Qais Yousef wrote:
On 08/26/2015 04:08 PM, Thomas Gleixner wrote:
IPI = Inter Processor Interrupt

As the name says that's an interrupt which goes from one cpu to
another. So an IPI has a very clear target.
OK understood. My interpretation of the processor here was the difference. I
was viewing the whole linux cpus as one unit with regard to its coprocessors.
You can only view it this way if you talk about peripheral interrupts
which are not used as per cpu interrupts and can be routed to a single
cpu or a set of cpus via set_affinity.

Whether the platform implements IPIs via general interrupts which are
made affine to a particular cpu or some other specialized mechanism is
completely irrelevant. An IPI is not subject to affinity settings,
period.

So if you want to use an IPI then you need a target cpu for that IPI.

If you want something which can be affined to any cpu, then you need a
general interrupt and not an IPI.
We are using IPIs to exchange interrupts. Affinity is not important to me.
That's a bold statement. If you chose CPU x as the target for the
interrupts received from the coprocessor, then you have pinned the
processing for this stuff on to CPU x. So you limit the freedom of
moving stuff around on the linux cpus.

I said that because I thought you were telling me if I'm expecting my IPIs to be movable then I must be using general interrupts. So what I was saying is that we use IPIs and if it's against the rule for them to have affinity we can live with that.


And if your root irq controller supports sending normal device
interrupts in the same or a similar way as it sends IPIs you can spare
quite some extra handling on the linux side for receiving the
coprocessor interrupt, i.e. you can use just the bog standard
request_irq() mechanism and have the ability to set the affinity of
that interrupt from user space so you can move it to the core on which
your processing happens. Definitely simpler and more flexible, so I
would go there if the hardware allows.

That's what I was trying to say but words failed me to explain it clearly maybe :(


But back to the IPIs. We need infrastructure and DT support to:

1) reserve an IPI

2) send an IPI

3) request/free an IPI

#1 We have no infrastructure for that, but we definitely need one.

    We can look at the IPI as a single linux irq number which is
    replicated on all cpu cores. The replication can happen in hardware
    or by software, but that depends on the underlying root irq
    controller. How that is implemented does not matter for the
    reservation.

    The most flexible and platform independent solution would be to
    describe the IPI space as a seperate irq domain. In most cases this
    would be a hierarchical domain stacked on the root irq domain:

    [IPI-domain] --> [GIC-MIPS-domain]

    on x86 this would be:

    [IPI-domain] --> [vector-domain]

    That needs some change how the IPIs which are used by the kernel
    (rescheduling, function call ..) are set up, but we get a proper
    management and collision avoidance that way. Depending on the
    platform we could actually remove the whole IPI compile time
    reservation and hand out IPIs at boot time on demand and
    dynamically.

    So the reservation function would be something like:

    unsigned int irq_reserve_ipi(const struct cpumask *dest, void *devid);

    @dest contains the possible targets for the IPI. So for generic
    linux IPIs this would be cpu_possible_mask. For your coprocessor
    the target would be a cpumask with just the bit of the coprocessor
    core set. If you need to use an IPI for sending an interrupt from
    the coprocessor to a specific linux core then @dest will contain
    just that target cpu.

    @devid is stored in the IPI domain for sanity checks during
    operation.

    The function returns a linux irq number or 0 if allocation fails.

    We need a complementary interface as well, so you can hand back the
    IPI to the core when the coprocessor is disabled:

    void irq_destroy_ipi(unsigned int irq, void *devid);


    To configure your coprocessor proper, we need a translation
    mechanism from the linux interrupt number to the magic value which
    needs to be written into the trigger register when the coprocessor
    wants to send an interrupt or an IPI.

    int irq_get_irq_hwcfg(unsigned int irq, struct irq_hwcfg *cfg);

    struct irq_hwcfg needs to be defined, but it might look like this:

      {
	/* Generic fields */
	x;
	...
	union {
	      mips_gic;
	      ...
	};
      };

    The actual hw specific value(s) need to be filled in from the irq
    domain specific code.


#2 We have no generic mechanism for that either.

    Something like this is needed:

    void irq_send_ipi(unsigned int irq, const struct cpumask *dest,
    		     void *devid);	

    @dest is for generic linux IPIs and can be NULL so the IPI is sent to
     	 the core(s) which have been handed in at reservation time

    @devid is used to sanity check the driver call.

    So that finally will call down via a irq chip callback into the
    code which sends the IPI.


#3 Now you get lucky, because we actually have an interface for this

    request_percpu_irq()
    free_percpu_irq()
    disable_percpu_irq()
    enable_percpu_irq()

    Though there is a caveat. enable/disable_percpu_irq() must be
    called from the target cpu, but that should be a solvable problem.

    And at the IPI-domain side we need sanity checks whether the cpu
    from which enable/disable is called is actually configured in the
    reservation mask.
There are a few other nasty details, but that's not important for
    the big picture.

    As I said above, I really would recommend to avoid that if possible
    because a bog standard device interrupt is way simpler to deal
    with.

Agreed. Something for us to think about and consider. But if not for AXD this kind of mechanism is important for us for other reasons so we probably want to see it through.


That's certainly not the quick and dirty solution you are looking for,
but exposing IPIs to drivers by anything else than a well thought out
infrastructure is not going to happen.

Thanks a lot for the detailed explanation. I wasn't looking for a quick and dirty solution but my view of the problem is much simpler than yours so my idea of a solution would look quick and dirty. I have a better appreciation of the problem now and a way to approach it :-)

From DT point of view are we OK with this form then

    coprocessor {
            interrupt-source = <&intc INT_SPEC COP_HWAFFINITY>;
            interrupt-sink = <&intc INT_SPEC CPU_HWAFFINITY>;
    }

and if the root controller sends normal IPI as it sends normal device interrupts then interrupt-sink can be a standard interrupts property (like in my case)

    coprocessor {
            interrupt-source = <&intc INT_SPEC COP_HWAFFINITY>;
            interrupts = <INT_SPEC>;
    }

Does this look right to you? Is there something else that needs to be covered still?

One more thing I can think of now is that the coprocessor will need the raw irq numbers that are picked by linux so that it can use them to trigger the IPI. Are we ok to add a function that returns this raw irq number (as opposed to linux irq number) directly from DT? The way this is communicated to the coprocessor will be platform specific.

Thanks,
Qais


Thanks,

	tglx





[Index of Archives]     [Linux MIPS Home]     [LKML Archive]     [Linux ARM Kernel]     [Linux ARM]     [Linux]     [Git]     [Yosemite News]     [Linux SCSI]     [Linux Hams]

  Powered by Linux