On Tue, Nov 19, 2019 at 11:32:48AM +0800, chenxiang (M) wrote:
>
>
> 在 2019/11/19 11:17, Ming Lei 写道:
> > On Tue, Nov 19, 2019 at 11:05:55AM +0800, chenxiang (M) wrote:
> > > Hi Ming,
> > >
> > > 在 2019/11/19 9:42, Ming Lei 写道:
> > > > On Tue, Nov 19, 2019 at 09:25:30AM +0800, chenxiang (M) wrote:
> > > > > Hi,
> > > > >
> > > > > There are 128 cpus and 16 irqs for SAS controller in my system, and there
> > > > > are 4 Nodes, every 32 cpus are for one node (cpu0-31 for node0, cpu32-63 for
> > > > > node1, cpu64-95 for node2, cpu96-127 for node3).
> > > > > We use function pci_alloc_irq_vectors_affinity() to set the affinity of
> > > > > irqs.
> > > > >
> > > > > I find that before the patch (Fixes: b1a5a73e64e9 ("genirq/affinity: Spread
> > > > > vectors on node according to nr_cpu ratio")), the relationship between irqs
> > > > > and cpus is: irq0 bind to cpu0-7, irq1 bind to cpu8-15,
> > > > > irq2 bind to cpu16-23, irq3 bind to cpu24-31,irq4 bind to cpu32-39... irq15
> > > > > bind to cpu120-127. But after the patch, the relationship is changed: irq0
> > > > > bind to cpu32-39,
> > > > > irq1 bind to cpu40-47, ..., irq11 bind to cpu120-127, irq12 bind to cpu0-7,
> > > > > irq13 bind to cpu8-15, irq14 bind to cpu16-23, irq15 bind to cpu24-31.
> > > > >
> > > > > I notice that before calling the sort() in function alloc_nodes_vectors(),
> > > > > the id of array node_vectors[] is from 0,1,2,3. But after function sort(),
> > > > > the index of array node_vectors[] is 1,2,3,0.
> > > > > But i think it sorts according to the numbers of cpus in those nodes, so it
> > > > > should be the same as before calling sort() as the numbers of cpus in every
> > > > > node are 32.
> > > > Maybe there are more non-present CPUs covered by node 0.
> > > >
> > > > Could you provide the following log?
> > > >
> > > > 1) lscpu
> > > >
> > > > 2) ./dump-io-irq-affinity $PCI_ID_SAS
> > > >
> > > > http://people.redhat.com/minlei/tests/tools/dump-io-irq-affinity
> > > >
> > > > You need to figure out the PCI ID(the 1st column of lspci output) of the SAS
> > > > controller via lspci.
> > > Sorry, I can't access the link you provide, but i can provide those irqs'
> > > affinity in the attachment.
> > > I also write a small testcase, and find id is 1, 2, 3, 0 after calling
> > > sort() .
> > Runtime log from /proc/interrupts isn't useful for investigating
> > affinity allocation issue, please use the attached script for collecting
> > log.
>
> Note: there are 32 irqs for SAS controller, irq0-15 are other interrupts
> (such as phy up/down/channel....), only irq 16-31 are cq interrupts which is
> processed by function pci_alloc_irq_vectors_affinity().
> The log is as follows:
>
> Euler:~ # ./dump-io-irq-affinity 74:02.0
> kernel version:
> Linux Euler 5.4.0-rc2-14683-g74684b1-dirty #224 SMP PREEMPT Mon Nov 18
> 18:54:27 CST 2019 aarch64 aarch64 aarch64 GNU/Linux
> PCI name is 74:02.0: sdd
> cat: /proc/irq/65/smp_affinity_list: No such file or directory
> cat: /proc/irq/65/effective_affinity_list: No such file or directory
> irq 65, cpu list , effective list
> irq 66, cpu list 0-31, effective list 0
> irq 67, cpu list 0-31, effective list 0
> cat: /proc/irq/68/smp_affinity_list: No such file or directory
> cat: /proc/irq/68/effective_affinity_list: No such file or directory
> irq 68, cpu list , effective list
> cat: /proc/irq/69/smp_affinity_list: No such file or directory
> cat: /proc/irq/69/effective_affinity_list: No such file or directory
> irq 69, cpu list , effective list
> cat: /proc/irq/70/smp_affinity_list: No such file or directory
> cat: /proc/irq/70/effective_affinity_list: No such file or directory
> irq 70, cpu list , effective list
> cat: /proc/irq/71/smp_affinity_list: No such file or directory
> cat: /proc/irq/71/effective_affinity_list: No such file or directory
> irq 71, cpu list , effective list
> cat: /proc/irq/72/smp_affinity_list: No such file or directory
> cat: /proc/irq/72/effective_affinity_list: No such file or directory
> irq 72, cpu list , effective list
> cat: /proc/irq/73/smp_affinity_list: No such file or directory
> cat: /proc/irq/73/effective_affinity_list: No such file or directory
> irq 73, cpu list , effective list
> cat: /proc/irq/74/smp_affinity_list: No such file or directory
> cat: /proc/irq/74/effective_affinity_list: No such file or directory
> irq 74, cpu list , effective list
> cat: /proc/irq/75/smp_affinity_list: No such file or directory
> cat: /proc/irq/75/effective_affinity_list: No such file or directory
> irq 75, cpu list , effective list
> irq 76, cpu list 0-31, effective list 0
> cat: /proc/irq/77/smp_affinity_list: No such file or directory
> cat: /proc/irq/77/effective_affinity_list: No such file or directory
> irq 77, cpu list , effective list
> cat: /proc/irq/78/smp_affinity_list: No such file or directory
> cat: /proc/irq/78/effective_affinity_list: No such file or directory
> irq 78, cpu list , effective list
> cat: /proc/irq/79/smp_affinity_list: No such file or directory
> cat: /proc/irq/79/effective_affinity_list: No such file or directory
> irq 79, cpu list , effective list
> cat: /proc/irq/80/smp_affinity_list: No such file or directory
> cat: /proc/irq/80/effective_affinity_list: No such file or directory
> irq 80, cpu list , effective list
> irq 81, cpu list 32-39, effective list 32
> irq 82, cpu list 40-47, effective list 40
> irq 83, cpu list 48-55, effective list 48
> irq 84, cpu list 56-63, effective list 56
> irq 85, cpu list 64-71, effective list 64
> irq 86, cpu list 72-79, effective list 72
> irq 87, cpu list 80-87, effective list 80
> irq 88, cpu list 88-95, effective list 88
> irq 89, cpu list 96-103, effective list 96
> irq 90, cpu list 104-111, effective list 104
> irq 91, cpu list 112-119, effective list 112
> irq 92, cpu list 120-127, effective list 120
> irq 93, cpu list 0-7, effective list 0
> irq 94, cpu list 8-15, effective list 8
> irq 95, cpu list 16-23, effective list 16
> irq 96, cpu list 24-31, effective list 24
OK, nothing is wrong.
That is the implementation of sort().
If you provide the following array to sort():
struct elem {
unsigned id;
unsigned val;
};
struct elem A[4] = {{0, 4}, {1, 4}, {2, 4}, {3, 4}};
And the output will becomes:
A = {{1, 4}, {2, 4}, {3, 4}, {0, 4}};
It can't be problem for users of pci_alloc_irq_vectors_affinity() cause
all nodes are symmetric, can it?
Thanks,
Ming
