在 2019/11/19 11:17, Ming Lei 写道:
On Tue, Nov 19, 2019 at 11:05:55AM +0800, chenxiang (M) wrote:
Hi Ming,
在 2019/11/19 9:42, Ming Lei 写道:
On Tue, Nov 19, 2019 at 09:25:30AM +0800, chenxiang (M) wrote:
Hi,
There are 128 cpus and 16 irqs for SAS controller in my system, and there
are 4 Nodes, every 32 cpus are for one node (cpu0-31 for node0, cpu32-63 for
node1, cpu64-95 for node2, cpu96-127 for node3).
We use function pci_alloc_irq_vectors_affinity() to set the affinity of
irqs.
I find that before the patch (Fixes: b1a5a73e64e9 ("genirq/affinity: Spread
vectors on node according to nr_cpu ratio")), the relationship between irqs
and cpus is: irq0 bind to cpu0-7, irq1 bind to cpu8-15,
irq2 bind to cpu16-23, irq3 bind to cpu24-31,irq4 bind to cpu32-39... irq15
bind to cpu120-127. But after the patch, the relationship is changed: irq0
bind to cpu32-39,
irq1 bind to cpu40-47, ..., irq11 bind to cpu120-127, irq12 bind to cpu0-7,
irq13 bind to cpu8-15, irq14 bind to cpu16-23, irq15 bind to cpu24-31.
I notice that before calling the sort() in function alloc_nodes_vectors(),
the id of array node_vectors[] is from 0,1,2,3. But after function sort(),
the index of array node_vectors[] is 1,2,3,0.
But i think it sorts according to the numbers of cpus in those nodes, so it
should be the same as before calling sort() as the numbers of cpus in every
node are 32.
Maybe there are more non-present CPUs covered by node 0.
Could you provide the following log?
1) lscpu
2) ./dump-io-irq-affinity $PCI_ID_SAS
http://people.redhat.com/minlei/tests/tools/dump-io-irq-affinity
You need to figure out the PCI ID(the 1st column of lspci output) of the SAS
controller via lspci.
Sorry, I can't access the link you provide, but i can provide those irqs'
affinity in the attachment.
I also write a small testcase, and find id is 1, 2, 3, 0 after calling
sort() .
Runtime log from /proc/interrupts isn't useful for investigating
affinity allocation issue, please use the attached script for collecting
log.
Note: there are 32 irqs for SAS controller, irq0-15 are other interrupts
(such as phy up/down/channel....), only irq 16-31 are cq interrupts
which is processed by function pci_alloc_irq_vectors_affinity().
The log is as follows:
Euler:~ # ./dump-io-irq-affinity 74:02.0
kernel version:
Linux Euler 5.4.0-rc2-14683-g74684b1-dirty #224 SMP PREEMPT Mon Nov 18
18:54:27 CST 2019 aarch64 aarch64 aarch64 GNU/Linux
PCI name is 74:02.0: sdd
cat: /proc/irq/65/smp_affinity_list: No such file or directory
cat: /proc/irq/65/effective_affinity_list: No such file or directory
irq 65, cpu list , effective list
irq 66, cpu list 0-31, effective list 0
irq 67, cpu list 0-31, effective list 0
cat: /proc/irq/68/smp_affinity_list: No such file or directory
cat: /proc/irq/68/effective_affinity_list: No such file or directory
irq 68, cpu list , effective list
cat: /proc/irq/69/smp_affinity_list: No such file or directory
cat: /proc/irq/69/effective_affinity_list: No such file or directory
irq 69, cpu list , effective list
cat: /proc/irq/70/smp_affinity_list: No such file or directory
cat: /proc/irq/70/effective_affinity_list: No such file or directory
irq 70, cpu list , effective list
cat: /proc/irq/71/smp_affinity_list: No such file or directory
cat: /proc/irq/71/effective_affinity_list: No such file or directory
irq 71, cpu list , effective list
cat: /proc/irq/72/smp_affinity_list: No such file or directory
cat: /proc/irq/72/effective_affinity_list: No such file or directory
irq 72, cpu list , effective list
cat: /proc/irq/73/smp_affinity_list: No such file or directory
cat: /proc/irq/73/effective_affinity_list: No such file or directory
irq 73, cpu list , effective list
cat: /proc/irq/74/smp_affinity_list: No such file or directory
cat: /proc/irq/74/effective_affinity_list: No such file or directory
irq 74, cpu list , effective list
cat: /proc/irq/75/smp_affinity_list: No such file or directory
cat: /proc/irq/75/effective_affinity_list: No such file or directory
irq 75, cpu list , effective list
irq 76, cpu list 0-31, effective list 0
cat: /proc/irq/77/smp_affinity_list: No such file or directory
cat: /proc/irq/77/effective_affinity_list: No such file or directory
irq 77, cpu list , effective list
cat: /proc/irq/78/smp_affinity_list: No such file or directory
cat: /proc/irq/78/effective_affinity_list: No such file or directory
irq 78, cpu list , effective list
cat: /proc/irq/79/smp_affinity_list: No such file or directory
cat: /proc/irq/79/effective_affinity_list: No such file or directory
irq 79, cpu list , effective list
cat: /proc/irq/80/smp_affinity_list: No such file or directory
cat: /proc/irq/80/effective_affinity_list: No such file or directory
irq 80, cpu list , effective list
irq 81, cpu list 32-39, effective list 32
irq 82, cpu list 40-47, effective list 40
irq 83, cpu list 48-55, effective list 48
irq 84, cpu list 56-63, effective list 56
irq 85, cpu list 64-71, effective list 64
irq 86, cpu list 72-79, effective list 72
irq 87, cpu list 80-87, effective list 80
irq 88, cpu list 88-95, effective list 88
irq 89, cpu list 96-103, effective list 96
irq 90, cpu list 104-111, effective list 104
irq 91, cpu list 112-119, effective list 112
irq 92, cpu list 120-127, effective list 120
irq 93, cpu list 0-7, effective list 0
irq 94, cpu list 8-15, effective list 8
irq 95, cpu list 16-23, effective list 16
irq 96, cpu list 24-31, effective list 24
Thanks,
Ming
