I'm trying to wrap my head around all this so please excuse me if this
is elementary.
I'm trying to mark and prioritize traffic. With pfifo and iptables,
this seems pretty straight forward and I have this working as I like.
I went to apply the config to our servers and found that instead of
using the pfifo qdisc by default it is using the mq qdisc. That lead
me on a researching spree for a few days to understand what it is, why
I would want it and how it works. Based on the research, the only
thing that need to change is to use mqprio instead of pfifo. That's
where things fall apart.
We have one server with dual 1 Gb NICs and dual 10 Gb NICs. I can't
get mqprio to load on the 1 Gb NICs, but it loads just fine on the 10
Gb NICs. I'm thinking that this is OK because we are only going to
need it on the faster interfaces anyway. I get to the real (to-be)
production hardware that has dual 10 Gb and dual 40 Gb and I can't get
mqprio to load on either. Of course mq loads just fine on all of them.
I see a message in dmesg saying that DCE is not enabled, but I was
under the impression that DCE is not required for 802.1q. I can't
figure out why it works on the 10 Gb NICs on the dev servers, but not
on the 10/40 Gb NICs of the production servers. I've tried enabling
DCB with dcbtool and lldptool, but nothing seems to enable DCB on some
of the NICs and even when I can get it enabled, it doesn't seem to
help.
[root@s1 ~]# dcbtool gc enp3s0f0 dcb
Command: Get Config
Feature: DCB State
Port: enp3s0f0
Status: Successful
DCB State: off
[root@s1 ~]# dcbtool sc enp3s0f0 dcb on
Command: Set Config
Feature: DCB State
Port: enp3s0f0
Status: Successful
[root@s1 ~]# dcbtool gc enp3s0f0 dcb
Command: Get
Config
Feature: DCB State
Port: enp3s0f0
Status: Successful
DCB State: off
[root@s1 ~]# dcbtool gc eth1 dcb
Command: Get Config
Feature: DCB State
Port: eth1
Status: Successful
DCB State: off
[root@s1 ~]# dcbtool sc eth1 dcb on
Command: Set
Config
Feature: DCB State
Port: eth1
Status: Successful
[root@s1 ~]# dcbtool gc eth1 dcb
Command: Get
Config
Feature: DCB State
Port: eth1
Status: Successful
DCB State: on
[root@s1 ~]# tc qdisc replace dev eth1 root mqprio
RTNETLINK answers: Invalid argument
[root@s1 ~]# tc qdisc replace dev eth1 root pfifo_fast
qdisc 'pfifo_fast' does not support option parsing
[root@s1 ~]# tc qdisc replace dev eth1 root pfifo
[root@s1 ~]# tc qdisc show dev eth1
qdisc pfifo 8016: root refcnt 65 limit 1000p
[root@s1 ~]# tc qdisc replace dev eth1 root mqprio
RTNETLINK answers: Invalid
argument
[root@s1 ~]# tc qdisc replace dev eth1 root mq
[root@s1 ~]# tc qdisc show dev eth1
qdisc mq 8018: root
In dmesg I get this when I try to enable mqprio:
[9849598.802105] ixgbe 0000:01:00.1: removed PHC on eth1
[9849599.210651] ixgbe 0000:01:00.1: irq 156 for MSI/MSI-X
[9849599.210664] ixgbe 0000:01:00.1: irq 157 for MSI/MSI-X
[9849599.210675] ixgbe 0000:01:00.1: irq 158 for MSI/MSI-X
[9849599.210685] ixgbe 0000:01:00.1: irq 159 for MSI/MSI-X
[9849599.210695] ixgbe 0000:01:00.1: irq 160 for MSI/MSI-X
[9849599.210705] ixgbe 0000:01:00.1: irq 161 for MSI/MSI-X
[9849599.210715] ixgbe 0000:01:00.1: irq 162 for MSI/MSI-X
[9849599.210726] ixgbe 0000:01:00.1: irq 163 for MSI/MSI-X
[9849599.210736] ixgbe 0000:01:00.1: irq 164 for MSI/MSI-X
[9849599.210746] ixgbe 0000:01:00.1: irq 165 for MSI/MSI-X
[9849599.210757] ixgbe 0000:01:00.1: irq 166 for MSI/MSI-X
[9849599.210767] ixgbe 0000:01:00.1: irq 167 for MSI/MSI-X
[9849599.210777] ixgbe 0000:01:00.1: irq 168 for MSI/MSI-X
[9849599.210791] ixgbe 0000:01:00.1: irq 169 for MSI/MSI-X
[9849599.210801] ixgbe 0000:01:00.1: irq 170 for MSI/MSI-X
[9849599.210811] ixgbe 0000:01:00.1: irq 171 for MSI/MSI-X
[9849599.210821] ixgbe 0000:01:00.1: irq 172 for MSI/MSI-X
[9849599.210832] ixgbe 0000:01:00.1: irq 173 for MSI/MSI-X
[9849599.210843] ixgbe 0000:01:00.1: irq 174 for MSI/MSI-X
[9849599.210854] ixgbe 0000:01:00.1: irq 175 for MSI/MSI-X
[9849599.210865] ixgbe 0000:01:00.1: irq 176 for MSI/MSI-X
[9849599.210875] ixgbe 0000:01:00.1: irq 177 for MSI/MSI-X
[9849599.210887] ixgbe 0000:01:00.1: irq 178 for MSI/MSI-X
[9849599.210897] ixgbe 0000:01:00.1: irq 179 for MSI/MSI-X
[9849599.210908] ixgbe 0000:01:00.1: irq 180 for MSI/MSI-X
[9849599.210919] ixgbe 0000:01:00.1: irq 181 for MSI/MSI-X
[9849599.210930] ixgbe 0000:01:00.1: irq 182 for MSI/MSI-X
[9849599.210941] ixgbe 0000:01:00.1: irq 183 for MSI/MSI-X
[9849599.210952] ixgbe 0000:01:00.1: irq 184 for MSI/MSI-X
[9849599.210965] ixgbe 0000:01:00.1: irq 185 for MSI/MSI-X
[9849599.210977] ixgbe 0000:01:00.1: irq 186 for MSI/MSI-X
[9849599.210988] ixgbe 0000:01:00.1: irq 187 for MSI/MSI-X
[9849599.210999] ixgbe 0000:01:00.1: irq 188 for MSI/MSI-X
[9849599.211108] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue
count = 32, Tx Queue count = 32
[9849599.350697] pps pps1: new PPS source ptp1
[9849599.350702] ixgbe 0000:01:00.1: registered PHC device on eth1
Can someone point me to what I may be missing or how to debug what is
wrong. Running strace on tc, there isn't much difference between a
successful NIC and an unsuccessful one.
This one does not work.
# strace tc qdisc replace dev eth1 root mqprio
execve("/usr/sbin/tc", ["tc", "qdisc", "replace", "dev", "eth1",
"root", "mqprio"], [/* 26 vars */]) = 0
brk(0) = 0x1cbb000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f075a5c5000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=32122, ...}) = 0
mmap(NULL, 32122, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f075a5bd000
close(3) = 0
open("/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260T\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1141552, ...}) = 0
mmap(NULL, 3150168, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7f075a0a4000
mmap(0x7f075a3a4000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x100000) = 0x7f075a3a4000
close(3) = 0
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19512, ...}) = 0
mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7f0759ea0000
mprotect(0x7f0759ea3000, 2093056, PROT_NONE) = 0
mmap(0x7f075a0a2000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f075a0a2000
close(3) = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\34\2\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2107600, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f075a5bc000
mmap(NULL, 3932736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7f0759adf000
mprotect(0x7f0759c95000, 2097152, PROT_NONE) = 0
mmap(0x7f0759e95000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b6000) = 0x7f0759e95000
mmap(0x7f0759e9b000, 16960, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f0759e9b000
close(3) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f075a5ba000
arch_prctl(ARCH_SET_FS, 0x7f075a5ba740) = 0
mprotect(0x7f0759e95000, 16384, PROT_READ) = 0
mprotect(0x7f075a0a2000, 4096, PROT_READ) = 0
mprotect(0x7f075a3a4000, 4096, PROT_READ) = 0
mprotect(0x63e000, 4096, PROT_READ) = 0
mprotect(0x7f075a5c6000, 4096, PROT_READ) = 0
munmap(0x7f075a5bd000, 32122) = 0
brk(0) = 0x1cbb000
brk(0x1cdc000) = 0x1cdc000
brk(0) = 0x1cdc000
open("/proc/net/psched", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f075a5c4000
read(3, "000003e8 00000040 000f4240 3b9ac"..., 1024) = 36
close(3) = 0
munmap(0x7f075a5c4000, 4096) = 0
socket(PF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, 0) = 3
setsockopt(3, SOL_SOCKET, SO_SNDBUF, [32768], 4) = 0
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [1048576], 4) = 0
bind(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
getsockname(3, {sa_family=AF_NETLINK, pid=9012, groups=00000000}, [12]) = 0
open("//usr/lib64/tc//q_mqprio.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT
(No such file or directory)
sendto(3, "(\0\0\0\22\0\1\3\307\33\345T\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
40, 0, NULL, 0) = 40
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0,
groups=00000000},
msg_iov(1)=[{"8\4\0\0\20\0\2\0\307\33\345T4#\0\0\0\0\4\3\1\0\0\0I\0\1\0\0\0\0\0"...,
16384}], msg_controllen=0, msg_flags=0}, 0) =
3264
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0,
groups=00000000},
msg_iov(1)=[{"T\4\0\0\20\0\2\0\307\33\345T4#\0\0\0\0\1\0\4\0\0\0C\30\1\0\0\0\0\0"...,
16384}], msg_controllen=0, msg_flags=0}, 0)
= 3312
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0,
groups=00000000},
msg_iov(1)=[{"\24\0\0\0\3\0\2\0\307\33\345T4#\0\0\0\0\0\0", 16384}],
msg_controllen=0, msg_flags=0}, 0) = 20
sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0,
groups=00000000},
msg_iov(1)=[{"\210\0\0\0$\0\5\5\310\33\345T\0\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\377\377\377\377"...,
136}], msg_controllen=0, msg_flag
s=0}, 0) = 136
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0,
groups=00000000},
msg_iov(1)=[{"\234\0\0\0\2\0\0\0\310\33\345T4#\0\0\352\377\377\377\210\0\0\0$\0\5\5\310\33\345T"...,
16384}], msg_controllen=0, ms
g_flags=0}, 0) = 156
write(2, "RTNETLINK answers: Invalid argum"..., 36RTNETLINK answers:
Invalid argument
) = 36
close(3) = 0
exit_group(2) = ?
+++ exited with 2 +++
This one works.
# strace tc qdisc replace dev enp3s0f1 root mqprio
execve("/usr/sbin/tc", ["tc", "qdisc", "replace", "dev", "enp3s0f1",
"root", "mqprio"], [/* 26 vars */]) = 0
brk(0) = 0xc62000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f1763075000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=32122, ...}) = 0
mmap(NULL, 32122, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f176306d000
close(3) = 0
open("/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260T\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1141552, ...}) = 0
mmap(NULL, 3150168, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7f1762b54000
mprotect(0x7f1762c55000, 2093056, PROT_NONE) = 0
mmap(0x7f1762e54000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x100000) = 0x7f1762e54000
close(3) = 0
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19512, ...}) = 0
mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7f1762950000
mprotect(0x7f1762953000, 2093056, PROT_NONE) = 0
mmap(0x7f1762b52000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f1762b52000
close(3) = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\34\2\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2107600, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f176306c000
mmap(NULL, 3932736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7f176258f000
mprotect(0x7f1762745000, 2097152, PROT_NONE) = 0
mmap(0x7f1762945000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b6000) = 0x7f1762945000
mmap(0x7f176294b000, 16960, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f176294b000
close(3) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f176306a000
arch_prctl(ARCH_SET_FS, 0x7f176306a740) = 0
mprotect(0x7f1762945000, 16384, PROT_READ) = 0
mprotect(0x7f1762b52000, 4096, PROT_READ) = 0
mprotect(0x7f1762e54000, 4096, PROT_READ) = 0
mprotect(0x63e000, 4096, PROT_READ) = 0
mprotect(0x7f1763076000, 4096, PROT_READ) = 0
munmap(0x7f176306d000, 32122) = 0
brk(0) = 0xc62000
brk(0xc83000) = 0xc83000
brk(0) = 0xc83000
open("/proc/net/psched", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f1763074000
read(3, "000003e8 00000040 000f4240 3b9ac"..., 1024) = 36
close(3) = 0
munmap(0x7f1763074000, 4096) = 0
socket(PF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, 0) = 3
setsockopt(3, SOL_SOCKET, SO_SNDBUF, [32768], 4) = 0
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [1048576], 4) = 0
bind(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
getsockname(3, {sa_family=AF_NETLINK, pid=9780, groups=00000000}, [12]) = 0
open("//usr/lib64/tc//q_mqprio.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT
(No such file or directory)
sendto(3, "(\0\0\0\22\0\1\3T\34\345T\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
40, 0, NULL, 0) = 40
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0,
groups=00000000},
msg_iov(1)=[{"8\4\0\0\20\0\2\0T\34\345T4&\0\0\0\0\4\3\1\0\0\0I\0\1\0\0\0\0\0"...,
16384}], msg_controllen=0, msg_flags=0}, 0) = 3264
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0,
groups=00000000},
msg_iov(1)=[{"T\4\0\0\20\0\2\0T\34\345T4&\0\0\0\0\1\0\4\0\0\0C\30\1\0\0\0\0\0"...,
16384}], msg_controllen=0, msg_flags=0}, 0) = 3312
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0,
groups=00000000},
msg_iov(1)=[{"\24\0\0\0\3\0\2\0T\34\345T4&\0\0\0\0\0\0", 16384}],
msg_controllen=0, msg_flags=0}, 0) = 20
sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0,
groups=00000000},
msg_iov(1)=[{"\210\0\0\0$\0\5\5U\34\345T\0\0\0\0\0\0\0\0\5\0\0\0\0\0\0\0\377\377\377\377"...,
136}], msg_controllen=0, msg_flags=0}, 0) = 136
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0,
groups=00000000},
msg_iov(1)=[{"$\0\0\0\2\0\0\0U\34\345T4&\0\0\0\0\0\0\210\0\0\0$\0\5\5U\34\345T"...,
16384}], msg_controllen=0, msg_flags=0}, 0) = 36
close(3) = 0
exit_group(0) = ?
+++ exited with 0 +++
[root@s1]# lspci | grep Ethernet
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller
10-Gigabit X540-AT2 (rev 01)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller
10-Gigabit X540-AT2 (rev 01)
03:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
SFI/SFP+ Network Connection (rev 01)
This is CentOS 7.0
[root@s1 ~]# uname -a
Linux s1 3.10.0-123.8.1.el7.x86_64 #1 SMP Mon Sep 22 19:06:58 UTC 2014
x86_64 x86_64 x86_64 GNU/Linux
I have similar issues with GRML 2014-11 (3.16.0-1-grml-amd64) on the
production hardware.
With Intel I350, 82599 and XL710 (i40e driver) adapters.
I'm at the end of my knowledge and could us a little push in the right
direction.
Thank you,
Robert LeBlanc
--
To unsubscribe from this list: send the line "unsubscribe lartc" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
