So I took a look at using several queues to spread load on several cpus,
and quickly found out some hot spots in our NFQUEUE processing, even
using cpu match, and correct cpu affinities.
iptables -N DISPATCH 2>/dev/null
iptables -F DISPATCH
for i in `seq 0 15`
do
iptables -A DISPATCH -m cpu --cpu $i -j NFQUEUE --queue-num $i
nfqnl_test -q $i &
done
iptables -A FORWARD -j DISPATCH
<stress load, 500.000 pps >
# cat /proc/net/netfilter/nfnetlink_queue
0 4855 0 2 65535 0 37638 886112 1
1 4858 0 2 65535 0 24530 886096 1
2 4862 0 2 65535 0 40666 886103 1
3 4866 0 2 65535 0 6480 830716 1
4 4870 0 2 65535 0 35957 886099 1
5 4872 0 2 65535 0 20001 886100 1
6 4875 0 2 65535 0 41656 886099 1
7 4877 0 2 65535 0 27196 886099 1
8 4879 0 2 65535 0 39173 886112 1
9 4881 0 2 65535 0 19840 886103 1
10 4883 0 2 65535 0 29339 850670 1
11 4885 0 2 65535 0 17933 886102 1
12 4887 0 2 65535 0 37680 886099 1
13 4889 0 2 65535 0 7327 830721 1
14 4891 0 2 65535 0 35577 886100 1
15 4893 0 2 65535 0 23854 886100 1
Number one offender is the nfnl_lock mutex hold each time we give a
verdict.
Number two is the nl_table_lock rwlock
# Events: 28K cycles
#
# Overhead Command Shared Object Symbol
# ........ ............ ........................... ...................................
#
10.60% nfqnl_test [kernel.kallsyms] [k] mutex_spin_on_owner
|
--- mutex_spin_on_owner
|
|--99.68%-- __mutex_lock_slowpath
| mutex_lock
| nfnl_lock
| nfnetlink_rcv
| netlink_unicast
| netlink_sendmsg
| sock_sendmsg
| __sys_sendmsg
| sys_sendmsg
| compat_sys_sendmsg
| compat_sys_socketcall
| sysenter_dispatch
| 0xffffe430
| nfnl_sendiov
| __set_verdict
| cb
| __nfnl_handle_msg
| nfnl_handle_packet
| nfq_handle_packet
| main
| __libc_start_main
| _start
--0.32%-- [...]
4.12% nfqnl_test [kernel.kallsyms] [k] __mutex_lock_slowpath
|
--- __mutex_lock_slowpath
|
|--99.68%-- mutex_lock
| nfnl_lock
Time to add RCU ;)
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
