On 11/27/24 10:51, Joe Damato wrote:
On Wed, Nov 27, 2024 at 09:43:54AM -0800, Guenter Roeck wrote:
Hi,
On Fri, Oct 11, 2024 at 06:45:00PM +0000, Joe Damato wrote:
Add a persistent NAPI config area for NAPI configuration to the core.
Drivers opt-in to setting the persistent config for a NAPI by passing an
index when calling netif_napi_add_config.
napi_config is allocated in alloc_netdev_mqs, freed in free_netdev
(after the NAPIs are deleted).
Drivers which call netif_napi_add_config will have persistent per-NAPI
settings: NAPI IDs, gro_flush_timeout, and defer_hard_irq settings.
Per-NAPI settings are saved in napi_disable and restored in napi_enable.
Co-developed-by: Martin Karsten <mkarsten@xxxxxxxxxxxx>
Signed-off-by: Martin Karsten <mkarsten@xxxxxxxxxxxx>
Signed-off-by: Joe Damato <jdamato@xxxxxxxxxx>
Reviewed-by: Jakub Kicinski <kuba@xxxxxxxxxx>
This patch triggers a lock inversion message on pcnet Ethernet adapters.
Thanks for the report. I am not familiar with the pcnet driver, but
took some time now to read the report below and the driver code.
I could definitely be reading the output incorrectly (if so please
let me know), but it seems like the issue can be triggered in this
case:
CPU 0:
pcnet32_open
lock(lp->lock)
napi_enable
napi_hash_add
lock(napi_hash_lock)
unlock(napi_hash_lock)
unlock(lp->lock)
Meanwhile on CPU 1:
pcnet32_close
napi_disable
napi_hash_del
lock(napi_hash_lock)
unlock(napi_hash_lock)
lock(lp->lock)
[... other code ...]
unlock(lp->lock)
[... other code ...]
lock(lp->lock)
[... other code ...]
unlock(lp->lock)
In other words: while the close path is holding napi_hash_lock (and
before it acquires lp->lock), the enable path takes lp->lock and
then napi_hash_lock.
It seems this was triggered because before the identified commit,
napi_enable did not call napi_hash_add (and thus did not take the
napi_hash_lock).
So, I agree there is an inversion; I can't say for sure if this
would cause a deadlock in certain situations. It seems like
napi_hash_del in the close path will return, so the inversion
doesn't seem like it'd lead to a deadlock, but I am not an expert in
this and could certainly be wrong.
I wonder if a potential fix for this would be in the driver's close
function?
In pcnet32_open the order is:
lock(lp->lock)
napi_enable
netif_start_queue
mod_timer(watchdog)
unlock(lp->lock)
Perhaps pcnet32_close should be the same?
I've included an example patch below for pcnet32_close and I've CC'd
the maintainer of pcnet32 that is not currently CC'd.
Guenter: Is there any change you might be able to test the proposed
patch below?
I moved the spinlock after del_timer_sync() because it is not a good idea
to hold a spinlock when calling that function. That results in:
[ 10.646956] BUG: sleeping function called from invalid context at net/core/dev.c:6775
[ 10.647142] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 1817, name: ip
[ 10.647237] preempt_count: 1, expected: 0
[ 10.647319] 2 locks held by ip/1817:
[ 10.647383] #0: ffffffff81ded990 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x22a/0x74c
[ 10.647880] #1: ff6000000498ccb0 (&lp->lock){-.-.}-{3:3}, at: pcnet32_close+0x40/0x126
[ 10.648050] irq event stamp: 3720
[ 10.648102] hardirqs last enabled at (3719): [<ffffffff80decaf4>] _raw_spin_unlock_irqrestore+0x54/0x62
[ 10.648204] hardirqs last disabled at (3720): [<ffffffff80dec8a2>] _raw_spin_lock_irqsave+0x5e/0x64
[ 10.648301] softirqs last enabled at (3712): [<ffffffff8001efca>] handle_softirqs+0x3e6/0x4a2
[ 10.648396] softirqs last disabled at (3631): [<ffffffff80ded6cc>] __do_softirq+0x12/0x1a
[ 10.648666] CPU: 0 UID: 0 PID: 1817 Comm: ip Tainted: G N 6.12.0-10313-g7d4050728c83-dirty #1
[ 10.648828] Tainted: [N]=TEST
[ 10.648879] Hardware name: riscv-virtio,qemu (DT)
[ 10.648978] Call Trace:
[ 10.649048] [<ffffffff80006d42>] dump_backtrace+0x1c/0x24
[ 10.649117] [<ffffffff80dc8d94>] show_stack+0x2c/0x38
[ 10.649180] [<ffffffff80de00b0>] dump_stack_lvl+0x74/0xac
[ 10.649246] [<ffffffff80de00fc>] dump_stack+0x14/0x1c
[ 10.649308] [<ffffffff8004da18>] __might_resched+0x23e/0x248
[ 10.649377] [<ffffffff8004da60>] __might_sleep+0x3e/0x62
[ 10.649441] [<ffffffff80b8d370>] napi_disable+0x24/0x10c
[ 10.649506] [<ffffffff809a06fe>] pcnet32_close+0x6c/0x126
...
This is due to might_sleep() at the beginning of napi_disable(). So it doesn't
work as intended, it just replaces one problem with another.
Don: Would you mind taking a look to see if this change is sensible?
Netdev maintainers: at a higher level, I'm not sure how many other
drivers might have locking patterns like this that commit
86e25f40aa1e ("net: napi: Add napi_config") will break in a similar
manner.
Do I:
- comb through drivers trying to identify these, and/or
Coccinelle, checking for napi_enable calls under spinlock, points to:
napi_enable called under spin_lock_irqsave from drivers/net/ethernet/via/via-velocity.c:2325
napi_enable called under spin_lock_irqsave from drivers/net/can/grcan.c:1076
napi_enable called under spin_lock from drivers/net/ethernet/marvell/mvneta.c:4388
napi_enable called under spin_lock_irqsave from drivers/net/ethernet/amd/pcnet32.c:2104
Guenter
- do we find a way to implement the identified commit with the
original lock ordering to avoid breaking any other driver?
I'd appreciate guidance/insight from the maintainers on how to best
proceed.
diff --git a/drivers/net/ethernet/amd/pcnet32.c b/drivers/net/ethernet/amd/pcnet32.c
index 72db9f9e7bee..ff56a308fec9 100644
--- a/drivers/net/ethernet/amd/pcnet32.c
+++ b/drivers/net/ethernet/amd/pcnet32.c
@@ -2623,13 +2623,13 @@ static int pcnet32_close(struct net_device *dev)
struct pcnet32_private *lp = netdev_priv(dev);
unsigned long flags;
+ spin_lock_irqsave(&lp->lock, flags);
+
del_timer_sync(&lp->watchdog_timer);
netif_stop_queue(dev);
napi_disable(&lp->napi);
- spin_lock_irqsave(&lp->lock, flags);
-
dev->stats.rx_missed_errors = lp->a->read_csr(ioaddr, 112);
netif_printk(lp, ifdown, KERN_DEBUG, dev,
