Hello Wei Liu,
On 27.06.2014 20:24, Philipp Hahn wrote:
> On 27.06.2014 19:48, Philipp Hahn wrote:> I guess we found the problem
> ourselves: For thus removed skb's the
>> reference counter on the associated vif was not decremented, as it is
>> normally done in two locations at the end of the function
>> xen_netbk_rx_action():
> ...
>> The test is currently running again for the weekend and on Monday we
>> will hopefully know more.
>
> FYI: The test VM survived the first reboot without locking up:
...
> Jun 27 19:49:23 xenmbint05b01 kernel: [ 2055.898349] UniDEBUG
> vif->mapped is false
The host survived the weekend with the problematic VM rebooting every 5
minutes; the log shows the shared ring being accessed unmapped, where
the kernel crashed previously.
So the attached patch fixes the bug (or at least prevents the OOPS).
@Wei Liu: You said that the patch is only a quick hack to detect, if my
analysis is correct and a proper fix would be needed. For us the
attached patch works, as the problem does not happen that often and is
hard to reproduce anyway, so spending more time on that issue is
probably not worth it. And that flag doesn't look that ugly.
@stable: at least 3.10 has the bug, but other long-term-stable kernels
have it too. The code in current is different as multi-queue was added,
so the patch wouldn't be in current.
Sincerely
Philipp Hahn
>From 45fd183c8faa5b7213aa4663cf224b414aacfc4b Mon Sep 17 00:00:00 2001
Message-Id: <45fd183c8faa5b7213aa4663cf224b414aacfc4b.1404286695.git.hahn@xxxxxxxxxxxxx>
From: Wei Liu <wei.liu2@xxxxxxxxxx>
Date: Wed, 2 Jul 2014 09:14:22 +0200
Subject: [PATCH] xen-netback: skip pending packets in unmapped ring
Organization: Univention GmbH, Bremen, Germany
1. The VM is receiving packets through bonding + bridge + netback +
netfront.
2. For some unknown reason at least one packet remains in the rx queue
and is not delivered to the domU immediately by netback.
3. The VM finishes shutting down.
4. The shared ring between dom0 and domU is freed.
5. then xen-netback continues processing the pending requests and tries
to put the packet into the now already released shared ring.
> [38551.547728] XXXlan0: port 9(vif26.0) entered disabled state
> [38551.549365] BUG: unable to handle kernel paging request at ffffc900108641d8
> [38551.549461] IP: [<ffffffffa04147dc>] xen_netbk_rx_action+0x18b/0x6f0
> [xen_netback]
> [38551.549551] PGD 57e20067 PUD 57e21067 PMD 571a7067 PTE 0
> [38551.549615] Oops: 0000 [#1] SMP
> [38551.549665] Modules linked in: tun xt_physdev xen_blkback xen_netback ip6_tables
> iptable_filter ip_tables ebtable_nat ebtables x_tables xen_gntdev nfsv3 nfsv4
> rpcsec_gss_krb5 nfsd nfs_acl auth_rpcgss oid_registry nfs fscache dns_resolver lockd
> sunrpc fuse loop xen_blkfront xen_evtchn blktap quota_v2 quota_tree xenfs xen_privcmd
> coretemp crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw gf128mul
> glue_helper aes_x86_64 snd_pcm snd_timer snd soundcore snd_page_alloc tpm_tis tpm lpc_ich
> tpm_bios i7core_edac i2c_i801 psmouse microcode edac_core serio_raw pcspkr mperf ioatdma
> mfd_core processor evdev thermal_sys ext4 jbd2 crc16 bonding bridge stp llc dm_snapshot
> dm_mirror dm_region_hash dm_log dm_mod sd_mod crc_t10dif ehci_pci uhci_hcd ehci_hcd mptsas
> mptscsih mptbase scsi_transport_sas usbcore usb_common igb dca i2c_algo_bit i2c_core ptp
> pps_core button
> [38551.550601] CPU: 0 PID: 12587 Comm: netback/0 Not tainted 3.10.0-ucs58-amd64 #1 Debian
> 3.10.11-1.58.201405060908
> [38551.550693] Hardware name: FUJITSU PRIMERGY BX620 S6/D3051, BIOS 080015 Rev.3C78.3051
> 07/22/2011
> [38551.550781] task: ffff880004b067c0 ti: ffff8800561ec000 task.ti: ffff8800561ec000
> [38551.550865] RIP: e030:[<ffffffffa04147dc>] [<ffffffffa04147dc>]
> xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
> [38551.550959] RSP: e02b:ffff8800561edce8 EFLAGS: 00010202
> [38551.551009] RAX: ffffc900104adac0 RBX: ffff8800541e95c0 RCX: ffffc90010864000
> [38551.551064] RDX: 000000000000003b RSI: 0000000000000000 RDI: ffff880040014380
> [38551.551120] RBP: ffff8800570e6800 R08: 0000000000000000 R09: ffff880004799800
> [38551.551175] R10: ffffffff813ca115 R11: ffff88005e4fdb08 R12: ffff880054e6f800
> [38551.551231] R13: ffff8800561edd58 R14: ffffc900104a1000 R15: 0000000000000000
> [38551.551289] FS: 00007f19a54a8700(0000) GS:ffff88005da00000(0000)
> knlGS:0000000000000000
> [38551.551374] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [38551.551425] CR2: ffffc900108641d8 CR3: 0000000054cb3000 CR4: 0000000000002660
> [38551.551481] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [38551.551537] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [38551.551592] Stack:
> [38551.551630] ffff880004b06ba0 0000000000000000 ffff88005da13ec0 ffff88005da13ec0
> [38551.551726] 0000000004b067c0 ffffc900104a8ac0 ffffc900104a1020 000000005da13ec0
> [38551.551823] 0000000000000000 0000000000000001 ffffc900104a8ac0 ffffc900104adac0
> [38551.551920] Call Trace:
> [38551.551966] [<ffffffff813ca32d>] ? _raw_spin_lock_irqsave+0x11/0x2f
> [38551.552021] [<ffffffffa0416033>] ? xen_netbk_kthread+0x174/0x841 [xen_netback]
> [38551.552106] [<ffffffff8105d373>] ? wake_up_bit+0x20/0x20
> [38551.560239] [<ffffffffa0415ebf>] ? xen_netbk_tx_build_gops+0xce8/0xce8 [xen_netback]
> [38551.560325] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [38551.560381] [<ffffffffa0415ebf>] ? xen_netbk_tx_build_gops+0xce8/0xce8 [xen_netback]
> [38551.560466] [<ffffffff8105ce1e>] ? kthread+0xab/0xb3
> [38551.560518] [<ffffffff81003638>] ? xen_end_context_switch+0xe/0x1c
> [38551.560572] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [38551.560628] [<ffffffff813cfbfc>] ? ret_from_fork+0x7c/0xb0
> [38551.560680] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [38551.560734] Code: 8b b3 d0 00 00 00 48 8b bb d8 00 00 00 0f b7 74 37 02 89 70 08 eb 07
> c7 40 08 00 00 00 00 89 d2 c7 40 04 00 00 00 00 48 83 c2 08 <0f> b7 34 d1 89 30 c7 44 24
> 60 00 00 00 00 8b 44 d1 04 89 44 24
> [38551.561151] RIP [<ffffffffa04147dc>] xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
> [38551.561238] RSP <ffff8800561edce8>
> [38551.561283] CR2: ffffc900108641d8
> [38551.561624] ---[ end trace 8c260c6af259c4aa ]---
Track the shared ring buffer being unmapped and drop those packets.
Signed-off-by: Wei Liu <wei.liu2@xxxxxxxxxx>
Signed-off-by: Philipp Hahn <hahn@xxxxxxxxxxxxx>
Tested-by: Philipp Hahn <hahn@xxxxxxxxxxxxx>
---
drivers/net/xen-netback/common.h | 1 +
drivers/net/xen-netback/interface.c | 1 +
drivers/net/xen-netback/netback.c | 10 ++++++++++
3 files changed, 12 insertions(+)
diff --git a/drivers/net/xen-netback/common.h b/drivers/net/xen-netback/common.h
index f2faa77..9239824 100644
--- a/drivers/net/xen-netback/common.h
+++ b/drivers/net/xen-netback/common.h
@@ -66,6 +66,7 @@ struct xenvif {
/* The shared rings and indexes. */
struct xen_netif_tx_back_ring tx;
struct xen_netif_rx_back_ring rx;
+ bool mapped;
/* Frontend feature information. */
u8 can_sg:1;
diff --git a/drivers/net/xen-netback/interface.c b/drivers/net/xen-netback/interface.c
index 540a796..5f11763 100644
--- a/drivers/net/xen-netback/interface.c
+++ b/drivers/net/xen-netback/interface.c
@@ -271,6 +271,7 @@ struct xenvif *xenvif_alloc(struct device *parent, domid_t domid,
vif->dev = dev;
INIT_LIST_HEAD(&vif->schedule_list);
INIT_LIST_HEAD(&vif->notify_list);
+ vif->mapped = false;
vif->credit_bytes = vif->remaining_credit = ~0UL;
vif->credit_usec = 0UL;
diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
index 70b830f..bab7043 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -720,6 +720,13 @@ static void xen_netbk_rx_action(struct xen_netbk *netbk)
vif = netdev_priv(skb->dev);
nr_frags = skb_shinfo(skb)->nr_frags;
+ if (!vif->mapped) {
+ netdev_err(vif->dev, "Shared ring is unmapped\n");
+ dev_kfree_skb(skb);
+ xenvif_put(vif);
+ continue;
+ }
+
sco = (struct skb_cb_overlay *)skb->cb;
sco->meta_slots_used = netbk_gop_skb(skb, &npo);
@@ -1864,6 +1871,8 @@ static int xen_netbk_kthread(void *data)
void xen_netbk_unmap_frontend_rings(struct xenvif *vif)
{
+ vif->mapped = false;
+
if (vif->tx.sring)
xenbus_unmap_ring_vfree(xenvif_to_xenbus_device(vif),
vif->tx.sring);
@@ -1899,6 +1908,7 @@ int xen_netbk_map_frontend_rings(struct xenvif *vif,
BACK_RING_INIT(&vif->rx, rxs, PAGE_SIZE);
vif->rx_req_cons_peek = 0;
+ vif->mapped = true;
return 0;
--
1.9.1