Peter Lieven wrote:
Am 03.10.2010 um 01:48 schrieb Peter Lieven:
Hi,
running 0.12.5 with a Ubuntu LTS 10.04.1 64-bit kernel I see the following error after a few days with severe load (constant 100-200mbps input).
# uname -a
Linux ubuntu-newsfeed 2.6.32-24-server #43-Ubuntu SMP Thu Sep 16 16:05:42 UTC 2010 x86_64 GNU/Linux
Linux newsfeed-spool 2.6.27.48-0.2-default seems not affected. I have a machine with even higher load (about 300-400mbits constant
network load) up for 4 months without a crash.
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] swapper: page allocation failure. order:0, mode:0x20
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Pid: 0, comm: swapper Not tainted 2.6.32-24-server #43-Ub
untu
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Call Trace:
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] <IRQ> [<ffffffff810fa7ae>] __alloc_pages_slowpath+0x56e
/0x580
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff810fa931>] __alloc_pages_nodemask+0x171/0x180
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff8112d817>] alloc_pages_current+0x87/0xd0
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff813db792>] try_fill_recv+0x182/0x200
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff813db9bd>] virtnet_poll+0x10d/0x160
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff81473c5f>] net_rx_action+0x10f/0x250
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff8106e317>] __do_softirq+0xb7/0x1e0
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff810c4dc0>] ? handle_IRQ_event+0x60/0x170
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff810142ec>] call_softirq+0x1c/0x30
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff81015cb5>] do_softirq+0x65/0xa0
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff8106e1b5>] irq_exit+0x85/0x90
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff8155fd25>] do_IRQ+0x75/0xf0
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff81013b13>] ret_from_intr+0x0/0x11
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] <EOI> [<ffffffff81038acb>] ? native_safe_halt+0xb/0x10
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff8101b69d>] ? default_idle+0x3d/0x90
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff81011e63>] ? cpu_idle+0xb3/0x110
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff8154285b>] ? rest_init+0x6b/0x80
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff8187edcc>] ? start_kernel+0x368/0x371
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff8187e33a>] ? x86_64_start_reservations+0x125/0x129
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] [<ffffffff8187e438>] ? x86_64_start_kernel+0xfa/0x109
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Mem-Info:
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Node 0 DMA per-cpu:
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] CPU 0: hi: 0, btch: 1 usd: 0
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] CPU 1: hi: 0, btch: 1 usd: 0
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] CPU 2: hi: 0, btch: 1 usd: 0
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] CPU 3: hi: 0, btch: 1 usd: 0
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Node 0 DMA32 per-cpu:
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] CPU 0: hi: 186, btch: 31 usd: 59
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] CPU 1: hi: 186, btch: 31 usd: 159
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] CPU 2: hi: 186, btch: 31 usd: 102
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] CPU 3: hi: 186, btch: 31 usd: 151
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Node 0 Normal per-cpu:
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] CPU 0: hi: 186, btch: 31 usd: 51
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] CPU 1: hi: 186, btch: 31 usd: 157
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] CPU 2: hi: 186, btch: 31 usd: 162
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] CPU 3: hi: 186, btch: 31 usd: 175
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] active_anon:1857 inactive_anon:1602 isolated_anon:0
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] active_file:458063 inactive_file:467825 isolated_file:0
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] unevictable:0 dirty:74195 writeback:18395 unstable:0
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] free:5214 slab_reclaimable:50506 slab_unreclaimable:3256
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] mapped:4151 shmem:1592 pagetables:400 bounce:0
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Node 0 DMA free:15896kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_fil
e:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB s
lab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable?
yes
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] lowmem_reserve[]: 0 3512 4017 4017
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Node 0 DMA32 free:4652kB min:7076kB low:8844kB high:10612kB active_anon:3832kB inactive_anon:110
8kB active_file:1642896kB inactive_file:1680672kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3596500kB mlocked:0kB dirty:262008kB wri
teback:61820kB mapped:10184kB shmem:2812kB slab_reclaimable:170892kB slab_unreclaimable:4164kB kernel_stack:8kB pagetables:512kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] lowmem_reserve[]: 0 0 505 505
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Node 0 Normal free:308kB min:1016kB low:1268kB high:1524kB active_anon:3596kB inactive_anon:5300
kB active_file:189356kB inactive_file:190628kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:517120kB mlocked:0kB dirty:34772kB writebac
k:11760kB mapped:6420kB shmem:3556kB slab_reclaimable:31132kB slab_unreclaimable:8852kB kernel_stack:1160kB pagetables:1088kB unstable:0kB bounce:0kB wr
iteback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] lowmem_reserve[]: 0 0 0 0
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Node 0 DMA: 2*4kB 4*8kB 5*16kB 3*32kB 3*64kB 1*128kB 2*256kB 1*512kB 2*1024kB 2*2048kB 2*4096kB
= 15896kB
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Node 0 DMA32: 607*4kB 72*8kB 0*16kB 1*32kB 0*64kB 1*128kB 3*256kB 1*512kB 0*1024kB 0*2048kB 0*40
96kB = 4444kB
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Node 0 Normal: 27*4kB 3*8kB 2*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*409
6kB = 356kB
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] 927861 total pagecache pages
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] 296 pages in swap cache
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Swap cache stats: add 797, delete 501, find 10963/10986
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Free swap = 3905004kB
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] Total swap = 3906552kB
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] 1048576 pages RAM
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] 34105 pages reserved
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] 668525 pages shared
Oct 2 10:09:35 ubuntu-newsfeed kernel: [248407.790029] 344067 pages non-shared
Is this anything known?
After this error, the network device hangs. However, it can be recovered by setting ip link down and up again.
BR,
Peter
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
anyone? willing to help with extensive debugging.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
