On 02/10/2011 05:23 PM, Ruben Kerkhof wrote:
This machine has been running for a week without problems, but then we started to get the following oopses again: 2011-02-06T19:45:35.221555+01:00 phy005 kernel: BUG: unable to handle kernel paging request at ffffea71929180e0 2011-02-06T19:45:35.222194+01:00 phy005 kernel: IP: [<ffffffff81034880>] gup_pte_range+0x94/0xd3 2011-02-06T19:45:35.222199+01:00 phy005 kernel: PGD 118600067 PUD 0 2011-02-06T19:45:35.222203+01:00 phy005 kernel: Oops: 0000 [#1] SMP 2011-02-06T19:45:35.222221+01:00 phy005 kernel: last sysfs file: /sys/devices/system/cpu/cpu15/topology/thread_siblings 2011-02-06T19:45:35.222224+01:00 phy005 kernel: CPU 4 2011-02-06T19:45:35.222229+01:00 phy005 kernel: Modules linked in: tun ipmi_devintf ipmi_si ipmi_msghandler bridge 8021q garp stp llc bonding xt_comment xt_recent ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 kvm_intel kvm i2c_i801 i2c_core iTCO_wdt serio_raw igb iTCO_vendor_support joydev ioatdma dca 3w_9xxx [last unloaded: scsi_wait_scan] 2011-02-06T19:45:35.222231+01:00 phy005 kernel: 2011-02-06T19:45:35.222233+01:00 phy005 kernel: Pid: 3650, comm: qemu-kvm Not tainted 2.6.34.7-66.tilaa.fc13.x86_64 #1 X8DTU/X8DTU 2011-02-06T19:45:35.222236+01:00 phy005 kernel: RIP: 0010:[<ffffffff81034880>] [<ffffffff81034880>] gup_pte_range+0x94/0xd3 2011-02-06T19:45:35.222239+01:00 phy005 kernel: RSP: 0018:ffff88060b9bda78 EFLAGS: 00010082 2011-02-06T19:45:35.222241+01:00 phy005 kernel: RAX: ffffea71929180e0 RBX: 00003ffffffff000 RCX: 0000000000000005 2011-02-06T19:45:35.222243+01:00 phy005 kernel: RDX: 00007fe54e400000 RSI: 00007fe54e3ff000 RDI: 1603a07305004067 2011-02-06T19:45:35.222245+01:00 phy005 kernel: RBP: ffff88060b9bda98 R08: ffff880b94384560 R09: ffff88060b9bdb44 2011-02-06T19:45:35.222248+01:00 phy005 kernel: R10: ffff880606b2fff8 R11: ffffea0000000000 R12: 0000000000000205 2011-02-06T19:45:35.222251+01:00 phy005 kernel: R13: ffffc00000000fff R14: 0000000000000005 R15: 0000000000000000 2011-02-06T19:45:35.222255+01:00 phy005 kernel: FS: 00007fe64cb0e700(0000) GS:ffff880655400000(0000) knlGS:0000000000000000 2011-02-06T19:45:35.222259+01:00 phy005 kernel: CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 2011-02-06T19:45:35.222263+01:00 phy005 kernel: CR2: ffffea71929180e0 CR3: 0000000bff06d000 CR4: 00000000000026e0 2011-02-06T19:45:35.222267+01:00 phy005 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2011-02-06T19:45:35.222271+01:00 phy005 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2011-02-06T19:45:35.222274+01:00 phy005 kernel: Process qemu-kvm (pid: 3650, threadinfo ffff88060b9bc000, task ffff880623ed2ee0) 2011-02-06T19:45:35.222278+01:00 phy005 kernel: Stack: 2011-02-06T19:45:35.222281+01:00 phy005 kernel: 00007fe54e400000 00007fe54e400000 00007fe54e400000 ffff88053a0d2388 2011-02-06T19:45:35.222285+01:00 phy005 kernel:<0> ffff88060b9bdaf8 ffffffff81034a15 00007fe54e3fffff 00007fe54e3fffff 2011-02-06T19:45:35.222289+01:00 phy005 kernel:<0> ffff88060b9bdb44 ffff880b94384560 ffff880bff06eca8 ffff880bff06d7f8 2011-02-06T19:45:35.222292+01:00 phy005 kernel: Call Trace: 2011-02-06T19:45:35.222296+01:00 phy005 kernel: [<ffffffff81034a15>] gup_pud_range+0x156/0x192 2011-02-06T19:45:35.222300+01:00 phy005 kernel: [<ffffffff81034b15>] get_user_pages_fast+0xc4/0x172 2011-02-06T19:45:35.222304+01:00 phy005 kernel: [<ffffffff81131fbc>] ? bio_add_page+0x36/0x38 2011-02-06T19:45:35.222308+01:00 phy005 kernel: [<ffffffff81134730>] dio_get_page+0x54/0x127 2011-02-06T19:45:35.222312+01:00 phy005 kernel: [<ffffffff81135317>] __blockdev_direct_IO+0x41d/0xa36 2011-02-06T19:45:35.222316+01:00 phy005 kernel: [<ffffffffa0080f69>] ? x86_emulate_insn+0x1ff8/0x2d61 [kvm] 2011-02-06T19:45:35.222320+01:00 phy005 kernel: [<ffffffff8113379b>] blkdev_direct_IO+0x4e/0x50 2011-02-06T19:45:35.222324+01:00 phy005 kernel: [<ffffffff81132c49>] ? blkdev_get_blocks+0x0/0x8d 2011-02-06T19:45:35.222328+01:00 phy005 kernel: [<ffffffff810cb516>] generic_file_direct_write+0xed/0x16d 2011-02-06T19:45:35.222331+01:00 phy005 kernel: [<ffffffff810cb72c>] __generic_file_aio_write+0x196/0x281 2011-02-06T19:45:35.222335+01:00 phy005 kernel: [<ffffffff811d5352>] ? file_has_perm+0xa4/0xc6 2011-02-06T19:45:35.222339+01:00 phy005 kernel: [<ffffffff81133043>] ? blkdev_aio_write+0x0/0x69 2011-02-06T19:45:35.222343+01:00 phy005 kernel: [<ffffffff8113306d>] blkdev_aio_write+0x2a/0x69 2011-02-06T19:45:35.222347+01:00 phy005 kernel: [<ffffffff81133043>] ? blkdev_aio_write+0x0/0x69 2011-02-06T19:45:35.222351+01:00 phy005 kernel: [<ffffffff8113d4eb>] aio_rw_vect_retry+0x85/0x18e 2011-02-06T19:45:35.222355+01:00 phy005 kernel: [<ffffffff8113e9b3>] aio_run_iocb+0x77/0x10f 2011-02-06T19:45:35.222359+01:00 phy005 kernel: [<ffffffff8113f508>] do_io_submit+0x558/0x7ce 2011-02-06T19:45:35.222363+01:00 phy005 kernel: [<ffffffff8113f78e>] sys_io_submit+0x10/0x12 2011-02-06T19:45:35.222366+01:00 phy005 kernel: [<ffffffff81009c72>] system_call_fastpath+0x16/0x1b 2011-02-06T19:45:35.222372+01:00 phy005 kernel: Code: 21 d8 49 01 c2 49 8b 3a 49 89 fe 4d 21 ee 4d 21 e6 49 39 ce 75 49 48 89 f8 0f 1f 40 00 48 21 d8 48 c1 e8 0c 48 6b c0 38 4c 01 d8<66> 83 38 00 48 89 c7 79 04 48 8b 78 10 f0 ff 47 08 49 63 39 48 2011-02-06T19:45:35.222376+01:00 phy005 kernel: RIP [<ffffffff81034880>] gup_pte_range+0x94/0xd3 2011-02-06T19:45:35.222379+01:00 phy005 kernel: RSP<ffff88060b9bda78> 2011-02-06T19:45:35.222382+01:00 phy005 kernel: CR2: ffffea71929180e0 2011-02-06T19:45:35.222386+01:00 phy005 kernel: ---[ end trace beed2b54d0bb8a00 ]---
Hm, outside any kvm code.
and 2011-02-06T19:47:15.023129+01:00 phy005 kernel: qemu-kvm: Corrupted page table at address 7fbde15ff64c 2011-02-06T19:47:15.023207+01:00 phy005 kernel: PGD 5ff58a067 PUD 612668067 PMD 5937b7067 PE 1603a07305008067
Again outside kvm, and again the magic pte 1603axxxxx.
followed by 2011-02-06T21:20:32.882972+01:00 phy005 kernel: BUG: unable to handle kernel paging request at fffff6b192918010 2011-02-06T21:20:32.883252+01:00 phy005 kernel: IP: [<ffffffffa0078826>] kvm_mmu_zap_page+0x28a/0x299 [kvm]
Well, after something goes bad, nothing good can come out of it.
after which we rebooted the machine and replaced the motherboard and cpus (we already replaced the memory before). But 2 days ago we got this oops: 2011-02-08T15:56:19.902104+01:00 phy005 kernel: BUG: unable to handle kernel paging request at ffffea71929181c0 2011-02-08T15:56:19.902686+01:00 phy005 kernel: IP: [<ffffffff81034880>] gup_pte_range+0x94/0xd3 2011-02-08T15:56:19.902693+01:00 phy005 kernel: PGD 118600067 PUD 0 2011-02-08T15:56:19.902699+01:00 phy005 kernel: Oops: 0000 [#1] SMP 2011-02-08T15:56:19.902703+01:00 phy005 kernel: last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_m ap 2011-02-08T15:56:19.902708+01:00 phy005 kernel: CPU 8 2011-02-08T15:56:19.902715+01:00 phy005 kernel: Modules linked in: tun ipmi_devintf ipmi_si ipmi_msghandler bridge 8021q garp stp llc bonding xt_comment xt_recent ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 kvm_intel kvm i gb i2c_i801 iTCO_wdt ioatdma i2c_core iTCO_vendor_support dca serio_raw joydev 3w_9xxx [last unloaded: scsi_wait_scan] 2011-02-08T15:56:19.902770+01:00 phy005 kernel: 2011-02-08T15:56:19.902775+01:00 phy005 kernel: Pid: 3346, comm: qemu-kvm Not tainted 2.6.34.7-66.tilaa.fc13.x86_64 #1 X 8DTU/X8DTU 2011-02-08T15:56:19.902781+01:00 phy005 kernel: RIP: 0010:[<ffffffff81034880>] [<ffffffff81034880>] gup_pte_range+0x94/ 0xd3 2011-02-08T15:56:19.902785+01:00 phy005 kernel: RSP: 0018:ffff880c21bc1a78 EFLAGS: 00010086 2011-02-08T15:56:19.902789+01:00 phy005 kernel: RAX: ffffea71929181c0 RBX: 00003ffffffff000 RCX: 0000000000000005 2011-02-08T15:56:19.902793+01:00 phy005 kernel: RDX: 00007fa2ca200000 RSI: 00007fa2ca1ff000 RDI: 1603a07305008067 2011-02-08T15:56:19.902797+01:00 phy005 kernel: RBP: ffff880c21bc1a98 R08: ffff88060fdfad60 R09: ffff880c21bc1b44 2011-02-08T15:56:19.902801+01:00 phy005 kernel: R10: ffff88061493fff8 R11: ffffea0000000000 R12: 0000000000000205 2011-02-08T15:56:19.902805+01:00 phy005 kernel: R13: ffffc00000000fff R14: 0000000000000005 R15: 0000000000000000 2011-02-08T15:56:19.902810+01:00 phy005 kernel: FS: 00007fa2d8724700(0000) GS:ffff880002080000(0000) knlGS:000000000000 0000 2011-02-08T15:56:19.902820+01:00 phy005 kernel: CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 2011-02-08T15:56:19.902825+01:00 phy005 kernel: CR2: ffffea71929181c0 CR3: 0000000c231f9000 CR4: 00000000000026e0 2011-02-08T15:56:19.902829+01:00 phy005 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2011-02-08T15:56:19.902833+01:00 phy005 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2011-02-08T15:56:19.902837+01:00 phy005 kernel: Process qemu-kvm (pid: 3346, threadinfo ffff880c21bc0000, task ffff880c2 264ddc0) 2011-02-08T15:56:19.902841+01:00 phy005 kernel: Stack: 2011-02-08T15:56:19.902844+01:00 phy005 kernel: 00007fa2ca200000 00007fa2ca201000 00007fa2ca201000 ffff880c22c3d280 2011-02-08T15:56:19.902848+01:00 phy005 kernel:<0> ffff880c21bc1af8 ffffffff81034a15 00007fa2ca200fff 00007fa2ca200fff 2011-02-08T15:56:19.902852+01:00 phy005 kernel:<0> ffff880c21bc1b44 ffff88060fdfad60 ffff880c2231a458 ffff880c231f97f8 2011-02-08T15:56:19.902855+01:00 phy005 kernel: Call Trace: 2011-02-08T15:56:19.902859+01:00 phy005 kernel: [<ffffffff81034a15>] gup_pud_range+0x156/0x192 2011-02-08T15:56:19.902863+01:00 phy005 kernel: [<ffffffff81034b15>] get_user_pages_fast+0xc4/0x172 2011-02-08T15:56:19.902867+01:00 phy005 kernel: [<ffffffff81131fbc>] ? bio_add_page+0x36/0x38 2011-02-08T15:56:19.902871+01:00 phy005 kernel: [<ffffffff81134730>] dio_get_page+0x54/0x127 2011-02-08T15:56:19.902875+01:00 phy005 kernel: [<ffffffff81135317>] __blockdev_direct_IO+0x41d/0xa36 2011-02-08T15:56:19.902880+01:00 phy005 kernel: [<ffffffffa008bf69>] ? x86_emulate_insn+0x1ff8/0x2d61 [kvm] 2011-02-08T15:56:19.902884+01:00 phy005 kernel: [<ffffffff8113379b>] blkdev_direct_IO+0x4e/0x50 2011-02-08T15:56:19.902888+01:00 phy005 kernel: [<ffffffff81132c49>] ? blkdev_get_blocks+0x0/0x8d 2011-02-08T15:56:19.902892+01:00 phy005 kernel: [<ffffffff810cb516>] generic_file_direct_write+0xed/0x16d 2011-02-08T15:56:19.902896+01:00 phy005 kernel: [<ffffffff810cb72c>] __generic_file_aio_write+0x196/0x281 2011-02-08T15:56:19.902899+01:00 phy005 kernel: [<ffffffff81133043>] ? blkdev_aio_write+0x0/0x69 2011-02-08T15:56:19.902909+01:00 phy005 kernel: [<ffffffff81133043>] ? blkdev_aio_write+0x0/0x69 2011-02-08T15:56:19.902914+01:00 phy005 kernel: [<ffffffff8113d4eb>] aio_rw_vect_retry+0x85/0x18e 2011-02-08T15:56:19.902919+01:00 phy005 kernel: [<ffffffff8113e9b3>] aio_run_iocb+0x77/0x10f 2011-02-08T15:56:19.902923+01:00 phy005 kernel: [<ffffffff8113f508>] do_io_submit+0x558/0x7ce 2011-02-08T15:56:19.902927+01:00 phy005 kernel: [<ffffffff8113f78e>] sys_io_submit+0x10/0x12 2011-02-08T15:56:19.902932+01:00 phy005 kernel: [<ffffffff81009c72>] system_call_fastpath+0x16/0x1b 2011-02-08T15:56:19.902938+01:00 phy005 kernel: Code: 21 d8 49 01 c2 49 8b 3a 49 89 fe 4d 21 ee 4d 21 e6 49 39 ce 75 49 48 89 f8 0f 1f 40 00 48 21 d8 48 c1 e8 0c 48 6b c0 38 4c 01 d8<66> 83 38 00 48 89 c7 79 04 48 8b 78 10 f0 ff 47 08 49 63 39 48 2011-02-08T15:56:19.903077+01:00 phy005 kernel: RIP [<ffffffff81034880>] gup_pte_range+0x94/0xd3 2011-02-08T15:56:19.903081+01:00 phy005 kernel: RSP<ffff880c21bc1a78> 2011-02-08T15:56:19.903084+01:00 phy005 kernel: CR2: ffffea71929181c0 2011-02-08T15:56:19.903088+01:00 phy005 kernel: ---[ end trace 174c28940e9fd0a7 ]---
Again outside kvm.
and yesterday this one: 2011-02-09T07:40:15.636528+01:00 phy005 kernel: BUG: unable to handle kernel NULL pointer dereference at (null) 2011-02-09T07:40:15.636635+01:00 phy005 kernel: IP: [<ffffffffa0082db8>] gfn_to_rmap+0x20/0x6e [kvm] 2011-02-09T07:40:15.636639+01:00 phy005 kernel: PGD 0 2011-02-09T07:40:15.636643+01:00 phy005 kernel: Oops: 0000 [#3] SMP 2011-02-09T07:40:15.636647+01:00 phy005 kernel: last sysfs file: /sys/devices/system/cpu/cpu15/topology/thread_siblings 2011-02-09T07:40:15.636650+01:00 phy005 kernel: CPU 2 2011-02-09T07:40:15.636656+01:00 phy005 kernel: Modules linked in: tun ipmi_devintf ipmi_si ipmi_msghandler bridge 8021q garp stp llc bonding xt_comment xt_recent ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 kvm_intel kvm igb i2c_i801 iTCO_wdt ioatdma i2c_core iTCO_vendor_support dca serio_raw joydev 3w_9xxx [last unloaded: scsi_wait_scan] 2011-02-09T07:40:15.636663+01:00 phy005 kernel: 2011-02-09T07:40:15.636666+01:00 phy005 kernel: Pid: 2572, comm: qemu-kvm Tainted: G D 2.6.34.7-66.tilaa.fc13.x86_64 #1 X8DTU/X8DTU 2011-02-09T07:40:15.636670+01:00 phy005 kernel: RIP: 0010:[<ffffffffa0082db8>] [<ffffffffa0082db8>] gfn_to_rmap+0x20/0x6e [kvm] 2011-02-09T07:40:15.636673+01:00 phy005 kernel: RSP: 0018:ffff88061cbcbcd8 EFLAGS: 00010246 2011-02-09T07:40:15.636677+01:00 phy005 kernel: RAX: 0000000000000000 RBX: 1603a07305004fff RCX: ffff88061cbcbd08 2011-02-09T07:40:15.636680+01:00 phy005 kernel: RDX: 0000000000000023 RSI: 1603a07305004fff RDI: 0000000000000000 2011-02-09T07:40:15.636683+01:00 phy005 kernel: RBP: ffff88061cbcbce8 R08: 0000000000000023 R09: 0000000000000000 2011-02-09T07:40:15.636686+01:00 phy005 kernel: R10: 0000000000000000 R11: ffffffffa0082c7f R12: 0000000000000001 2011-02-09T07:40:15.636689+01:00 phy005 kernel: R13: 0000000000311763 R14: ffff8809b8b01ce0 R15: 0000000000000000 2011-02-09T07:40:15.636692+01:00 phy005 kernel: FS: 0000000000000000(0000) GS:ffff880002040000(0000) knlGS:0000000000000000 2011-02-09T07:40:15.636695+01:00 phy005 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b 2011-02-09T07:40:15.636699+01:00 phy005 kernel: CR2: 0000000000000000 CR3: 0000000001a42000 CR4: 00000000000026e0 2011-02-09T07:40:15.636702+01:00 phy005 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2011-02-09T07:40:15.636705+01:00 phy005 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2011-02-09T07:40:15.636709+01:00 phy005 kernel: Process qemu-kvm (pid: 2572, threadinfo ffff88061cbca000, task ffff88061cf04650) 2011-02-09T07:40:15.636711+01:00 phy005 kernel: Stack: 2011-02-09T07:40:15.636715+01:00 phy005 kernel: ffff88036c471ff8 ffff880c23984000 ffff88061cbcbd18 ffffffffa0082ea9 2011-02-09T07:40:15.636718+01:00 phy005 kernel:<0> ffff8809b8b01ce0 ffff880c23984000 ffff88036c471ff8 00000000000001ff 2011-02-09T07:40:15.636721+01:00 phy005 kernel:<0> ffff88061cbcbd58 ffffffffa008363b 0000000000000200 ffff880c23984000 2011-02-09T07:40:15.636724+01:00 phy005 kernel: Call Trace: 2011-02-09T07:40:15.636728+01:00 phy005 kernel: [<ffffffffa0082ea9>] rmap_remove+0xa3/0x1a0 [kvm] 2011-02-09T07:40:15.636731+01:00 phy005 kernel: [<ffffffffa008363b>] kvm_mmu_zap_page+0x9f/0x299 [kvm] 2011-02-09T07:40:15.636734+01:00 phy005 kernel: [<ffffffffa0083a42>] kvm_mmu_zap_all+0x35/0x60 [kvm] 2011-02-09T07:40:15.636738+01:00 phy005 kernel: [<ffffffffa0078cde>] kvm_arch_flush_shadow+0x16/0x22 [kvm] 2011-02-09T07:40:15.636741+01:00 phy005 kernel: [<ffffffffa006eb0a>] kvm_mmu_notifier_release+0x31/0x44 [kvm] 2011-02-09T07:40:15.636744+01:00 phy005 kernel: [<ffffffff810fac37>] __mmu_notifier_release+0x4f/0x7b 2011-02-09T07:40:15.636748+01:00 phy005 kernel: [<ffffffff810e735d>] exit_mmap+0x2c/0x132 2011-02-09T07:40:15.636751+01:00 phy005 kernel: [<ffffffff8104ad7a>] mmput+0x5e/0xca 2011-02-09T07:40:15.636754+01:00 phy005 kernel: [<ffffffff8104f0d5>] exit_mm+0x114/0x121 2011-02-09T07:40:15.636757+01:00 phy005 kernel: [<ffffffff81050bf5>] do_exit+0x254/0x752 2011-02-09T07:40:15.636760+01:00 phy005 kernel: [<ffffffff8100a60e>] ? apic_timer_interrupt+0xe/0x20 2011-02-09T07:40:15.636764+01:00 phy005 kernel: [<ffffffff81051174>] do_group_exit+0x81/0xab 2011-02-09T07:40:15.636767+01:00 phy005 kernel: [<ffffffff810511b5>] sys_exit_group+0x17/0x1b 2011-02-09T07:40:15.636771+01:00 phy005 kernel: [<ffffffff81009c72>] system_call_fastpath+0x16/0x1b 2011-02-09T07:40:15.636777+01:00 phy005 kernel: Code: 88 ff ff ff b8 01 00 00 00 c9 c3 55 48 89 e5 41 54 53 0f 1f 44 00 00 41 89 d4 48 89 f3 e8 7b c7 fe ff 41 83 fc 01 48 89 c7 75 0d<48> 2b 18 48 c1 e3 03 48 03 58 18 eb 39 41 8d 4c 24 ff be 01 00 2011-02-09T07:40:15.636785+01:00 phy005 kernel: RIP [<ffffffffa0082db8>] gfn_to_rmap+0x20/0x6e [kvm] 2011-02-09T07:40:15.636788+01:00 phy005 kernel: RSP<ffff88061cbcbcd8> 2011-02-09T07:40:15.636791+01:00 phy005 kernel: CR2: 0000000000000000 2011-02-09T07:40:15.637743+01:00 phy005 kernel: ---[ end trace 174c28940e9fd0a9 ]--- 2011-02-09T07:40:15.637751+01:00 phy005 kernel: Fixing recursive fault but reboot is needed!
In kvm. Was there a reboot between the two?
So it doesn't seem to be a hardware problem since we replaced all that.
I agree. And your other machines are stable? When you say "identical software", are those exactly the same binaries? copying Andrea for possible insight into the non-kvm oopses. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
