Hello, we're having problem one of our kvm guests. According to storage summary mail Christopher sent few days ago, I switched my disk model to SCSI which should be the safest choice. But now we've stumbled upon the problem that the whole guest crashes when we run one specific postgres query which heavily loads it. I can 100% reproduce this problem on both our production nodes (each 8cores, 16GB RAM), on my testing machine (4cores, 3GB RAM) this causes the HOST to reboot (which is even worse). We haven't experienced this problem with virtio. I tried both qemu-kvm-0.10.6 and kvm-88, the host is running 2.6.30.5, I tried 2.6.29.x and 2.6.30.5 for guest. Guest backtrace follows (it's a bit mangled as it's obtained using netconsole) Sep 2 19:01:20 sql2 [ 1564.795629] BUG: unable to handle kernel Sep 2 19:01:20 sql2 NULL pointer dereference Sep 2 19:01:20 sql2 NULL pointer dereference Sep 2 19:01:20 sql2 at 0000000000000358 Sep 2 19:01:20 sql2 [ 1564.797727] IP: Sep 2 19:01:20 sql2 at 0000000000000358 Sep 2 19:01:20 sql2 [ 1564.797727] IP: Sep 2 19:01:20 sql2 [<ffffffffa018beff>] sym_int_sir+0x2bf/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [<ffffffffa018beff>] sym_int_sir+0x2bf/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.798969] PGD 2d5a2067 Sep 2 19:01:20 sql2 PUD 2d4ad067 Sep 2 19:01:20 sql2 [ 1564.798969] PGD 2d5a2067 Sep 2 19:01:20 sql2 PMD 0 Sep 2 19:01:20 sql2 PUD 2d4ad067 Sep 2 19:01:20 sql2 PMD 0 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.799505] Oops: 0000 [#1] Sep 2 19:01:20 sql2 PREEMPT Sep 2 19:01:20 sql2 [ 1564.799505] Oops: 0000 [#1] Sep 2 19:01:20 sql2 SMP Sep 2 19:01:20 sql2 PREEMPT Sep 2 19:01:20 sql2 SMP Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.800773] last sysfs file: /sys/block/md0/dev Sep 2 19:01:20 sql2 [ 1564.800773] CPU 3 Sep 2 19:01:20 sql2 [ 1564.800773] last sysfs file: /sys/block/md0/dev Sep 2 19:01:20 sql2 [ 1564.800773] CPU 3 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.800773] Modules linked in: Sep 2 19:01:20 sql2 netconsole Sep 2 19:01:20 sql2 [ 1564.800773] Modules linked in: Sep 2 19:01:20 sql2 ipv6 Sep 2 19:01:20 sql2 netconsole Sep 2 19:01:20 sql2 ipv6 Sep 2 19:01:20 sql2 reiserfs Sep 2 19:01:20 sql2 dm_mirror Sep 2 19:01:20 sql2 reiserfs Sep 2 19:01:20 sql2 dm_mirror Sep 2 19:01:20 sql2 dm_region_hash Sep 2 19:01:20 sql2 dm_log Sep 2 19:01:20 sql2 dm_region_hash Sep 2 19:01:20 sql2 dm_log Sep 2 19:01:20 sql2 dm_mod Sep 2 19:01:20 sql2 video Sep 2 19:01:20 sql2 dm_mod Sep 2 19:01:20 sql2 video Sep 2 19:01:20 sql2 backlight Sep 2 19:01:20 sql2 output Sep 2 19:01:20 sql2 backlight Sep 2 19:01:20 sql2 output Sep 2 19:01:20 sql2 sbs Sep 2 19:01:20 sql2 sbshc Sep 2 19:01:20 sql2 sbs Sep 2 19:01:20 sql2 sbshc Sep 2 19:01:20 sql2 fan Sep 2 19:01:20 sql2 container Sep 2 19:01:20 sql2 fan Sep 2 19:01:20 sql2 battery Sep 2 19:01:20 sql2 container Sep 2 19:01:20 sql2 battery Sep 2 19:01:20 sql2 ac Sep 2 19:01:20 sql2 ac Sep 2 19:01:20 sql2 parport_pc Sep 2 19:01:20 sql2 lp Sep 2 19:01:20 sql2 parport_pc Sep 2 19:01:20 sql2 lp Sep 2 19:01:20 sql2 parport Sep 2 19:01:20 sql2 nvram Sep 2 19:01:20 sql2 parport Sep 2 19:01:20 sql2 nvram Sep 2 19:01:20 sql2 sg Sep 2 19:01:20 sql2 sr_mod Sep 2 19:01:20 sql2 sg Sep 2 19:01:20 sql2 sr_mod Sep 2 19:01:20 sql2 cdrom Sep 2 19:01:20 sql2 thermal Sep 2 19:01:20 sql2 cdrom Sep 2 19:01:20 sql2 thermal Sep 2 19:01:20 sql2 processor Sep 2 19:01:20 sql2 thermal_sys Sep 2 19:01:20 sql2 processor Sep 2 19:01:20 sql2 button Sep 2 19:01:20 sql2 thermal_sys Sep 2 19:01:20 sql2 button Sep 2 19:01:20 sql2 virtio_balloon Sep 2 19:01:20 sql2 virtio_balloon Sep 2 19:01:20 sql2 i2c_piix4 Sep 2 19:01:20 sql2 e1000 Sep 2 19:01:20 sql2 i2c_piix4 Sep 2 19:01:20 sql2 i2c_core Sep 2 19:01:20 sql2 e1000 Sep 2 19:01:20 sql2 piix Sep 2 19:01:20 sql2 i2c_core Sep 2 19:01:20 sql2 piix Sep 2 19:01:20 sql2 sym53c8xx Sep 2 19:01:20 sql2 scsi_transport_spi Sep 2 19:01:20 sql2 sym53c8xx Sep 2 19:01:20 sql2 pcspkr Sep 2 19:01:20 sql2 scsi_transport_spi Sep 2 19:01:20 sql2 pcspkr Sep 2 19:01:20 sql2 virtio_blk Sep 2 19:01:20 sql2 virtio_net Sep 2 19:01:20 sql2 virtio_blk Sep 2 19:01:20 sql2 virtio_pci Sep 2 19:01:20 sql2 virtio_net Sep 2 19:01:20 sql2 virtio_ring Sep 2 19:01:20 sql2 virtio_pci Sep 2 19:01:20 sql2 virtio Sep 2 19:01:20 sql2 virtio_ring Sep 2 19:01:20 sql2 8139cp Sep 2 19:01:20 sql2 virtio Sep 2 19:01:20 sql2 8139cp Sep 2 19:01:20 sql2 mii Sep 2 19:01:20 sql2 bitrev Sep 2 19:01:20 sql2 mii Sep 2 19:01:20 sql2 bitrev Sep 2 19:01:20 sql2 pata_acpi Sep 2 19:01:20 sql2 pata_acpi Sep 2 19:01:20 sql2 ide_pci_generic Sep 2 19:01:20 sql2 ide_core Sep 2 19:01:20 sql2 ide_pci_generic Sep 2 19:01:20 sql2 ata_piix Sep 2 19:01:20 sql2 ide_core Sep 2 19:01:20 sql2 ata_generic Sep 2 19:01:20 sql2 ata_piix Sep 2 19:01:20 sql2 libata Sep 2 19:01:20 sql2 ata_generic Sep 2 19:01:20 sql2 sd_mod Sep 2 19:01:20 sql2 libata Sep 2 19:01:20 sql2 scsi_mod Sep 2 19:01:20 sql2 sd_mod Sep 2 19:01:20 sql2 ext4 Sep 2 19:01:20 sql2 scsi_mod Sep 2 19:01:20 sql2 jbd2 Sep 2 19:01:20 sql2 ext4 Sep 2 19:01:20 sql2 jbd2 Sep 2 19:01:20 sql2 crc32 Sep 2 19:01:20 sql2 crc16 Sep 2 19:01:20 sql2 crc32 Sep 2 19:01:20 sql2 uhci_hcd Sep 2 19:01:20 sql2 crc16 Sep 2 19:01:20 sql2 ohci_hcd Sep 2 19:01:20 sql2 uhci_hcd Sep 2 19:01:20 sql2 ehci_hcd Sep 2 19:01:20 sql2 ohci_hcd Sep 2 19:01:20 sql2 ehci_hcd Sep 2 19:01:20 sql2 [last unloaded: scsi_wait_scan] Sep 2 19:01:20 sql2 [last unloaded: scsi_wait_scan] Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.800773] Pid: 0, comm: swapper Not tainted 2.6.30lb.06 #1 Sep 2 19:01:20 sql2 [ 1564.800773] Pid: 0, comm: swapper Not tainted 2.6.30lb.06 #1 Sep 2 19:01:20 sql2 [ 1564.800773] RIP: 0010:[<ffffffffa018beff>] Sep 2 19:01:20 sql2 [ 1564.800773] RIP: 0010:[<ffffffffa018beff>] Sep 2 19:01:20 sql2 [<ffffffffa018beff>] sym_int_sir+0x2bf/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [<ffffffffa018beff>] sym_int_sir+0x2bf/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.800773] RSP: 0018:ffff88000244edf8 EFLAGS: 00010087 Sep 2 19:01:20 sql2 [ 1564.800773] RSP: 0018:ffff88000244edf8 EFLAGS: 00010087 Sep 2 19:01:20 sql2 [ 1564.800773] RAX: 000000000000000b RBX: 000000000000000b RCX: 0000000000000084 Sep 2 19:01:20 sql2 [ 1564.800773] RAX: 000000000000000b RBX: 000000000000000b RCX: 0000000000000084 Sep 2 19:01:20 sql2 [ 1564.800773] RDX: ffff88002e8dc800 RSI: 000000002d46b168 RDI: ffffc2000021a006 Sep 2 19:01:20 sql2 [ 1564.800773] RDX: ffff88002e8dc800 RSI: 000000002d46b168 RDI: ffffc2000021a006 Sep 2 19:01:20 sql2 [ 1564.800773] RBP: ffff88000244ee68 R08: ffffffffa00b243c R09: 00017fd30f1b243c Sep 2 19:01:20 sql2 [ 1564.800773] RBP: ffff88000244ee68 R08: ffffffffa00b243c R09: 00017fd30f1b243c Sep 2 19:01:20 sql2 [ 1564.800773] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88002d46b168 Sep 2 19:01:20 sql2 [ 1564.800773] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88002d46b168 Sep 2 19:01:20 sql2 [ 1564.800773] R13: 0000000000000000 R14: ffff88002d46b000 R15: 0000000000000000 Sep 2 19:01:20 sql2 [ 1564.800773] R13: 0000000000000000 R14: ffff88002d46b000 R15: 0000000000000000 Sep 2 19:01:20 sql2 [ 1564.800773] FS: 0000000000000000(0000) GS:ffff88000244b000(0000) knlGS:0000000000000000 Sep 2 19:01:20 sql2 [ 1564.800773] FS: 0000000000000000(0000) GS:ffff88000244b000(0000) knlGS:0000000000000000 Sep 2 19:01:20 sql2 [ 1564.800773] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Sep 2 19:01:20 sql2 [ 1564.800773] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Sep 2 19:01:20 sql2 [ 1564.800773] CR2: 0000000000000358 CR3: 000000002dd34000 CR4: 00000000000006e0 Sep 2 19:01:20 sql2 [ 1564.800773] CR2: 0000000000000358 CR3: 000000002dd34000 CR4: 00000000000006e0 Sep 2 19:01:20 sql2 [ 1564.800773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Sep 2 19:01:20 sql2 [ 1564.800773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Sep 2 19:01:20 sql2 [ 1564.800773] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Sep 2 19:01:20 sql2 [ 1564.800773] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Sep 2 19:01:20 sql2 [ 1564.800773] Process swapper (pid: 0, threadinfo ffff88002f8c2000, task ffff88002f8c8000) Sep 2 19:01:20 sql2 [ 1564.800773] Process swapper (pid: 0, threadinfo ffff88002f8c2000, task ffff88002f8c8000) Sep 2 19:01:20 sql2 [ 1564.800773] Stack: Sep 2 19:01:20 sql2 [ 1564.800773] 0000000000000040 Sep 2 19:01:20 sql2 ffff88000244eec0 Sep 2 19:01:20 sql2 [ 1564.800773] 0000000000000040 Sep 2 19:01:20 sql2 0000000000000001 Sep 2 19:01:20 sql2 ffff88000244eec0 Sep 2 19:01:20 sql2 0000000000000001 Sep 2 19:01:20 sql2 0000000000000000 Sep 2 19:01:20 sql2 0000000000000000 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 ffff88000245bfc0 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 0000000000000046 Sep 2 19:01:20 sql2 ffff88000245bfc0 Sep 2 19:01:20 sql2 0000000000000000 Sep 2 19:01:20 sql2 0000000000000001 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 0000000000000000 Sep 2 19:01:20 sql2 0000000000000001 Sep 2 19:01:20 sql2 0000000000000000 Sep 2 19:01:20 sql2 0000000000000000 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.800773] Call Trace: Sep 2 19:01:20 sql2 [ 1564.800773] <IRQ> Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [<ffffffffa018e11c>] sym_interrupt+0x43c/0x720 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffffa0186b61>] sym53c8xx_intr+0x41/0x90 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff8028ecac>] handle_IRQ_event+0x4c/0x1c0 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff80290eb9>] handle_fasteoi_irq+0x89/0x110 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff8020e05f>] handle_irq+0x1f/0x30 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff8020d81e>] do_IRQ+0x6e/0x100 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff8020bd53>] ret_from_intr+0x0/0xa Sep 2 19:01:20 sql2 [ 1564.800773] <EOI> Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [<ffffffff802130c5>] ? default_idle+0x45/0x150 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff8025da81>] ? atomic_notifier_call_chain+0x11/0x20 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff8020a20e>] ? cpu_idle+0x4e/0xa0 Sep 2 19:01:20 sql2 [ 1564.800773] Sep 2 19:01:20 sql2 [<ffffffff804e5aa3>] ? start_secondary+0x173/0x1d0 Sep 2 19:01:20 sql2 [ 1564.800773] Code: Sep 2 19:01:20 sql2 ad Sep 2 19:01:20 sql2 07 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 2c Sep 2 19:01:20 sql2 01 Sep 2 19:01:20 sql2 0f Sep 2 19:01:20 sql2 84 Sep 2 19:01:20 sql2 5d Sep 2 19:01:20 sql2 09 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 0f Sep 2 19:01:20 sql2 ae Sep 2 19:01:20 sql2 f8 Sep 2 19:01:20 sql2 49 Sep 2 19:01:20 sql2 8b Sep 2 19:01:20 sql2 b6 Sep 2 19:01:20 sql2 50 Sep 2 19:01:20 sql2 05 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 41 Sep 2 19:01:20 sql2 8b Sep 2 19:01:20 sql2 be Sep 2 19:01:20 sql2 60 Sep 2 19:01:20 sql2 06 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 48 Sep 2 19:01:20 sql2 83 Sep 2 19:01:20 sql2 c6 Sep 2 19:01:20 sql2 2c Sep 2 19:01:20 sql2 e8 Sep 2 19:01:20 sql2 e6 Sep 2 19:01:20 sql2 64 Sep 2 19:01:20 sql2 1f Sep 2 19:01:20 sql2 e0 Sep 2 19:01:20 sql2 e9 Sep 2 19:01:20 sql2 4d Sep 2 19:01:20 sql2 fe Sep 2 19:01:20 sql2 ff Sep 2 19:01:20 sql2 ff Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 8b Sep 2 19:01:20 sql2 97 Sep 2 19:01:20 sql2 58 Sep 2 19:01:20 sql2 03 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 48 Sep 2 19:01:20 sql2 8b Sep 2 19:01:20 sql2 82 Sep 2 19:01:20 sql2 80 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 4c Sep 2 19:01:20 sql2 8b Sep 2 19:01:20 sql2 a0 Sep 2 19:01:20 sql2 c8 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 00 Sep 2 19:01:20 sql2 Sep 2 19:01:20 sql2 [ 1564.878205] RIP Sep 2 19:01:20 sql2 [<ffffffffa018beff>] sym_int_sir+0x2bf/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.878205] RSP <ffff88000244edf8> Sep 2 19:01:20 sql2 [ 1564.878205] CR2: 0000000000000358 Sep 2 19:01:20 sql2 [ 1564.878205] ---[ end trace 1af2f1cc92726b74 ]--- Sep 2 19:01:20 sql2 [ 1564.878205] Kernel panic - not syncing: Fatal exception in interrupt Sep 2 19:01:20 sql2 [ 1564.878205] Pid: 0, comm: swapper Tainted: G D 2.6.30lb.06 #1 Sep 2 19:01:20 sql2 [ 1564.878205] Call Trace: Sep 2 19:01:20 sql2 [ 1564.878205] <IRQ> Sep 2 19:01:20 sql2 [<ffffffff80240cda>] panic+0xaa/0x180 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff804ecee0>] ? _spin_unlock_irqrestore+0x50/0x60 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8024146f>] ? release_console_sem+0x1df/0x200 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff80241707>] ? console_unblank+0x67/0x80 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8024095e>] ? print_oops_end_marker+0x1e/0x20 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8020f74f>] oops_end+0x9f/0xb0 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8022871a>] no_context+0x15a/0x250 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff80232730>] ? __enqueue_entity+0x80/0x90 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff802289ab>] __bad_area_nosemaphore+0xdb/0x1c0 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff80234499>] ? find_busiest_group+0x1a9/0x940 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff80228b2e>] bad_area_nosemaphore+0xe/0x10 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff80228e4b>] do_page_fault+0xab/0x270 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff804ed36f>] page_fault+0x1f/0x30 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffffa018beff>] ? sym_int_sir+0x2bf/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffffa018bc9e>] ? sym_int_sir+0x5e/0x1590 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffffa018e11c>] sym_interrupt+0x43c/0x720 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffffa0186b61>] sym53c8xx_intr+0x41/0x90 [sym53c8xx] Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8028ecac>] handle_IRQ_event+0x4c/0x1c0 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff80290eb9>] handle_fasteoi_irq+0x89/0x110 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8020e05f>] handle_irq+0x1f/0x30 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8020d81e>] do_IRQ+0x6e/0x100 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8020bd53>] ret_from_intr+0x0/0xa Sep 2 19:01:20 sql2 [ 1564.878205] <EOI> Sep 2 19:01:20 sql2 [<ffffffff802130c5>] ? default_idle+0x45/0x150 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8025da81>] ? atomic_notifier_call_chain+0x11/0x20 Sep 2 19:01:20 sql2 [ 1564.878205] [<ffffffff8020a20e>] ? cpu_idle+0x4e/0xa0 Any idea on what could be wrong? I'll gladly provide any further information which might help to track this bug down... thanks a lot regards nik -- ------------------------------------- Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis@xxxxxxxxxxx ------------------------------------- -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
