Is there a possibility to track this further down? I've problems on two other KVMs since a few weeks which I think that they're related to this. Host for this KVMs are kernel 2.6.32. Guests until today were also running 2.6.32. Inside the KVMs we're using GlusterFS, NFSv4 and Apache with PHP. From time to time the httpd-processes are "hanging". When this happens then we're seeing a lot of soft lockups. This hosts are running Xeon X5560 processors. Until today I suspected that this problems only happens on older Xeon's but this doesn't seems to be true. I've attached the output from /var/log/messages (https://bugzilla.kernel.org/attachment.cgi?id=26048) from one of the hosts with GlusterFS. I've now downgraded to kernel 2.6.30 in the guests. But since this problem also exists in 2.6.34-rc3 I suspect that we're never ever will be able to do a kernel update in the guests when they're using NFS :-( But what I definitely can say is that all the problems only happens with guests running kernel >= 2.6.31 and with a remote file system (NFS, GlusterFS). Some days ago another KVM have had a network shutdown using kernel 2.6.32 in host and guest + NFSv4. But this only happend once until now and there isn't so much traffic running through the interfaces of that host. All other guests with kernel 2.6.30 (about 80 guests on 18 hosts) with NFS and KVM 0.12.3 are really running perfectly. Thanks! Robert On 04/13/10 10:51, Robert Wimmer wrote: > I've tried to do my very best. In general I can > say: All 2.6.30 versions work, all 2.6.31 fail. 2.6.31-rc3 > fails with "soft lockup" and is the only one which > don't show any "swapper page allocation failure". > But the result is finally the same... 2.6.31-rc4 > don't show "soft lockups" but "swapper page allocation failure". > Here is the dmesg output for 2.6.31-rc3: > https://bugzilla.kernel.org/attachment.cgi?id=25986 > > So here is what I've done. Started with a fresh tree > and my 2.6.30 .config: > > rm -fr /usr/src/linux > cd /usr/src > git clone > git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux > cd linux > git checkout 0b4f2928f14c4a9770b0866923fc81beb7f4aa57 > > Here is the "git bisect log" output: > > # bad: [74fca6a42863ffacaf7ba6f1936a9f228950f657] Linux 2.6.31 > # good: [07a2039b8eb0af4ff464efd3dfd95de5c02648c6] Linux 2.6.30 > git bisect start 'v2.6.31' 'v2.6.30' '--' 'drivers/virtio/' > 'drivers/net/virtio_net.c' > # good: [e3353853730eb99c56b7b0aed1667d51c0e3699a] virtio: enhance > id_matching for virtio drivers > git bisect good e3353853730eb99c56b7b0aed1667d51c0e3699a > # good: [9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb] Merge branch 'master' > of master.kernel.org:/pub/scm/linux/kernel/git/torvalds/linux-2.6 > git bisect good 9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb > # bad: [ff52c3fc7188855ede75d87b022271f0da309e5b] virtio: fix memory > leak on device removal > git bisect bad ff52c3fc7188855ede75d87b022271f0da309e5b > # good: [31278e71471399beaff9280737e52b47db4dc345] net: group address > list and its count > git bisect good 31278e71471399beaff9280737e52b47db4dc345 > # bad: [4b892e6582e3a4fe01f623aea386907270d5bf83] virtio-pci: correctly > unregister root device on error > git bisect bad 4b892e6582e3a4fe01f623aea386907270d5bf83 > > Hopefully this gives you some hints. The problem > for me is that I don't know what commit I should > consider good or bad. Should I consider the > commit with the "soft lockup" as good because it > don't show the allocation failure? Currently it is > marked as bad (4b892e6582e3a4fe01f623aea386907270d5bf83). > What should I do next? > > Thanks! > Robert > > On 04/12/10 15:52, Michael S. Tsirkin wrote: > >> On Mon, Apr 12, 2010 at 03:50:31PM +0200, Robert Wimmer wrote: >> >> >>> Sorry but I need some more git help. Here is what I've done. >>> Started with a fresh clone of the kernel: >>> >>> cd /usr/src >>> git clone >>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux >>> cd linux >>> git checkout 0b4f2928f14c4a9770b0866923fc81beb7f4aa57 >>> >>> Since I already knew that this commit wasn't good I did >>> >>> git bisect start >>> git bisect bad >>> >>> >> I think what you miss is marking the good commit. >> bisect does a binary search but it needs to know >> both good and bad commits to search in the range. >> >> Optionally, you can use '-- drivers/virtio/ drivers/net/virtio_net.c' >> what this does is limit bisect to commits that touch files in >> question. This way you get much less tests to run >> (about 4) but after you find a first problematic commit >> you must verify that a commit just before it does not have the issue. >> >> If this turns out not to be the case, you'll have to >> fallback on full bisect, and we will now this is some >> other change in kernel that triggered the regression. >> >> >> >> >>> compiled and started over. As expected the problem returns. >>> So I've done another >>> >>> git bisect bad >>> >>> but I always get the same commit: >>> >>> kabul:/usr/src/linux # git bisect log >>> git bisect start >>> # bad: [0b4f2928f14c4a9770b0866923fc81beb7f4aa57] smc91x: fix >>> compilation on SMP >>> git bisect bad 0b4f2928f14c4a9770b0866923fc81beb7f4aa57 >>> # bad: [0b4f2928f14c4a9770b0866923fc81beb7f4aa57] smc91x: fix >>> compilation on SMP >>> git bisect bad 0b4f2928f14c4a9770b0866923fc81beb7f4aa57 >>> # bad: [0b4f2928f14c4a9770b0866923fc81beb7f4aa57] smc91x: fix >>> compilation on SMP >>> git bisect bad 0b4f2928f14c4a9770b0866923fc81beb7f4aa57 >>> >>> I've expected that after each "git bisect bad" I get the previous >>> commit before the "bad" one. How can get the previous commit? >>> The bisect documentation couldn't help me. >>> >>> Thanks! >>> Robert >>> >>> >>> >>> On 04/12/10 13:23, Michael S. Tsirkin wrote: >>> >>> >>>> On Mon, Apr 12, 2010 at 11:25:26AM +0200, Robert Wimmer wrote: >>>> >>>> >>>> >>>>> server10:/usr/src/linux # git bisect start v2.6.31 v2.6.30 -- >>>>> drivers/virtio/ drivers/net/virtio_net.c >>>>> Bisecting: 12 revisions left to test after this (roughly 4 steps) >>>>> [e3353853730eb99c56b7b0aed1667d51c0e3699a] virtio: enhance id_matching >>>>> for virtio drivers >>>>> >>>>> >>>>> >>>>> >>>> Sorry I wasn't clear. the way to use bisect is as follows: >>>> - first start as you did now. >>>> 1. now build kernel, install and test >>>> 2. if bug is there, type 'git bisect bad' >>>> 3. if bug is not there, type 'git bisect good' >>>> 4. The above will give you another kernel version to test >>>> if so go back to step 1 >>>> 6. this will be repeated about 4 times (number of steps above) >>>> 7. in the end you will get the first revision which has the >>>> problem. Let's assume it is revision ABCDEF. >>>> >>>> Type git bisect log to see your history. >>>> >>>> 8. Now git reset --hard ABCDEF~1 and try again. >>>> >>>> If you see the problem with ABCDEF but not ABCDEF~1 >>>> then we will have a good guess at the culprit. >>>> >>>> Some more tips here: >>>> http://www.kernel.org/pub/software/scm/git/docs/git-bisect.html >>>> >>>> >>>> >>>> >>>> >>>>> Today I've upgraded to qemu-kvm-0.12.3-r1 (Gentoo package) >>>>> but doesn't help. Still getting "page allocation failure" with >>>>> 2.6.31-rc5. >>>>> >>>>> Does it makes sense to use the same 2.6.31-rc5 kernel >>>>> in the host and guest for testing? Currently I'm still using 2.6.32 >>>>> in host and testing 2.6.31-rc5 in guest until "crashes". >>>>> Then I start the guest with 2.6.30 again which works >>>>> without trouble with 2.6.32 as host. >>>>> >>>>> This is really strange. I have hosts with 2.6.32 running >>>>> guests with 2.6.32 which works perfectly. These hosts >>>>> and guests running on HP DL 380 G6 with Intel Xeon X5560. >>>>> The guests which don't work with 2.6.32 (and 2.6.32 >>>>> as host) running on HP DL 380 G5 with Intel Xeon L5420. >>>>> >>>>> >>>>> >>>> Hmm. Some subtle race? >>>> >>>> >>>> >>>> >>>>> (All guests) and (all hosts) have the same packages >>>>> and the same versions installed and the same kernel >>>>> configs (hosts and guests using different .config but the >>>>> difference is very small e.g. CONFIG_PARAVIRT_SPINLOCKS=y, >>>>> CONFIG_PARAVIRT_GUEST=y in guests but not in hosts >>>>> .config). >>>>> >>>>> I've had problems with qemu-kvm 0.12.2 with high network >>>>> traffic which was solved by a patch submitted by Tom >>>>> Lendacky: >>>>> >>>>> "Fix a race condition where qemu finds that there are not enough virtio >>>>> ring buffers available and the guest make more buffers available before >>>>> qemu can enable notifications." >>>>> http://www.mail-archive.com/kvm@xxxxxxxxxxxxxxx/msg28667.html >>>>> >>>>> It was a real lifesaver for the HP DL 380 G6 mentioned >>>>> above but maybe this is now causing the problems with the G5 machines. >>>>> The symptoms are the same. I can still log into the guest >>>>> via VNC but the network is down. >>>>> >>>>> Thanks! >>>>> Robert >>>>> >>>>> >>>>> >>>>> >>>> For now the only thing we seem to know for sure is that on >>>> specific hardware there's a regression between 2.6.30 and >>>> 2.6.31-rc5. Yes, it is possible that all it does >>>> is expose a qemu bug, but it's hard to say. >>>> Let's find out what change >>>> does that, this should give us a hint. >>>> >>>> >>>> >>>> >>>>> On 04/11/10 13:03, Michael S. Tsirkin wrote: >>>>> >>>>> >>>>> >>>>>> On Fri, Apr 09, 2010 at 12:15:01PM +0200, Robert Wimmer wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> I'm not really a git hero so here is what I've done: >>>>>>> >>>>>>> cd /usr/src >>>>>>> git clone >>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux >>>>>>> cd linux >>>>>>> git checkout -b mykernel 0b4f2928f14c4a9770b0866923fc81beb7f4aa57 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> Looks right. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Then I've checked >>>>>>> >>>>>>> drivers/net/virtio_net.c >>>>>>> drivers/net/smc91x.c >>>>>>> >>>>>>> if the changes commited where not in there. >>>>>>> Next I build my kernel as usual. I used my .config >>>>>>> from 2.6.30 (which is working fine in a several >>>>>>> guests / .config see here: >>>>>>> https://bugzilla.kernel.org/attachment.cgi?id=25925) >>>>>>> and build the kernel >>>>>>> >>>>>>> genkernel --menuconfig --lvm --oldconfig all >>>>>>> >>>>>>> which finally gave me a 2.6.31-rc5. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> That's right. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> I should mention >>>>>>> that 2.6.30 was using SLUB. So here is the output >>>>>>> from the 2.6.31-rc5 kernel running about 20 min.: >>>>>>> https://bugzilla.kernel.org/attachment.cgi?id=25926 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> Hmm, so we see the error here as well? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Seems not very usefull to me. I'm currently compiling >>>>>>> the same kernel with SLAB. >>>>>>> >>>>>>> Please let me know if the git commands above are >>>>>>> right and/or if you need other kernel options enabled. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> Looks right. You don't have to add -b flag if you don't >>>>>> want to. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Thanks! >>>>>>> Robert >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> Hmm, I do not see anything else that seems related. >>>>>> Could you please try to bisect? >>>>>> >>>>>> git bisect start v2.6.31 v2.6.30 -- drivers/virtio/ drivers/net/virtio_net.c >>>>>> >>>>>> should help assuming the change that triggers this is in virtio. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On 04/08/10 22:04, Michael S. Tsirkin wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Thu, Apr 08, 2010 at 10:39:34PM +0300, Avi Kivity wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> cc: mst >>>>>>>>> >>>>>>>>> On 04/08/2010 10:34 PM, Andrew Morton wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> (switched to email. Please respond via emailed reply-to-all, not via the >>>>>>>>>> bugzilla web interface). >>>>>>>>>> >>>>>>>>>> On Wed, 7 Apr 2010 10:29:20 GMT >>>>>>>>>> bugzilla-daemon@xxxxxxxxxxxxxxxxxxx wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=15709 >>>>>>>>>>> >>>>>>>>>>> Summary: swapper page allocation failure >>>>>>>>>>> Product: Memory Management >>>>>>>>>>> Version: 2.5 >>>>>>>>>>> Kernel Version: 2.6.32 and 2.6.33 >>>>>>>>>>> Platform: All >>>>>>>>>>> OS/Version: Linux >>>>>>>>>>> Tree: Mainline >>>>>>>>>>> Status: NEW >>>>>>>>>>> Severity: normal >>>>>>>>>>> Priority: P1 >>>>>>>>>>> Component: Slab Allocator >>>>>>>>>>> AssignedTo: akpm@xxxxxxxxxxxxxxxxxxxx >>>>>>>>>>> ReportedBy: kernel@xxxxxxxxxxx >>>>>>>>>>> Regression: No >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Created an attachment (id=25903) >>>>>>>>>>> --> (https://bugzilla.kernel.org/attachment.cgi?id=25903) >>>>>>>>>>> dmesg output >>>>>>>>>>> >>>>>>>>>>> I'm having problems with "swapper page allocation failure's" since upgrading >>>>>>>>>>> from kernel 2.6.30 to 2.6.32/2.6.33. The problems occur inside a kernel virtual >>>>>>>>>>> maschine (KVM). Running Gentoo with kernel 2.6.32 as host which works fine. As >>>>>>>>>>> long as kernel 2.6.30 is used as guest kernel the guest runs fine. But after >>>>>>>>>>> upgrading to 2.6.32 and 2.6.33 I get "swapper page allocation failure's" (see >>>>>>>>>>> attachment of dmesg output). The guest is only running a Apache webserver and >>>>>>>>>>> serves files from a NFS share. It has 1 GB RAM and 2 virtual CPUs. I've tried >>>>>>>>>>> different kernel configurations (e.g. a unmodified version from Sabayon Linux >>>>>>>>>>> Distribution) but doesn't help. Load of the guest (and host) is very low. >>>>>>>>>>> Network traffic is about 20-50 MBit/s. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> hm, this is a regression. >>>>>>>>>> >>>>>>>>>> : [ 454.006706] users: page allocation failure. order:0, mode:0x20 >>>>>>>>>> : [ 454.006712] Pid: 7992, comm: users Not tainted 2.6.34-rc3-git6 #2 >>>>>>>>>> : [ 454.006714] Call Trace: >>>>>>>>>> : [ 454.006717]<IRQ> [<ffffffff8109dff7>] __alloc_pages_nodemask+0x5c8/0x615 >>>>>>>>>> : [ 454.006796] [<ffffffff817860ce>] ? ip_local_deliver+0x65/0x6d >>>>>>>>>> : [ 454.006820] [<ffffffff810c39c4>] alloc_pages_current+0x96/0x9f >>>>>>>>>> : [ 454.006842] [<ffffffff8167f2c7>] try_fill_recv+0x5e/0x20f >>>>>>>>>> : [ 454.006846] [<ffffffff8167fe13>] virtnet_poll+0x52a/0x5c7 >>>>>>>>>> : [ 454.006858] [<ffffffff8104fe74>] ? run_timer_softirq+0x1dc/0x1f4 >>>>>>>>>> : [ 454.006873] [<ffffffff8176035d>] net_rx_action+0xad/0x1a5 >>>>>>>>>> : [ 454.006882] [<ffffffff8104b6cd>] __do_softirq+0x9c/0x127 >>>>>>>>>> : [ 454.006897] [<ffffffff81008ffc>] call_softirq+0x1c/0x30 >>>>>>>>>> : [ 454.006901] [<ffffffff8100af01>] do_softirq+0x41/0x7e >>>>>>>>>> : [ 454.006904] [<ffffffff8104b3e3>] irq_exit+0x36/0x75 >>>>>>>>>> : [ 454.006907] [<ffffffff8100a5ee>] do_IRQ+0xaa/0xc1 >>>>>>>>>> : [ 454.006926] [<ffffffff8183bc13>] ret_from_intr+0x0/0x11 >>>>>>>>>> : [ 454.006928]<EOI> [<ffffffff81026b25>] ? kvm_deferred_mmu_op+0x5e/0xe7 >>>>>>>>>> : [ 454.006942] [<ffffffff81026b19>] ? kvm_deferred_mmu_op+0x52/0xe7 >>>>>>>>>> : [ 454.006946] [<ffffffff81026c03>] kvm_mmu_write+0x2e/0x35 >>>>>>>>>> : [ 454.006949] [<ffffffff81026c7d>] kvm_set_pte_at+0x19/0x1b >>>>>>>>>> : [ 454.006953] [<ffffffff810aba67>] __do_fault+0x3c4/0x492 >>>>>>>>>> : [ 454.006957] [<ffffffff810adcf4>] handle_mm_fault+0x478/0x9d8 >>>>>>>>>> : [ 454.006966] [<ffffffff810deb59>] ? path_put+0x2c/0x30 >>>>>>>>>> : [ 454.006975] [<ffffffff8102f162>] do_page_fault+0x2f6/0x31a >>>>>>>>>> : [ 454.006979] [<ffffffff8183b81e>] ? _raw_spin_lock+0x9/0xd >>>>>>>>>> : [ 454.006982] [<ffffffff8183bef5>] page_fault+0x25/0x30 >>>>>>>>>> : [ 454.006985] Mem-Info: >>>>>>>>>> : [ 454.006987] Node 0 DMA per-cpu: >>>>>>>>>> : [ 454.006990] CPU 0: hi: 0, btch: 1 usd: 0 >>>>>>>>>> : [ 454.006992] CPU 1: hi: 0, btch: 1 usd: 0 >>>>>>>>>> : [ 454.006993] Node 0 DMA32 per-cpu: >>>>>>>>>> : [ 454.006996] CPU 0: hi: 186, btch: 31 usd: 185 >>>>>>>>>> : [ 454.006998] CPU 1: hi: 186, btch: 31 usd: 112 >>>>>>>>>> : [ 454.007003] active_anon:8308 inactive_anon:8544 isolated_anon:0 >>>>>>>>>> : [ 454.007005] active_file:4882 inactive_file:205902 isolated_file:0 >>>>>>>>>> : [ 454.007006] unevictable:0 dirty:11 writeback:0 unstable:0 >>>>>>>>>> : [ 454.007007] free:1385 slab_reclaimable:2445 slab_unreclaimable:4466 >>>>>>>>>> : [ 454.007008] mapped:1895 shmem:113 pagetables:1370 bounce:0 >>>>>>>>>> : [ 454.007010] Node 0 DMA free:4000kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:11844kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15768kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:64kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no >>>>>>>>>> : [ 454.007021] lowmem_reserve[]: 0 994 994 994 >>>>>>>>>> : [ 454.007025] Node 0 DMA32 free:1540kB min:4000kB low:5000kB high:6000kB active_anon:33232kB inactive_anon:34176kB active_file:19528kB inactive_file:811764kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1018068kB mlocked:0kB dirty:44kB writeback:0kB mapped:7580kB shmem:452kB slab_reclaimable:9716kB slab_unreclaimable:17832kB kernel_stack:1144kB pagetables:5480kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no >>>>>>>>>> : [ 454.007036] lowmem_reserve[]: 0 0 0 0 >>>>>>>>>> : [ 454.007040] Node 0 DMA: 0*4kB 4*8kB 6*16kB 5*32kB 6*64kB 4*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 4000kB >>>>>>>>>> : [ 454.007050] Node 0 DMA32: 13*4kB 2*8kB 3*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1556kB >>>>>>>>>> : [ 454.007059] 210914 total pagecache pages >>>>>>>>>> : [ 454.007061] 0 pages in swap cache >>>>>>>>>> : [ 454.007063] Swap cache stats: add 0, delete 0, find 0/0 >>>>>>>>>> : [ 454.007065] Free swap = 1959924kB >>>>>>>>>> : [ 454.007067] Total swap = 1959924kB >>>>>>>>>> : [ 454.014238] 262140 pages RAM >>>>>>>>>> : [ 454.014241] 7489 pages reserved >>>>>>>>>> : [ 454.014242] 21430 pages shared >>>>>>>>>> : [ 454.014244] 247174 pages non-shared >>>>>>>>>> >>>>>>>>>> Either page reclaim got worse or kvm/virtio-net got more aggressive. >>>>>>>>>> >>>>>>>>>> Avi, Rusty: can you think of any changes in the KVM/virtio area in the >>>>>>>>>> 2.6.30 -> 2.6.32 timeframe which may have increased the GFP_ATOMIC >>>>>>>>>> demands upon the page allocator? >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> On the contrary, with commit >>>>>>>> 3161e453e496eb5643faad30fff5a5ab183da0fe >>>>>>>> we should be using GFP_ATOMIC less. >>>>>>>> But maybe there's a bug and it has the reverse effect somehow ... >>>>>>>> >>>>>>>> Robert, could you pls try 3161e453e496eb5643faad30fff5a5ab183da0fe >>>>>>>> and if that *does* have the problem, >>>>>>>> 0b4f2928f14c4a9770b0866923fc81beb7f4aa57? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxxx For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>