Re: [PATCH 00/49] Automatic NUMA Balancing v10

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Dec 07, 2012 at 12:01:13PM +0100, Ingo Molnar wrote:
> 
> * Mel Gorman <mgorman@xxxxxxx> wrote:
> 
> > This is a full release of all the patches so apologies for the 
> > flood. [...]
> 
> I have yet to process all your mails, but assuming I address all 
> your review feedback and the latest unified tree in tip:master 
> shows no regression in your testing, would you be willing to 
> start using it for ongoing work?
> 

Ingo,

If you had read the second paragraph of the mail you just responded to or
the results at the end then you would have seen that I had problems with
the performance. You would also know that tip/master testing for the last
week was failing due to a boot problem (issue was in mainline not tip and
has been already fixed) and would have known that since the -v18 release
that numacore was effectively disabled on my test machine.

Clearly you are not reading the bug reports you are receiving and you're not
seeing the small bit of review feedback or answering the review questions
you have received either. Why would I be more forthcoming when I feel that
it'll simply be ignored?  You simply assume that each batch of patches
you place on top must be fixing all known regressions and ignoring any
evidence to the contrary.

If you had read my mail from last Tuesday you would even know which patch
was causing the problem that effectively disabled numacore although not
why. The comment about p->numa_faults was completely off the mark (long
journey, was tired, assumed numa_faults was a counter and not a pointer
which was careless).  If you had called me on it then I would have spotted
the actual problem sooner. The problem was indeed with the nr_cpus_allowed
== num_online_cpus()s check which I had pointed out was a suspicious check
although for different reasons. As it turns out, a printk() bodge showed
that nr_cpus_allowed == 80 set in sched_init_smp() while num_online_cpus()
== 48. This effectively disabling numacore. If you had responded to the
bug report, this would likely have been found last Wednesday.

As for my ongoing work, I have not actually changed much in the last
two weeks or so -- build fixes and your scalability patches. As I've
said multiple times, my primary objective was to build something minimal
that did something better than mainline although not necessarily as good
as the kernel potentially if either numacore or autonuma were rebased
on top. I left the tree so that other testing might validate it was
correct and avoid changing the tree too much prior to the merge window.
I deliberately avoided working on anything that would directly collide
with what numacore was trying to achieve.

> It would make it much easier for me to pick up your 
> enhancements, fixes, etc.
> 
> > Changelog since V9
> >   o Migration scalability                                             (mingo)
> 
> To *really* see migration scalability bottlenecks you need to 
> remove the migration-bandwidth throttling kludge from your tree 
> (or configure it up very high if you want to do it simple).
> 

Why is it a kludge? I already explained what the rational behind the rate
limiting was. It's not about scalability, it's about mitigating worse-case
behaviour and the amount of time the kernel spends moving data around which
a deliberately adverse workload can trigger. It is unacceptable if during a
phase change that a process would stall potentially for milliseconds (seconds
if the node is large enough I guess) while the data is being migrated. Here
is it again -- http://www.spinics.net/lists/linux-mm/msg47440.html . You
either ignored the mail or simply could not be bothered explaining why
you thought this was the incorrect decision or why the concerns about an
adverse workload were unimportant.

I have a vague suspicion actually that when you are modelling the task->data
relationship that you make an implicit assumption that moving data has
zero or near-zero cost. In such a model it would always make sense to move
quickly and immediately but in practice the cost of moving can exceed the
performance benefit of accessing local data and lead to regressions. It
becomes more pronounced if the nodes are not fully connected.

> Some (certainly not all) of the performance regressions you 
> reported were certainly due to numa/core code hitting the 
> migration codepaths as aggressively as the workload demanded - 
> and hitting scalability bottlenecks.
> 

How are you so certain? How do you not know it's because your code is
migrating excessively for no good reason because the algorithm has a flaw
in it? Or that the cost of excessive migration is not being offset by
local data accesses? The critical point to note is that if it really was
only scalability problems then autonuma would suffer the same problems
and would be impossible to autonumas performance to exceed numacores.
This isn't the case making it unlikely the scalability is your only problem.

Either way, last night I applied a patch on top of latest tip/master to
remove the nr_cpus_allowed check so that numacore would be enabled again
and tested that. In some places it has indeed much improved. In others
it is still regressing badly and in two case, it's corrupting memory --
specjbb when THP is enabled crashes when running for single or multiple
JVMs. It is likely that a zero page is being inserted due to a race with
migration and causes the JVM to throw a null pointer exception. Here is
the comparison on the rough off-chance you actually read it this time.

stats-v8r6		Same collection of TLB flush fixes and stats
numacore-20121130	Roughly numacore v17
numafix-20121209	numacore as of December 9th with the nr_cpus_allowed check removed.
			Note that this is a 3.7-rc8 based test because that's what tip/master
			is.
autonuma-v28fastr4	Autonuma v28fast with THP patch on top
balancenuma-v9r2	Balance numa v9
balancenuma-v10r3	V9 + the migration scalability patches

AutoNUMA Benchmark
==================

                                      3.7.0-rc7             3.7.0-rc6             3.7.0-rc8             3.7.0-rc7             3.7.0-rc7             3.7.0-rc7
                                     stats-v8r6     numacore-20121130      numafix-20121209    autonuma-v28fastr4      balancenuma-v9r2     balancenuma-v10r3
User    NUMA01               65230.85 (  0.00%)    24835.22 ( 61.93%)    21882.80 ( 66.45%)    30410.22 ( 53.38%)    52436.65 ( 19.61%)    59949.95 (  8.10%)
User    NUMA01_THEADLOCAL    60794.67 (  0.00%)    17856.17 ( 70.63%)    18367.20 ( 69.79%)    17185.34 ( 71.73%)    17829.96 ( 70.67%)    17501.83 ( 71.21%)
User    NUMA02                7031.50 (  0.00%)     2084.38 ( 70.36%)     2391.47 ( 65.99%)     2238.73 ( 68.16%)     2079.48 ( 70.43%)     2094.68 ( 70.21%)
User    NUMA02_SMT            2916.19 (  0.00%)     1009.28 ( 65.39%)     1046.49 ( 64.11%)     1037.07 ( 64.44%)      997.57 ( 65.79%)     1010.15 ( 65.36%)
System  NUMA01                  39.66 (  0.00%)      926.55 (-2236.23%)      134.00 (-237.87%)      236.83 (-497.15%)      275.09 (-593.62%)      265.02 (-568.23%)
System  NUMA01_THEADLOCAL       42.33 (  0.00%)      513.99 (-1114.25%)      201.65 (-376.38%)       70.90 (-67.49%)      110.82 (-161.80%)      130.30 (-207.82%)
System  NUMA02                   1.25 (  0.00%)       18.57 (-1385.60%)       13.00 (-940.00%)        6.39 (-411.20%)        6.42 (-413.60%)        9.17 (-633.60%)
System  NUMA02_SMT              16.66 (  0.00%)       12.32 ( 26.05%)        7.26 ( 56.42%)        3.17 ( 80.97%)        3.58 ( 78.51%)        6.21 ( 62.73%)
Elapsed NUMA01                1511.76 (  0.00%)      575.93 ( 61.90%)      475.26 ( 68.56%)      701.62 ( 53.59%)     1185.53 ( 21.58%)     1352.74 ( 10.52%)
Elapsed NUMA01_THEADLOCAL     1387.17 (  0.00%)      398.55 ( 71.27%)      405.25 ( 70.79%)      378.47 ( 72.72%)      397.37 ( 71.35%)      387.93 ( 72.03%)
Elapsed NUMA02                 176.81 (  0.00%)       51.14 ( 71.08%)       62.08 ( 64.89%)       53.45 ( 69.77%)       49.51 ( 72.00%)       49.77 ( 71.85%)
Elapsed NUMA02_SMT             163.96 (  0.00%)       48.92 ( 70.16%)       54.45 ( 66.79%)       48.17 ( 70.62%)       47.71 ( 70.90%)       48.63 ( 70.34%)
CPU     NUMA01                4317.00 (  0.00%)     4473.00 ( -3.61%)     4632.00 ( -7.30%)     4368.00 ( -1.18%)     4446.00 ( -2.99%)     4451.00 ( -3.10%)
CPU     NUMA01_THEADLOCAL     4385.00 (  0.00%)     4609.00 ( -5.11%)     4582.00 ( -4.49%)     4559.00 ( -3.97%)     4514.00 ( -2.94%)     4545.00 ( -3.65%)
CPU     NUMA02                3977.00 (  0.00%)     4111.00 ( -3.37%)     3873.00 (  2.62%)     4200.00 ( -5.61%)     4212.00 ( -5.91%)     4226.00 ( -6.26%)
CPU     NUMA02_SMT            1788.00 (  0.00%)     2087.00 (-16.72%)     1935.00 ( -8.22%)     2159.00 (-20.75%)     2098.00 (-17.34%)     2089.00 (-16.83%)

Latest numacore has improved on the numa01 case quite a bit. However, this is
the adverse workload.  For the workloads that actually do something sensible,
autonuma and balancenuma are both beating numacore by a good margin.

numacores system CPU usage continues to be excessive -- over triple
balancenumas in the numa01 case. Over quadruple in the numa01_threadlocal
case. Double in numa02 and over double in the numa02_smt case.

Duration and vmstats showed nothing interesting so I excluded them this time.

SpecJBB, Multiple JVMs, THP is enabled
======================================

There is no latest numacore figures available because the JVM in two
separate tests crashed with this report

Input Properties:
  per_jvm_warehouse_rampup = 3.0
  per_jvm_warehouse_rampdown = 20.0
  jvm_instances = 4
  deterministic_random_seed = false
  ramp_up_seconds = 30
  measurement_seconds = 240
  starting_number_warehouses = 1
  increment_number_warehouses = 1
  ending_number_warehouses = 24
  expected_peak_warehouse = 12
Waiting on instance 1 pid 4028 to finish.
Accepted client /127.0.0.1:59130
Accepted client /127.0.0.1:58393
Accepted client /127.0.0.1:53374
Accepted client /127.0.0.1:40128
java.lang.NullPointerException: error
/root/git-private/autonuma-test/shellpacks/shellpack-bench-specjbb: line 203:  4028 Aborted                 java $USE_HUGEPAGE $SPECJBB_MAXHEAP spec.jbb.JBBmain -propfile SPECjbb.props -id $INSTANCE > $SHELLPACK_TEMP/jvm-instance-$INSTANCE.log
Waiting on instance 1 pid 4029 to finish.
Exception in thread "main" java.lang.NullPointerException
	at spec.jbb.Company.displayResultTotals(Unknown Source)
	at spec.jbb.JBBmain.DoARun(Unknown Source)
	at spec.jbb.JBBmain.runWarehouse(Unknown Source)
	at spec.jbb.JBBmain.doIt(Unknown Source)
	at spec.jbb.JBBmain.main(Unknown Source)
Exception in thread "main" java.lang.NullPointerException
	at spec.jbb.Company.displayResultTotals(Unknown Source)
	at spec.jbb.JBBmain.DoARun(Unknown Source)
	at spec.jbb.JBBmain.runWarehouse(Unknown Source)
	at spec.jbb.JBBmain.doIt(Unknown Source)
	at spec.jbb.JBBmain.main(Unknown Source)
Exception in thread "main" java.lang.NullPointerException
	at spec.jbb.Company.displayResultTotals(Unknown Source)
	at spec.jbb.JBBmain.DoARun(Unknown Source)
	at spec.jbb.JBBmain.runWarehouse(Unknown Source)
	at spec.jbb.JBBmain.doIt(Unknown Source)
	at spec.jbb.JBBmain.main(Unknown Source)
Read from remote host compass: Connection reset by peer

Here are the results for the kernels that succeeded

                      3.7.0-rc7             3.7.0-rc6             3.7.0-rc7             3.7.0-rc7             3.7.0-rc7
                     stats-v8r6     numacore-20121130    autonuma-v28fastr4      balancenuma-v9r2     balancenuma-v10r3
Mean   1      31311.75 (  0.00%)     27938.00 (-10.77%)     31474.25 (  0.52%)     31112.00 ( -0.64%)     31281.50 ( -0.10%)
Mean   2      62972.75 (  0.00%)     51899.00 (-17.58%)     66654.00 (  5.85%)     62937.50 ( -0.06%)     62483.50 ( -0.78%)
Mean   3      91292.00 (  0.00%)     80908.00 (-11.37%)     97177.50 (  6.45%)     90665.50 ( -0.69%)     90667.00 ( -0.68%)
Mean   4     115768.75 (  0.00%)     99497.25 (-14.06%)    125596.00 (  8.49%)    116812.50 (  0.90%)    116193.50 (  0.37%)
Mean   5     137248.50 (  0.00%)     92837.75 (-32.36%)    152795.25 ( 11.33%)    139037.75 (  1.30%)    139055.50 (  1.32%)
Mean   6     155528.50 (  0.00%)    105554.50 (-32.13%)    177455.25 ( 14.10%)    155769.25 (  0.15%)    159129.50 (  2.32%)
Mean   7     156747.50 (  0.00%)    122582.25 (-21.80%)    184578.75 ( 17.76%)    157103.25 (  0.23%)    163234.00 (  4.14%)
Mean   8     152069.50 (  0.00%)    122439.00 (-19.48%)    186619.25 ( 22.72%)    157631.00 (  3.66%)    163077.75 (  7.24%)
Mean   9     146609.75 (  0.00%)    112410.00 (-23.33%)    186165.00 ( 26.98%)    152561.00 (  4.06%)    159656.00 (  8.90%)
Mean   10    142819.00 (  0.00%)    111456.00 (-21.96%)    182569.75 ( 27.83%)    145320.00 (  1.75%)    153414.25 (  7.42%)
Mean   11    128292.25 (  0.00%)     98027.00 (-23.59%)    176104.75 ( 37.27%)    138599.50 (  8.03%)    147194.25 ( 14.73%)
Mean   12    128769.75 (  0.00%)    129469.50 (  0.54%)    169003.00 ( 31.24%)    131994.75 (  2.50%)    140049.75 (  8.76%)
Mean   13    126488.50 (  0.00%)    110133.75 (-12.93%)    162725.75 ( 28.65%)    130005.25 (  2.78%)    139109.75 (  9.98%)
Mean   14    123400.00 (  0.00%)    117929.75 ( -4.43%)    163781.25 ( 32.72%)    126340.75 (  2.38%)    137883.00 ( 11.74%)
Mean   15    122139.50 (  0.00%)    122404.25 (  0.22%)    160800.25 ( 31.65%)    128612.75 (  5.30%)    136624.00 ( 11.86%)
Mean   16    116413.50 (  0.00%)    124573.50 (  7.01%)    160882.75 ( 38.20%)    117793.75 (  1.19%)    134005.75 ( 15.11%)
Mean   17    117263.25 (  0.00%)    121937.25 (  3.99%)    159069.75 ( 35.65%)    121991.75 (  4.03%)    133444.50 ( 13.80%)
Mean   18    117277.00 (  0.00%)    116633.75 ( -0.55%)    158694.75 ( 35.32%)    119089.75 (  1.55%)    129650.75 ( 10.55%)
Mean   19    113231.00 (  0.00%)    111035.75 ( -1.94%)    155563.25 ( 37.39%)    119699.75 (  5.71%)    123403.25 (  8.98%)
Mean   20    113628.75 (  0.00%)    113451.25 ( -0.16%)    154779.75 ( 36.22%)    118400.75 (  4.20%)    126041.25 ( 10.92%)
Mean   21    110982.50 (  0.00%)    107660.50 ( -2.99%)    151147.25 ( 36.19%)    115663.25 (  4.22%)    121906.50 (  9.84%)
Mean   22    107660.25 (  0.00%)    104771.50 ( -2.68%)    151180.50 ( 40.42%)    111038.00 (  3.14%)    125519.00 ( 16.59%)
Mean   23    105320.50 (  0.00%)     88275.25 (-16.18%)    147032.00 ( 39.60%)    112817.50 (  7.12%)    124148.25 ( 17.88%)
Mean   24    110900.50 (  0.00%)     85169.00 (-23.20%)    147407.00 ( 32.92%)    109556.50 ( -1.21%)    122544.00 ( 10.50%)
Stddev 1        720.83 (  0.00%)       982.31 (-36.28%)       942.80 (-30.79%)      1170.23 (-62.35%)       539.84 ( 25.11%)
Stddev 2        466.00 (  0.00%)      1770.75 (-279.99%)      1327.32 (-184.83%)      1368.51 (-193.67%)      2103.32 (-351.35%)
Stddev 3        509.61 (  0.00%)      4849.62 (-851.63%)      1803.72 (-253.94%)      1088.04 (-113.50%)       410.73 ( 19.40%)
Stddev 4       1750.10 (  0.00%)     10708.16 (-511.86%)      2010.11 (-14.86%)      1456.90 ( 16.75%)      1370.22 ( 21.71%)
Stddev 5        700.05 (  0.00%)     16497.79 (-2256.66%)      2354.70 (-236.36%)       759.38 ( -8.48%)      1869.54 (-167.06%)
Stddev 6       2259.33 (  0.00%)     24221.98 (-972.09%)      1516.32 ( 32.89%)      1032.39 ( 54.31%)      1720.87 ( 23.83%)
Stddev 7       3390.99 (  0.00%)      4721.80 (-39.25%)      2398.34 ( 29.27%)      2487.08 ( 26.66%)      4327.85 (-27.63%)
Stddev 8       7533.18 (  0.00%)      8609.90 (-14.29%)      2895.55 ( 61.56%)      3902.53 ( 48.20%)      2536.68 ( 66.33%)
Stddev 9       9223.98 (  0.00%)     10731.70 (-16.35%)      4726.23 ( 48.76%)      5673.20 ( 38.50%)      3377.59 ( 63.38%)
Stddev 10      4578.09 (  0.00%)     11136.27 (-143.25%)      6705.48 (-46.47%)      5516.47 (-20.50%)      7227.58 (-57.87%)
Stddev 11      8201.30 (  0.00%)      3580.27 ( 56.35%)     10915.90 (-33.10%)      4757.42 ( 41.99%)      4056.02 ( 50.54%)
Stddev 12      5713.70 (  0.00%)     13923.12 (-143.68%)     16555.64 (-189.75%)      4573.05 ( 19.96%)      3678.89 ( 35.61%)
Stddev 13      5878.95 (  0.00%)     10471.09 (-78.11%)     18628.01 (-216.86%)      1680.65 ( 71.41%)      3947.39 ( 32.86%)
Stddev 14      4783.95 (  0.00%)      4051.35 ( 15.31%)     18324.63 (-283.04%)      2637.82 ( 44.86%)      4806.09 ( -0.46%)
Stddev 15      6281.48 (  0.00%)      3357.07 ( 46.56%)     17654.58 (-181.06%)      2003.38 ( 68.11%)      3005.22 ( 52.16%)
Stddev 16      6948.12 (  0.00%)      3763.32 ( 45.84%)     18280.52 (-163.10%)      3526.10 ( 49.25%)      3309.24 ( 52.37%)
Stddev 17      5603.77 (  0.00%)      1452.04 ( 74.09%)     18230.53 (-225.33%)      1712.95 ( 69.43%)      3516.09 ( 37.25%)
Stddev 18      6200.90 (  0.00%)      1870.12 ( 69.84%)     18486.73 (-198.13%)       751.36 ( 87.88%)      2412.60 ( 61.09%)
Stddev 19      6726.31 (  0.00%)      1045.21 ( 84.46%)     18465.25 (-174.52%)      1750.49 ( 73.98%)      4482.82 ( 33.35%)
Stddev 20      5713.58 (  0.00%)      2066.90 ( 63.82%)     19947.77 (-249.13%)      1892.91 ( 66.87%)      2612.62 ( 54.27%)
Stddev 21      4566.92 (  0.00%)      2460.40 ( 46.13%)     21189.08 (-363.97%)      3639.75 ( 20.30%)      1963.17 ( 57.01%)
Stddev 22      6168.05 (  0.00%)      2770.81 ( 55.08%)     20033.82 (-224.80%)      3682.20 ( 40.30%)      1159.17 ( 81.21%)
Stddev 23      6295.45 (  0.00%)      1337.32 ( 78.76%)     22610.91 (-259.16%)      2013.53 ( 68.02%)      3842.61 ( 38.96%)
Stddev 24      3108.17 (  0.00%)      1381.20 ( 55.56%)     21243.56 (-583.47%)      4044.16 (-30.11%)      2673.39 ( 13.99%)
TPut   1     125247.00 (  0.00%)    111752.00 (-10.77%)    125897.00 (  0.52%)    124448.00 ( -0.64%)    125126.00 ( -0.10%)
TPut   2     251891.00 (  0.00%)    207596.00 (-17.58%)    266616.00 (  5.85%)    251750.00 ( -0.06%)    249934.00 ( -0.78%)
TPut   3     365168.00 (  0.00%)    323632.00 (-11.37%)    388710.00 (  6.45%)    362662.00 ( -0.69%)    362668.00 ( -0.68%)
TPut   4     463075.00 (  0.00%)    397989.00 (-14.06%)    502384.00 (  8.49%)    467250.00 (  0.90%)    464774.00 (  0.37%)
TPut   5     548994.00 (  0.00%)    371351.00 (-32.36%)    611181.00 ( 11.33%)    556151.00 (  1.30%)    556222.00 (  1.32%)
TPut   6     622114.00 (  0.00%)    422218.00 (-32.13%)    709821.00 ( 14.10%)    623077.00 (  0.15%)    636518.00 (  2.32%)
TPut   7     626990.00 (  0.00%)    490329.00 (-21.80%)    738315.00 ( 17.76%)    628413.00 (  0.23%)    652936.00 (  4.14%)
TPut   8     608278.00 (  0.00%)    489756.00 (-19.48%)    746477.00 ( 22.72%)    630524.00 (  3.66%)    652311.00 (  7.24%)
TPut   9     586439.00 (  0.00%)    449640.00 (-23.33%)    744660.00 ( 26.98%)    610244.00 (  4.06%)    638624.00 (  8.90%)
TPut   10    571276.00 (  0.00%)    445824.00 (-21.96%)    730279.00 ( 27.83%)    581280.00 (  1.75%)    613657.00 (  7.42%)
TPut   11    513169.00 (  0.00%)    392108.00 (-23.59%)    704419.00 ( 37.27%)    554398.00 (  8.03%)    588777.00 ( 14.73%)
TPut   12    515079.00 (  0.00%)    517878.00 (  0.54%)    676012.00 ( 31.24%)    527979.00 (  2.50%)    560199.00 (  8.76%)
TPut   13    505954.00 (  0.00%)    440535.00 (-12.93%)    650903.00 ( 28.65%)    520021.00 (  2.78%)    556439.00 (  9.98%)
TPut   14    493600.00 (  0.00%)    471719.00 ( -4.43%)    655125.00 ( 32.72%)    505363.00 (  2.38%)    551532.00 ( 11.74%)
TPut   15    488558.00 (  0.00%)    489617.00 (  0.22%)    643201.00 ( 31.65%)    514451.00 (  5.30%)    546496.00 ( 11.86%)
TPut   16    465654.00 (  0.00%)    498294.00 (  7.01%)    643531.00 ( 38.20%)    471175.00 (  1.19%)    536023.00 ( 15.11%)
TPut   17    469053.00 (  0.00%)    487749.00 (  3.99%)    636279.00 ( 35.65%)    487967.00 (  4.03%)    533778.00 ( 13.80%)
TPut   18    469108.00 (  0.00%)    466535.00 ( -0.55%)    634779.00 ( 35.32%)    476359.00 (  1.55%)    518603.00 ( 10.55%)
TPut   19    452924.00 (  0.00%)    444143.00 ( -1.94%)    622253.00 ( 37.39%)    478799.00 (  5.71%)    493613.00 (  8.98%)
TPut   20    454515.00 (  0.00%)    453805.00 ( -0.16%)    619119.00 ( 36.22%)    473603.00 (  4.20%)    504165.00 ( 10.92%)
TPut   21    443930.00 (  0.00%)    430642.00 ( -2.99%)    604589.00 ( 36.19%)    462653.00 (  4.22%)    487626.00 (  9.84%)
TPut   22    430641.00 (  0.00%)    419086.00 ( -2.68%)    604722.00 ( 40.42%)    444152.00 (  3.14%)    502076.00 ( 16.59%)
TPut   23    421282.00 (  0.00%)    353101.00 (-16.18%)    588128.00 ( 39.60%)    451270.00 (  7.12%)    496593.00 ( 17.88%)
TPut   24    443602.00 (  0.00%)    340676.00 (-23.20%)    589628.00 ( 32.92%)    438226.00 ( -1.21%)    490176.00 ( 10.50%)

numacore v17 regressed but we knew that already.

autonuma does the best overall

balancenuma does all right and the scalability patches help quite a bit.

SPECJBB PEAKS
                                   3.7.0-rc7                  3.7.0-rc6                  3.7.0-rc7                  3.7.0-rc7                  3.7.0-rc7
                                  stats-v8r6          numacore-20121130         autonuma-v28fastr4           balancenuma-v9r2          balancenuma-v10r3
 Expctd Warehouse            12.00 (  0.00%)            12.00 (  0.00%)            12.00 (  0.00%)            12.00 (  0.00%)            12.00 (  0.00%)
 Expctd Peak Bops        515079.00 (  0.00%)        517878.00 (  0.54%)        676012.00 ( 31.24%)        527979.00 (  2.50%)        560199.00 (  8.76%)
 Actual Warehouse             7.00 (  0.00%)            12.00 ( 71.43%)             8.00 ( 14.29%)             8.00 ( 14.29%)             7.00 (  0.00%)
 Actual Peak Bops        626990.00 (  0.00%)        517878.00 (-17.40%)        746477.00 ( 19.06%)        630524.00 (  0.56%)        652936.00 (  4.14%)
 SpecJBB Bops            465685.00 (  0.00%)        447214.00 ( -3.97%)        628328.00 ( 34.93%)        480925.00 (  3.27%)        521332.00 ( 11.95%)
 SpecJBB Bops/JVM        116421.00 (  0.00%)        111804.00 ( -3.97%)        157082.00 ( 34.93%)        120231.00 (  3.27%)        130333.00 ( 11.95%)

numacore is pretty old here so ignore the regression.

autonuma is the best but balancenuma sees some of the performance gain.

MMTests Statistics: vmstat
                             3.7.0-rc7   3.7.0-rc6   3.7.0-rc7   3.7.0-rc7   3.7.0-rc7
                            stats-v8r6numacore-20121130autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
Page Ins                         37116       36404       36740       35664       34832
Page Outs                        30340       33624       29428       29656       30320
Swap Ins                             0           0           0           0           0
Swap Outs                            0           0           0           0           0
Direct pages scanned                 0           0           0           0           0
Kswapd pages scanned                 0           0           0           0           0
Kswapd pages reclaimed               0           0           0           0           0
Direct pages reclaimed               0           0           0           0           0
Kswapd efficiency                 100%        100%        100%        100%        100%
Kswapd velocity                  0.000       0.000       0.000       0.000       0.000
Direct efficiency                 100%        100%        100%        100%        100%
Direct velocity                  0.000       0.000       0.000       0.000       0.000
Percentage direct scans             0%          0%          0%          0%          0%
Page writes by reclaim               0           0           0           0           0
Page writes file                     0           0           0           0           0
Page writes anon                     0           0           0           0           0
Page reclaim immediate               0           0           0           0           0
Page rescued immediate               0           0           0           0           0
Slabs scanned                        0           0           0           0           0
Direct inode steals                  0           0           0           0           0
Kswapd inode steals                  0           0           0           0           0
Kswapd skipped wait                  0           0           0           0           0
THP fault alloc                  63322       49889       52514       65794       66963
THP collapse alloc                 130          53         463         128         121
THP splits                         355         192         376         371         362
THP fault fallback                   0           0           0           0           0
THP collapse fail                    0           0           0           0           0
Compaction stalls                    0           0           0           0           0
Compaction success                   0           0           0           0           0
Compaction failures                  0           0           0           0           0
Page migrate success                 0           0           0    51424061    50195011
Page migrate failure                 0           0           0           0           0
Compaction pages isolated            0           0           0           0           0
Compaction migrate scanned           0           0           0           0           0
Compaction free scanned              0           0           0           0           0
Compaction cost                      0           0           0       53378       52102
NUMA PTE updates                     0           0           0   411047238   404964644
NUMA hint faults                     0           0           0     3077302     3075026
NUMA hint local faults               0           0           0      958617      870171
NUMA pages migrated                  0           0           0    51424061    50195011
AutoNUMA cost                        0           0           0       19240       19163

All it shows really is that THP is enabled and that balancenuma is migrating
more than I'd like -- 48MB/sec on average throughout the test.

SpecJBB, Multiple JVMs, THP is disabled
=======================================
                      3.7.0-rc7             3.7.0-rc6             3.7.0-rc8             3.7.0-rc7             3.7.0-rc7             3.7.0-rc7
                     stats-v8r6     numacore-20121130      numafix-20121209    autonuma-v28fastr4      balancenuma-v9r2     balancenuma-v10r3
Mean   1      26036.50 (  0.00%)     19595.00 (-24.74%)     23791.25 ( -8.62%)     24738.25 ( -4.99%)     25595.00 ( -1.70%)     25610.50 ( -1.64%)
Mean   2      53629.75 (  0.00%)     38481.50 (-28.25%)     46966.75 (-12.42%)     55646.75 (  3.76%)     53045.25 ( -1.09%)     53383.00 ( -0.46%)
Mean   3      77385.00 (  0.00%)     53685.50 (-30.63%)     66913.25 (-13.53%)     82714.75 (  6.89%)     76596.00 ( -1.02%)     76502.75 ( -1.14%)
Mean   4     100097.75 (  0.00%)     68253.50 (-31.81%)     72186.50 (-27.88%)    107883.25 (  7.78%)     98618.00 ( -1.48%)     99786.50 ( -0.31%)
Mean   5     119012.75 (  0.00%)     74164.50 (-37.68%)     72126.50 (-39.40%)    130260.25 (  9.45%)    119354.50 (  0.29%)    121741.75 (  2.29%)
Mean   6     137419.25 (  0.00%)     86158.50 (-37.30%)     52123.00 (-62.07%)    154244.50 ( 12.24%)    136901.75 ( -0.38%)    136990.50 ( -0.31%)
Mean   7     138018.25 (  0.00%)     96059.25 (-30.40%)     55582.50 (-59.73%)    159501.00 ( 15.57%)    138265.50 (  0.18%)    139398.75 (  1.00%)
Mean   8     136774.00 (  0.00%)     97003.50 (-29.08%)     30208.25 (-77.91%)    162868.00 ( 19.08%)    138554.50 (  1.30%)    137340.75 (  0.41%)
Mean   9     127966.50 (  0.00%)     95261.00 (-25.56%)    125900.50 ( -1.61%)    163008.00 ( 27.38%)    137954.00 (  7.80%)    134200.50 (  4.87%)
Mean   10    124628.75 (  0.00%)     96202.25 (-22.81%)     73809.00 (-40.78%)    159696.50 ( 28.14%)    131322.25 (  5.37%)    126927.50 (  1.84%)
Mean   11    117269.00 (  0.00%)     95924.25 (-18.20%)    127804.25 (  8.98%)    154701.50 ( 31.92%)    125032.75 (  6.62%)    122925.00 (  4.82%)
Mean   12    111962.25 (  0.00%)     94247.25 (-15.82%)    146580.25 ( 30.92%)    150936.50 ( 34.81%)    118119.50 (  5.50%)    119931.75 (  7.12%)
Mean   13    111595.50 (  0.00%)    106538.50 ( -4.53%)    134462.75 ( 20.49%)    147193.25 ( 31.90%)    116398.75 (  4.30%)    117349.75 (  5.16%)
Mean   14    110881.00 (  0.00%)    103549.00 ( -6.61%)    137573.25 ( 24.07%)    144584.00 ( 30.40%)    114934.50 (  3.66%)    115838.25 (  4.47%)
Mean   15    109337.50 (  0.00%)    101729.00 ( -6.96%)    139722.50 ( 27.79%)    143333.00 ( 31.09%)    115523.75 (  5.66%)    115151.25 (  5.32%)
Mean   16    107031.75 (  0.00%)    101983.75 ( -4.72%)    121221.75 ( 13.26%)    141907.75 ( 32.58%)    113666.00 (  6.20%)    113673.50 (  6.21%)
Mean   17    105491.25 (  0.00%)    100205.75 ( -5.01%)    129429.75 ( 22.69%)    140691.00 ( 33.37%)    112751.50 (  6.88%)    113221.25 (  7.33%)
Mean   18    101102.75 (  0.00%)     96635.50 ( -4.42%)    115086.50 ( 13.83%)    137784.25 ( 36.28%)    112582.50 ( 11.35%)    111533.50 ( 10.32%)
Mean   19    103907.25 (  0.00%)     94578.25 ( -8.98%)    126392.75 ( 21.64%)    135719.25 ( 30.62%)    110152.25 (  6.01%)    113959.25 (  9.67%)
Mean   20    100496.00 (  0.00%)     92683.75 ( -7.77%)    123318.75 ( 22.71%)    135264.25 ( 34.60%)    108861.50 (  8.32%)    113746.00 ( 13.18%)
Mean   21     99570.00 (  0.00%)     92955.75 ( -6.64%)    111293.00 ( 11.77%)    133891.00 ( 34.47%)    110094.00 ( 10.57%)    109462.50 (  9.94%)
Mean   22     98611.75 (  0.00%)     89781.75 ( -8.95%)    118218.50 ( 19.88%)    132399.75 ( 34.26%)    109322.75 ( 10.86%)    110502.75 ( 12.06%)
Mean   23     98173.00 (  0.00%)     88846.00 ( -9.50%)    118210.00 ( 20.41%)    130726.00 ( 33.16%)    106046.25 (  8.02%)    107304.25 (  9.30%)
Mean   24     92074.75 (  0.00%)     88581.00 ( -3.79%)    111965.00 ( 21.60%)    127552.25 ( 38.53%)    102362.00 ( 11.17%)    107119.25 ( 16.34%)
Stddev 1        735.13 (  0.00%)       538.24 ( 26.78%)       854.37 (-16.22%)       121.08 ( 83.53%)       906.62 (-23.33%)       788.06 ( -7.20%)
Stddev 2        406.26 (  0.00%)      3458.87 (-751.39%)      4220.03 (-938.75%)       477.32 (-17.49%)      1322.57 (-225.55%)       468.57 (-15.34%)
Stddev 3        644.20 (  0.00%)      1360.89 (-111.25%)      2573.27 (-299.45%)       922.47 (-43.20%)       609.27 (  5.42%)       599.26 (  6.98%)
Stddev 4        743.93 (  0.00%)      2149.34 (-188.92%)     14533.01 (-1853.53%)      1385.42 (-86.23%)      1119.02 (-50.42%)       801.13 ( -7.69%)
Stddev 5        898.53 (  0.00%)      2521.01 (-180.57%)     15303.97 (-1603.23%)       763.24 ( 15.06%)       942.52 ( -4.90%)      1718.19 (-91.22%)
Stddev 6       1126.61 (  0.00%)      3818.22 (-238.91%)     23616.59 (-1996.26%)      1527.03 (-35.54%)      2445.69 (-117.08%)      1754.32 (-55.72%)
Stddev 7       2907.61 (  0.00%)      4419.29 (-51.99%)     29664.97 (-920.25%)      1536.66 ( 47.15%)      4881.65 (-67.89%)      4863.83 (-67.28%)
Stddev 8       3200.64 (  0.00%)       382.01 ( 88.06%)     10743.99 (-235.68%)      1228.09 ( 61.63%)      5459.06 (-70.56%)      5583.95 (-74.46%)
Stddev 9       2907.92 (  0.00%)      1813.39 ( 37.64%)     11763.90 (-304.55%)      1502.61 ( 48.33%)      2501.16 ( 13.99%)      2525.02 ( 13.17%)
Stddev 10      5093.23 (  0.00%)      1313.58 ( 74.21%)     34926.95 (-585.75%)      2763.19 ( 45.75%)      2973.78 ( 41.61%)      2005.95 ( 60.62%)
Stddev 11      4982.41 (  0.00%)      1163.02 ( 76.66%)     13792.07 (-176.81%)      4776.28 (  4.14%)      6068.34 (-21.80%)      4256.77 ( 14.56%)
Stddev 12      3051.38 (  0.00%)      2117.59 ( 30.60%)      5819.48 (-90.72%)      9252.59 (-203.23%)      3885.96 (-27.35%)      2580.44 ( 15.43%)
Stddev 13      2918.03 (  0.00%)      2252.11 ( 22.82%)      8340.05 (-185.81%)      9384.83 (-221.62%)      1833.07 ( 37.18%)      2523.28 ( 13.53%)
Stddev 14      3178.97 (  0.00%)      2337.49 ( 26.47%)      6166.98 (-93.99%)      9353.03 (-194.22%)      1072.60 ( 66.26%)      1140.55 ( 64.12%)
Stddev 15      2438.31 (  0.00%)      1707.72 ( 29.96%)     10687.74 (-338.33%)     10494.03 (-330.38%)      2295.76 (  5.85%)      1213.75 ( 50.22%)
Stddev 16      2682.25 (  0.00%)       840.47 ( 68.67%)     10963.32 (-308.74%)     10343.25 (-285.62%)      2416.09 (  9.92%)      1697.27 ( 36.72%)
Stddev 17      2807.66 (  0.00%)      1546.16 ( 44.93%)     10755.81 (-283.09%)     11446.15 (-307.68%)      2484.08 ( 11.52%)       563.50 ( 79.93%)
Stddev 18      3049.27 (  0.00%)       934.11 ( 69.37%)      8523.80 (-179.54%)     11779.80 (-286.31%)      1472.27 ( 51.72%)      1533.68 ( 49.70%)
Stddev 19      2782.65 (  0.00%)       735.28 ( 73.58%)      9045.84 (-225.08%)     11416.35 (-310.27%)       514.78 ( 81.50%)      1283.38 ( 53.88%)
Stddev 20      2379.12 (  0.00%)       956.25 ( 59.81%)      3789.62 (-59.29%)     10511.63 (-341.83%)      1641.25 ( 31.01%)      1758.22 ( 26.10%)
Stddev 21      2975.22 (  0.00%)       438.31 ( 85.27%)      8160.39 (-174.28%)     11292.91 (-279.57%)      1087.60 ( 63.44%)       434.51 ( 85.40%)
Stddev 22      2260.61 (  0.00%)       718.23 ( 68.23%)     10418.90 (-360.89%)     11993.84 (-430.56%)       909.16 ( 59.78%)       322.32 ( 85.74%)
Stddev 23      2900.85 (  0.00%)       275.47 ( 90.50%)      9829.57 (-238.85%)     12234.80 (-321.77%)       701.39 ( 75.82%)      1444.19 ( 50.21%)
Stddev 24      2578.98 (  0.00%)       481.68 ( 81.32%)      7696.37 (-198.43%)     12769.61 (-395.14%)       732.56 ( 71.60%)      1777.60 ( 31.07%)
TPut   1     104146.00 (  0.00%)     78380.00 (-24.74%)     95165.00 ( -8.62%)     98953.00 ( -4.99%)    102380.00 ( -1.70%)    102442.00 ( -1.64%)
TPut   2     214519.00 (  0.00%)    153926.00 (-28.25%)    187867.00 (-12.42%)    222587.00 (  3.76%)    212181.00 ( -1.09%)    213532.00 ( -0.46%)
TPut   3     309540.00 (  0.00%)    214742.00 (-30.63%)    267653.00 (-13.53%)    330859.00 (  6.89%)    306384.00 ( -1.02%)    306011.00 ( -1.14%)
TPut   4     400391.00 (  0.00%)    273014.00 (-31.81%)    288746.00 (-27.88%)    431533.00 (  7.78%)    394472.00 ( -1.48%)    399146.00 ( -0.31%)
TPut   5     476051.00 (  0.00%)    296658.00 (-37.68%)    288506.00 (-39.40%)    521041.00 (  9.45%)    477418.00 (  0.29%)    486967.00 (  2.29%)
TPut   6     549677.00 (  0.00%)    344634.00 (-37.30%)    208492.00 (-62.07%)    616978.00 ( 12.24%)    547607.00 ( -0.38%)    547962.00 ( -0.31%)
TPut   7     552073.00 (  0.00%)    384237.00 (-30.40%)    222330.00 (-59.73%)    638004.00 ( 15.57%)    553062.00 (  0.18%)    557595.00 (  1.00%)
TPut   8     547096.00 (  0.00%)    388014.00 (-29.08%)    120833.00 (-77.91%)    651472.00 ( 19.08%)    554218.00 (  1.30%)    549363.00 (  0.41%)
TPut   9     511866.00 (  0.00%)    381044.00 (-25.56%)    503602.00 ( -1.61%)    652032.00 ( 27.38%)    551816.00 (  7.80%)    536802.00 (  4.87%)
TPut   10    498515.00 (  0.00%)    384809.00 (-22.81%)    295236.00 (-40.78%)    638786.00 ( 28.14%)    525289.00 (  5.37%)    507710.00 (  1.84%)
TPut   11    469076.00 (  0.00%)    383697.00 (-18.20%)    511217.00 (  8.98%)    618806.00 ( 31.92%)    500131.00 (  6.62%)    491700.00 (  4.82%)
TPut   12    447849.00 (  0.00%)    376989.00 (-15.82%)    586321.00 ( 30.92%)    603746.00 ( 34.81%)    472478.00 (  5.50%)    479727.00 (  7.12%)
TPut   13    446382.00 (  0.00%)    426154.00 ( -4.53%)    537851.00 ( 20.49%)    588773.00 ( 31.90%)    465595.00 (  4.30%)    469399.00 (  5.16%)
TPut   14    443524.00 (  0.00%)    414196.00 ( -6.61%)    550293.00 ( 24.07%)    578336.00 ( 30.40%)    459738.00 (  3.66%)    463353.00 (  4.47%)
TPut   15    437350.00 (  0.00%)    406916.00 ( -6.96%)    558890.00 ( 27.79%)    573332.00 ( 31.09%)    462095.00 (  5.66%)    460605.00 (  5.32%)
TPut   16    428127.00 (  0.00%)    407935.00 ( -4.72%)    484887.00 ( 13.26%)    567631.00 ( 32.58%)    454664.00 (  6.20%)    454694.00 (  6.21%)
TPut   17    421965.00 (  0.00%)    400823.00 ( -5.01%)    517719.00 ( 22.69%)    562764.00 ( 33.37%)    451006.00 (  6.88%)    452885.00 (  7.33%)
TPut   18    404411.00 (  0.00%)    386542.00 ( -4.42%)    460346.00 ( 13.83%)    551137.00 ( 36.28%)    450330.00 ( 11.35%)    446134.00 ( 10.32%)
TPut   19    415629.00 (  0.00%)    378313.00 ( -8.98%)    505571.00 ( 21.64%)    542877.00 ( 30.62%)    440609.00 (  6.01%)    455837.00 (  9.67%)
TPut   20    401984.00 (  0.00%)    370735.00 ( -7.77%)    493275.00 ( 22.71%)    541057.00 ( 34.60%)    435446.00 (  8.32%)    454984.00 ( 13.18%)
TPut   21    398280.00 (  0.00%)    371823.00 ( -6.64%)    445172.00 ( 11.77%)    535564.00 ( 34.47%)    440376.00 ( 10.57%)    437850.00 (  9.94%)
TPut   22    394447.00 (  0.00%)    359127.00 ( -8.95%)    472874.00 ( 19.88%)    529599.00 ( 34.26%)    437291.00 ( 10.86%)    442011.00 ( 12.06%)
TPut   23    392692.00 (  0.00%)    355384.00 ( -9.50%)    472840.00 ( 20.41%)    522904.00 ( 33.16%)    424185.00 (  8.02%)    429217.00 (  9.30%)
TPut   24    368299.00 (  0.00%)    354324.00 ( -3.79%)    447860.00 ( 21.60%)    510209.00 ( 38.53%)    409448.00 ( 11.17%)    428477.00 ( 16.34%)

Latest numacore has improved dramatically here. In v17, it was regressing
heavily across the board. The latest figures show that it regresses heavily
for small numbers of warehouses and shows very large performance gains
for larger numbers of warehouses. This problem with regressions for smaller
numbers of warehouses has been reported repeatedly and it has been pointed out
multiple times that specjbb by default ignores these results which can be
very misleading.

autonuma shows large gains even for small numbers of warehouses and larger
performnace gains than numacore does. This is without the TLB optimisations.

balancenuma is not great, but it's better than mainline.


SPECJBB PEAKS
                                   3.7.0-rc7                  3.7.0-rc6                  3.7.0-rc8                  3.7.0-rc7                  3.7.0-rc7                  3.7.0-rc7
                                  stats-v8r6          numacore-20121130           numafix-20121209         autonuma-v28fastr4           balancenuma-v9r2          balancenuma-v10r3
 Expctd Warehouse            12.00 (  0.00%)            12.00 (  0.00%)            12.00 (  0.00%)            12.00 (  0.00%)            12.00 (  0.00%)            12.00 (  0.00%)
 Expctd Peak Bops        447849.00 (  0.00%)        376989.00 (-15.82%)        586321.00 ( 30.92%)        603746.00 ( 34.81%)        472478.00 (  5.50%)        479727.00 (  7.12%)
 Actual Warehouse             7.00 (  0.00%)            13.00 ( 85.71%)            12.00 ( 71.43%)             9.00 ( 28.57%)             8.00 ( 14.29%)             7.00 (  0.00%)
 Actual Peak Bops        552073.00 (  0.00%)        426154.00 (-22.81%)        586321.00 (  6.20%)        652032.00 ( 18.11%)        554218.00 (  0.39%)        557595.00 (  1.00%)
 SpecJBB Bops            415458.00 (  0.00%)        385328.00 ( -7.25%)        502608.00 ( 20.98%)        554456.00 ( 33.46%)        446405.00 (  7.45%)        451937.00 (  8.78%)
 SpecJBB Bops/JVM        103865.00 (  0.00%)         96332.00 ( -7.25%)        125652.00 ( 20.98%)        138614.00 ( 33.46%)        111601.00 (  7.45%)        112984.00 (  8.78%)

numacore is showing good performance gains both at the peak and in the
specjbb score. Note that the specjbb score ignored the regressions for
smaller numbers of warehouses.

autonuma was still better.

balancenuma was all right, better than mainline.

MMTests Statistics: duration
           3.7.0-rc7   3.7.0-rc6   3.7.0-rc8   3.7.0-rc7   3.7.0-rc7   3.7.0-rc7
          stats-v8r6numacore-20121130numafix-20121209autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
User       177832.71   148340.09   165197.46   177337.90   176411.93   176466.36
System         89.07    28052.02    12438.18      287.31     1464.93     1467.74
Elapsed      4035.81     4041.26     4038.34     4028.05     4041.53     4031.74

numacores system CPU usage is incredibly high -- over 8 times higher
than balancenumas.

balancenumas system CPU usage also sucks to be honest.

MMTests Statistics: vmstat
                             3.7.0-rc7   3.7.0-rc6   3.7.0-rc8   3.7.0-rc7   3.7.0-rc7   3.7.0-rc7
                            stats-v8r6numacore-20121130numafix-20121209autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
Page Ins                         37380       66040       34576       36416       35452       34948
Page Outs                        29224       46900       31972       29584       29612       30892
Swap Ins                             0           0           0           0           0           0
Swap Outs                            0           0           0           0           0           0
Direct pages scanned                 0           0           0           0           0           0
Kswapd pages scanned                 0           0           0           0           0           0
Kswapd pages reclaimed               0           0           0           0           0           0
Direct pages reclaimed               0           0           0           0           0           0
Kswapd efficiency                 100%        100%        100%        100%        100%        100%
Kswapd velocity                  0.000       0.000       0.000       0.000       0.000       0.000
Direct efficiency                 100%        100%        100%        100%        100%        100%
Direct velocity                  0.000       0.000       0.000       0.000       0.000       0.000
Percentage direct scans             0%          0%          0%          0%          0%          0%
Page writes by reclaim               0           0           0           0           0           0
Page writes file                     0           0           0           0           0           0
Page writes anon                     0           0           0           0           0           0
Page reclaim immediate               0           0           0           0           0           0
Page rescued immediate               0           0           0           0           0           0
Slabs scanned                        0           0           0           0           0           0
Direct inode steals                  0           0           0           0           0           0
Kswapd inode steals                  0           0           0           0           0           0
Kswapd skipped wait                  0           0           0           0           0           0
THP fault alloc                      2           3           1           2           2           2
THP collapse alloc                   0           0           0           0           0           0
THP splits                           0           0           0           0           0           0
THP fault fallback                   0           0           0           0           0           0
THP collapse fail                    0           0           0           0           0           0
Compaction stalls                    0           0           0           0           0           0
Compaction success                   0           0           0           0           0           0
Compaction failures                  0           0           0           0           0           0
Page migrate success                 0           0   193988041           0    37611432    39796961
Page migrate failure                 0           0           0           0           0           0
Compaction pages isolated            0           0           0           0           0           0
Compaction migrate scanned           0           0           0           0           0           0
Compaction free scanned              0           0           0           0           0           0
Compaction cost                      0           0      201359           0       39040       41309
NUMA PTE updates                     0           0   904384590           0   288455303   286931926
NUMA hint faults                     0           0           0           0   270103189   269176121
NUMA hint local faults               0           0           0           0    70822016    70400386
NUMA pages migrated                  0           0   193988041           0    37611432    39796961
AutoNUMA cost                        0           0       10016           0     1353249     1348645

According to this, numacore never had a NUMA fault. This is completely broken
obviously and it's because PTE NUMA hinting faults are not accounted for
by numacore because that path does not call numa_migration_target(). The
consequences are not that great, it just means that the notional "AutoNUMA
cost" is meaningless for numacore.

What is interesting is numacores migration rate -- 187MB/sec on average. This
is over quadruple balancenumas migration rate of 38MB/sec on average.

SpecJBB, Single JVM, THP is enabled
===================================

As with the Multiple JVM test with THP enabled, numacore crashes. This
time the message is

Timing Measurement began Sun Dec 09 17:12:53 GMT 2012 for 0.5 minutes
Exception in thread "Thread-1040" java.lang.NullPointerException
        at java.util.TreeMap.access$100(Unknown Source)
        at java.util.TreeMap$PrivateEntryIterator.nextEntry(Unknown Source)
        at java.util.TreeMap$ValueIterator.next(Unknown Source)
        at spec.jbb.DeliveryTransaction.preprocess(Unknown Source)
        at spec.jbb.DeliveryHandler.handleDelivery(Unknown Source)
        at spec.jbb.DeliveryTransaction.process(Unknown Source)
        at spec.jbb.TransactionManager.runTxn(Unknown Source)
        at spec.jbb.TransactionManager.goManual(Unknown Source)
        at spec.jbb.TransactionManager.go(Unknown Source)
        at spec.jbb.JBBmain.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Timing Measurement ended Sun Dec 09 17:13:23 GMT 2012

Here are the rest of the resutls

                    3.7.0-rc7             3.7.0-rc6             3.7.0-rc7             3.7.0-rc7             3.7.0-rc7
                   stats-v8r6     numacore-20121130    autonuma-v28fastr4      balancenuma-v9r2     balancenuma-v10r3
TPut 1      25550.00 (  0.00%)     25491.00 ( -0.23%)     24233.00 ( -5.15%)     24913.00 ( -2.49%)     26480.00 (  3.64%)
TPut 2      55943.00 (  0.00%)     51630.00 ( -7.71%)     55312.00 ( -1.13%)     55042.00 ( -1.61%)     56920.00 (  1.75%)
TPut 3      87707.00 (  0.00%)     74497.00 (-15.06%)     88569.00 (  0.98%)     86135.00 ( -1.79%)     88608.00 (  1.03%)
TPut 4     117911.00 (  0.00%)     98435.00 (-16.52%)    118561.00 (  0.55%)    117486.00 ( -0.36%)    117953.00 (  0.04%)
TPut 5     143285.00 (  0.00%)    133964.00 ( -6.51%)    145703.00 (  1.69%)    142821.00 ( -0.32%)    144926.00 (  1.15%)
TPut 6     171208.00 (  0.00%)    152795.00 (-10.75%)    171006.00 ( -0.12%)    170635.00 ( -0.33%)    169394.00 ( -1.06%)
TPut 7     195635.00 (  0.00%)    162517.00 (-16.93%)    198699.00 (  1.57%)    196108.00 (  0.24%)    196491.00 (  0.44%)
TPut 8     222655.00 (  0.00%)    168679.00 (-24.24%)    224903.00 (  1.01%)    223494.00 (  0.38%)    225978.00 (  1.49%)
TPut 9     244787.00 (  0.00%)    193394.00 (-20.99%)    248313.00 (  1.44%)    251858.00 (  2.89%)    251569.00 (  2.77%)
TPut 10    271565.00 (  0.00%)    237987.00 (-12.36%)    272148.00 (  0.21%)    275869.00 (  1.58%)    279049.00 (  2.76%)
TPut 11    298270.00 (  0.00%)    207908.00 (-30.30%)    303749.00 (  1.84%)    301763.00 (  1.17%)    301399.00 (  1.05%)
TPut 12    320867.00 (  0.00%)    257937.00 (-19.61%)    327808.00 (  2.16%)    329681.00 (  2.75%)    330506.00 (  3.00%)
TPut 13    343514.00 (  0.00%)    248474.00 (-27.67%)    349080.00 (  1.62%)    340606.00 ( -0.85%)    350817.00 (  2.13%)
TPut 14    365321.00 (  0.00%)    298876.00 (-18.19%)    370026.00 (  1.29%)    379939.00 (  4.00%)    361752.00 ( -0.98%)
TPut 15    377071.00 (  0.00%)    296562.00 (-21.35%)    329847.00 (-12.52%)    395421.00 (  4.87%)    396091.00 (  5.04%)
TPut 16    404979.00 (  0.00%)    287964.00 (-28.89%)    411066.00 (  1.50%)    420551.00 (  3.85%)    411673.00 (  1.65%)
TPut 17    420593.00 (  0.00%)    342590.00 (-18.55%)    428242.00 (  1.82%)    437461.00 (  4.01%)    428270.00 (  1.83%)
TPut 18    440178.00 (  0.00%)    377508.00 (-14.24%)    440392.00 (  0.05%)    455014.00 (  3.37%)    447671.00 (  1.70%)
TPut 19    448876.00 (  0.00%)    397727.00 (-11.39%)    462036.00 (  2.93%)    479223.00 (  6.76%)    461881.00 (  2.90%)
TPut 20    460513.00 (  0.00%)    411831.00 (-10.57%)    476437.00 (  3.46%)    493176.00 (  7.09%)    474824.00 (  3.11%)
TPut 21    474161.00 (  0.00%)    442153.00 ( -6.75%)    487513.00 (  2.82%)    505246.00 (  6.56%)    468938.00 ( -1.10%)
TPut 22    474493.00 (  0.00%)    429921.00 ( -9.39%)    487920.00 (  2.83%)    527360.00 ( 11.14%)    475208.00 (  0.15%)
TPut 23    489559.00 (  0.00%)    460354.00 ( -5.97%)    508298.00 (  3.83%)    534820.00 (  9.25%)    490743.00 (  0.24%)
TPut 24    495378.00 (  0.00%)    486826.00 ( -1.73%)    514403.00 (  3.84%)    545294.00 ( 10.08%)    493974.00 ( -0.28%)
TPut 25    491795.00 (  0.00%)    520474.00 (  5.83%)    507373.00 (  3.17%)    543526.00 ( 10.52%)    489850.00 ( -0.40%)
TPut 26    490038.00 (  0.00%)    465587.00 ( -4.99%)    376322.00 (-23.21%)    545175.00 ( 11.25%)    491352.00 (  0.27%)
TPut 27    491233.00 (  0.00%)    469764.00 ( -4.37%)    366225.00 (-25.45%)    536927.00 (  9.30%)    489611.00 ( -0.33%)
TPut 28    489058.00 (  0.00%)    489561.00 (  0.10%)    414027.00 (-15.34%)    543127.00 ( 11.06%)    473835.00 ( -3.11%)
TPut 29    471539.00 (  0.00%)    492496.00 (  4.44%)    400529.00 (-15.06%)    541615.00 ( 14.86%)    486009.00 (  3.07%)
TPut 30    480343.00 (  0.00%)    488349.00 (  1.67%)    405612.00 (-15.56%)    542904.00 ( 13.02%)    478384.00 ( -0.41%)
TPut 31    478109.00 (  0.00%)    460043.00 ( -3.78%)    401471.00 (-16.03%)    529079.00 ( 10.66%)    466457.00 ( -2.44%)
TPut 32    475736.00 (  0.00%)    472007.00 ( -0.78%)    401075.00 (-15.69%)    532423.00 ( 11.92%)    467866.00 ( -1.65%)
TPut 33    470758.00 (  0.00%)    474348.00 (  0.76%)    399592.00 (-15.12%)    518811.00 ( 10.21%)    464764.00 ( -1.27%)
TPut 34    467304.00 (  0.00%)    475878.00 (  1.83%)    394589.00 (-15.56%)    518334.00 ( 10.92%)    446719.00 ( -4.41%)
TPut 35    466391.00 (  0.00%)    487411.00 (  4.51%)    382799.00 (-17.92%)    513591.00 ( 10.12%)    447071.00 ( -4.14%)
TPut 36    452722.00 (  0.00%)    478050.00 (  5.59%)    381120.00 (-15.82%)    503801.00 ( 11.28%)    452243.00 ( -0.11%)
TPut 37    447878.00 (  0.00%)    478467.00 (  6.83%)    382803.00 (-14.53%)    494555.00 ( 10.42%)    442751.00 ( -1.14%)
TPut 38    447907.00 (  0.00%)    455542.00 (  1.70%)    341693.00 (-23.71%)    482758.00 (  7.78%)    444023.00 ( -0.87%)
TPut 39    428322.00 (  0.00%)    367921.00 (-14.10%)    404210.00 ( -5.63%)    464550.00 (  8.46%)    440482.00 (  2.84%)
TPut 40    429157.00 (  0.00%)    394277.00 ( -8.13%)    378554.00 (-11.79%)    467767.00 (  9.00%)    411807.00 ( -4.04%)
TPut 41    424339.00 (  0.00%)    415413.00 ( -2.10%)    399220.00 ( -5.92%)    457669.00 (  7.85%)    428273.00 (  0.93%)
TPut 42    397440.00 (  0.00%)    421027.00 (  5.93%)    372161.00 ( -6.36%)    458156.00 ( 15.28%)    422535.00 (  6.31%)
TPut 43    405391.00 (  0.00%)    433900.00 (  7.03%)    383936.00 ( -5.29%)    438929.00 (  8.27%)    410196.00 (  1.19%)
TPut 44    400692.00 (  0.00%)    427504.00 (  6.69%)    374757.00 ( -6.47%)    423538.00 (  5.70%)    399471.00 ( -0.30%)
TPut 45    399623.00 (  0.00%)    372622.00 ( -6.76%)    379797.00 ( -4.96%)    407255.00 (  1.91%)    374068.00 ( -6.39%)
TPut 46    391920.00 (  0.00%)    351205.00 (-10.39%)    368042.00 ( -6.09%)    411353.00 (  4.96%)    384363.00 ( -1.93%)
TPut 47    378199.00 (  0.00%)    358150.00 ( -5.30%)    368744.00 ( -2.50%)    408739.00 (  8.08%)    385670.00 (  1.98%)
TPut 48    379346.00 (  0.00%)    387287.00 (  2.09%)    373581.00 ( -1.52%)    423791.00 ( 11.72%)    380665.00 (  0.35%)
TPut 49    373614.00 (  0.00%)    395793.00 (  5.94%)    372621.00 ( -0.27%)    423024.00 ( 13.22%)    377985.00 (  1.17%)
TPut 50    372494.00 (  0.00%)    366488.00 ( -1.61%)    388778.00 (  4.37%)    410647.00 ( 10.24%)    378831.00 (  1.70%)
TPut 51    382195.00 (  0.00%)    381771.00 ( -0.11%)    387687.00 (  1.44%)    423249.00 ( 10.74%)    402233.00 (  5.24%)
TPut 52    369118.00 (  0.00%)    429441.00 ( 16.34%)    390226.00 (  5.72%)    410023.00 ( 11.08%)    396558.00 (  7.43%)
TPut 53    366453.00 (  0.00%)    445744.00 ( 21.64%)    399257.00 (  8.95%)    405937.00 ( 10.77%)    383916.00 (  4.77%)
TPut 54    366571.00 (  0.00%)    375762.00 (  2.51%)    395098.00 (  7.78%)    402220.00 (  9.72%)    395417.00 (  7.87%)
TPut 55    367580.00 (  0.00%)    336113.00 ( -8.56%)    400550.00 (  8.97%)    420978.00 ( 14.53%)    398098.00 (  8.30%)
TPut 56    367056.00 (  0.00%)    375635.00 (  2.34%)    385743.00 (  5.09%)    412685.00 ( 12.43%)    384029.00 (  4.62%)
TPut 57    359163.00 (  0.00%)    354001.00 ( -1.44%)    389827.00 (  8.54%)    394688.00 (  9.89%)    381032.00 (  6.09%)
TPut 58    360552.00 (  0.00%)    353312.00 ( -2.01%)    394099.00 (  9.30%)    388655.00 (  7.79%)    378132.00 (  4.88%)
TPut 59    354967.00 (  0.00%)    368534.00 (  3.82%)    390746.00 ( 10.08%)    399086.00 ( 12.43%)    387101.00 (  9.05%)
TPut 60    362976.00 (  0.00%)    388472.00 (  7.02%)    383073.00 (  5.54%)    399713.00 ( 10.12%)    390635.00 (  7.62%)
TPut 61    368072.00 (  0.00%)    399476.00 (  8.53%)    380807.00 (  3.46%)    372060.00 (  1.08%)    383187.00 (  4.11%)
TPut 62    356938.00 (  0.00%)    385648.00 (  8.04%)    387736.00 (  8.63%)    377183.00 (  5.67%)    378484.00 (  6.04%)
TPut 63    357491.00 (  0.00%)    404325.00 ( 13.10%)    396672.00 ( 10.96%)    384221.00 (  7.48%)    378907.00 (  5.99%)
TPut 64    357322.00 (  0.00%)    389552.00 (  9.02%)    386826.00 (  8.26%)    378601.00 (  5.96%)    369852.00 (  3.51%)
TPut 65    341262.00 (  0.00%)    394964.00 ( 15.74%)    380271.00 ( 11.43%)    382896.00 ( 12.20%)    382897.00 ( 12.20%)
TPut 66    357807.00 (  0.00%)    384846.00 (  7.56%)    362723.00 (  1.37%)    361530.00 (  1.04%)    380023.00 (  6.21%)
TPut 67    345092.00 (  0.00%)    376842.00 (  9.20%)    364193.00 (  5.54%)    374449.00 (  8.51%)    373877.00 (  8.34%)
TPut 68    350334.00 (  0.00%)    358330.00 (  2.28%)    359368.00 (  2.58%)    384920.00 (  9.87%)    381888.00 (  9.01%)
TPut 69    348372.00 (  0.00%)    356188.00 (  2.24%)    364449.00 (  4.61%)    395611.00 ( 13.56%)    375892.00 (  7.90%)
TPut 70    335077.00 (  0.00%)    359313.00 (  7.23%)    356418.00 (  6.37%)    375448.00 ( 12.05%)    372358.00 ( 11.13%)
TPut 71    341197.00 (  0.00%)    364168.00 (  6.73%)    343847.00 (  0.78%)    376113.00 ( 10.23%)    384292.00 ( 12.63%)
TPut 72    345032.00 (  0.00%)    356934.00 (  3.45%)    345007.00 ( -0.01%)    375313.00 (  8.78%)    381504.00 ( 10.57%)

numacore v17 was doing reasonably well but we knew that already.

autonuma does not do great on this test.

balancenuma does all right. The scalability patches actually hurt in this case
but it's likely down to varability in the decisions made by the scheduler as much
as anything else.

SPECJBB PEAKS
                                   3.7.0-rc7                  3.7.0-rc6                  3.7.0-rc7                  3.7.0-rc7                  3.7.0-rc7
                                  stats-v8r6          numacore-20121130         autonuma-v28fastr4           balancenuma-v9r2          balancenuma-v10r3
 Expctd Warehouse            48.00 (  0.00%)            48.00 (  0.00%)            48.00 (  0.00%)            48.00 (  0.00%)            48.00 (  0.00%)
 Expctd Peak Bops        379346.00 (  0.00%)        387287.00 (  2.09%)        373581.00 ( -1.52%)        423791.00 ( 11.72%)        380665.00 (  0.35%)
 Actual Warehouse            24.00 (  0.00%)            25.00 (  4.17%)            24.00 (  0.00%)            24.00 (  0.00%)            24.00 (  0.00%)
 Actual Peak Bops        495378.00 (  0.00%)        520474.00 (  5.07%)        514403.00 (  3.84%)        545294.00 ( 10.08%)        493974.00 ( -0.28%)
 SpecJBB Bops            183389.00 (  0.00%)        193652.00 (  5.60%)        193461.00 (  5.49%)        201083.00 (  9.65%)        195465.00 (  6.58%)
 SpecJBB Bops/JVM        183389.00 (  0.00%)        193652.00 (  5.60%)        193461.00 (  5.49%)        201083.00 (  9.65%)        195465.00 (  6.58%)

Balancenuma does all right on its specjbb score but the peak score with
the migration scalability patches applied is hurt. At least it's still
comparable to mainline.

MMTests Statistics: duration
           3.7.0-rc7   3.7.0-rc6   3.7.0-rc8   3.7.0-rc7   3.7.0-rc7   3.7.0-rc7
          stats-v8r6numacore-20121130numafix-20121209autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
User       316340.52   311420.23    31308.52   314589.64   316061.23   315584.37
System        102.08     3067.27      803.23      352.70      428.76      450.71
Elapsed      7433.22     7436.63     1398.05     7434.74     7432.60     7435.03

Usual comments about system CPU usage. You actually see latest numacore
figures here because they are based on what happened up until the crash.

MMTests Statistics: vmstat
                             3.7.0-rc7   3.7.0-rc6   3.7.0-rc8   3.7.0-rc7   3.7.0-rc7   3.7.0-rc7
                            stats-v8r6numacore-20121130numafix-20121209autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
Page Ins                         66212       36180       31560       36152       36188       63852
Page Outs                        31248       35544       12016       28388       28024       42360
Swap Ins                             0           0           0           0           0           0
Swap Outs                            0           0           0           0           0           0
Direct pages scanned                 0           0           0           0           0           0
Kswapd pages scanned                 0           0           0           0           0           0
Kswapd pages reclaimed               0           0           0           0           0           0
Direct pages reclaimed               0           0           0           0           0           0
Kswapd efficiency                 100%        100%        100%        100%        100%        100%
Kswapd velocity                  0.000       0.000       0.000       0.000       0.000       0.000
Direct efficiency                 100%        100%        100%        100%        100%        100%
Direct velocity                  0.000       0.000       0.000       0.000       0.000       0.000
Percentage direct scans             0%          0%          0%          0%          0%          0%
Page writes by reclaim               0           0           0           0           0           0
Page writes file                     0           0           0           0           0           0
Page writes anon                     0           0           0           0           0           0
Page reclaim immediate               0           0           0           0           0           0
Page rescued immediate               0           0           0           0           0           0
Slabs scanned                        0           0           0           0           0           0
Direct inode steals                  0           0           0           0           0           0
Kswapd inode steals                  0           0           0           0           0           0
Kswapd skipped wait                  0           0           0           0           0           0
THP fault alloc                  48874       45657       34986       48296       48697       47056
THP collapse alloc                  51           2           9         157          53          69
THP splits                          70          37          28          83          78          56
THP fault fallback                   0           0           0           0           0           0
THP collapse fail                    0           0           0           0           0           0
Compaction stalls                    0           0           0           0           0           0
Compaction success                   0           0           0           0           0           0
Compaction failures                  0           0           0           0           0           0
Page migrate success                 0           0   110442307           0    45908125    46995604
Page migrate failure                 0           0           0           0           0           0
Compaction pages isolated            0           0           0           0           0           0
Compaction migrate scanned           0           0           0           0           0           0
Compaction free scanned              0           0           0           0           0           0
Compaction cost                      0           0      114639           0       47652       48781
NUMA PTE updates                     0           0   391813174           0   351907231   361308027
NUMA hint faults                     0           0      796717           0     2010327     1867697
NUMA hint local faults               0           0      261885           0      677602      572742
NUMA pages migrated                  0           0   110442307           0    45908125    46995604
AutoNUMA cost                        0           0        8824           0       13387       12760

THP was certainly enabled.

numacores migration rate is extremely high until it crashed -- 308MB/sec
as opposed to balancenumas 24MB/sec on average.

SpecJBB, Single JVM, THP is disabled
====================================

                    3.7.0-rc7             3.7.0-rc6             3.7.0-rc8             3.7.0-rc7             3.7.0-rc7             3.7.0-rc7
                   stats-v8r6     numacore-20121130      numafix-20121209    autonuma-v28fastr4      balancenuma-v9r2     balancenuma-v10r3
TPut 1      19861.00 (  0.00%)     18255.00 ( -8.09%)     20169.00 (  1.55%)     19636.00 ( -1.13%)     19838.00 ( -0.12%)     20650.00 (  3.97%)
TPut 2      47613.00 (  0.00%)     37136.00 (-22.00%)     45050.00 ( -5.38%)     47153.00 ( -0.97%)     47481.00 ( -0.28%)     48199.00 (  1.23%)
TPut 3      72438.00 (  0.00%)     55692.00 (-23.12%)     64075.00 (-11.55%)     69394.00 ( -4.20%)     72029.00 ( -0.56%)     72932.00 (  0.68%)
TPut 4      98455.00 (  0.00%)     81301.00 (-17.42%)     93595.00 ( -4.94%)     98577.00 (  0.12%)     98437.00 ( -0.02%)     99748.00 (  1.31%)
TPut 5     120831.00 (  0.00%)     89067.00 (-26.29%)    115796.00 ( -4.17%)    120805.00 ( -0.02%)    117218.00 ( -2.99%)    121254.00 (  0.35%)
TPut 6     140013.00 (  0.00%)    108349.00 (-22.62%)    116704.00 (-16.65%)    125079.00 (-10.67%)    139878.00 ( -0.10%)    145360.00 (  3.82%)
TPut 7     163553.00 (  0.00%)    116192.00 (-28.96%)    118711.00 (-27.42%)    164368.00 (  0.50%)    167133.00 (  2.19%)    169539.00 (  3.66%)
TPut 8     190148.00 (  0.00%)    125955.00 (-33.76%)    118079.00 (-37.90%)    188906.00 ( -0.65%)    183058.00 ( -3.73%)    188936.00 ( -0.64%)
TPut 9     211343.00 (  0.00%)    144068.00 (-31.83%)    170067.00 (-19.53%)    206645.00 ( -2.22%)    205699.00 ( -2.67%)    217322.00 (  2.83%)
TPut 10    233190.00 (  0.00%)    148098.00 (-36.49%)    133365.00 (-42.81%)    234533.00 (  0.58%)    233632.00 (  0.19%)    227292.00 ( -2.53%)
TPut 11    253333.00 (  0.00%)    146043.00 (-42.35%)    108866.00 (-57.03%)    254167.00 (  0.33%)    251938.00 ( -0.55%)    259924.00 (  2.60%)
TPut 12    270661.00 (  0.00%)    131739.00 (-51.33%)    146170.00 (-46.00%)    271490.00 (  0.31%)    271393.00 (  0.27%)    272536.00 (  0.69%)
TPut 13    299807.00 (  0.00%)    169396.00 (-43.50%)    134946.00 (-54.99%)    299758.00 ( -0.02%)    270594.00 ( -9.74%)    299110.00 ( -0.23%)
TPut 14    319243.00 (  0.00%)    150705.00 (-52.79%)    145135.00 (-54.54%)    318481.00 ( -0.24%)    318566.00 ( -0.21%)    325133.00 (  1.84%)
TPut 15    339054.00 (  0.00%)    116872.00 (-65.53%)    127277.00 (-62.46%)    331534.00 ( -2.22%)    344672.00 (  1.66%)    318119.00 ( -6.17%)
TPut 16    354315.00 (  0.00%)    124346.00 (-64.91%)     86657.00 (-75.54%)    352600.00 ( -0.48%)    316761.00 (-10.60%)    364648.00 (  2.92%)
TPut 17    371306.00 (  0.00%)    118493.00 (-68.09%)     93297.00 (-74.87%)    368260.00 ( -0.82%)    328888.00 (-11.42%)    371088.00 ( -0.06%)
TPut 18    386361.00 (  0.00%)    138571.00 (-64.13%)    208447.00 (-46.05%)    374358.00 ( -3.11%)    356148.00 ( -7.82%)    399913.00 (  3.51%)
TPut 19    401827.00 (  0.00%)    118855.00 (-70.42%)    155803.00 (-61.23%)    399476.00 ( -0.59%)    393918.00 ( -1.97%)    405771.00 (  0.98%)
TPut 20    411130.00 (  0.00%)    144024.00 (-64.97%)    116524.00 (-71.66%)    407799.00 ( -0.81%)    377706.00 ( -8.13%)    406038.00 ( -1.24%)
TPut 21    425352.00 (  0.00%)    154264.00 (-63.73%)    144766.00 (-65.97%)    429226.00 (  0.91%)    431677.00 (  1.49%)    431583.00 (  1.46%)
TPut 22    438150.00 (  0.00%)    153892.00 (-64.88%)    222211.00 (-49.28%)    385827.00 (-11.94%)    440379.00 (  0.51%)    438861.00 (  0.16%)
TPut 23    438425.00 (  0.00%)    146506.00 (-66.58%)    213367.00 (-51.33%)    433963.00 ( -1.02%)    361427.00 (-17.56%)    445293.00 (  1.57%)
TPut 24    461598.00 (  0.00%)    138869.00 (-69.92%)    189745.00 (-58.89%)    439691.00 ( -4.75%)    471567.00 (  2.16%)    488259.00 (  5.78%)
TPut 25    459475.00 (  0.00%)    141698.00 (-69.16%)    105196.00 (-77.11%)    431373.00 ( -6.12%)    487921.00 (  6.19%)    447353.00 ( -2.64%)
TPut 26    452651.00 (  0.00%)    142844.00 (-68.44%)    125573.00 (-72.26%)    447517.00 ( -1.13%)    425336.00 ( -6.03%)    469793.00 (  3.79%)
TPut 27    450436.00 (  0.00%)    140870.00 (-68.73%)     68802.00 (-84.73%)    430805.00 ( -4.36%)    456114.00 (  1.26%)    461172.00 (  2.38%)
TPut 28    459770.00 (  0.00%)    143078.00 (-68.88%)    144373.00 (-68.60%)    432260.00 ( -5.98%)    478317.00 (  4.03%)    452144.00 ( -1.66%)
TPut 29    450347.00 (  0.00%)    142076.00 (-68.45%)    221760.00 (-50.76%)    440423.00 ( -2.20%)    388175.00 (-13.81%)    473273.00 (  5.09%)
TPut 30    449252.00 (  0.00%)    146900.00 (-67.30%)    139971.00 (-68.84%)    435082.00 ( -3.15%)    440795.00 ( -1.88%)    435189.00 ( -3.13%)
TPut 31    446802.00 (  0.00%)    148008.00 (-66.87%)    195143.00 (-56.32%)    418684.00 ( -6.29%)    417343.00 ( -6.59%)    437562.00 ( -2.07%)
TPut 32    439701.00 (  0.00%)    149591.00 (-65.98%)    159107.00 (-63.81%)    421866.00 ( -4.06%)    438719.00 ( -0.22%)    469763.00 (  6.84%)
TPut 33    434477.00 (  0.00%)    142801.00 (-67.13%)    110758.00 (-74.51%)    420631.00 ( -3.19%)    454673.00 (  4.65%)    451224.00 (  3.85%)
TPut 34    423014.00 (  0.00%)    152308.00 (-63.99%)    111701.00 (-73.59%)    415202.00 ( -1.85%)    415194.00 ( -1.85%)    446735.00 (  5.61%)
TPut 35    429012.00 (  0.00%)    154116.00 (-64.08%)    118968.00 (-72.27%)    402395.00 ( -6.20%)    425151.00 ( -0.90%)    434230.00 (  1.22%)
TPut 36    421097.00 (  0.00%)    157571.00 (-62.58%)    174626.00 (-58.53%)    404770.00 ( -3.88%)    430480.00 (  2.23%)    425324.00 (  1.00%)
TPut 37    414815.00 (  0.00%)    150771.00 (-63.65%)    238764.00 (-42.44%)    388842.00 ( -6.26%)    393351.00 ( -5.17%)    405824.00 ( -2.17%)
TPut 38    412361.00 (  0.00%)    157070.00 (-61.91%)    173206.00 (-58.00%)    398947.00 ( -3.25%)    401555.00 ( -2.62%)    432074.00 (  4.78%)
TPut 39    402234.00 (  0.00%)    161487.00 (-59.85%)    119790.00 (-70.22%)    382645.00 ( -4.87%)    423106.00 (  5.19%)    401091.00 ( -0.28%)
TPut 40    380278.00 (  0.00%)    165947.00 (-56.36%)    309375.00 (-18.65%)    394039.00 (  3.62%)    405371.00 (  6.60%)    410739.00 (  8.01%)
TPut 41    393204.00 (  0.00%)    160540.00 (-59.17%)    146153.00 (-62.83%)    385605.00 ( -1.93%)    403383.00 (  2.59%)    372466.00 ( -5.27%)
TPut 42    380622.00 (  0.00%)    151946.00 (-60.08%)    269523.00 (-29.19%)    374843.00 ( -1.52%)    380797.00 (  0.05%)    396227.00 (  4.10%)
TPut 43    371566.00 (  0.00%)    162369.00 (-56.30%)    344584.00 ( -7.26%)    347951.00 ( -6.36%)    386765.00 (  4.09%)    345633.00 ( -6.98%)
TPut 44    365538.00 (  0.00%)    161127.00 (-55.92%)    147195.00 (-59.73%)    355070.00 ( -2.86%)    344701.00 ( -5.70%)    391276.00 (  7.04%)
TPut 45    359305.00 (  0.00%)    159062.00 (-55.73%)    102716.00 (-71.41%)    350973.00 ( -2.32%)    370666.00 (  3.16%)    331191.00 ( -7.82%)
TPut 46    343160.00 (  0.00%)    163889.00 (-52.24%)    309203.00 ( -9.90%)    347960.00 (  1.40%)    380147.00 ( 10.78%)    323176.00 ( -5.82%)
TPut 47    346983.00 (  0.00%)    168666.00 (-51.39%)    330345.00 ( -4.80%)    313612.00 ( -9.62%)    362189.00 (  4.38%)    343154.00 ( -1.10%)
TPut 48    338143.00 (  0.00%)    153448.00 (-54.62%)    291944.00 (-13.66%)    341809.00 (  1.08%)    365342.00 (  8.04%)    354348.00 (  4.79%)
TPut 49    333941.00 (  0.00%)    142784.00 (-57.24%)    252850.00 (-24.28%)    336174.00 (  0.67%)    371700.00 ( 11.31%)    353148.00 (  5.75%)
TPut 50    334001.00 (  0.00%)    135713.00 (-59.37%)    252350.00 (-24.45%)    322489.00 ( -3.45%)    367963.00 ( 10.17%)    355823.00 (  6.53%)
TPut 51    338310.00 (  0.00%)    133402.00 (-60.57%)    232361.00 (-31.32%)    354805.00 (  4.88%)    372592.00 ( 10.13%)    351194.00 (  3.81%)
TPut 52    322897.00 (  0.00%)    150293.00 (-53.45%)    193895.00 (-39.95%)    353169.00 (  9.38%)    363024.00 ( 12.43%)    344846.00 (  6.80%)
TPut 53    329801.00 (  0.00%)    160792.00 (-51.25%)    180672.00 (-45.22%)    353588.00 (  7.21%)    365359.00 ( 10.78%)    355499.00 (  7.79%)
TPut 54    336610.00 (  0.00%)    164696.00 (-51.07%)    248332.00 (-26.23%)    361189.00 (  7.30%)    377851.00 ( 12.25%)    363987.00 (  8.13%)
TPut 55    325920.00 (  0.00%)    172380.00 (-47.11%)    271331.00 (-16.75%)    365678.00 ( 12.20%)    375735.00 ( 15.28%)    363697.00 ( 11.59%)
TPut 56    318997.00 (  0.00%)    176071.00 (-44.80%)    155354.00 (-51.30%)    367048.00 ( 15.06%)    380588.00 ( 19.31%)    362614.00 ( 13.67%)
TPut 57    321776.00 (  0.00%)    174531.00 (-45.76%)    279294.00 (-13.20%)    341874.00 (  6.25%)    378996.00 ( 17.78%)    360366.00 ( 11.99%)
TPut 58    308532.00 (  0.00%)    174202.00 (-43.54%)    170351.00 (-44.79%)    348156.00 ( 12.84%)    361623.00 ( 17.21%)    369693.00 ( 19.82%)
TPut 59    318974.00 (  0.00%)    175343.00 (-45.03%)    243463.00 (-23.67%)    358252.00 ( 12.31%)    360457.00 ( 13.01%)    364556.00 ( 14.29%)
TPut 60    325465.00 (  0.00%)    173694.00 (-46.63%)    222867.00 (-31.52%)    360808.00 ( 10.86%)    362745.00 ( 11.45%)    354232.00 (  8.84%)
TPut 61    319151.00 (  0.00%)    172320.00 (-46.01%)    218542.00 (-31.52%)    350597.00 (  9.85%)    371277.00 ( 16.33%)    352478.00 ( 10.44%)
TPut 62    320837.00 (  0.00%)    172312.00 (-46.29%)    251630.00 (-21.57%)    359062.00 ( 11.91%)    361009.00 ( 12.52%)    352930.00 ( 10.00%)
TPut 63    318198.00 (  0.00%)    172297.00 (-45.85%)    172040.00 (-45.93%)    356137.00 ( 11.92%)    347637.00 (  9.25%)    335322.00 (  5.38%)
TPut 64    321438.00 (  0.00%)    171894.00 (-46.52%)    151337.00 (-52.92%)    347376.00 (  8.07%)    346756.00 (  7.88%)    351410.00 (  9.32%)
TPut 65    314482.00 (  0.00%)    169147.00 (-46.21%)    143487.00 (-54.37%)    351726.00 ( 11.84%)    357429.00 ( 13.66%)    351236.00 ( 11.69%)
TPut 66    316802.00 (  0.00%)    170234.00 (-46.26%)    230207.00 (-27.33%)    344548.00 (  8.76%)    362143.00 ( 14.31%)    347058.00 (  9.55%)
TPut 67    312139.00 (  0.00%)    168180.00 (-46.12%)    148468.00 (-52.44%)    329030.00 (  5.41%)    353305.00 ( 13.19%)    345903.00 ( 10.82%)
TPut 68    323918.00 (  0.00%)    168392.00 (-48.01%)    184696.00 (-42.98%)    319985.00 ( -1.21%)    344250.00 (  6.28%)    345703.00 (  6.73%)
TPut 69    307506.00 (  0.00%)    167082.00 (-45.67%)    221855.00 (-27.85%)    340673.00 ( 10.79%)    339346.00 ( 10.35%)    336071.00 (  9.29%)
TPut 70    306799.00 (  0.00%)    165764.00 (-45.97%)    246518.00 (-19.65%)    331678.00 (  8.11%)    349583.00 ( 13.95%)    341944.00 ( 11.46%)
TPut 71    304232.00 (  0.00%)    165289.00 (-45.67%)    225582.00 (-25.85%)    319824.00 (  5.13%)    335238.00 ( 10.19%)    343396.00 ( 12.87%)
TPut 72    301619.00 (  0.00%)    163909.00 (-45.66%)    154552.00 (-48.76%)    326875.00 (  8.37%)    345999.00 ( 14.71%)    343949.00 ( 14.03%)

Latest numacore is regressing really badly here.

autonuma is all right.

balancenuma is all right. Migration scalability patches actually seem to
hurt a little.

SPECJBB PEAKS
                                   3.7.0-rc7                  3.7.0-rc6                  3.7.0-rc8                  3.7.0-rc7                  3.7.0-rc7                  3.7.0-rc7
                                  stats-v8r6          numacore-20121130           numafix-20121209         autonuma-v28fastr4           balancenuma-v9r2          balancenuma-v10r3
 Expctd Warehouse            48.00 (  0.00%)            48.00 (  0.00%)            48.00 (  0.00%)            48.00 (  0.00%)            48.00 (  0.00%)            48.00 (  0.00%)
 Expctd Peak Bops        338143.00 (  0.00%)        153448.00 (-54.62%)        291944.00 (-13.66%)        341809.00 (  1.08%)        365342.00 (  8.04%)        354348.00 (  4.79%)
 Actual Warehouse            24.00 (  0.00%)            56.00 (133.33%)            43.00 ( 79.17%)            26.00 (  8.33%)            25.00 (  4.17%)            24.00 (  0.00%)
 Actual Peak Bops        461598.00 (  0.00%)        176071.00 (-61.86%)        344584.00 (-25.35%)        447517.00 ( -3.05%)        487921.00 (  5.70%)        488259.00 (  5.78%)
 SpecJBB Bops            163683.00 (  0.00%)         83963.00 (-48.70%)        109061.00 (-33.37%)        176379.00 (  7.76%)        184040.00 ( 12.44%)        179621.00 (  9.74%)
 SpecJBB Bops/JVM        163683.00 (  0.00%)         83963.00 (-48.70%)        109061.00 (-33.37%)        176379.00 (  7.76%)        184040.00 ( 12.44%)        179621.00 (  9.74%)

numacore regresses 25.35% at the peak and 33.37% on its specjbb score.

balancenuma does all right -- 5.78% gain at the peak, 9.74% on its overall
specjbb score.

MMTests Statistics: duration
           3.7.0-rc7   3.7.0-rc6   3.7.0-rc8   3.7.0-rc7   3.7.0-rc7   3.7.0-rc7
          stats-v8r6numacore-20121130numafix-20121209autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
User       316751.91   167098.56   227496.59   307598.67   309109.47   313644.48
System         60.28   122511.08    72477.33     4411.81     1820.70     2654.77
Elapsed      7434.08     7451.36     7476.09     7437.52     7438.28     7438.19

numacores system CPu usage has improved but it's still insane -- 27 times
higher than balancenumas which itself is high. Put another way, numacore
is using over 1000 times more system CPU than the mainline kernel is.

MMTests Statistics: vmstat
                             3.7.0-rc7   3.7.0-rc6   3.7.0-rc8   3.7.0-rc7   3.7.0-rc7   3.7.0-rc7
                            stats-v8r6numacore-20121130numafix-20121209autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
Page Ins                         37112       36416       34572       37436       35400       34708
Page Outs                        29252       35664       29788       28120       28504       28292
Swap Ins                             0           0           0           0           0           0
Swap Outs                            0           0           0           0           0           0
Direct pages scanned                 0           0           0           0           0           0
Kswapd pages scanned                 0           0           0           0           0           0
Kswapd pages reclaimed               0           0           0           0           0           0
Direct pages reclaimed               0           0           0           0           0           0
Kswapd efficiency                 100%        100%        100%        100%        100%        100%
Kswapd velocity                  0.000       0.000       0.000       0.000       0.000       0.000
Direct efficiency                 100%        100%        100%        100%        100%        100%
Direct velocity                  0.000       0.000       0.000       0.000       0.000       0.000
Percentage direct scans             0%          0%          0%          0%          0%          0%
Page writes by reclaim               0           0           0           0           0           0
Page writes file                     0           0           0           0           0           0
Page writes anon                     0           0           0           0           0           0
Page reclaim immediate               0           0           0           0           0           0
Page rescued immediate               0           0           0           0           0           0
Slabs scanned                        0           0           0           0           0           0
Direct inode steals                  0           0           0           0           0           0
Kswapd inode steals                  0           0           0           0           0           0
Kswapd skipped wait                  0           0           0           0           0           0
THP fault alloc                      3           2           3           2           2           2
THP collapse alloc                   0           0           0           4           0           0
THP splits                           0           0           0           1           0           0
THP fault fallback                   0           0           0           0           0           0
THP collapse fail                    0           0           0           0           0           0
Compaction stalls                    0           0           0           0           0           0
Compaction success                   0           0           0           0           0           0
Compaction failures                  0           0           0           0           0           0
Page migrate success                 0           0   472734998           0    24675369    36216149
Page migrate failure                 0           0           0           0           0           0
Compaction pages isolated            0           0           0           0           0           0
Compaction migrate scanned           0           0           0           0           0           0
Compaction free scanned              0           0           0           0           0           0
Compaction cost                      0           0      490698           0       25613       37592
NUMA PTE updates                     0           0  2978374076           0   200854895   256255594
NUMA hint faults                     0           0           0           0   195451244   250219588
NUMA hint local faults               0           0           0           0    50377035    63739483
NUMA pages migrated                  0           0   472734998           0    24675369    36216149
AutoNUMA cost                        0           0       29830           0      979131     1253579

numacore is migrating on average 247MB/sec. balancenuma is migrating
19MB/sec on average.

I ran the other normal benchmarks too. kernbench and aim9 are more or less ok. The impact is on hackbench

HACKBENCH PIPES
                     3.7.0-rc7             3.7.0-rc6             3.7.0-rc8             3.7.0-rc7             3.7.0-rc7             3.7.0-rc7
                    stats-v8r6     numacore-20121130      numafix-20121209    autonuma-v28fastr4      balancenuma-v9r2     balancenuma-v10r3
Procs 1       0.0250 (  0.00%)      0.0260 ( -4.00%)      0.0246 (  1.48%)      0.0261 ( -4.27%)      0.0325 (-30.07%)      0.0353 (-41.14%)
Procs 4       0.0696 (  0.00%)      0.0702 ( -0.84%)      0.0602 ( 13.57%)      0.0707 ( -1.65%)      0.0760 ( -9.20%)      0.0738 ( -5.98%)
Procs 8       0.0836 (  0.00%)      0.0973 (-16.43%)      0.0949 (-13.53%)      0.1030 (-23.21%)      0.0887 ( -6.15%)      0.1031 (-23.36%)
Procs 12      0.0971 (  0.00%)      0.0969 (  0.21%)      0.1447 (-49.00%)      0.1235 (-27.19%)      0.0953 (  1.88%)      0.1394 (-43.56%)
Procs 16      0.1218 (  0.00%)      0.1286 ( -5.52%)      0.2214 (-81.70%)      0.1775 (-45.69%)      0.1105 (  9.33%)      0.2188 (-79.57%)
Procs 20      0.1472 (  0.00%)      0.1508 ( -2.48%)      0.2744 (-86.43%)      0.1584 ( -7.64%)      0.1378 (  6.38%)      0.2567 (-74.37%)
Procs 24      0.1684 (  0.00%)      0.1823 ( -8.20%)      0.3602 (-113.82%)      0.4648 (-175.96%)      0.1623 (  3.68%)      0.3118 (-85.12%)
Procs 28      0.1919 (  0.00%)      0.1969 ( -2.61%)      0.4632 (-141.39%)      0.5287 (-175.57%)      0.1900 (  0.96%)      0.4326 (-125.48%)
Procs 32      0.2256 (  0.00%)      0.2163 (  4.12%)      0.5040 (-123.40%)      0.4607 (-104.23%)      0.2163 (  4.13%)      0.4583 (-103.16%)
Procs 36      0.2228 (  0.00%)      0.2658 (-19.29%)      0.5481 (-145.98%)      0.6190 (-177.83%)      0.2570 (-15.33%)      0.5267 (-136.38%)
Procs 40      0.2811 (  0.00%)      0.2906 ( -3.37%)      0.6223 (-121.36%)      0.2595 (  7.69%)      0.2638 (  6.15%)      0.5941 (-111.35%)

HACKBENCH SOCKETS
                     3.7.0-rc7             3.7.0-rc6             3.7.0-rc8             3.7.0-rc7             3.7.0-rc7             3.7.0-rc7
                    stats-v8r6     numacore-20121130      numafix-20121209    autonuma-v28fastr4      balancenuma-v9r2     balancenuma-v10r3
Procs 1       0.0220 (  0.00%)      0.0220 (  0.00%)      0.0229 ( -4.20%)      0.0283 (-28.66%)      0.0216 (  1.89%)      0.0256 (-16.36%)
Procs 4       0.0456 (  0.00%)      0.0513 (-12.51%)      0.0559 (-22.50%)      0.0820 (-79.73%)      0.0407 ( 10.76%)      0.0627 (-37.46%)
Procs 8       0.0679 (  0.00%)      0.0714 ( -5.20%)      0.1472 (-116.82%)      0.2772 (-308.32%)      0.0697 ( -2.60%)      0.1715 (-152.63%)
Procs 12      0.0940 (  0.00%)      0.0973 ( -3.56%)      0.2259 (-140.32%)      0.1155 (-22.87%)      0.0973 ( -3.55%)      0.2459 (-161.55%)
Procs 16      0.1181 (  0.00%)      0.1263 ( -6.96%)      0.3248 (-174.92%)      0.4467 (-278.19%)      0.1234 ( -4.46%)      0.3231 (-173.55%)
Procs 20      0.1504 (  0.00%)      0.1531 ( -1.83%)      0.4039 (-168.54%)      0.4917 (-226.94%)      0.1534 ( -1.97%)      0.4172 (-177.36%)
Procs 24      0.1757 (  0.00%)      0.1826 ( -3.92%)      0.3965 (-125.60%)      0.5142 (-192.57%)      0.1826 ( -3.89%)      0.4759 (-170.78%)
Procs 28      0.2044 (  0.00%)      0.2166 ( -5.93%)      0.5438 (-165.99%)      0.6600 (-222.85%)      0.2164 ( -5.88%)      0.5455 (-166.83%)
Procs 32      0.2456 (  0.00%)      0.2501 ( -1.86%)      0.6261 (-154.93%)      0.6391 (-160.22%)      0.2449 (  0.27%)      0.6093 (-148.11%)
Procs 36      0.2649 (  0.00%)      0.2747 ( -3.70%)      0.7066 (-166.71%)      0.5775 (-117.97%)      0.2815 ( -6.27%)      0.6840 (-158.19%)
Procs 40      0.3067 (  0.00%)      0.3114 ( -1.56%)      0.7588 (-147.42%)      0.7517 (-145.12%)      0.3081 ( -0.48%)      0.8871 (-189.27%)

Latest numacore, autonuma and balancenuma are all butchering hackbench
performance. Considering that balancenuma started hurting performance with
the migration scalability patches leads me to conclude that they might be
directly or indirectly responsible.

MMTests Statistics: vmstat
                             3.7.0-rc7   3.7.0-rc6   3.7.0-rc8   3.7.0-rc7   3.7.0-rc7   3.7.0-rc7
                            stats-v8r6numacore-20121130numafix-20121209autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
Page Ins                             4           4           4           4           4           4
Page Outs                         1540        1636        2568        2264        1548        2484
Swap Ins                             0           0           0           0           0           0
Swap Outs                            0           0           0           0           0           0
Direct pages scanned                 0           0           0           0           0           0
Kswapd pages scanned                 0           0           0           0           0           0
Kswapd pages reclaimed               0           0           0           0           0           0
Direct pages reclaimed               0           0           0           0           0           0
Kswapd efficiency                 100%        100%        100%        100%        100%        100%
Kswapd velocity                  0.000       0.000       0.000       0.000       0.000       0.000
Direct efficiency                 100%        100%        100%        100%        100%        100%
Direct velocity                  0.000       0.000       0.000       0.000       0.000       0.000
Percentage direct scans             0%          0%          0%          0%          0%          0%
Page writes by reclaim               0           0           0           0           0           0
Page writes file                     0           0           0           0           0           0
Page writes anon                     0           0           0           0           0           0
Page reclaim immediate               0           0           0           0           0           0
Page rescued immediate               0           0           0           0           0           0
Slabs scanned                        0           0           0           0           0           0
Direct inode steals                  0           0           0           0           0           0
Kswapd inode steals                  0           0           0           0           0           0
Kswapd skipped wait                  0           0           0           0           0           0
THP fault alloc                      5           0           0           0           6           5
THP collapse alloc                   0           0           0           0           0           0
THP splits                           0           0           0           0           0           0
THP fault fallback                   0           0           0           0           0           0
THP collapse fail                    0           0           0           0           0           0
Compaction stalls                    0           0           0           0           0           0
Compaction success                   0           0           0           0           0           0
Compaction failures                  0           0           0           0           0           0
Page migrate success                 0           0           0           0        1649          49
Page migrate failure                 0           0           0           0           0           0
Compaction pages isolated            0           0           0           0           0           0
Compaction migrate scanned           0           0           0           0           0           0
Compaction free scanned              0           0           0           0           0           0
Compaction cost                      0           0           0           0           1           0
NUMA PTE updates                     0           0           0           0       21646       22884
NUMA hint faults                     0           0           0           0        1045        2131
NUMA hint local faults               0           0           0           0          40        1218
NUMA pages migrated                  0           0           0           0        1649          49
AutoNUMA cost                        0           0           0           0           5          10

Based on this, I believe the migration patches are only indirectly
responsible. No way should hackbench be migrating or receiving a PTE update
at all. Rather than withdrawing the scalability patches it might make more
sense to either increase the length of time before a PTE takes place or
to delay NUMA PTE updates until the RSS reaches a particular size instead
of just relying on where the task gets scheduled.

So overall, I still believe that balancenuma should be merged at this point
based on these results. Nothing stops you doing a rebase of numacore on
top afterwards and introduce it in parts validating at each point it's
actually improving performance and not just assuming it does.

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>


[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]