Re: Some performance figure questions.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Sorry for sending this email twice. There was a problem with
character encoding "=" was replaced by "=3D".

Hi,

it looks reasonable if you are using only a single core to do the test.

The interconnect bandwidth is usually lower then the
memory controller bandwidth, so there is always this bottleneck.

With many/most architectures (especially Nehalem / Westmere) you cannot
saturate all the memory bandwith with only a single core, so you
need to do the test with multiple or better with all the cores.

On my machine I have a bandwidth delta of around 35 %, when
accessing memory on node 1 and running the application on node 0.

This is what I've tried out:

wget www.cs.virginia.edu/stream/FTP/Code/stream.c
gcc -fopenmp -mcmodel=medium -O -DSTREAM_ARRAY_SIZE=134217728
-DNTIMES=20 stream.c -o stream.1GiB.20

##############################
###########################

numactl --cpunodebind=0 --membind=0  ./stream.1GiB.20
-------------------------------------------------------------
STREAM version $Revision: 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array size = 134217728 (elements), Offset = 0 (elements)
Memory per array = 1024.0 MiB (= 1.0 GiB).
Total memory required = 3072.0 MiB (= 3.0 GiB).
Each kernel will be executed 20 times.
 The *best* time for each kernel (excluding the first iteration)
 will be used to compute the reported bandwidth.
-------------------------------------------------------------
Number of Threads requested = 20
Number of Threads counted = 20
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 104100 microseconds.
   (= 104100 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           14258.8     0.150689     0.150608     0.150890
Scale:          14255.4     0.150742     0.150643     0.150935
Add:            15889.5     0.202965     0.202727     0.203927
Triad:          15904.0     0.202771     0.202542     0.203169
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------

#########################################################

numactl --cpunodebind=0 --membind=1  ./stream.1GiB.20
-------------------------------------------------------------
STREAM version $Revision: 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array size = 134217728 (elements), Offset = 0 (elements)
Memory per array = 1024.0 MiB (= 1.0 GiB).
Total memory required = 3072.0 MiB (= 3.0 GiB).
Each kernel will be executed 20 times.
 The *best* time for each kernel (excluding the first iteration)
 will be used to compute the reported bandwidth.
-------------------------------------------------------------
Number of Threads requested = 20
Number of Threads counted = 20
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 146572 microseconds.
   (= 146572 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:            9763.3     0.220143     0.219955     0.220324
Scale:           9758.8     0.220277     0.220056     0.220557
Add:            10442.6     0.308598     0.308469     0.308825
Triad:          10453.1     0.308435     0.308159     0.308759
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------


It can get even worse if the local and the remote CPUs are accessing
memory on  single node. I get arround 50 % of the bandwidth, which
makes sense using 1 instead of 2 memory controllers.

numactl --cpunodebind=0,1 --membind=1  ./stream.1GiB.20
-------------------------------------------------------------
STREAM version $Revision: 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array size = 134217728 (elements), Offset = 0 (elements)
Memory per array = 1024.0 MiB (= 1.0 GiB).
Total memory required = 3072.0 MiB (= 3.0 GiB).
Each kernel will be executed 20 times.
 The *best* time for each kernel (excluding the first iteration)
 will be used to compute the reported bandwidth.
-------------------------------------------------------------
Number of Threads requested = 40
Number of Threads counted = 40
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 121801 microseconds.
   (= 121801 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           12337.7     0.174139     0.174059     0.174325
Scale:          12296.9     0.174744     0.174637     0.174885
Add:            14211.8     0.226803     0.226659     0.226928
Triad:          14251.7     0.226201     0.226024     0.226418
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------

#########################################################

numactl --cpunodebind=0,1 --membind=0,1  ./stream.1GiB.20
-------------------------------------------------------------
STREAM version $Revision: 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array size = 134217728 (elements), Offset = 0 (elements)
Memory per array = 1024.0 MiB (= 1.0 GiB).
Total memory required = 3072.0 MiB (= 3.0 GiB).
Each kernel will be executed 20 times.
 The *best* time for each kernel (excluding the first iteration)
 will be used to compute the reported bandwidth.
-------------------------------------------------------------
Number of Threads requested = 40
Number of Threads counted = 40
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 56900 microseconds.
   (= 56900 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           27330.0     0.078743     0.078576     0.079084
Scale:          27239.9     0.079228     0.078836     0.080867
Add:            30466.9     0.105910     0.105729     0.106404
Triad:          30504.9     0.105749     0.105597     0.106209
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------

Regards,
Andreas

2014-11-25 16:11 GMT+01:00 Serge A <serge.ayoun@xxxxxxxxx>:
> I am new to this list and hope this is the right place to ask this question.
>
>
>
> We are on process to check performance of a new hardware feature on NUMA
> system.
>
> I did a few experiments to check memory performance behavior on a two node
> NUMA system with the following SLIT:
>
>
>
> 10 21
>
> 21 10
>
>
>
> The test does the following: allocate 1 GB buffer and access the buffer
> sequentially with a 256bytes stride, i.e. accessing
>
> Offset 0, 256, 512, …..
>
> This test was performed while running on node 0 (using numa_run_on_node()
> API) done while accessing local node memory 0 and also remote node 1 memory
> (using(numa_alloc_onnode()) API).
>
>
>
> Surprisingly, the performance delta is about 15%: local memory test took
> 1800ms while the remote one took 2100ms. I expected much more.
>
>
>
> Two questions:
>
> Do these results look reasonable to you?
>
> Is there some kind of recommended and standard memory test for NUMA system?
>
>
>
> Thanks,
>
> Serge
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-numa" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-numa" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Linux Kernel]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux SCSI]     [Devices]

  Powered by Linux