Hello, I have rather strange test results with different numa settings. My application uses multiple threads and (simplified) accesses data in a memory mapped file. I tried to find optimal settings and see the following effects with 40 threads (on a NUMA hardware with 8 nodes with 6 CPUs per node): Independently of the number of assigned nodes, the runtime is nearly the same (25 +/- 3 seconds). The used CPU (showed in top) is nearly 100% * number of processors (e.g. 600% for one node up to 4000% for 8 nodes, here not 4800, because I use only 40 threads). The user CPU (measured with time) is around 1m20 (+/-5s). The system CPU is 1m20 with 1 node and increases (3min for 2 nodes, 6min for 3 nodes etc. up to 15min for 8 nodes). So every gain with additional nodes/cpus is completely wasted in system CPU. I made some other tests with only CPU assignments. If I assign only 1 CPU, the test needs 47s. Optimum is reached at 4 CPUs (on 2 different nodes) with 16s. This is even much better than any of the results where I limited to 1, 2, ... or 8 nodes. The system is a SUSE Linux Enterprise Server 11 SP1 with kernel 2.6.32.12. I will not exclude any problems in our application or my test constellation, but we have some background in optimizing it for multithreading and had good scaling results on non-NUMA hardware. It seems to me that either scheduling or memory access (esp. to memory mapped files) does not really scale in the (this?) kernel. Could pthread mutexes be the problem (scaling bad on NUMA) or rather some internal kernel mutexes when accessing memory that is located on another node? Has anyone similar effects and/or some hints that could improve the situation? Are there better NUMA optimized kernels available (for Suse or Red hat Enterprise Linux)? Thank you in advance! Regards, Andreas -- To unsubscribe from this list: send the line "unsubscribe linux-numa" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
