| What "idle" driver are you using? /dev/cpu_dma_latency might not be sufficient if the OS uses certain idle instructions, IMO mwait is still issued and its latency might not be 1 on Atoms. What is in /sys/devices/system/cpu/cpu0/cpuidle/state*/latency on Atoms?
Btw disabling all power management is IMO not a good idea. This disables TurboBoost (do Atoms have it?) which gives a huge gain if the TDP is low enough. Kernel scheduler in recent kernels should be smart enough to keep some cores busy to an extent and not wake up all cores and this will give you better performance than using all the cores concurrently unless you really use all of their CPU time.
Jan
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Nick,
I've been trying to replicate your results without success. Can you help me understand what I'm doing that is not the same as your test?
My setup is two boxes, one is a client and the other is a server. The server has Intel(R) Atom(TM) CPU C2750 @ 2.40GHz, 32 GB RAM and 2 Intel S3500 240 GB SSD drives. The boxes have Infiniband FDR cards connected to a QDR switch using IPoIB. I set up OSDs on the 2 SSDs and set pool size=1. I mapped a 200GB RBD using the kernel module ran fio on the RBD. I adjusted the number of cores, clock speed and C-states of the server and here are my results:
Adjusted core number and set the processor to a set frequency using the userspace governor.
8 jobs 8 depth Cores
1 2 3 4 5 6 7 8
Frequency 2.4 387 762 1121 1432 1657 1900 2092 2260
GHz 2 386 758 1126 1428 1657 1890 2090 2232
1.6 382 756 1127 1428 1656 1894 2083 2201
1.2 385 756 1125 1431 1656 1885 2093 2244
I then adjusted the processor to not go in a deeper sleep state than C1 and also tested setting the highest CPU frequency with the ondemand governor.
1 job 1 depth
Cores 1
<=C1, feq range C0-C6, freq range C0-C6, static freq <=C1, static freq
Frequency 2.4 381 381 379 381
GHz 2 382 380 381 381
1.6 380 381 379 382
1.2 383 378 379 383
Cores 8
<=C1, feq range C0-C6, freq range C0-C6, static freq <=C1, static freq
Frequency 2.4 629 580 584 629
GHz 2 630 579 584 634
1.6 630 579 584 634
1.2 632 581 582 634
Here I'm see a correlation between # cores and C-states, but not frequency.
Frequency was controlled with:
cpupower frequency-set -d 1.2GHz -u 1.2GHz -g userspace
and
cpupower frequency-set -d 1.2GHz -u 2.0GHz -g ondemand
Core count adjusted by:
for i in {1..7}; do echo 0 > /sys/devices/system/cpu/cpu$i/online; done
C-states controlled by:
# python
Python 2.7.5 (default, Jun 24 2015, 00:41:19)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> fd = open('/dev/cpu_dma_latency','wb')
>>> fd.write('1')
>>> fd.flush()
>>> fd.close() # Don't run this until the tests are completed (the handle has to stay open).
>>>
I'd like to replicate your results. I'd also like if you can verify some of mine in your set-up around C-States and cores.
Thanks,
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.0.2
Comment: https://www.mailvelope.com
wsFcBAEBCAAQBQJV5g8GCRDmVDuy+mK58QAAe6YP/j+SNGFI2z7ndnbOk87D
UjxG+hiZT5bkdt2/wVfI6QiH0UGDA3rLBsttOHPgfxP6/CEy801q8/fO0QOk
tLxIgX01K4ECls2uhiFAM3bhKalFsKDM6rHYFx96tIGWonQeou36ouDG8pfz
YsprvQ2XZEX1+G4dfZZ4lc3A3mfIY6Wsn7DC0tup9eRp3cl9hQLXEu4Zg8CZ
7867FNaud4S4f6hYV0KUC0fv+hZvyruMCt/jgl8gVr8bAdNgiW5u862gsk5b
sO9mb7H679G8t47m3xd89jTh9siMshbcakF9PXKzrN7DxBb/sBuN3GykesZA
+5jdUTzPCxFu+LocJ91by8FybatpLwxycmfP2gRxd/owclXk5BqqJUnrdYVm
n2GcHobdHVv9k/s+iBVV0xbwqOY+IO9UNUfLAKNy7E1xtpXdTpQBuokmu/4D
WXg3C4u+DsZNvcziO4s/edQ1koOQm1Fcj5VnbouSqmsHpB5nHeJbGmiKNTBA
9pE/hTph56YRqOE3bq3X/ohjtziL7/e/MVF3VUisDJieaLxV9weLxKIf0W9t
L7NMhX7iUIMps5ulA9qzd8qJK6yBa65BVXtk5M0A5oTA/VvxHQT6e5nSZS+Z
WLjavMnmSSJT1BQZ5GkVbVqo4UVjndcXEvkBm3+McaGKliO2xvxP+U3nCKpZ
js+h
=4WAa
-----END PGP SIGNATURE-----
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxxhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
|
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
