On 11/07/14 20:22, Andres Freund wrote:
On 2014-07-11 12:40:15 +1200, Mark Kirkwood wrote:Full report http://paste.ubuntu.com/7777886/# 8.82% postgres [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave | |--75.69%-- pagevec_lru_move_fn | __lru_cache_add | lru_cache_add | putback_lru_page | migrate_pages | migrate_misplaced_page | do_numa_page | handle_mm_fault | __do_page_fault | do_page_fault | page_faultSo, the majority of the time is spent in numa page migration. Can you disable numa_balancing? I'm not sure if your kernel version does that at runtime or whether you need to reboot. The kernel.numa_balancing sysctl might work. Otherwise you probably need to boot with numa_balancing=0. It'd also be worthwhile to test this with numactl --interleave.
Trying out with numa_balancing=0 seemed to get essentially the same performance. Similarly wrapping postgres startup with --interleave.
All this made me want to try with numa *really* disabled. So rebooted the box with "numa=off" appended to the kernel cmdline. Somewhat surprisingly (to me anyway), the numbers were essentially identical. The profile, however is quite different:
Full report at http://paste.ubuntu.com/7806285/4.56% postgres [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
|
|--41.89%-- try_to_wake_up
| |
| |--96.12%-- default_wake_function
| | |
| | |--99.96%-- pollwake
| | | __wake_up_common
| | | __wake_up_sync_key
| | | sock_def_readable
| | | |
| | | |--99.94%--
unix_stream_sendmsg
| | | |
sock_sendmsg
| | | |
SYSC_sendto
| | | |
sys_sendto
| | | | tracesys
| | | |
__libc_send
| | | | pq_flush
| | | |
ReadyForQuery
| | | |
PostgresMain
| | | |
ServerLoop
| | | |
PostmasterMain
| | | | main
| | | |
__libc_start_main
| | | --0.06%-- [...]
| | --0.04%-- [...]
| |
| |--2.87%-- wake_up_process
| | |
| | |--95.71%--
wake_up_sem_queue_do
| | | SYSC_semtimedop
| | | sys_semop
| | | tracesys
| | | __GI___semop
| | | |
| | | |--99.75%--
LWLockRelease
| | | | |
| | | |
|--25.09%-- RecordTransactionCommit
| | | | |
CommitTransaction
| | | | |
CommitTransactionCommand
| | | | |
finish_xact_command.part.4
| | | | |
PostgresMain
| | | | |
ServerLoop
| | | | |
PostmasterMain
| | | | |
main
| | | | |
__libc_start_main
regards Mark
