* Ingo Molnar <mingo@xxxxxxx> wrote:
4> The place for the sock_rfree() hit looks a bit weird, and i'll
> investigate it now a bit more to place the real overhead point
> properly. (i already mapped the test-bit overhead: that comes from
> napi_disable_pending())
ok, here's a new set of profiles. (again for tbench 64-thread on a
16-way box, with v2.6.28-rc5-19-ge14c8bf and with the kernel config i
posted before.)
Here are the per major subsystem percentages:
NET overhead ( 5786945/10096751): 57.31%
security overhead ( 925933/10096751): 9.17%
usercopy overhead ( 837887/10096751): 8.30%
sched overhead ( 753662/10096751): 7.46%
syscall overhead ( 268809/10096751): 2.66%
IRQ overhead ( 266500/10096751): 2.64%
slab overhead ( 180258/10096751): 1.79%
timer overhead ( 92986/10096751): 0.92%
pagealloc overhead ( 87381/10096751): 0.87%
VFS overhead ( 53295/10096751): 0.53%
PID overhead ( 44469/10096751): 0.44%
pagecache overhead ( 33452/10096751): 0.33%
gtod overhead ( 11064/10096751): 0.11%
IDLE overhead ( 0/10096751): 0.00%
---------------------------------------------------------
left ( 753878/10096751): 7.47%
The breakdown is very similar to what i sent before, within noise.
[ 'left' is random overhead from all around the place - i categorized
the 500 most expensive functions in the profile per subsystem.
I stopped short of doing it for all 1300+ functions: it's rather
laborous manual work even with hefty use of regex patterns.
It's also less meaningful in practice: the trend in the first 500
functions is present in the remaining 800 functions as well. I
watched the breakdown evolve as i increased the coverage - in
practice it is the first 100 functions that matter - it just doesnt
change after that. ]
The readprofile output below seems structured in a more useful way now
- i tweaked compiler options to have the profiler hits spread out in a
more meaningful way. I collected 10 million NMI profiler hits, and
normalized the readprofile output up to 100%.
[ I'll post per function analysis as i complete them, as a reply to
this mail. ]
Ingo
100.000000 total
................
7.253355 copy_user_generic_string
3.934833 avc_has_perm_noaudit
3.356152 ip_queue_xmit
3.038025 skb_release_data
2.118525 skb_release_head_state
1.997533 tcp_ack
1.833688 tcp_recvmsg
1.717771 eth_type_trans
1.673249 __inet_lookup_established
1.508888 system_call
1.469183 tcp_current_mss
1.431553 tcp_transmit_skb
1.385125 tcp_sendmsg
1.327643 tcp_v4_rcv
1.292328 nf_hook_thresh
1.203205 schedule
1.059501 nf_hook_slow
1.027373 constant_test_bit
0.945183 sock_rfree
0.922748 __switch_to
0.911605 netif_rx
0.876270 register_gifconf
0.788200 ip_local_deliver_finish
0.781467 dev_queue_xmit
0.766530 constant_test_bit
0.758208 _local_bh_enable_ip
0.747184 load_cr3
0.704341 memset_c
0.671260 sysret_check
0.651845 ip_finish_output2
0.620204 audit_free_names
0.617781 audit_syscall_exit
0.615149 skb_copy_datagram_iovec
0.613848 selinux_socket_sock_rcv_skb
0.606995 constant_test_bit
0.593936 __tcp_push_pending_frames
0.592198 tcp_cleanup_rbuf
0.574093 ip_rcv
0.567886 netif_receive_skb
0.563377 get_page_from_freelist
0.557657 tcp_event_data_recv
0.539274 ip_local_deliver
0.534130 sys_recvfrom
0.512321 __tcp_select_window
0.498427 tcp_rcv_established
0.494862 sys_sendto
0.487473 audit_syscall_entry
0.478495 sched_clock_cpu
0.474861 kfree
0.466310 tcp_established_options
0.461384 net_rx_action
0.447162 __mod_timer
0.442078 ip_rcv_finish
0.441631 find_pid_ns
0.441124 sk_wait_data
0.423943 __sock_recvmsg
0.422126 selinux_parse_skb
0.417975 __napi_schedule
0.414082 __do_softirq
0.403604 task_rq_lock
0.380792 nf_iterate
0.377614 select_task_rq_fair
0.374973 sock_sendmsg
0.374635 kmem_cache_alloc_node
0.368775 avc_has_perm
0.368706 local_bh_disable
0.361834 release_sock
0.346400 sock_common_recvmsg
0.342825 skb_clone
0.338704 __alloc_skb
0.326488 do_softirq
0.323410 lock_sock_nested
0.322129 __copy_skb_header
0.316835 put_page
0.310966 selinux_ip_postroute
0.306229 sel_netport_sid
0.299863 try_to_wake_up
0.296288 process_backlog
0.294818 __inet_lookup
0.294778 thread_return
0.293219 cfs_rq_of
0.292315 internal_add_timer
0.292305 tcp_rcv_space_adjust
0.281053 constant_test_bit
0.278779 local_bh_enable
0.272910 *unknown*
0.269593 schedule_timeout
0.261846 tcp_v4_md5_lookup
0.260992 __ip_local_out
0.255868 __enqueue_entity
0.253931 avc_audit
0.252004 finish_task_switch
0.249263 audit_get_context
0.248290 sockfd_lookup_light
0.247416 virt_to_head_page
0.244149 tcp_options_write
0.243603 memcpy_toiovec
0.243434 sock_recvmsg
0.242599 call_softirq
0.242391 __unlazy_fpu
0.236412 fput_light
0.235628 ret_from_sys_call
0.234933 sk_reset_timer
0.228358 math_state_restore
0.227117 socket_has_perm
0.223492 virt_to_cache
0.219063 __cache_free
0.216401 update_curr
0.216232 tcp_v4_send_check
0.213978 audit_free_aux
0.213223 tcp_v4_do_rcv
0.212975 __kfree_skb
0.211137 dev_hard_start_xmit
0.209052 tcp_rtt_estimator
0.207999 netif_needs_gso
0.207662 __update_sched_clock
0.207284 rb_erase
0.204861 enqueue_task_fair
0.203490 skb_release_all
0.203252 tcp_send_delayed_ack
0.203232 inet_ehashfn
0.199846 sel_netport_find
0.195396 system_call_after_swapgs
0.186756 lock_timer_base
0.186687 pick_next_task_fair
0.183986 mod_timer
0.182982 loopback_xmit
0.182605 native_read_tsc
0.181195 skb_set_owner_r
0.179248 switch_mm
0.175584 set_next_entity
0.173329 raw_local_deliver
0.171641 sys_kill
0.164510 dequeue_task_fair
0.161938 clear_bit
0.160528 sock_def_readable
0.157628 __tcp_ack_snd_check
0.156893 skb_can_coalesce
0.156556 tcp_snd_wnd_test
0.155662 ip_output
0.150627 sk_stream_alloc_skb
0.150219 cpu_sdc
0.149425 sysret_careful
0.148760 tcp_data_snd_check
0.147816 auditsys
0.147419 pskb_may_pull
0.147151 fget_light
0.143774 tcp_cwnd_test
0.143029 rb_insert_color
0.142265 __wake_up
0.141808 tcp_bound_to_half_wnd
0.138600 __sk_dst_check
0.138431 free_hot_cold_page
0.137954 unroll_tree_refs
0.137080 __skb_unlink
0.135124 __sock_sendmsg
0.135064 get_pageblock_flags_group
0.132701 kmem_cache_free
0.128152 bictcp_cong_avoid
0.127874 __napi_complete
0.127527 ____cache_alloc
0.127368 tcp_is_cwnd_limited
0.127278 find_vpid
0.126941 constant_test_bit
0.126504 sk_mem_charge
0.126255 __alloc_pages_internal
0.125977 dst_release
0.125521 hash_64
0.124895 put_prev_task_fair
0.123802 netlbl_enabled
0.122829 sched_clock
0.122640 skb_push
0.122035 __phys_addr
0.121161 dput
0.120515 tcp_prequeue_process
0.118916 __skb_dequeue
0.117715 selinux_socket_sendmsg
0.117536 __inc_zone_state
0.115907 sk_wake_async
0.113504 selinux_ipv4_output
0.113017 sel_netif_sid
0.112431 skb_reset_network_header
0.111170 check_preempt_wakeup
0.111061 bictcp_acked
0.110882 sel_netnode_find
0.109978 update_min_vruntime
0.109889 resched_task
0.109879 current_kernel_time
0.109432 tcp_checksum_complete_user
0.107476 ip_dont_fragment
0.107386 sysret_audit
0.106979 inet_csk_reset_xmit_timer
0.106006 skb_entail
0.105777 sysret_signal
0.105420 avc_hash
0.105251 __skb_clone
0.105211 tcp_init_tso_segs
0.103523 __dequeue_entity
0.101715 PageLRU
0.101378 tcp_parse_aligned_timestamp
0.101219 __xchg
0.100544 constant_test_bit
0.097991 __kmalloc
0.097584 test_tsk_thread_flag
0.097475 autoremove_wake_function
0.095747 selinux_task_kill
0.094416 get_page
0.093353 dequeue_task
0.092728 __local_bh_disable
0.091943 selinux_netlbl_sock_rcv_skb
0.091655 path_put
0.090970 skb_headroom
0.090950 PageTail
0.090642 dst_destroy
0.090523 netpoll_rx
0.089589 skb_header_pointer
0.085935 security_socket_recvmsg
0.084008 alloc_pages_current
0.083184 compare_ether_addr
0.082479 rb_next
0.082439 sk_wmem_schedule
0.081635 next_zones_zonelist
0.080135 tcp_cwnd_validate
0.079877 tcp_event_new_data_sent
0.079817 fcheck_files
0.079082 ip_skb_dst_mtu
0.078804 ip_finish_output
0.078278 wakeup_preempt_entity
0.077026 sel_netif_find
0.076788 __skb_queue_tail
0.076570 sock_flag
0.076520 tcp_win_from_space
0.076510 zone_watermark_ok
0.076282 sel_netnode_sid
0.076162 policy_zonelist
0.074732 __wake_up_common
0.074613 compound_head
0.074593 task_has_perm
0.073243 __find_general_cachep
0.073064 tcp_push
0.072925 skb_cloned
0.072309 pskb_may_pull
0.071852 TCP_ECN_check_ce
0.071495 cap_task_to_inode
0.070770 default_wake_function
0.069429 xfrm4_policy_check
0.069091 tcp_parse_md5sig_option
0.068287 tcp_v4_md5_do_lookup
0.068059 tcp_v4_tw_remember_stamp
0.067344 tcp_ca_event
0.067125 tcp_ca_event
0.065457 place_entity
0.065318 write_seqlock
0.065089 device_not_available
0.065069 test_ti_thread_flag
0.063878 tcp_set_skb_tso_segs
0.063550 selinux_netlbl_inode_permission
0.063391 sock_wfree
0.063311 prepare_to_wait
0.058872 pid_vnr
0.058803 __cycles_2_ns
0.057631 ip_local_out
0.057333 tcp_ack_saw_tstamp
0.056896 copy_to_user
0.056628 set_bit
0.055913 free_pages_check
0.054969 tcp_rcv_rtt_measure_ts
0.053797 init_rootdomain
0.053708 selinux_socket_recvmsg
0.053698 pid_nr_ns
0.053629 sk_eat_skb
0.052814 _local_bh_enable
0.052645 nf_hook_thresh
0.052516 sched_info_queued
0.052457 enqueue_task
0.052228 sk_filter
0.052159 __cpu_clear
0.051980 local_bh_enable_ip
0.050292 update_rq_clock
0.048981 task_tgid_vnr
0.048881 copy_from_user
0.048782 tcp_parse_options
0.048484 lock_sock
0.047779 net_timestamp
0.047044 open_softirq
0.046955 tcp_win_from_space
0.045981 __skb_dequeue
0.043846 getboottime
0.043777 account_group_exec_runtime
0.043519 can_checksum_protocol
0.043469 set_user_nice
0.042784 skb_fill_page_desc
0.042247 security_socket_sendmsg
0.041989 read_profile
0.041930 tcp_validate_incoming
0.041612 check_preempt_curr
0.041413 skb_pull
0.041026 generic_smp_call_function_interrupt
0.041016 calc_delta_fair
0.040936 clear_buddies
0.040768 tcp_data_queue
0.040698 page_count
0.039695 lock_sock
0.039099 skb_headroom
0.038851 system_call_fastpath
0.038622 zone_statistics
0.037500 tcp_sack_extend
0.037381 __kmalloc_node
0.036587 first_zones_zonelist
0.036497 mntput
0.036179 pick_next_task
0.035991 kmap
0.035911 sock_put
0.035613 deactivate_task
0.035027 __nr_to_section
0.033985 page_zone
0.033190 native_load_tls
0.032882 netif_tx_queue_stopped
0.032713 __skb_insert
0.032187 sock_flag
0.031988 check_kill_permission
0.031790 policy_nodemask
0.031621 detach_timer
0.030558 inet_csk_clear_xmit_timer
0.030469 task_rq_unlock
0.029883 tcp_nagle_test
0.029744 tracesys
0.028383 virt_to_slab
0.028115 tcp_v4_check
0.028046 __cpu_set
0.027658 page_get_cache
0.027063 tcp_store_ts_recent
0.027053 __skb_pull
0.026953 gfp_zone
0.026586 sock_rcvlowat
0.026576 csum_partial
0.026397 init_waitqueue_head
0.026109 finish_wait
0.026040 kill_pid_info
0.025404 tcp_full_space
0.024888 __skb_queue_before
0.024550 dst_confirm
0.022603 inet_ehash_bucket
0.021888 activate_task
0.021650 tcp_rto_min
0.021283 d_callback
0.020965 signal_pending
0.020925 avc_node_free
0.020915 empty_bucket
0.020746 group_send_sig_info
0.020657 skb_reset_transport_header
0.020061 sock_put
0.019992 signal_pending_state
0.019684 tcp_sync_mss
0.019346 skb_network_offset
0.019276 skb_split
0.018988 tcp_adjust_fackets_out
0.018204 tcp_fast_path_check
0.017727 __skb_unlink
0.017687 napi_disable_pending
0.017678 sg_set_page
0.017022 get_pageblock_bitmap
0.016972 tcp_cong_avoid
0.016962 pid_task
0.016754 skb_set_tail_pointer
0.016039 selinux_ipv4_postroute
0.015930 idle_cpu
0.015632 skb_reset_network_header
0.015552 __count_vm_events
0.015483 source_load
0.014867 __skb_unlink
0.014738 skb_reset_transport_header
0.014599 set_bit
0.014241 audit_zero_context
0.014231 zone_page_state
0.014152 clear_bit
0.013874 PageSlab
0.013546 __memset
0.013238 get_pageblock_migratetype
0.012623 __rb_rotate_right
0.012543 kmem_find_general_cachep
0.012414 __kprobes_text_start
0.012344 security_sock_rcv_skb
0.012344 node_zonelist
0.012335 dnotify_parent
0.012096 skb_headroom
0.011778 tcp_push_one
0.011540 mnt_want_write
0.011143 kmalloc
0.011073 retint_swapgs
0.010954 __rb_rotate_left
0.010805 check_pgd_range
0.010785 tcp_mss_split_point
0.010755 migrate_timer_list
0.010338 __send_IPI_dest_field
0.010229 reschedule_interrupt
0.010179 sock_flag
0.009882 smp_call_function_mask
0.009673 test_tsk_need_resched
0.009564 tcp_urg
0.009504 generic_file_aio_read
0.009176 PageReserved
0.009147 net_invalid_timestamp
0.009087 __node_set
0.008749 do_tcp_setsockopt
0.008730 set_tsk_thread_flag
0.008720 tcp_enter_loss
0.008422 sock_error
0.008362 target_load
0.008302 crypto_hash_update
0.008104 PageReadahead
0.008044 tcp_poll
0.007915 tcp_checksum_complete
0.007329 tcp_snd_test
0.007309 selinux_file_permission
0.007290 sel_netif_destroy
0.007220 put_pages_list
0.006992 dst_output
0.006743 prepare_to_copy
0.006694 tcp_init_cwnd
0.006555 clear_bit
0.006535 set_bit
0.006425 normal_prio
0.006366 msleep
0.006346 error_sti
0.006336 tcp_rcv_rtt_update
0.006167 tcp_send_ack
0.005989 tcp_init_nondata_skb
0.005720 kfree_skb
0.005502 call_function_interrupt
0.005413 __count_vm_event
0.005403 __skb_checksum_complete_head
0.005363 page_cache_get_speculative
0.005323 dev_kfree_skb_irq
0.005174 skb_store_bits
0.004956 cpu_avg_load_per_task
0.004916 dev_cpu_callback
0.004807 __kmem_cache_destroy
0.004777 tcp_init_metrics
0.004777 io_schedule
0.004777 find_get_page
0.004707 eth_header_parse
0.004688 cap_task_kill
0.004678 error_exit
0.004668 rb_prev
0.004658 tso_fragment
0.004648 mmdrop
0.004628 skb_reset_tail_pointer
0.004598 apic_timer_interrupt
0.004588 clear_bit
0.004519 tcp_simple_retransmit
0.004449 get_max_files
0.004370 sk_stop_timer
0.004340 tcp_reset
0.004251 netlbl_cache_add
0.004201 tcp_add_reno_sack
0.004151 __pskb_trim_head
0.004102 __profile_flip_buffers
0.004092 sk_common_release
0.004052 audit_copy_inode
0.003953 eth_change_mtu
0.003943 vfs_read
0.003923 run_timer_softirq
0.003843 mnt_drop_write
0.003814 clear_page_c
0.003804 do_sync_read
0.003744 unset_migratetype_isolate
0.003714 sk_stream_moderate_sndbuf
0.003545 tcp_try_rmem_schedule
0.003476 native_apic_mem_write
0.003466 sys_read
0.003446 skb_checksum
0.003436 timer_set_base
0.003426 security_task_kill
0.003416 __flow_cache_shrink
0.003406 __skb_checksum_complete
0.003277 alloc_skb
0.003267 physflat_send_IPI_mask
0.003218 skb_gso_ok
0.003178 constant_test_bit
0.003168 find_next_bit
0.003158 selinux_netlbl_skbuff_getsid
0.003118 constant_test_bit
0.003099 pull_task
0.003079 hrtimer_run_queues
0.003049 free_hot_page
0.003009 scheduler_tick
0.002900 set_32bit_tls
0.002890 tcp_acceptable_seq
0.002811 rw_verify_area
0.002751 radix_tree_lookup_slot
0.002731 zero_user_segment
0.002731 sock_common_setsockopt
0.002612 __load_balance_iterator
0.002473 run_posix_cpu_timers
0.002264 task_utime
0.002254 switched_to_fair
0.002185 fsnotify_access
0.002145 __rmqueue_smallest
0.002125 __schedule_bug
0.002095 __task_rq_lock
0.002086 tcp_may_update_window
0.002076 restore_args
0.002066 hrtimer_run_pending
0.002056 generic_segment_checks
0.002026 getnstimeofday
0.002006 idle_task
0.001976 touch_atime
0.001956 __wake_up_locked
0.001927 sk_mem_charge
0.001877 smp_apic_timer_interrupt
0.001827 native_smp_send_reschedule
0.001798 __tcp_fast_path_on
0.001788 file_read_actor
0.001768 _cond_resched
0.001738 avc_policy_seqno
0.001718 tcp_ack_snd_check
0.001629 ip_send_check
0.001619 account_system_time
0.001579 __xapic_wait_icr_idle
0.001579 get_stats
0.001539 tcp_set_state
0.001539 bictcp_state
0.001529 tcp_fast_path_on
0.001519 file_accessed
0.001480 get_seconds
0.001450 kernel_math_error
0.001410 ktime_set
0.001331 kmap_atomic
0.001281 printk_tick
0.001281 __next_cpu_nr
0.001271 account_group_system_time
0.001261 __mod_zone_page_state
0.001222 weighted_cpuload
0.001192 security_file_permission
0.001162 ack_APIC_irq
0.001152 __free_one_page
0.001142 rcu_pending
0.001142 drain_array
0.001122 sched_clock_tick
0.001122 csum_fold
0.001102 ret_from_intr
0.001083 retint_careful
0.001073 need_resched
0.001073 calc_delta_mine
0.001043 tcp_v4_md5_do_del
0.001043 PageActive
0.001033 mark_page_accessed
0.001033 ktime_get_ts
0.001023 tcp_insert_write_queue_after
0.001013 tcp_delack_timer
0.001013 task_tick_fair
0.000973 delay_tsc
0.000963 nv_nic_irq_optimized
0.000904 tick_periodic
0.000894 skb_reserve
0.000884 cache_reap
0.000874 timespec_trunc
0.000864 skb_header_release
0.000854 zone_page_state_add
0.000844 update_process_times
0.000834 sk_rmem_schedule
0.000824 find_busiest_group
0.000804 current_fs_time
0.000785 tick_handle_periodic
0.000785 __sk_mem_schedule
0.000785 irq_enter
0.000755 use_cpu_writer_for_mount
0.000755 tcp_ratehalving_spur_to_response
0.000745 update_wall_time
0.000745 tcp_sendpage
0.000745 __alloc_pages_nodemask
0.000725 ktime_get
0.000725 irq_exit
0.000705 inotify_inode_queue_event
0.000665 set_pageblock_flags_group
0.000646 inotify_dentry_parent_queue_event
0.000626 ack_APIC_irq
0.000606 write_profile
0.000566 set_normalized_timespec
0.000566 raise_softirq
0.000526 task_cputime_zero
0.000516 smp_reschedule_interrupt
0.000516 __skb_insert
0.000497 page_fault
0.000497 __copy_user_nocache
0.000487 run_local_timers
0.000487 read_tsc
0.000487 nf_unregister_hook
0.000477 __rcu_pending
0.000477 jiffies_to_usecs
0.000457 timespec_to_ktime
0.000437 __skb_trim
0.000427 __call_rcu
0.000417 free_pages_bulk
0.000407 smp_call_function_interrupt
0.000397 set_irq_regs
0.000397 radix_tree_deref_slot
0.000397 expand
0.000387 handle_mm_fault
0.000387 handle_IRQ_event
0.000387 fput_light
0.000377 refresh_cpu_vm_stats
0.000377 n_tty_write
0.000367 get_page
0.000358 run_rebalance_domains
0.000358 get_cpu_mask
0.000348 task_hot
0.000348 __skb_queue_after
0.000348 retint_check
0.000348 do_select
0.000338 PageUptodate
0.000338 copy_page_c
0.000328 cond_resched
0.000318 unmap_vmas
0.000318 sk_mem_reclaim
0.000318 rmqueue_bulk
0.000318 reciprocal_value
0.000318 irq_return
0.000308 rb_first
0.000308 alloc_skb
0.000308 account_process_tick
0.000298 net_enable_timestamp
0.000298 clocksource_read
0.000298 account_system_time_scaled
0.000288 sched_slice
0.000278 ip_compute_csum
0.000278 constant_test_bit
0.000278 constant_test_bit
0.000268 set_curr_task_fair
0.000268 note_interrupt
0.000268 exit_idle
0.000258 native_apic_mem_write
0.000258 exit_intr
0.000248 PageReferenced
0.000238 usb_hcd_irq
0.000238 __mnt_is_readonly
0.000238 constant_test_bit
0.000218 IRQ0xba_interrupt
0.000218 handle_fasteoi_irq
0.000209 raise_softirq_irqoff
0.000209 __find_get_block
0.000199 tcp_current_ssthresh
0.000199 n_tty_receive_buf
0.000189 wake_up_page
0.000189 vgacon_save_screen
0.000189 free_block
0.000189 constant_test_bit
0.000179 pagefault_disable
0.000169 clocksource_get_next
0.000169 __bitmap_weight
0.000159 tty_ldisc_deref
0.000159 tcp_write_timer
0.000159 kmem_cache_alloc
0.000159 free_alien_cache
0.000159 ext3_mark_iloc_dirty
0.000159 constant_test_bit
0.000159 __bitmap_equal
0.000149 transfer_objects
0.000149 __rcu_process_callbacks
0.000149 page_waitqueue
0.000149 constant_test_bit
0.000139 __rmqueue
0.000139 release_pages
0.000139 constant_test_bit
0.000129 __tcp_checksum_complete
0.000129 run_workqueue
0.000129 poll_freewait
0.000129 n_tty_read
0.000129 iommu_area_free
0.000129 generic_file_llseek
0.000129 __cpus_setall
0.000129 cond_resched_softirq
0.000129 avc_node_populate
0.000129 add_to_page_cache_lru
0.000129 account_user_time
0.000119 wait_consider_task
0.000119 sys_select
0.000119 round_jiffies_common
0.000119 nv_start_xmit_optimized
0.000119 core_sys_select
0.000109 tcp_tso_segment
0.000109 sigprocmask
0.000109 proc_reg_read
0.000109 path_to_nameidata
0.000109 PageBuddy
0.000109 ohci_irq
0.000109 nv_tx_done_optimized
0.000109 nv_msi_workaround
0.000109 IRQ0xc2_interrupt
0.000109 __ext3_get_inode_loc
0.000109 account_group_user_time
0.000099 __wake_up_sync
0.000099 __up_read
0.000099 update_vsyscall
0.000099 memmove
0.000099 kmalloc
0.000099 ext3_get_blocks_handle
0.000099 do_device_not_available
0.000099 constant_test_bit
0.000089 tcp_incr_quickack
0.000089 smp_send_reschedule
0.000089 remove_from_page_cache
0.000089 rcu_process_callbacks
0.000089 prepare_to_wait_exclusive
0.000089 pde_users_dec
0.000089 find_first_bit
0.000089 constant_test_bit
0.000089 common_interrupt
0.000089 add_wait_queue
0.000079 task_gtime
0.000079 sys_lseek
0.000079 start_this_handle
0.000079 schedule_hrtimeout_range
0.000079 __sched_fork
0.000079 journal_put_journal_head
0.000079 find_first_zero_bit
0.000079 do_syslog
0.000079 do_sync_write
0.000079 constant_test_bit
0.000079 ack_apic_level
0.000070 write_seqlock
0.000070 slab_get_obj
0.000070 remove_wait_queue
0.000070 pty_chars_in_buffer
0.000070 ____pagevec_lru_add
0.000070 lock_hrtimer_base
0.000070 kstat_incr_irqs_this_cpu
0.000070 journal_dirty_data
0.000070 journal_add_journal_head
0.000070 find_lock_page
0.000070 copy_from_read_buf
0.000070 bit_waitqueue
0.000070 alloc_page_vma
0.000060 vfs_write
0.000060 tty_write
0.000060 __strnlen_user
0.000060 sk_mem_uncharge
0.000060 rt_worker_func
0.000060 radix_tree_preload
0.000060 poll_select_copy_remaining
0.000060 pagefault_enable
0.000060 __mark_inode_dirty
0.000060 lru_add_drain_all
0.000060 lock_page
0.000060 list_replace_init
0.000060 journal_stop
0.000060 iowrite8
0.000060 hrtimer_forward
0.000060 gart_unmap_single
0.000060 find_vma
0.000060 __down_read_trylock
0.000060 do_page_fault
0.000060 do_IRQ
0.000060 create_empty_buffers
0.000060 constant_test_bit
0.000060 constant_test_bit
0.000060 alloc_iommu
0.000060 add_to_page_cache_locked
0.000050 zero_fd_set
0.000050 vsnprintf
0.000050 unlock_page
0.000050 tty_read
0.000050 tty_poll
0.000050 sock_poll
0.000050 sock_def_error_report
0.000050 set_wq_data
0.000050 rcu_check_callbacks
0.000050 radix_tree_node_rcu_free
0.000050 pipe_poll
0.000050 opost
0.000050 n_tty_chars_in_buffer
0.000050 __next_cpu
0.000050 mutex_trylock
0.000050 msecs_to_jiffies
0.000050 mempool_alloc_slab
0.000050 load_elf_binary
0.000050 __link_path_walk
0.000050 __journal_remove_journal_head
0.000050 journal_commit_transaction
0.000050 journal_cancel_revoke
0.000050 irq_complete_move
0.000050 irq_cfg
0.000050 fsnotify_modify
0.000050 __first_cpu
0.000050 file_update_time
0.000050 filemap_fault
0.000050 ext3_new_blocks
0.000050 ext3_mark_inode_dirty
0.000050 do_wp_page
0.000050 __do_fault
0.000050 buffer_dirty
0.000050 anon_vma_prepare
0.000040 yield
0.000040 wq_per_cpu
0.000040 walk_page_buffers
0.000040 __wake_up_bit
0.000040 vma_adjust
0.000040 tty_put_char
0.000040 tty_paranoia_check
0.000040 tcp_current_ssthresh
0.000040 sys_write
0.000040 sys_rt_sigprocmask
0.000040 sock_no_bind
0.000040 show_stat
0.000040 SetPageSwapBacked
0.000040 set_irq_regs
0.000040 set_buffer_write_io_error
0.000040 recalc_sigpending
0.000040 radix_tree_delete
0.000040 queue_delayed_work_on
0.000040 pty_write
0.000040 __pollwait
0.000040 physflat_send_IPI_allbutself
0.000040 page_zone
0.000040 page_remove_rmap
0.000040 page_is_file_cache
0.000040 page_evictable
0.000040 nv_get_empty_tx_slots
0.000040 n_tty_poll
0.000040 next_zone
0.000040 next_online_pgdat
0.000040 need_resched
0.000040 mutex_unlock
0.000040 mpol_needs_cond_ref
0.000040 __lookup
0.000040 journal_invalidatepage
0.000040 journal_dirty_metadata
0.000040 ioread8
0.000040 input_available_p
0.000040 inet_csk_reset_xmit_timer
0.000040 get_fd_set
0.000040 generic_write_checks
0.000040 free_poll_entry
0.000040 fput
0.000040 __ext3_journal_stop
0.000040 ext3_get_group_desc
0.000040 ext3_get_block
0.000040 do_mpage_readpage
0.000040 __d_lookup
0.000040 del_page_from_lru
0.000040 __dec_zone_state
0.000040 copy_user_generic
0.000040 __bitmap_and
0.000040 add_page_to_lru_list
0.000040 account_user_time_scaled
0.000040 account_steal_time
0.000030 worker_thread
0.000030 wake_up_bit
0.000030 vmstat_update
0.000030 vm_normal_page
0.000030 tty_write_unlock
0.000030 tty_write_lock
0.000030 tty_wakeup
0.000030 tty_ldisc_try
0.000030 tty_ioctl
0.000030 tag_get
0.000030 sys_pread64
0.000030 submit_bh
0.000030 stop_this_cpu
0.000030 sock_aio_write
0.000030 sk_mem_reclaim
0.000030 sk_backlog_rcv
0.000030 show_interrupts
0.000030 sg_next
0.000030 seq_printf
0.000030 send_remote_softirq
0.000030 remove_vma
0.000030 reg_delay
0.000030 radix_tree_lookup
0.000030 radix_tree_insert
0.000030 proc_lookup_de
0.000030 pipe_write
0.000030 __percpu_counter_add
0.000030 pci_map_single
0.000030 nv_napi_poll
0.000030 __next_node
0.000030 native_send_call_func_ipi
0.000030 mpage_readpages
0.000030 mix_pool_bytes_extract
0.000030 mii_rw
0.000030 mempool_alloc
0.000030 __make_request
0.000030 jbd_lock_bh_state
0.000030 iov_iter_copy_from_user_atomic
0.000030 insert_work
0.000030 hrtimer_try_to_cancel
0.000030 get_dma_ops
0.000030 __generic_file_aio_write_nolock
0.000030 gart_map_sg
0.000030 __fput
0.000030 fixup_irqs
0.000030 __find_get_block_slow
0.000030 filp_close
0.000030 ext3_get_branch
0.000030 ext3_dirty_inode
0.000030 ext3_block_to_path
0.000030 do_get_write_access
0.000030 delayed_work_timer_fn
0.000030 csum_block_add
0.000030 copy_process
0.000030 copy_page_range
0.000030 constant_test_bit
0.000030 constant_test_bit
0.000030 check_irqs_on
0.000030 call_rcu
0.000030 __brelse
0.000030 _atomic_dec_and_lock
0.000020 __xchg
0.000020 vm_stat_account
0.000020 vma_prio_tree_remove
0.000020 tty_mode_ioctl
0.000020 tty_audit_add_data
0.000020 try_to_free_buffers
0.000020 truncate_inode_pages_range
0.000020 tcp_slow_start
0.000020 task_curr
0.000020 sys_setpgid
0.000020 sys_rt_sigreturn
0.000020 sys_getppid
0.000020 strncpy_from_user
0.000020 sock_put
0.000020 smp_call_function
0.000020 __sk_mem_reclaim
0.000020 signal_wake_up
0.000020 signal_pending
0.000020 set_termios
0.000020 SetPageUptodate
0.000020 SetPageLRU
0.000020 set_fd_set
0.000020 set_bit
0.000020 __send_IPI_shortcut
0.000020 security_inode_need_killpriv
0.000020 scsi_request_fn
0.000020 sb_bread
0.000020 restore_i387_xstate
0.000020 __qdisc_run
0.000020 pud_alloc
0.000020 pmd_alloc
0.000020 pfn_pte
0.000020 pfifo_fast_enqueue
0.000020 pfifo_fast_dequeue
0.000020 pci_map_page
0.000020 path_get
0.000020 __pagevec_free
0.000020 pagevec_add
0.000020 PageUnevictable
0.000020 page_mapping
0.000020 nv_get_hw_stats
0.000020 number
0.000020 normalize_rt_tasks
0.000020 __netif_tx_lock
0.000020 mk_pid
0.000020 memscan
0.000020 memcpy_c
0.000020 __lru_cache_add
0.000020 __lookup_mnt
0.000020 load_balance_rt
0.000020 kthread_should_stop
0.000020 journal_start
0.000020 journal_remove_journal_head
0.000020 __journal_file_buffer
0.000020 jbd_unlock_bh_journal_head
0.000020 itimer_get_remtime
0.000020 irq_to_desc
0.000020 iowrite32
0.000020 inotify_remove_watch_locked
0.000020 inode_permission
0.000020 inode_has_perm
0.000020 init_timer
0.000020 goal_in_my_reservation
0.000020 get_vma_policy
0.000020 __get_free_pages
0.000020 generic_sync_sb_inodes
0.000020 gart_map_single
0.000020 freezing
0.000020 free_pgtables
0.000020 free_pages_and_swap_cache
0.000020 free_buffer_head
0.000020 __follow_mount
0.000020 flush_tlb_page
0.000020 find_busiest_queue
0.000020 file_has_perm
0.000020 ext3_try_to_allocate
0.000020 ext3_journal_start
0.000020 __ext3_journal_dirty_metadata
0.000020 ext3_file_write
0.000020 enqueue_hrtimer
0.000020 dup_mm
0.000020 do_wait
0.000020 do_vfs_ioctl
0.000020 do_path_lookup
0.000020 do_munmap
0.000020 do_machine_check
0.000020 do_lookup
0.000020 do_follow_link
0.000020 dma_unmap_single
0.000020 __dec_zone_page_state
0.000020 count_vm_event
0.000020 constant_test_bit
0.000020 constant_test_bit
0.000020 compound_head
0.000020 clear_buffer_jbddirty
0.000020 clear_buffer_delay
0.000020 claim_block
0.000020 cascade
0.000020 cancel_dirty_page
0.000020 cache_grow
0.000020 brelse
0.000020 __block_prepare_write
0.000020 __blocking_notifier_call_chain
0.000020 blk_rq_map_sg
0.000020 __bitmap_empty
0.000020 __bitmap_andnot
0.000020 anon_vma_unlink
0.000010 zone_page_state
0.000010 zero_user_segments
0.000010 __xchg
0.000010 __vma_link_rb
0.000010 vma_link
0.000010 vfs_llseek
0.000010 __up_write
0.000010 update_xtime_cache
0.000010 unmap_underlying_metadata
0.000010 unmap_region
0.000010 unix_poll
0.000010 tty_write_room
0.000010 tty_unthrottle
0.000010 tty_ldisc_ref_wait
0.000010 tty_ldisc_ref
0.000010 tty_fasync
0.000010 tty_check_change
0.000010 tty_chars_in_buffer
0.000010 tty_audit_fork
0.000010 truncate_complete_page
0.000010 test_tsk_thread_flag
0.000010 taskstats_exit
0.000010 sys_writev
0.000010 sys_readahead
0.000010 sys_poll
0.000010 sys_newstat
0.000010 sys_nanosleep
0.000010 sys_ioctl
0.000010 syscall_trace_leave
0.000010 sync_supers
0.000010 stub_execve
0.000010 split_page
0.000010 sock_kfree_s
0.000010 __sleep_on_page_lock
0.000010 skip_atoi
0.000010 signal_pending
0.000010 signal_pending
0.000010 sg_init_table
0.000010 set_task_cpu
0.000010 __set_page_dirty
0.000010 SetPageActive
0.000010 set_bit
0.000010 seq_puts
0.000010 selinux_task_setpgid
0.000010 selinux_secctx_to_secid
0.000010 selinux_sb_show_options
0.000010 selinux_inode_permission
0.000010 selinux_inode_need_killpriv
0.000010 selinux_inode_free_security
0.000010 selinux_inode_alloc_security
0.000010 selinux_d_instantiate
0.000010 security_vm_enough_memory
0.000010 second_overflow
0.000010 scsi_run_queue
0.000010 __scsi_put_command
0.000010 scsi_init_sgtable
0.000010 scsi_end_request
0.000010 schedule_tail
0.000010 schedule_delayed_work
0.000010 sb_any_quota_enabled
0.000010 rt_hash
0.000010 round_jiffies_relative
0.000010 remove_hrtimer
0.000010 __remove_hrtimer
0.000010 __remove_from_page_cache
0.000010 rcu_bh_qsctr_inc
0.000010 radix_tree_tag_clear
0.000010 radix_tree_gang_lookup_tag_slot
0.000010 radix_tree_gang_lookup_slot
0.000010 queue_delayed_work
0.000010 qdisc_run
0.000010 put_tty_queue_nolock
0.000010 put_io_context
0.000010 pty_write_room
0.000010 pty_open
0.000010 ptep_set_access_flags
0.000010 profile_munmap
0.000010 proc_pident_lookup
0.000010 proc_get_inode
0.000010 prio_tree_replace
0.000010 prio_tree_remove
0.000010 prio_tree_insert
0.000010 pmd_none_or_clear_bad
0.000010 pipe_release
0.000010 pipe_read
0.000010 pid_revalidate
0.000010 pgd_alloc
0.000010 pci_unmap_single
0.000010 pci_read_config_dword
0.000010 pci_conf1_write
0.000010 pci_bus_read_config_dword
0.000010 path_walk
0.000010 page_zone
0.000010 PageSwapCache
0.000010 PageSwapCache
0.000010 PageSwapCache
0.000010 __page_set_anon_rmap
0.000010 PagePrivate
0.000010 PagePrivate
0.000010 PagePrivate
0.000010 page_add_file_rmap
0.000010 on_each_cpu
0.000010 nv_do_interrupt
0.000010 net_tx_action
0.000010 netif_start_queue
0.000010 netif_carrier_ok
0.000010 need_resched
0.000010 need_iommu
0.000010 native_pte_clear
0.000010 native_io_delay
0.000010 mutex_lock
0.000010 mprotect_fixup
0.000010 mod_zone_page_state
0.000010 mntput_no_expire
0.000010 mm_init
0.000010 mmap_region
0.000010 mempool_free
0.000010 memcmp
0.000010 mcheck_check_cpu
0.000010 may_open
0.000010 __lookup_tag
0.000010 locks_remove_posix
0.000010 locks_remove_flock
0.000010 lock_buffer
0.000010 load_elf_binary
0.000010 load_balance_fair
0.000010 ll_back_merge_fn
0.000010 kzalloc
0.000010 ktime_add_safe
0.000010 kill_fasync
0.000010 __journal_temp_unlink_buffer
0.000010 journal_switch_revoke_table
0.000010 __journal_remove_checkpoint
0.000010 journal_get_write_access
0.000010 journal_get_undo_access
0.000010 journal_get_descriptor_buffer
0.000010 journal_bmap
0.000010 jbd_unlock_bh_state
0.000010 jbd_unlock_bh_state
0.000010 IRQ0xd2_interrupt
0.000010 ip_append_data
0.000010 iov_iter_advance
0.000010 iov_fault_in_pages_read
0.000010 iommu_area_alloc
0.000010 inode_sub_bytes
0.000010 inode_doinit_with_dentry
0.000010 inode_add_bytes
0.000010 __inc_zone_page_state
0.000010 inc_zone_page_state
0.000010 hweight_long
0.000010 hweight64
0.000010 hrtimer_wakeup
0.000010 hrtimer_init
0.000010 hash_64
0.000010 half_md4_transform
0.000010 __grab_cache_page
0.000010 get_user_pages
0.000010 get_signal_to_deliver
0.000010 get_random_int
0.000010 getname
0.000010 get_empty_filp
0.000010 __getblk
0.000010 generic_permission
0.000010 generic_make_request
0.000010 generic_fillattr
0.000010 generic_file_open
0.000010 generic_file_llseek_unlocked
0.000010 generic_file_buffered_write
0.000010 generic_file_aio_write
0.000010 generic_cont_expand_simple
0.000010 generic_block_bmap
0.000010 freezing
0.000010 free_swap_cache
0.000010 free_pid
0.000010 free_pgd_range
0.000010 free_pages
0.000010 flush_old_exec
0.000010 first_online_pgdat
0.000010 find_vma_prepare
0.000010 find_task_by_pid_type_ns
0.000010 find_next_zero_bit
0.000010 find_inode_fast
0.000010 file_remove_suid
0.000010 file_mask_to_av
0.000010 file_free_rcu
0.000010 __FD_CLR
0.000010 ext3_write_begin
0.000010 ext3_try_to_allocate_with_rsv
0.000010 ext3_ordered_write_end
0.000010 ext3_journalled_set_page_dirty
0.000010 ext3_invalidatepage
0.000010 ext3_iget_acl
0.000010 ext3_get_inode_flags
0.000010 ext3_free_data
0.000010 ext3_discard_reservation
0.000010 exit_thread
0.000010 exit_task_namespaces
0.000010 exit_sem
0.000010 end_that_request_last
0.000010 end_buffer_write_sync
0.000010 end_buffer_async_write
0.000010 elv_rb_del
0.000010 elv_queue_empty
0.000010 elv_merged_request
0.000010 elv_completed_request
0.000010 elf_map
0.000010 echo_char
0.000010 e1000_watchdog
0.000010 e1000_read_phy_reg
0.000010 __drain_alien_cache
0.000010 __d_path
0.000010 __down_write_nested
0.000010 __down_write
0.000010 double_rq_lock
0.000010 do_timer
0.000010 do_sys_open
0.000010 do_sigaltstack
0.000010 do_sigaction
0.000010 do_setitimer
0.000010 do_pipe_flags
0.000010 __do_page_cache_readahead
0.000010 do_notify_parent
0.000010 do_filp_open
0.000010 do_exit
0.000010 dnotify_flush
0.000010 d_kill
0.000010 destroy_inode
0.000010 dequeue_signal
0.000010 de_put
0.000010 delayacct_end
0.000010 create_write_pipe
0.000010 create_workqueue_thread
0.000010 __cpus_equal
0.000010 cpu_quiet
0.000010 __cpu_clear
0.000010 __cpu_clear
0.000010 count
0.000010 copy_thread
0.000010 copy_namespaces
0.000010 constant_test_bit
0.000010 constant_test_bit
0.000010 constant_test_bit
0.000010 constant_test_bit
0.000010 constant_test_bit
0.000010 __cond_resched
0.000010 clocksource_forward_now
0.000010 __clear_user
0.000010 clear_inode
0.000010 clear_buffer_new
0.000010 clear_bit
0.000010 clear_bit
0.000010 check_for_bios_corruption
0.000010 __cfq_slice_expired
0.000010 cfq_set_request
0.000010 cfq_dispatch_requests
0.000010 cfq_completed_request
0.000010 cap_set_effective
0.000010 can_share_swap_page
0.000010 bvec_alloc_bs
0.000010 buffer_uptodate
0.000010 buffer_mapped
0.000010 buffer_locked
0.000010 buffer_jbd
0.000010 buffer_jbd
0.000010 brelse
0.000010 __bread
0.000010 blk_invoke_request_fn
0.000010 __blk_complete_request
0.000010 blk_add_trace_generic
0.000010 blk_add_trace_bio
0.000010 bit_spin_lock
0.000010 bio_put
0.000010 bio_alloc_bioset
0.000010 bdi_read_congested
0.000010 balance_runtime
0.000010 balance_dirty_pages_ratelimited_nr
0.000010 audit_log_task_context
0.000010 ata_sff_qc_prep
0.000010 ata_scsi_queuecmd
0.000010 ata_link_max_devices
0.000010 ata_get_xlat_func
0.000010 arp_process
0.000010 arch_pick_mmap_layout
0.000010 arch_irq_stat_cpu
0.000010 arch_dup_task_struct
0.000010 alloc_pid
0.000010 alloc_fdtable
0.000010 alloc_fd
0.000010 add_mm_rss
0.000010 acct_collect
--
To unsubscribe from this list: send the line "unsubscribe kernel-testers" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
