Hi Roman,
I ran the test for ~24 hours.
In this time MemAvailable dropped from 31111424 bytes to 29223928 bytes
(So a drop of ~1.8GB)
At this time Percpu is occupying 1600512 Kb (~1.5GB)
Here are the percpu stats at the current time (I also collected them
throughout most of the test so can send that too if required)
->
thanks,
Dan McGinnes
IBM Cloud - Containers performance
Int Tel: 247359 Ext Tel: 01962 817359
Notes: Daniel McGinnes/UK/IBM
Email: MCGINNES@xxxxxxxxxx
IBM (UK) Ltd, Hursley Park,Winchester,Hampshire, SO21 2JN
From: Roman Gushchin <guro@xxxxxx>
To: Daniel McGinnes <MCGINNES@xxxxxxxxxx>
Cc: "cgroups@xxxxxxxxxxxxxxx" <cgroups@xxxxxxxxxxxxxxx>, Nathaniel
Rockwell <nrockwell@xxxxxxxxxx>
Date: 02/10/2018 12:07
Subject: Re: PROBLEM: Memory leaking when running kubernetes
cronjobs
Hi Daniel!
/sys/kernel/debug/percpu_stats
Thanks!
From: Daniel McGinnes <MCGINNES@xxxxxxxxxx>
Sent: Tuesday, October 2, 2018 3:52:32 AM
To: Roman Gushchin
Cc: cgroups@xxxxxxxxxxxxxxx; Nathaniel Rockwell
Subject: Re: PROBLEM: Memory leaking when running kubernetes cronjobs
Hi Roman,
I have set the config option, rebuilt and installed. Where can I find the
percpu allocator statistics? I'velooked in /proc and can't see anything
obvious..
thanks,
Dan McGinnes
IBM Cloud - Containers performance
Int Tel: 247359 Ext Tel: 01962 817359
Notes: Daniel McGinnes/UK/IBM
Email: MCGINNES@xxxxxxxxxx
IBM (UK) Ltd, Hursley Park,Winchester,Hampshire, SO21 2JN
From: Roman Gushchin <guro@xxxxxx>
To: Daniel McGinnes <MCGINNES@xxxxxxxxxx>
Cc: "cgroups@xxxxxxxxxxxxxxx" <cgroups@xxxxxxxxxxxxxxx>, Nathaniel
Rockwell <nrockwell@xxxxxxxxxx>
Date: 01/10/2018 12:14
Subject: Re: PROBLEM: Memory leaking when running kubernetes
cronjobs
Hi Daniel!
Can you, please, rebuild your kernel with the CONFIG_PERCPU_STATS config
option,
and post percpu allocator statistics after the test?
Sorry, I missed this option earlier.
Thanks!
From: Daniel McGinnes <MCGINNES@xxxxxxxxxx>
Sent: Monday, October 1, 2018 4:00:55 AM
To: Roman Gushchin
Cc: cgroups@xxxxxxxxxxxxxxx; Nathaniel Rockwell
Subject: Re: PROBLEM: Memory leaking when running kubernetes cronjobs
Hi Roman,
I let the test run over the weekend and Percpu memory has grown to 1.7GB
Here is the latest output from the modified memleak tool (after I did
drop_caches):
[09:56:31] Top 10 stacks with outstanding allocations:
17232 bytes in 2154 allocations from stack
pcpu_alloc+0x39f [kernel]
__alloc_percpu_gfp+0x12 [kernel]
fib6_info_alloc+0x47 [kernel]
ip6_route_info_create+0x25d [kernel]
ip6_route_add+0x1d [kernel]
addrconf_add_mroute+0xb5 [kernel]
addrconf_add_dev+0x4f [kernel]
addrconf_dev_config+0x83 [kernel]
addrconf_notify+0x104 [kernel]
notifier_call_chain+0x4c [kernel]
raw_notifier_call_chain+0x16 [kernel]
call_netdevice_notifiers_info+0x2d [kernel]
netdev_state_change+0x5a [kernel]
linkwatch_do_dev+0x38 [kernel]
__linkwatch_run_queue+0x10a [kernel]
linkwatch_event+0x25 [kernel]
process_one_work+0x1fd [kernel]
worker_thread+0x34 [kernel]
kthread+0x121 [kernel]
ret_from_fork+0x35 [kernel]
28536 bytes in 29 allocations from stack
pcpu_alloc+0x39f [kernel]
__alloc_percpu+0x15 [kernel]
mem_cgroup_css_alloc+0x75 [kernel]
cgroup_apply_control_enable+0x13e [kernel]
cgroup_mkdir+0x22b [kernel]
kernfs_iop_mkdir+0x5f [kernel]
vfs_mkdir+0x108 [kernel]
do_mkdirat+0xe8 [kernel]
__x64_sys_mkdir+0x1b [kernel]
do_syscall_64+0x5a [kernel]
entry_SYSCALL_64_after_hwframe+0x44 [kernel]
59584 bytes in 266 allocations from stack
pcpu_alloc+0x39f [kernel]
__alloc_percpu+0x15 [kernel]
mem_cgroup_css_alloc+0xa7 [kernel]
cgroup_apply_control_enable+0x13e [kernel]
cgroup_mkdir+0x22b [kernel]
kernfs_iop_mkdir+0x5f [kernel]
vfs_mkdir+0x108 [kernel]
do_mkdirat+0xe8 [kernel]
__x64_sys_mkdirat+0x1a [kernel]
do_syscall_64+0x5a [kernel]
entry_SYSCALL_64_after_hwframe+0x44 [kernel]
139232 bytes in 4351 allocations from stack
pcpu_alloc+0x39f [kernel]
__alloc_percpu+0x15 [kernel]
__kmem_cache_create+0x3dc [kernel]
create_cache+0xd2 [kernel]
memcg_create_kmem_cache+0x109 [kernel]
memcg_kmem_cache_create_func+0x20 [kernel]
process_one_work+0x1fd [kernel]
worker_thread+0x34 [kernel]
kthread+0x121 [kernel]
ret_from_fork+0x35 [kernel]
261744 bytes in 266 allocations from stack
pcpu_alloc+0x39f [kernel]
__alloc_percpu+0x15 [kernel]
mem_cgroup_css_alloc+0x75 [kernel]
cgroup_apply_control_enable+0x13e [kernel]
cgroup_mkdir+0x22b [kernel]
kernfs_iop_mkdir+0x5f [kernel]
vfs_mkdir+0x108 [kernel]
do_mkdirat+0xe8 [kernel]
__x64_sys_mkdirat+0x1a [kernel]
do_syscall_64+0x5a [kernel]
entry_SYSCALL_64_after_hwframe+0x44 [kernel]
279016 bytes in 34877 allocations from stack
pcpu_alloc+0x39f [kernel]
__alloc_percpu_gfp+0x12 [kernel]
fib6_info_alloc+0x47 [kernel]
addrconf_f6i_alloc+0x36 [kernel]
ipv6_add_addr+0x120 [kernel]
addrconf_add_linklocal+0x73 [kernel]
addrconf_addr_gen+0xfe [kernel]
addrconf_dev_config+0x9f [kernel]
addrconf_notify+0x104 [kernel]
notifier_call_chain+0x4c [kernel]
raw_notifier_call_chain+0x16 [kernel]
call_netdevice_notifiers_info+0x2d [kernel]
__dev_notify_flags+0x65 [kernel]
dev_change_flags+0x52 [kernel]
do_setlink+0x2eb [kernel]
rtnl_newlink+0x51e [kernel]
rtnetlink_rcv_msg+0x27c [kernel]
netlink_rcv_skb+0x54 [kernel]
rtnetlink_rcv+0x15 [kernel]
netlink_unicast+0x1ab [kernel]
netlink_sendmsg+0x2d1 [kernel]
sock_sendmsg+0x3e [kernel]
__sys_sendto+0x13f [kernel]
__x64_sys_sendto+0x28 [kernel]
do_syscall_64+0x5a [kernel]
entry_SYSCALL_64_after_hwframe+0x44 [kernel]
279016 bytes in 34877 allocations from stack
pcpu_alloc+0x39f [kernel]
__alloc_percpu_gfp+0x12 [kernel]
fib6_info_alloc+0x47 [kernel]
ip6_route_info_create+0x25d [kernel]
ip6_route_add+0x1d [kernel]
addrconf_add_mroute+0xb5 [kernel]
addrconf_add_dev+0x4f [kernel]
addrconf_dev_config+0x83 [kernel]
addrconf_notify+0x104 [kernel]
notifier_call_chain+0x4c [kernel]
raw_notifier_call_chain+0x16 [kernel]
call_netdevice_notifiers_info+0x2d [kernel]
__dev_notify_flags+0x65 [kernel]
dev_change_flags+0x52 [kernel]
do_setlink+0x2eb [kernel]
rtnl_newlink+0x51e [kernel]
rtnetlink_rcv_msg+0x27c [kernel]
netlink_rcv_skb+0x54 [kernel]
rtnetlink_rcv+0x15 [kernel]
netlink_unicast+0x1ab [kernel]
netlink_sendmsg+0x2d1 [kernel]
sock_sendmsg+0x3e [kernel]
__sys_sendto+0x13f [kernel]
__x64_sys_sendto+0x28 [kernel]
do_syscall_64+0x5a [kernel]
entry_SYSCALL_64_after_hwframe+0x44 [kernel]
279016 bytes in 34877 allocations from stack
pcpu_alloc+0x39f [kernel]
__alloc_percpu_gfp+0x12 [kernel]
fib6_info_alloc+0x47 [kernel]
ip6_route_info_create+0x25d [kernel]
ip6_route_add+0x1d [kernel]
addrconf_prefix_route+0x10d [kernel]
addrconf_add_linklocal+0x9a [kernel]
addrconf_addr_gen+0xfe [kernel]
addrconf_dev_config+0x9f [kernel]
addrconf_notify+0x104 [kernel]
notifier_call_chain+0x4c [kernel]
raw_notifier_call_chain+0x16 [kernel]
call_netdevice_notifiers_info+0x2d [kernel]
__dev_notify_flags+0x65 [kernel]
dev_change_flags+0x52 [kernel]
do_setlink+0x2eb [kernel]
rtnl_newlink+0x51e [kernel]
rtnetlink_rcv_msg+0x27c [kernel]
netlink_rcv_skb+0x54 [kernel]
rtnetlink_rcv+0x15 [kernel]
netlink_unicast+0x1ab [kernel]
netlink_sendmsg+0x2d1 [kernel]
sock_sendmsg+0x3e [kernel]
__sys_sendto+0x13f [kernel]
__x64_sys_sendto+0x28 [kernel]
do_syscall_64+0x5a [kernel]
entry_SYSCALL_64_after_hwframe+0x44 [kernel]
308248 bytes in 38531 allocations from stack
pcpu_alloc+0x39f [kernel]
__alloc_percpu_gfp+0x12 [kernel]
fib6_info_alloc+0x47 [kernel]
addrconf_f6i_alloc+0x36 [kernel]
ipv6_add_addr+0x120 [kernel]
add_addr+0x73 [kernel]
addrconf_notify+0x50f [kernel]
notifier_call_chain+0x4c [kernel]
raw_notifier_call_chain+0x16 [kernel]
call_netdevice_notifiers_info+0x2d [kernel]
__dev_notify_flags+0x65 [kernel]
dev_change_flags+0x52 [kernel]
do_setlink+0x2eb [kernel]
rtnl_newlink+0x51e [kernel]
rtnetlink_rcv_msg+0x27c [kernel]
netlink_rcv_skb+0x54 [kernel]
rtnetlink_rcv+0x15 [kernel]
netlink_unicast+0x1ab [kernel]
netlink_sendmsg+0x2d1 [kernel]
sock_sendmsg+0x3e [kernel]
__sys_sendto+0x13f [kernel]
__x64_sys_sendto+0x28 [kernel]
do_syscall_64+0x5a [kernel]
entry_SYSCALL_64_after_hwframe+0x44 [kernel]
1015808 bytes in 248 allocations from stack
pcpu_alloc+0x39f [kernel]
__alloc_percpu+0x15 [kernel]
xt_percpu_counter_alloc+0x68 [kernel]
find_check_entry.isra.7+0x55 [kernel]
translate_table+0x475 [kernel]
do_ipt_set_ctl+0x11a [kernel]
nf_setsockopt+0x4c [kernel]
ip_setsockopt+0x8a [kernel]
raw_setsockopt+0x34 [kernel]
sock_common_setsockopt+0x1a [kernel]
__sys_setsockopt+0x86 [kernel]
__x64_sys_setsockopt+0x24 [kernel]
do_syscall_64+0x5a [kernel]
entry_SYSCALL_64_after_hwframe+0x44 [kernel]
Although the stack with "1015808 bytes in 248 allocations from stack" is
the largest this does not seem to have changed significantly since the
start of the test - where it showed:
1019904 bytes in 249 allocations from stack
pcpu_alloc+0x39f [kernel]
__alloc_percpu+0x15 [kernel]
xt_percpu_counter_alloc+0x68 [kernel]
find_check_entry.isra.7+0x55 [kernel]
translate_table+0x475 [kernel]
do_ipt_set_ctl+0x11a [kernel]
nf_setsockopt+0x4c [kernel]
ip_setsockopt+0x8a [kernel]
raw_setsockopt+0x34 [kernel]
sock_common_setsockopt+0x1a [kernel]
__sys_setsockopt+0x86 [kernel]
__x64_sys_setsockopt+0x24 [kernel]
do_syscall_64+0x5a [kernel]
entry_SYSCALL_64_after_hwframe+0x44 [kernel]
The ones that look maybe interesting are the ones that look related to
ipv6 - e.g.:
279016 bytes in 34877 allocations from stack
pcpu_alloc+0x39f [kernel]
__alloc_percpu_gfp+0x12 [kernel]
fib6_info_alloc+0x47 [kernel]
addrconf_f6i_alloc+0x36 [kernel]
ipv6_add_addr+0x120 [kernel]
addrconf_add_linklocal+0x73 [kernel]
addrconf_addr_gen+0xfe [kernel]
addrconf_dev_config+0x9f [kernel]
addrconf_notify+0x104 [kernel]
notifier_call_chain+0x4c [kernel]
raw_notifier_call_chain+0x16 [kernel]
call_netdevice_notifiers_info+0x2d [kernel]
__dev_notify_flags+0x65 [kernel]
dev_change_flags+0x52 [kernel]
do_setlink+0x2eb [kernel]
rtnl_newlink+0x51e [kernel]
rtnetlink_rcv_msg+0x27c [kernel]
netlink_rcv_skb+0x54 [kernel]
rtnetlink_rcv+0x15 [kernel]
netlink_unicast+0x1ab [kernel]
netlink_sendmsg+0x2d1 [kernel]
sock_sendmsg+0x3e [kernel]
__sys_sendto+0x13f [kernel]
__x64_sys_sendto+0x28 [kernel]
do_syscall_64+0x5a [kernel]
entry_SYSCALL_64_after_hwframe+0x44 [kernel]
These show a steady climb in both the number of bytes & number of
allocations throughout the test.. The one thing I don't quite understand
is that the amount of memory seems really small - only ~270KB - which
doesn't seem to tie up with the fact Percpu memory has increased to 1.7GB
(From about 300MB at the start of the test).
Here is the full output from meleak in case it's of use ->
Dan McGinnes
IBM Cloud - Containers performance
Int Tel: 247359 Ext Tel: 01962 817359
Notes: Daniel McGinnes/UK/IBM
Email: MCGINNES@xxxxxxxxxx
IBM (UK) Ltd, Hursley Park,Winchester,Hampshire, SO21 2JN
From: Roman Gushchin <guro@xxxxxx>
To: Daniel McGinnes <MCGINNES@xxxxxxxxxx>
Cc: "cgroups@xxxxxxxxxxxxxxx" <cgroups@xxxxxxxxxxxxxxx>, Nathaniel
Rockwell <nrockwell@xxxxxxxxxx>
Date: 27/09/2018 18:37
Subject: Re: PROBLEM: Memory leaking when running kubernetes
cronjobs
On Thu, Sep 27, 2018 at 05:13:43PM +0100, Daniel McGinnes wrote:
> Hi Roman,
>
> that seems to be working. I've kicked off a test collecting memleak info
> with a 10 min interval.
>
> Here is example output it is showing:
>
> [16:03:07] Top 10 stacks with outstanding allocations:
> 1256 bytes in 157 allocations from stack
> pcpu_alloc+0x39f [kernel]
> __alloc_percpu_gfp+0x12 [kernel]
> fib6_info_alloc+0x47 [kernel]
> ip6_route_info_create+0x25d [kernel]
> ip6_route_add+0x1d [kernel]
> addrconf_prefix_route+0x10d [kernel]
> addrconf_add_linklocal+0x9a [kernel]
> addrconf_addr_gen+0xfe [kernel]
> addrconf_dev_config+0x9f [kernel]
> addrconf_notify+0x104 [kernel]
> notifier_call_chain+0x4c [kernel]
> raw_notifier_call_chain+0x16 [kernel]
> call_netdevice_notifiers_info+0x2d [kernel]
> __dev_notify_flags+0x65 [kernel]
> dev_change_flags+0x52 [kernel]
> do_setlink+0x2eb [kernel]
> rtnl_newlink+0x51e [kernel]
> rtnetlink_rcv_msg+0x27c [kernel]
> netlink_rcv_skb+0x54 [kernel]
> rtnetlink_rcv+0x15 [kernel]
> netlink_unicast+0x1ab [kernel]
> netlink_sendmsg+0x2d1 [kernel]
> sock_sendmsg+0x3e [kernel]
> __sys_sendto+0x13f [kernel]
> __x64_sys_sendto+0x28 [kernel]
> do_syscall_64+0x5a [kernel]
> entry_SYSCALL_64_after_hwframe+0x44 [kernel]
> 1256 bytes in 157 allocations from stack
> pcpu_alloc+0x39f [kernel]
> __alloc_percpu_gfp+0x12 [kernel]
> fib6_info_alloc+0x47 [kernel]
> ip6_route_info_create+0x25d [kernel]
> ip6_route_add+0x1d [kernel]
> addrconf_add_mroute+0xb5 [kernel]
> addrconf_add_dev+0x4f [kernel]
> addrconf_dev_config+0x83 [kernel]
> addrconf_notify+0x104 [kernel]
> notifier_call_chain+0x4c [kernel]
> raw_notifier_call_chain+0x16 [kernel]
> call_netdevice_notifiers_info+0x2d [kernel]
> __dev_notify_flags+0x65 [kernel]
> dev_change_flags+0x52 [kernel]
> do_setlink+0x2eb [kernel]
> rtnl_newlink+0x51e [kernel]
> rtnetlink_rcv_msg+0x27c [kernel]
> netlink_rcv_skb+0x54 [kernel]
> rtnetlink_rcv+0x15 [kernel]
> netlink_unicast+0x1ab [kernel]
> netlink_sendmsg+0x2d1 [kernel]
> sock_sendmsg+0x3e [kernel]
> __sys_sendto+0x13f [kernel]
> __x64_sys_sendto+0x28 [kernel]
> do_syscall_64+0x5a [kernel]
> entry_SYSCALL_64_after_hwframe+0x44 [kernel]
> 1260 bytes in 315 allocations from stack
> pcpu_alloc+0x39f [kernel]
> __alloc_percpu_gfp+0x12 [kernel]
> __percpu_counter_init+0x23 [kernel]
> fprop_global_init+0x22 [kernel]
> wb_domain_init+0x6e [kernel]
> mem_cgroup_css_alloc+0x19c [kernel]
> cgroup_apply_control_enable+0x13e [kernel]
> cgroup_mkdir+0x22b [kernel]
> kernfs_iop_mkdir+0x5f [kernel]
> vfs_mkdir+0x108 [kernel]
> do_mkdirat+0xe8 [kernel]
> __x64_sys_mkdirat+0x1a [kernel]
> do_syscall_64+0x5a [kernel]
> entry_SYSCALL_64_after_hwframe+0x44 [kernel]
> 1400 bytes in 175 allocations from stack
> pcpu_alloc+0x39f [kernel]
> __alloc_percpu_gfp+0x12 [kernel]
> fib6_info_alloc+0x47 [kernel]
> addrconf_f6i_alloc+0x36 [kernel]
> ipv6_add_addr+0x120 [kernel]
> add_addr+0x73 [kernel]
> addrconf_notify+0x50f [kernel]
> notifier_call_chain+0x4c [kernel]
> raw_notifier_call_chain+0x16 [kernel]
> call_netdevice_notifiers_info+0x2d [kernel]
> __dev_notify_flags+0x65 [kernel]
> dev_change_flags+0x52 [kernel]
> do_setlink+0x2eb [kernel]
> rtnl_newlink+0x51e [kernel]
> rtnetlink_rcv_msg+0x27c [kernel]
> netlink_rcv_skb+0x54 [kernel]
> rtnetlink_rcv+0x15 [kernel]
> netlink_unicast+0x1ab [kernel]
> netlink_sendmsg+0x2d1 [kernel]
> sock_sendmsg+0x3e [kernel]
> __sys_sendto+0x13f [kernel]
> __x64_sys_sendto+0x28 [kernel]
> do_syscall_64+0x5a [kernel]
> entry_SYSCALL_64_after_hwframe+0x44 [kernel]
> 2520 bytes in 315 allocations from stack
> pcpu_alloc+0x39f [kernel]
> __alloc_percpu_gfp+0x12 [kernel]
> percpu_ref_init+0x23 [kernel]
> cgroup_apply_control_enable+0x183 [kernel]
> cgroup_mkdir+0x22b [kernel]
> kernfs_iop_mkdir+0x5f [kernel]
> vfs_mkdir+0x108 [kernel]
> do_mkdirat+0xe8 [kernel]
> __x64_sys_mkdirat+0x1a [kernel]
> do_syscall_64+0x5a [kernel]
> entry_SYSCALL_64_after_hwframe+0x44 [kernel]
> 2520 bytes in 315 allocations from stack
> pcpu_alloc+0x39f [kernel]
> __alloc_percpu_gfp+0x12 [kernel]
> percpu_ref_init+0x23 [kernel]
> cgroup_mkdir+0x106 [kernel]
> kernfs_iop_mkdir+0x5f [kernel]
> vfs_mkdir+0x108 [kernel]
> do_mkdirat+0xe8 [kernel]
> __x64_sys_mkdirat+0x1a [kernel]
> do_syscall_64+0x5a [kernel]
> entry_SYSCALL_64_after_hwframe+0x44 [kernel]
> 70560 bytes in 315 allocations from stack
> pcpu_alloc+0x39f [kernel]
> __alloc_percpu+0x15 [kernel]
> mem_cgroup_css_alloc+0xa7 [kernel]
> cgroup_apply_control_enable+0x13e [kernel]
> cgroup_mkdir+0x22b [kernel]
> kernfs_iop_mkdir+0x5f [kernel]
> vfs_mkdir+0x108 [kernel]
> do_mkdirat+0xe8 [kernel]
> __x64_sys_mkdirat+0x1a [kernel]
> do_syscall_64+0x5a [kernel]
> entry_SYSCALL_64_after_hwframe+0x44 [kernel]
> 184640 bytes in 5770 allocations from stack
> pcpu_alloc+0x39f [kernel]
> __alloc_percpu+0x15 [kernel]
> __kmem_cache_create+0x3dc [kernel]
> create_cache+0xd2 [kernel]
> memcg_create_kmem_cache+0x109 [kernel]
> memcg_kmem_cache_create_func+0x20 [kernel]
> process_one_work+0x1fd [kernel]
> worker_thread+0x34 [kernel]
> kthread+0x121 [kernel]
> ret_from_fork+0x35 [kernel]
> 309960 bytes in 315 allocations from stack
> pcpu_alloc+0x39f [kernel]
> __alloc_percpu+0x15 [kernel]
> mem_cgroup_css_alloc+0x75 [kernel]
> cgroup_apply_control_enable+0x13e [kernel]
> cgroup_mkdir+0x22b [kernel]
> kernfs_iop_mkdir+0x5f [kernel]
> vfs_mkdir+0x108 [kernel]
> do_mkdirat+0xe8 [kernel]
> __x64_sys_mkdirat+0x1a [kernel]
> do_syscall_64+0x5a [kernel]
> entry_SYSCALL_64_after_hwframe+0x44 [kernel]
> 1015808 bytes in 248 allocations from stack
> pcpu_alloc+0x39f [kernel]
> __alloc_percpu+0x15 [kernel]
> xt_percpu_counter_alloc+0x68 [kernel]
> find_check_entry.isra.7+0x55 [kernel]
> translate_table+0x475 [kernel]
> do_ipt_set_ctl+0x11a [kernel]
> nf_setsockopt+0x4c [kernel]
> ip_setsockopt+0x8a [kernel]
> raw_setsockopt+0x34 [kernel]
> sock_common_setsockopt+0x1a [kernel]
> __sys_setsockopt+0x86 [kernel]
> __x64_sys_setsockopt+0x24 [kernel]
> do_syscall_64+0x5a [kernel]
> entry_SYSCALL_64_after_hwframe+0x44 [kernel]
Perfect!
Given that cgroups are not "leaking", I'd bet on the last stack.
I haven't seen anything similar in out case. Doesn't seem to be
related to cgroups directly.
Also, it might be interesting to gather data for longer periods of time,
which also include manual dropping of caches; to be sure that it's
a real leak.
Thanks!
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Percpu Memory Statistics
Allocation Info:
----------------------------------------
unit_size : 262144
static_size : 155648
reserved_size : 8192
dyn_size : 28672
atom_size : 2097152
alloc_size : 2097152
Global Stats:
----------------------------------------
nr_alloc : 7243527
nr_dealloc : 7189359
nr_cur_alloc : 54168
nr_max_alloc : 100329
nr_chunks : 50
nr_max_chunks : 50
min_alloc_size : 4
max_alloc_size : 4096
empty_pop_pages : 272
Per Chunk Stats:
----------------------------------------
Chunk: <- Reserved Chunk
nr_alloc : 2
max_alloc_size : 320
empty_pop_pages : 1
first_bit : 81
free_bytes : 7868
contig_bytes : 7868
sum_frag : 0
max_frag : 0
cur_min_alloc : 4
cur_med_alloc : 4
cur_max_alloc : 320
Chunk: <- First Chunk
nr_alloc : 561
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 7168
free_bytes : 0
contig_bytes : 0
sum_frag : 0
max_frag : 0
cur_min_alloc : 4
cur_med_alloc : 4
cur_max_alloc : 4096
Chunk:
nr_alloc : 10203
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 7738
free_bytes : 912
contig_bytes : 32
sum_frag : 912
max_frag : 32
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 2214
max_alloc_size : 4096
empty_pop_pages : 13
first_bit : 582
free_bytes : 94016
contig_bytes : 52264
sum_frag : 41752
max_frag : 4096
cur_min_alloc : 4
cur_med_alloc : 32
cur_max_alloc : 4096
Chunk:
nr_alloc : 2624
max_alloc_size : 4096
empty_pop_pages : 1
first_bit : 24
free_bytes : 105560
contig_bytes : 6176
sum_frag : 105432
max_frag : 6176
cur_min_alloc : 4
cur_med_alloc : 32
cur_max_alloc : 984
Chunk:
nr_alloc : 1180
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 251864
contig_bytes : 4376
sum_frag : 249880
max_frag : 4376
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 224
Chunk:
nr_alloc : 1315
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 4
free_bytes : 239884
contig_bytes : 5864
sum_frag : 238812
max_frag : 5864
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 984
Chunk:
nr_alloc : 5612
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 210852
contig_bytes : 2984
sum_frag : 210852
max_frag : 2984
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 984
Chunk:
nr_alloc : 1803
max_alloc_size : 4096
empty_pop_pages : 4
first_bit : 0
free_bytes : 245744
contig_bytes : 8544
sum_frag : 245744
max_frag : 8544
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 984
Chunk:
nr_alloc : 309
max_alloc_size : 4096
empty_pop_pages : 13
first_bit : 0
free_bytes : 258696
contig_bytes : 21800
sum_frag : 258688
max_frag : 21800
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 984
Chunk:
nr_alloc : 2558
max_alloc_size : 4096
empty_pop_pages : 6
first_bit : 0
free_bytes : 229416
contig_bytes : 6744
sum_frag : 229416
max_frag : 6744
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 860
max_alloc_size : 4096
empty_pop_pages : 8
first_bit : 0
free_bytes : 230736
contig_bytes : 21376
sum_frag : 230736
max_frag : 21376
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 280
max_alloc_size : 4096
empty_pop_pages : 1
first_bit : 0
free_bytes : 157704
contig_bytes : 6792
sum_frag : 153608
max_frag : 6792
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 708
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 244216
contig_bytes : 5296
sum_frag : 243944
max_frag : 5296
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 361
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 189760
contig_bytes : 5168
sum_frag : 189760
max_frag : 5168
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 1521
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 249976
contig_bytes : 4232
sum_frag : 249512
max_frag : 4912
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 8
Chunk:
nr_alloc : 294
max_alloc_size : 4096
empty_pop_pages : 1
first_bit : 0
free_bytes : 133064
contig_bytes : 4760
sum_frag : 128968
max_frag : 4760
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 1553
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 147520
contig_bytes : 4960
sum_frag : 147520
max_frag : 4960
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 2563
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 233464
contig_bytes : 4072
sum_frag : 233464
max_frag : 4072
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 394
max_alloc_size : 4096
empty_pop_pages : 1
first_bit : 0
free_bytes : 140440
contig_bytes : 5248
sum_frag : 140408
max_frag : 5248
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 1836
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 212128
contig_bytes : 4984
sum_frag : 212016
max_frag : 4984
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 879
max_alloc_size : 4096
empty_pop_pages : 23
first_bit : 0
free_bytes : 226496
contig_bytes : 95192
sum_frag : 226496
max_frag : 95192
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 459
max_alloc_size : 4096
empty_pop_pages : 27
first_bit : 0
free_bytes : 246208
contig_bytes : 52208
sum_frag : 244640
max_frag : 52208
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 295
max_alloc_size : 4096
empty_pop_pages : 31
first_bit : 0
free_bytes : 259784
contig_bytes : 93392
sum_frag : 258176
max_frag : 93392
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 8
Chunk:
nr_alloc : 1402
max_alloc_size : 4096
empty_pop_pages : 1
first_bit : 0
free_bytes : 233292
contig_bytes : 5416
sum_frag : 233196
max_frag : 5416
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 984
Chunk:
nr_alloc : 187
max_alloc_size : 4096
empty_pop_pages : 28
first_bit : 0
free_bytes : 260604
contig_bytes : 69232
sum_frag : 259252
max_frag : 69232
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 32
Chunk:
nr_alloc : 348
max_alloc_size : 4096
empty_pop_pages : 28
first_bit : 0
free_bytes : 256496
contig_bytes : 25768
sum_frag : 252232
max_frag : 25768
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 984
Chunk:
nr_alloc : 330
max_alloc_size : 4096
empty_pop_pages : 23
first_bit : 0
free_bytes : 258528
contig_bytes : 42472
sum_frag : 258384
max_frag : 42472
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 984
Chunk:
nr_alloc : 1395
max_alloc_size : 4096
empty_pop_pages : 1
first_bit : 0
free_bytes : 250852
contig_bytes : 6776
sum_frag : 250852
max_frag : 6776
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 32
Chunk:
nr_alloc : 571
max_alloc_size : 4096
empty_pop_pages : 6
first_bit : 0
free_bytes : 248632
contig_bytes : 9376
sum_frag : 248592
max_frag : 9376
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 232
max_alloc_size : 4096
empty_pop_pages : 20
first_bit : 0
free_bytes : 242828
contig_bytes : 38680
sum_frag : 242204
max_frag : 38680
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 1048
max_alloc_size : 4096
empty_pop_pages : 6
first_bit : 0
free_bytes : 240332
contig_bytes : 14144
sum_frag : 240332
max_frag : 14144
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 782
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 246204
contig_bytes : 4848
sum_frag : 244604
max_frag : 4848
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 371
max_alloc_size : 4096
empty_pop_pages : 1
first_bit : 0
free_bytes : 236360
contig_bytes : 5896
sum_frag : 235320
max_frag : 5896
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 326
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 224620
contig_bytes : 6488
sum_frag : 224300
max_frag : 6488
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 439
max_alloc_size : 4096
empty_pop_pages : 2
first_bit : 1024
free_bytes : 232048
contig_bytes : 6560
sum_frag : 231376
max_frag : 6560
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 287
max_alloc_size : 4096
empty_pop_pages : 3
first_bit : 0
free_bytes : 234912
contig_bytes : 11520
sum_frag : 234912
max_frag : 11520
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 419
max_alloc_size : 4096
empty_pop_pages : 4
first_bit : 0
free_bytes : 237928
contig_bytes : 8376
sum_frag : 237928
max_frag : 8376
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 730
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 251136
contig_bytes : 5280
sum_frag : 249432
max_frag : 5280
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 2048
Chunk:
nr_alloc : 598
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 237936
contig_bytes : 4744
sum_frag : 237760
max_frag : 4744
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 497
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 239920
contig_bytes : 5768
sum_frag : 239912
max_frag : 5768
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 368
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 224800
contig_bytes : 5144
sum_frag : 224240
max_frag : 5144
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 403
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 229116
contig_bytes : 5872
sum_frag : 225572
max_frag : 5872
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 577
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 1024
free_bytes : 239220
contig_bytes : 5440
sum_frag : 239180
max_frag : 5440
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 391
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 1024
free_bytes : 250560
contig_bytes : 6376
sum_frag : 249208
max_frag : 6376
cur_min_alloc : 8
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 302
max_alloc_size : 4096
empty_pop_pages : 0
first_bit : 0
free_bytes : 241152
contig_bytes : 5464
sum_frag : 240800
max_frag : 5464
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 444
max_alloc_size : 4096
empty_pop_pages : 1
first_bit : 0
free_bytes : 233024
contig_bytes : 7344
sum_frag : 233024
max_frag : 7344
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 432
max_alloc_size : 4096
empty_pop_pages : 1
first_bit : 0
free_bytes : 228948
contig_bytes : 5280
sum_frag : 226660
max_frag : 5280
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 400
max_alloc_size : 4096
empty_pop_pages : 2
first_bit : 1024
free_bytes : 244692
contig_bytes : 6160
sum_frag : 244676
max_frag : 6160
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 499
max_alloc_size : 4096
empty_pop_pages : 5
first_bit : 0
free_bytes : 253184
contig_bytes : 7360
sum_frag : 252592
max_frag : 7360
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
Chunk:
nr_alloc : 6
max_alloc_size : 4096
empty_pop_pages : 11
first_bit : 0
free_bytes : 237568
contig_bytes : 192512
sum_frag : 45056
max_frag : 24576
cur_min_alloc : 4096
cur_med_alloc : 4096
cur_max_alloc : 4096
