Re: [PATCH bpf] bpf: Adjust insufficient default bpf_jit_limit

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



From:   Daniel Borkmann <daniel@xxxxxxxxxxxxx>
Date:   Mon, 20 Mar 2023 15:37:25 +0100
> We've seen recent AWS EKS (Kubernetes) user reports like the following:
> 
>   After upgrading EKS nodes from v20230203 to v20230217 on our 1.24 EKS
>   clusters after a few days a number of the nodes have containers stuck
>   in ContainerCreating state or liveness/readiness probes reporting the
>   following error:
> 
>     Readiness probe errored: rpc error: code = Unknown desc = failed to
>     exec in container: failed to start exec "4a11039f730203ffc003b7[...]":
>     OCI runtime exec failed: exec failed: unable to start container process:
>     unable to init seccomp: error loading seccomp filter into kernel:
>     error loading seccomp filter: errno 524: unknown
> 
>   However, we had not been seeing this issue on previous AMIs and it only
>   started to occur on v20230217 (following the upgrade from kernel 5.4 to
>   5.10) with no other changes to the underlying cluster or workloads.
> 
>   We tried the suggestions from that issue (sysctl net.core.bpf_jit_limit=452534528)
>   which helped to immediately allow containers to be created and probes to
>   execute but after approximately a day the issue returned and the value
>   returned by cat /proc/vmallocinfo | grep bpf_jit | awk '{s+=$2} END {print s}'
>   was steadily increasing.
> 
> I tested bpf tree to observe bpf_jit_charge_modmem, bpf_jit_uncharge_modmem
> their sizes passed in as well as bpf_jit_current under tcpdump BPF filter,
> seccomp BPF and native (e)BPF programs, and the behavior all looks sane
> and expected, that is nothing "leaking" from an upstream perspective.
> 
> The bpf_jit_limit knob was originally added in order to avoid a situation
> where unprivileged applications loading BPF programs (e.g. seccomp BPF
> policies) consuming all the module memory space via BPF JIT such that loading
> of kernel modules would be prevented. The default limit was defined back in
> 2018 and while good enough back then, we are generally seeing far more BPF
> consumers today.
> 
> Adjust the limit for the BPF JIT pool from originally 1/4 to now 1/2 of the
> module memory space to better reflect today's needs and avoid more users
> running into potentially hard to debug issues.
> 
> Fixes: fdadd04931c2 ("bpf: fix bpf_jit_limit knob for PAGE_SIZE >= 64K")
> Reported-by: Stephen Haynes <sh@xxxxxxxx>
> Reported-by: Lefteris Alexakis <lefteris.alexakis@xxxxxxx>
> Signed-off-by: Daniel Borkmann <daniel@xxxxxxxxxxxxx>

Hi Daniel,

Thanks for tha patch.

Reviewed-by: Kuniyuki Iwashima <kuniyu@xxxxxxxxxx>


> Link: https://github.com/awslabs/amazon-eks-ami/issues/1179
> Link: https://github.com/awslabs/amazon-eks-ami/issues/1219

I'm investigating these issues with EKS folks.  On the issue 1179, the
customer was using our 5.4 kernel, and on 1219, 5.10 kernel.

Then, I found my memleak fix commit a1140cb215fa ("seccomp: Move
copy_seccomp() to no failure path.") was not backported to upstream 5.10
stable trees.  We'll test if the issue can be reproduced with/without
the fix.

Anyway, I'll backport this patch to our all trees.

Thanks,
Kuniyuki


> ---
>  kernel/bpf/core.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> index b297e9f60ca1..e2d256c82072 100644
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c
> @@ -972,7 +972,7 @@ static int __init bpf_jit_charge_init(void)
>  {
>  	/* Only used as heuristic here to derive limit. */
>  	bpf_jit_limit_max = bpf_jit_alloc_exec_limit();
> -	bpf_jit_limit = min_t(u64, round_up(bpf_jit_limit_max >> 2,
> +	bpf_jit_limit = min_t(u64, round_up(bpf_jit_limit_max >> 1,
>  					    PAGE_SIZE), LONG_MAX);
>  	return 0;
>  }
> -- 
> 2.27.0



[Index of Archives]     [Linux Samsung SoC]     [Linux Rockchip SoC]     [Linux Actions SoC]     [Linux for Synopsys ARC Processors]     [Linux NFS]     [Linux NILFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]


  Powered by Linux