Re: [PATCH bpf-next v3 1/2] bpf: Track aligned st store as imprecise spilled registers

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



From: Yonghong Song <yonghong.song@xxxxxxxxx>
Date: Mon,  8 Jan 2024 20:05:24 -0800
> With patch set [1], precision backtracing supports register spill/fill
> to/from the stack. The patch [2] allows initial imprecise register spill
> with content 0. This is a common case for cpuv3 and lower for
> initializing the stack variables with pattern
>   r1 = 0
>   *(u64 *)(r10 - 8) = r1
> and the [2] has demonstrated good verification improvement.
> 
> For cpuv4, the initialization could be
>   *(u64 *)(r10 - 8) = 0
> The current verifier marks the r10-8 contents with STACK_ZERO.
> Similar to [2], let us permit the above insn to behave like
> imprecise register spill which can reduce number of verified states.
> The change is in function check_stack_write_fixed_off().
> 
> Before this patch, spilled zero will be marked as STACK_ZERO
> which can provide precise values. In check_stack_write_var_off(),
> STACK_ZERO will be maintained if writing a const zero
> so later it can provide precise values if needed.
> 
> The above handling of '*(u64 *)(r10 - 8) = 0' as a spill
> will have issues in check_stack_write_var_off() as the spill
> will be converted to STACK_MISC and the precise value 0
> is lost. To fix this issue, if the spill slots with const
> zero and the BPF_ST write also with const zero, the spill slots
> are preserved, which can later provide precise values
> if needed. Without the change in check_stack_write_var_off(),
> the test_verifier subtest 'BPF_ST_MEM stack imm zero, variable offset'
> will fail.
> 
> I checked cpuv3 and cpuv4 with and without this patch with veristat.
> There is no state change for cpuv3 since '*(u64 *)(r10 - 8) = 0'
> is only generated with cpuv4.
> 
> For cpuv4:
> $ ../veristat -C old.cpuv4.csv new.cpuv4.csv -e file,prog,insns,states -f 'insns_diff!=0'
> File                                        Program              Insns (A)  Insns (B)  Insns    (DIFF)  States (A)  States (B)  States (DIFF)
> ------------------------------------------  -------------------  ---------  ---------  ---------------  ----------  ----------  -------------
> local_storage_bench.bpf.linked3.o           get_local                  228        168    -60 (-26.32%)          17          14   -3 (-17.65%)
> pyperf600_bpf_loop.bpf.linked3.o            on_event                  6066       4889  -1177 (-19.40%)         403         321  -82 (-20.35%)
> test_cls_redirect.bpf.linked3.o             cls_redirect             35483      35387     -96 (-0.27%)        2179        2177    -2 (-0.09%)
> test_l4lb_noinline.bpf.linked3.o            balancer_ingress          4494       4522     +28 (+0.62%)         217         219    +2 (+0.92%)
> test_l4lb_noinline_dynptr.bpf.linked3.o     balancer_ingress          1432       1455     +23 (+1.61%)          92          94    +2 (+2.17%)
> test_xdp_noinline.bpf.linked3.o             balancer_ingress_v6       3462       3458      -4 (-0.12%)         216         216    +0 (+0.00%)
> verifier_iterating_callbacks.bpf.linked3.o  widening                    52         41    -11 (-21.15%)           4           3   -1 (-25.00%)
> xdp_synproxy_kern.bpf.linked3.o             syncookie_tc             12412      11719    -693 (-5.58%)         345         330   -15 (-4.35%)
> xdp_synproxy_kern.bpf.linked3.o             syncookie_xdp            12478      11794    -684 (-5.48%)         346         331   -15 (-4.34%)
> 
> test_l4lb_noinline and test_l4lb_noinline_dynptr has minor regression, but
> pyperf600_bpf_loop and local_storage_bench gets pretty good improvement.
> 
>   [1] https://lore.kernel.org/all/20231205184248.1502704-1-andrii@xxxxxxxxxx/
>   [2] https://lore.kernel.org/all/20231205184248.1502704-9-andrii@xxxxxxxxxx/
> 
> Cc: Kuniyuki Iwashima <kuniyu@xxxxxxxxxx>
> Cc: Martin KaFai Lau <kafai@xxxxxx>
> Signed-off-by: Yonghong Song <yonghong.song@xxxxxxxxx>

Tested-by: Kuniyuki Iwashima <kuniyu@xxxxxxxxxx>

I confirmed that the selftest of synproxy kfunc ran successfully
for cpuv4 with your patch.

https://lore.kernel.org/bpf/20231221012806.37137-7-kuniyu@xxxxxxxxxx/

---8<---
$ sudo ./test_progs-cpuv4 -t tcp_custom
#349/1   tcp_custom_syncookie/IPv4 TCP  :OK
#349/2   tcp_custom_syncookie/IPv6 TCP  :OK
#349     tcp_custom_syncookie:OK
Summary: 1/2 PASSED, 0 SKIPPED, 0 FAILED
---8<---

Thank you so much!


> ---
>  kernel/bpf/verifier.c                            | 16 ++++++++++++++--
>  .../selftests/bpf/progs/verifier_spill_fill.c    | 16 ++++++++--------
>  2 files changed, 22 insertions(+), 10 deletions(-)
> 
> Changelogs:
>   v2 -> v3:
>     - add precision checking to the spilled zero value register in
>       check_stack_write_var_off().
>     - check spill slot-by-slot instead of in a bunch within a spi.
>   v1 -> v2:
>     - Preserve with-const-zero spill if writing is also zero
>       in check_stack_write_var_off().
>     - Add a test with not-8-byte-aligned BPF_ST store.
> 
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index adbf330d364b..54da1045e078 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -4493,7 +4493,7 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
>  		if (fls64(reg->umax_value) > BITS_PER_BYTE * size)
>  			state->stack[spi].spilled_ptr.id = 0;
>  	} else if (!reg && !(off % BPF_REG_SIZE) && is_bpf_st_mem(insn) &&
> -		   insn->imm != 0 && env->bpf_capable) {
> +		   env->bpf_capable) {
>  		struct bpf_reg_state fake_reg = {};
>  
>  		__mark_reg_known(&fake_reg, insn->imm);
> @@ -4615,6 +4615,7 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
>  
>  	/* Variable offset writes destroy any spilled pointers in range. */
>  	for (i = min_off; i < max_off; i++) {
> +		struct bpf_reg_state *spill_reg;
>  		u8 new_type, *stype;
>  		int slot, spi;
>  
> @@ -4640,7 +4641,18 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
>  			return -EINVAL;
>  		}
>  
> -		/* Erase all spilled pointers. */
> +		/* If writing_zero and the the spi slot contains a spill of value 0,
> +		 * maintain the spill type.
> +		 */
> +		if (writing_zero && is_spilled_scalar_reg(&state->stack[spi])) {
> +			spill_reg = &state->stack[spi].spilled_ptr;
> +			if (tnum_is_const(spill_reg->var_off) && spill_reg->var_off.value == 0) {
> +				zero_used = true;
> +				continue;
> +			}
> +		}
> +
> +		/* Erase all other spilled pointers. */
>  		state->stack[spi].spilled_ptr.type = NOT_INIT;
>  
>  		/* Update the slot type. */
> diff --git a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
> index 39fe3372e0e0..d4b3188afe07 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
> @@ -495,14 +495,14 @@ char single_byte_buf[1] SEC(".data.single_byte_buf");
>  SEC("raw_tp")
>  __log_level(2)
>  __success
> -/* make sure fp-8 is all STACK_ZERO */
> -__msg("2: (7a) *(u64 *)(r10 -8) = 0          ; R10=fp0 fp-8_w=00000000")
> +/* fp-8 is spilled IMPRECISE value zero (represented by a zero value fake reg) */
> +__msg("2: (7a) *(u64 *)(r10 -8) = 0          ; R10=fp0 fp-8_w=0")
>  /* but fp-16 is spilled IMPRECISE zero const reg */
>  __msg("4: (7b) *(u64 *)(r10 -16) = r0        ; R0_w=0 R10=fp0 fp-16_w=0")
> -/* validate that assigning R2 from STACK_ZERO doesn't mark register
> +/* validate that assigning R2 from STACK_SPILL with zero value  doesn't mark register
>   * precise immediately; if necessary, it will be marked precise later
>   */
> -__msg("6: (71) r2 = *(u8 *)(r10 -1)          ; R2_w=0 R10=fp0 fp-8_w=00000000")
> +__msg("6: (71) r2 = *(u8 *)(r10 -1)          ; R2_w=0 R10=fp0 fp-8_w=0")
>  /* similarly, when R2 is assigned from spilled register, it is initially
>   * imprecise, but will be marked precise later once it is used in precise context
>   */
> @@ -520,14 +520,14 @@ __msg("mark_precise: frame0: regs=r0 stack= before 3: (b7) r0 = 0")
>  __naked void partial_stack_load_preserves_zeros(void)
>  {
>  	asm volatile (
> -		/* fp-8 is all STACK_ZERO */
> +		/* fp-8 is value zero (represented by a zero value fake reg) */
>  		".8byte %[fp8_st_zero];" /* LLVM-18+: *(u64 *)(r10 -8) = 0; */
>  
>  		/* fp-16 is const zero register */
>  		"r0 = 0;"
>  		"*(u64 *)(r10 -16) = r0;"
>  
> -		/* load single U8 from non-aligned STACK_ZERO slot */
> +		/* load single U8 from non-aligned spilled value zero slot */
>  		"r1 = %[single_byte_buf];"
>  		"r2 = *(u8 *)(r10 -1);"
>  		"r1 += r2;"
> @@ -539,7 +539,7 @@ __naked void partial_stack_load_preserves_zeros(void)
>  		"r1 += r2;"
>  		"*(u8 *)(r1 + 0) = r2;" /* this should be fine */
>  
> -		/* load single U16 from non-aligned STACK_ZERO slot */
> +		/* load single U16 from non-aligned spilled value zero slot */
>  		"r1 = %[single_byte_buf];"
>  		"r2 = *(u16 *)(r10 -2);"
>  		"r1 += r2;"
> @@ -551,7 +551,7 @@ __naked void partial_stack_load_preserves_zeros(void)
>  		"r1 += r2;"
>  		"*(u8 *)(r1 + 0) = r2;" /* this should be fine */
>  
> -		/* load single U32 from non-aligned STACK_ZERO slot */
> +		/* load single U32 from non-aligned spilled value zero slot */
>  		"r1 = %[single_byte_buf];"
>  		"r2 = *(u32 *)(r10 -4);"
>  		"r1 += r2;"
> -- 
> 2.34.1




[Index of Archives]     [Linux Samsung SoC]     [Linux Rockchip SoC]     [Linux Actions SoC]     [Linux for Synopsys ARC Processors]     [Linux NFS]     [Linux NILFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]


  Powered by Linux