> On Jan 11, 2024, at 2:48 PM, Daniel Xu <dxu@xxxxxxxxx> wrote:
>
> Hi Song,
>
> On Thu, Jan 11, 2024 at 09:51:05PM +0000, Song Liu wrote:
>> The problem
>>
>> Inlining can cause surprises to tracing users, especially when the tool
>> appears to be working. For example, with
>>
>> [root@ ~]# bpftrace -e 'kprobe:switch_mm {}'
>> Attaching 1 probe...
>>
>> The user may not realize switch_mm() is inlined by leave_mm(), and we are
>> not tracing the code path leave_mm => switch_mm. (This is x86_64, and both
>> functions are in arch/x86/mm/tlb.c.)
>>
>> We have folks working on ideas to create offline tools to detect such
>> issues for critical use cases at compile time. However, I think it is
>> necessary to handle it at program load/attach time.
>
> Could you clarify what offline means?
The idea is to keep a list of kernel functions used by key services. At
kernel build time, we check whether any of these functions are inlined.
If so, the tool will catch it, and we need to add noinline to the function.
>
> I wonder if libbpf should just give a way for applications to query
> inline status. Seems most flexible. And maybe also a special SEC() def?
API to query inline status makes sense. What do we do with SEC()?
>
> Although I suspect end users might want a flag to "attach anyways; I'm
> aware", so a more generic api (eg `btf__is_inlined(..)`) results in a
> smaller and maximally flexible result.
>
>> Detect "inlined by some callers" functions
>>
>> This appears to be straightforward in pahole. Something like the following
>> should do the work:
>>
>> diff --git i/btf_encoder.c w/btf_encoder.c
>> index fd040086827e..e546a059eb4b 100644
>> --- i/btf_encoder.c
>> +++ w/btf_encoder.c
>> @@ -885,6 +885,15 @@ static int32_t btf_encoder__add_func(struct btf_encoder *encoder, struct functio
>> struct llvm_annotation *annot;
>> const char *name;
>>
>> + if (function__inlined(fn)) {
>> + /* This function is inlined by some callers. */
>> + }
>> +
>> btf_fnproto_id = btf_encoder__add_func_proto(encoder, &fn->proto);
>> name = function__name(fn);
>> btf_fn_id = btf_encoder__add_ref_type(encoder, BTF_KIND_FUNC, btf_fnproto_id, name, false);
>>
>>
>> Mark "inlined by some callers" functions
>>
>> We have a few options to mark these functions.
>>
>> 1. We can set struct btf_type.info.kind_flag for inlined function. Or we
>> can use a bit from info.vlen.
>
> Seems reasonable. Decl tag is another option but probably too heavy.
>
>> 2. We can simply not generate btf for these functions. This is similar to
>> --skip_encoding_btf_inconsistent_proto.
>
> This option seems a bit confusing. Basically you're suggesting that:
>
> func_in_avilable_filter_functions && !func_in_btf
>
> implies partially inlined function, right? Seems a bit non-obvious.
I didn't think about available_filter_functions. My original idea was to
not generate BTF for these functions, and thus disallow attaching (some)
BPF programs to them. I guess that will not stop regular kprobe attach.
Thanks,
Song
