On Sun, 2022-11-13 at 23:52 -0800, Yonghong Song wrote:
>
> On 11/11/22 1:55 PM, Eduard Zingerman wrote:
> > On Fri, 2022-10-28 at 11:56 -0700, Yonghong Song wrote:
> > > > > [...]
> > > >
> > > > Ok, could we change the problem to detecting if some type is defined.
> > > > Would it be possible to have something like
> > > >
> > > > #if !__is_type_defined(struct abc)
> > > > struct abc {
> > > > };
> > > > #endif
> > > >
> > > > I think we talked about this and there were problems with this
> > > > approach, but I don't remember details and how insurmountable the
> > > > problem is. Having a way to check whether some type is defined would
> > > > be very useful even outside of -target bpf parlance, though, so maybe
> > > > it's the problem worth attacking?
> > >
> > > Yes, we discussed this before. This will need to add additional work
> > > in preprocessor. I just made a discussion topic in llvm discourse
> > >
> > > https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268
> > >
> > > Let us see whether we can get some upstream agreement or not.
> >
> > I did a small investigation of this feature.
> >
> > The main pre-requirement is construction of the symbol table during
> > source code pre-processing, which implies necessity to parse the
> > source code at the same time. It is technically possible in clang, as
> > lexing, pre-processing and AST construction happens at the same time
> > when in compilation mode.
> >
> > The prototype is available here [1], it includes:
> > - Change in the pre-processor that adds an optional callback
> > "IsTypeDefinedFn" & necessary parsing of __is_type_defined
> > construct.
> > - Change in Sema module (responsible for parsing/AST & symbol table)
> > that installs the appropriate "IsTypeDefinedFn" in the pre-processor
> > instance.
> >
> > However, this prototype builds a backward dependency between
> > pre-processor and semantic analysis. There are currently no such
> > dependencies in the clang code base.
> >
> > This makes it impossible to do pre-processing and compilation
> > separately, e.g. consider the following example:
> >
> > $ cat test.c
> >
> > struct foo { int x; };
> >
> > #if __is_type_defined(foo)
> > const int x = 1;
> > #else
> > const int x = 2;
> > #endif
> >
> > $ clang -cc1 -ast-print test.c -o -
> >
> > struct foo {
> > int x;
> > };
> > const int x = 1;
> >
> > $ clang -E test.c -o -
> >
> > # ... some line directives ...
> > struct foo { int x; };
> > const int x = 2;
>
> Is it any chance '-E' could output the same one as '-cc1 -ast-print'?
> That is, even with -E we could do some semantics analysis
> as well, using either current clang semantics analysis or creating
> an minimal version of sema analysis in preprocessor itself?
Sema drives consumption of tokens from Preprocessor. Calls to
Preprocessor are done on a parsing recursive descent. Extracting a
stream of tokens would require an incremental parser instead.
A minimal version of such parser is possible to implement for C.
It might be the case that matching open / closing braces and
identifiers following 'struct' / 'union' / 'enum' keywords might be
almost sufficient but I need to try to be sure (e.g. it is more
complex for 'typedef').
I can work on it but I don't think there is a chance to upstream this work.
Thanks,
Eduard
>
> >
> > Note that __is_type_defined is computed to different value in the
> > first and second calls. This is so because semantic analysis (AST,
> > symbol table) is not done for -E.
> >
> > It also breaks that C11 standard which clearly separates
> > pre-processing and semantic analysis phases, see [2] 5.1.1.2.
> >
> > So, my conclusion is as follows: this is technically possible in clang
> > but has no chance to reach llvm upstream.
> >
> > Thanks,
> > Eduard
> >
> > [1] https://github.com/llvm/llvm-project/compare/main...eddyz87:llvm-project:is-type-defined-experiment
> > [2] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf
> >
> >
> > >
> > > >
> > > > >
> > > > > >
> > > > > > BTW, I suggest splitting libbpf btf_dedup and btf_dump changes into a
> > > > > > separate series and sending them as non-RFC sooner. Those improvements
> > > > > > are independent of all the header guards stuff, let's get them landed
> > > > > > sooner.
> > > > > >
> > > > > > > After some discussion with Alexei and Yonghong I'd like to request
> > > > > > > your comments regarding a somewhat brittle and partial solution to
> > > > > > > this issue that relies on adding `#ifndef FOO_H ... #endif` guards in
> > > > > > > the generated `vmlinux.h`.
> > > > > > >
> > > > > >
> > > > > > [...]
> > > > > >
> > > > > > > Eduard Zingerman (12):
> > > > > > > libbpf: Deduplicate unambigous standalone forward declarations
> > > > > > > selftests/bpf: Tests for standalone forward BTF declarations
> > > > > > > deduplication
> > > > > > > libbpf: Support for BTF_DECL_TAG dump in C format
> > > > > > > selftests/bpf: Tests for BTF_DECL_TAG dump in C format
> > > > > > > libbpf: Header guards for selected data structures in vmlinux.h
> > > > > > > selftests/bpf: Tests for header guards printing in BTF dump
> > > > > > > bpftool: Enable header guards generation
> > > > > > > kbuild: Script to infer header guard values for uapi headers
> > > > > > > kbuild: Header guards for types from include/uapi/*.h in kernel BTF
> > > > > > > selftests/bpf: Script to verify uapi headers usage with vmlinux.h
> > > > > > > selftests/bpf: Known good uapi headers for test_uapi_headers.py
> > > > > > > selftests/bpf: script for infer_header_guards.pl testing
> > > > > > >
> > > > > > > scripts/infer_header_guards.pl | 191 +++++
> > > > > > > scripts/link-vmlinux.sh | 13 +-
> > > > > > > tools/bpf/bpftool/btf.c | 4 +-
> > > > > > > tools/lib/bpf/btf.c | 178 ++++-
> > > > > > > tools/lib/bpf/btf.h | 7 +-
> > > > > > > tools/lib/bpf/btf_dump.c | 232 +++++-
> > > > > > > .../selftests/bpf/good_uapi_headers.txt | 677 ++++++++++++++++++
> > > > > > > tools/testing/selftests/bpf/prog_tests/btf.c | 152 ++++
> > > > > > > .../selftests/bpf/prog_tests/btf_dump.c | 11 +-
> > > > > > > .../bpf/progs/btf_dump_test_case_decl_tag.c | 39 +
> > > > > > > .../progs/btf_dump_test_case_header_guards.c | 94 +++
> > > > > > > .../bpf/test_uapi_header_guards_infer.sh | 33 +
> > > > > > > .../selftests/bpf/test_uapi_headers.py | 197 +++++
> > > > > > > 13 files changed, 1816 insertions(+), 12 deletions(-)
> > > > > > > create mode 100755 scripts/infer_header_guards.pl
> > > > > > > create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt
> > > > > > > create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c
> > > > > > > create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c
> > > > > > > create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh
> > > > > > > create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py
> > > > > > >
> > > > > > > --
> > > > > > > 2.34.1
> > > > > > >
> >
