On Fri, 2023-08-25 at 13:57 -0700, Andrii Nakryiko wrote:
> On Thu, Aug 24, 2023 at 4:02 PM Eduard Zingerman <eddyz87@xxxxxxxxx> wrote:
> >
> > Add a section on CO-RE relocations to llvm_relo.rst.
> > Describe relevant .BTF.ext structure, `enum bpf_core_relo_kind`
> > and `struct bpf_core_relo` in some detail.
> > Description is based on doc-string from include/uapi/linux/bpf.h.
> >
> > Signed-off-by: Eduard Zingerman <eddyz87@xxxxxxxxx>
> > ---
>
> Looks great overall, thanks a lot for adding this!
>
> > Documentation/bpf/btf.rst | 27 ++++-
> > Documentation/bpf/llvm_reloc.rst | 178 +++++++++++++++++++++++++++++++
> > 2 files changed, 201 insertions(+), 4 deletions(-)
> >
> > diff --git a/Documentation/bpf/btf.rst b/Documentation/bpf/btf.rst
> > index f32db1f44ae9..c0530211c3c1 100644
> > --- a/Documentation/bpf/btf.rst
> > +++ b/Documentation/bpf/btf.rst
> > @@ -726,8 +726,8 @@ same as the one describe in :ref:`BTF_Type_String`.
> > 4.2 .BTF.ext section
> > --------------------
> >
> > -The .BTF.ext section encodes func_info and line_info which needs loader
> > -manipulation before loading into the kernel.
> > +The .BTF.ext section encodes func_info, line_info and CO-RE relocations
> > +which needs loader manipulation before loading into the kernel.
> >
> > The specification for .BTF.ext section is defined at ``tools/lib/bpf/btf.h``
> > and ``tools/lib/bpf/btf.c``.
> > @@ -745,11 +745,16 @@ The current header of .BTF.ext section::
> > __u32 func_info_len;
> > __u32 line_info_off;
> > __u32 line_info_len;
> > +
> > + /* optional part of .BTF.ext header */
> > + __u32 core_relo_off;
> > + __u32 core_relo_len;
> > };
> >
> > It is very similar to .BTF section. Instead of type/string section, it
> > -contains func_info and line_info section. See :ref:`BPF_Prog_Load` for details
> > -about func_info and line_info record format.
> > +contains func_info, line_info and core_relo sub-sections.
> > +See :ref:`BPF_Prog_Load` for details about func_info and line_info
> > +record format.
> >
> > The func_info is organized as below.::
> >
> > @@ -787,6 +792,20 @@ kernel API, the ``insn_off`` is the instruction offset in the unit of ``struct
> > bpf_insn``. For ELF API, the ``insn_off`` is the byte offset from the
> > beginning of section (``btf_ext_info_sec->sec_name_off``).
> >
> > +The core_relo is organized as below.::
> > +
> > + core_relo_rec_size
>
> nit: should we specify that this is __u32 value? Same for func_info
> and line_info. I'm not sure we ever explicitly mention this this
> record size is 4 byte long.
Yeap, we don't mention it anywhere, I will add __u32 to the
description.
>
> > + btf_ext_info_sec for section #1 /* core_relo for section #1 */
> > + btf_ext_info_sec for section #2 /* core_relo for section #2 */
> > +
> > +``core_relo_rec_size`` specifies the size of ``bpf_core_relo``
> > +structure when .BTF.ext is generated. All ``bpf_core_relo`` structures
> > +within a single ``btf_ext_info_sec`` describe relocations applied to
> > +section named by ``btf_ext_info_sec::sec_name_off``.
> > +
> > +See :ref:`Documentation/bpf/llvm_reloc <btf-co-re-relocations>`
> > +for more information on CO-RE relocations.
> > +
> > 4.2 .BTF_ids section
> > --------------------
> >
> > diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst
> > index 450e6403fe3d..efe0b6ea4921 100644
> > --- a/Documentation/bpf/llvm_reloc.rst
> > +++ b/Documentation/bpf/llvm_reloc.rst
> > @@ -240,3 +240,181 @@ The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations::
> > Offset Info Type Symbol's Value Symbol's Name
> > 000000000000002c 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text
> > 0000000000000040 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text
> > +
> > +.. _btf-co-re-relocations:
> > +
> > +=================
> > +CO-RE Relocations
> > +=================
> > +
> > +From object file point of view CO-RE mechanism is implemented as a set
> > +of CO-RE specific relocation records. These relocation records are not
> > +related to ELF relocations and are encoded in .BTF.ext section.
> > +See :ref:`Documentation/bpf/btf <BTF_Ext_Section>` for more
> > +information on .BTF.ext structure.
> > +
> > +
> > +CO-RE relocations are applied to BPF instructions to update immediate
> > +or offset fields of the instruction at load time with information
> > +relevant for target kernel.
> > +
> > +Relocation kinds
> > +================
> > +
> > +There are several kinds of CO-RE relocations that could be split in
> > +three groups:
> > +
> > +* Field-based - patch instruction with field related information, e.g.
> > + change offset field of the BPF_LD instruction to reflect offset
> > + of a specific structure field in the target kernel.
> > +
> > +* Type-based - patch instruction with type related information, e.g.
> > + change immediate field of the BPF_MOV instruction to 0 or 1 to
> > + reflect if specific type is present in the target kernel.
> > +
> > +* Enum-based - patch instruction with enum related information, e.g.
> > + change immediate field of the BPF_MOV instruction to reflect value
> > + of a specific enum literal in the target kernel.
> > +
>
> Instead of referencing BPF_MOV specifically, would it be useful to
> incorporate all the different instructions that can be relocated?
> bpf_core_patch_insn comment has a nice summary, maybe we can somehow
> reuse it in this doc as well?
>
> * Currently supported classes of BPF instruction are:
> * 1. rX = <imm> (assignment with immediate operand);
> * 2. rX += <imm> (arithmetic operations with immediate operand);
> * 3. rX = <imm64> (load with 64-bit immediate value);
> * 4. rX = *(T *)(rY + <off>), where T is one of {u8, u16, u32, u64};
> * 5. *(T *)(rX + <off>) = rY, where T is one of {u8, u16, u32, u64};
> * 6. *(T *)(rX + <off>) = <imm>, where T is one of {u8, u16, u32, u64}.
Good point. I will keep the BPF_MOV as an example for relocation kind
groups description and add this comment describing all relocatable
instructions.
> > +The complete list of relocation kinds is represented by the following enum:
> > +
> > +.. code-block:: c
> > +
> > + enum bpf_core_relo_kind {
> > + BPF_CORE_FIELD_BYTE_OFFSET = 0, /* field byte offset */
> > + BPF_CORE_FIELD_BYTE_SIZE = 1, /* field size in bytes */
> > + BPF_CORE_FIELD_EXISTS = 2, /* field existence in target kernel */
> > + BPF_CORE_FIELD_SIGNED = 3, /* field signedness (0 - unsigned, 1 - signed) */
> > + BPF_CORE_FIELD_LSHIFT_U64 = 4, /* bitfield-specific left bitshift */
> > + BPF_CORE_FIELD_RSHIFT_U64 = 5, /* bitfield-specific right bitshift */
> > + BPF_CORE_TYPE_ID_LOCAL = 6, /* type ID in local BPF object */
> > + BPF_CORE_TYPE_ID_TARGET = 7, /* type ID in target kernel */
> > + BPF_CORE_TYPE_EXISTS = 8, /* type existence in target kernel */
> > + BPF_CORE_TYPE_SIZE = 9, /* type size in bytes */
> > + BPF_CORE_ENUMVAL_EXISTS = 10, /* enum value existence in target kernel */
> > + BPF_CORE_ENUMVAL_VALUE = 11, /* enum value integer value */
> > + BPF_CORE_TYPE_MATCHES = 12, /* type match in target kernel */
> > + };
> > +
> > +CO-RE Relocation Record
> > +=======================
> > +
>
> [...]
>
> > +CO-RE Relocation Examples
> > +=========================
> > +
> > +For the following C code:
> > +
> > +.. code-block:: c
> > +
> > + struct foo {
> > + int a;
> > + int b;
> > + } __attribute__((preserve_access_index));
> > +
> > + enum bar { U, V };
> > +
> > + void buz(struct foo *s, volatile unsigned long *g) {
> > + s->a = 1;
> > + *g = __builtin_preserve_field_info(s->b, 1);
> > + *g = __builtin_preserve_type_info(*s, 1);
> > + *g = __builtin_preserve_enum_value(*(enum bar *)V, 1);
> > + }
> > +
> > +With the following BTF definititions:
>
> Gmail points to typo in "definitions"
Ouch.
> > +
> > +.. code-block::
> > +
> > + ...
> > + [2] STRUCT 'foo' size=8 vlen=2
> > + 'a' type_id=3 bits_offset=0
> > + 'b' type_id=3 bits_offset=32
> > + [3] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
> > + ...
> > + [9] ENUM 'bar' encoding=UNSIGNED size=4 vlen=2
> > + 'U' val=0
> > + 'V' val=1
> > +
> > +The following relocation entries would be generated:
> > +
> > +.. code-block:: c
> > +
> > + <buz>:
> > + 0: *(u32 *)(r1 + 0x0) = 0x1
> > + 00: CO-RE <byte_off> [2] struct foo::a (0:0)
> > + 1: r1 = 0x4
> > + 08: CO-RE <byte_sz> [2] struct foo::b (0:1)
> > + 2: *(u64 *)(r2 + 0x0) = r1
> > + 3: r1 = 0x8
> > + 18: CO-RE <type_size> [2] struct foo
> > + 4: *(u64 *)(r2 + 0x0) = r1
> > + 5: r1 = 0x1 ll
> > + 28: CO-RE <enumval_value> [9] enum bar::V = 1
> > + 7: *(u64 *)(r2 + 0x0) = r1
> > + 8: exit
> > +
> > +Note: modifications for llvm-objdump to show these relocation entries
> > +are currently work in progress.
>
> Do we need this note here? Doesn't seem like you have any other
> reference to llvm-objdump?
Idk, I hope that llvm-objdump update would be merged eventually and
I'll replace the note with correct llvm-objdump incantation.
