Jesper Dangaard Brouer wrote: > On Wed, 26 May 2021 12:12:09 -0700 > Andrii Nakryiko <andrii.nakryiko@xxxxxxxxx> wrote: > > > On Wed, May 26, 2021 at 3:59 AM Jesper Dangaard Brouer > > <brouer@xxxxxxxxxx> wrote: > > > > > > Hi All, > > > > > > I see a need for a driver to use different XDP metadata layout on a per > > > packet basis. E.g. PTP packets contains a hardware timestamp. E.g. VLAN > > > offloading and associated metadata as only relevant for packets using > > > VLANs. (Reserving room for every possible HW-hint is against the idea > > > of BTF). > > > > > > The question is how to support multiple BTF types on per packet basis? > > > (I need input from BTF experts, to tell me if I'm going in the wrong > > > direction with below ideas). > > > > I'm trying to follow all three threads, but still, can someone dumb it > > down for me and use few very specific examples to show how all this is > > supposed to work end-to-end. I.e., how the C definition for those > > custom BTF layouts might look like and how they are used in BPF > > programs, etc. I'm struggling to put all the pieces together, even > > ignoring all the netdev-specific configuration questions. > > I admit that this thread is pushing the boundaries and "ask" too much. > I think we need some steps in-between to get the ball rolling first. I > myself need to learn more of what is possible today with BTF, before I > ask for more features and multiple simultaneous BTF IDs. Dang, I replied to the original thread before I saw this. In short I don't think you are asking for much, most of it already exists. > > I will go read Andrii's excellent docs [1]+[2] *again*, and perhaps[3]. > Do you recommend other BTF docs? > > [1] https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html > [2] https://nakryiko.com/posts/bpf-portability-and-co-re/ > [3] https://facebookmicrosites.github.io/bpf/blog/2018/11/14/btf-enhancement.html Take a look over the selftests and libbpf BTF headers that should give you the gist. > > > As for BTF on a per-packet basis. This means that BTF itself is not > > known at the BPF program verification time, so there will be some sort > > of if/else if/else conditions to handle all recognized BTF IDs? Is > > that right? > > I do want libbpf CO-RE and BPF program verification to work. I'm > asking for a BPF-program that can supply multiple BTF struct layouts > and get all of them CO-RE offset adjusted. See other thread, In C we call this a union right? > > The XDP/BPF-prog itself have if/else conditions on BPF-IDs to handle > all the BPF IDs it knows. When loading the BPF-prog the offset > relocation are done for the code (as usual I presume). On kprobe side we have lots of code with if (exists(...)) { } else { } Same should be doable from XDP side. > > Maybe it is worth pointing out, that the reason for requiring the > BPF-prog to check the BPF-ID match, is to solve the netdev HW feature > update problem. I'm basically evil and say we can update the netdev HW > features anytime, because it is the BPF programmers responsibility to > check if BTF info changed (after prog was loaded). (The BPF programmer > can solve this via requesting all the possible BTF IDs the driver can > change between, or choose she is only interested in a single variant). I don't think this is evil, it should be expected over time. But, this is two or three steps ahead of where we are. As long as you can get updated BTF either out of band or through kernel interface this works as expected as far as I can tell. > > By this, I'm trying to avoid loading an XDP-prog locks down what > hardware features can be enabled/disabled. It would be sad running > tcpdump (-j adapter_unsynced) that request for HW RX-timestamp is > blocked due to XDP being loaded. Same as above if (exists(...)) {do better version} else {do slow version} > > > > Fake but specific code would help (at least me) to actually join the > > discussion. Thanks. > > I agree, I actually want to code-up a simple example that use BTF CO-RE > and then try to follow the libbpf code that adjust the offsets. I > admit I need to understand BTF better myself, before I ask for > new/advanced features ;-) > > Thanks Andrii for giving us feedback, I do need to learn more about BTF > myself to join the discussion myself. I suspect you will find with some minor user side changes this should already work. > > > > > > > > Let me describe a possible/proposed packet flow (feel free to > > > disagree): > > > > > > When driver RX e.g. a PTP packet it knows HW is configured for > > > PTP-TS and when it sees a TS is available, then it chooses a code > > > path that use the BTF layout that contains RX-TS. To communicate > > > what BTF-type the XDP-metadata contains, it simply store the BTF-ID > > > in xdp_buff->btf_id. > > > > > > When redirecting the xdp_buff is converted to xdp_frame, and also > > > contains the btf_id member. When converting xdp_frame to SKB, then > > > netcore-code checks if this BTF-ID have been registered, if so > > > there is a (callback or BPF-hook) registered to handle this > > > BTF-type that transfer the fields from XDP-metadata area into SKB > > > fields. > > > > > > The XDP-prog also have access to this ctx->btf_id and can > > > multiplex on this in the BPF-code itself. Or use other methods like > > > parsing PTP packet and extract TS as expected BTF offset in XDP > > > metadata (perhaps add a sanity check if metadata-size match). > > > > > > > > > I talked to AF_XDP people (Magnus, Bjørn and William) about this > > > idea, and they pointed out that AF_XDP also need to know what > > > BTF-layout is used. As Magnus wrote in other thread; there is only > > > 32-bit left in AF_XDP descriptor option. We could store the BTF-ID > > > in this field, but it would block for other use-cases. Bjørn came > > > up with the idea of storing the BTF-ID in the BTF-layout itself, > > > but as the last-member (to have fixed offset to check in userspace > > > AF_XDP program). Then we only need to use a single bit in AF_XDP > > > descriptor option to say XDP-metadata is BTF described. > > > > > > In the AF_XDP userspace program, the programmers can have a similar > > > callback system per known BTF-ID. This way they can compile > > > efficient code per ID via requesting the BTF layout from the > > > kernel. (Hint: `bpftool btf dump id 42 format c`). > > > > > > Please let me know if this it the right or wrong direction? > > -- > Best regards, > Jesper Dangaard Brouer > MSc.CS, Principal Kernel Engineer at Red Hat > LinkedIn: http://www.linkedin.com/in/brouer >
