On 21/02/17 04:53PM, Toke Høiland-Jørgensen wrote: > "Brian G. Merrell" <brian.g.merrell@xxxxxxxxx> writes: > > > On 21/02/15 01:47PM, Toke Høiland-Jørgensen wrote: > >> "Brian G. Merrell" <brian.g.merrell@xxxxxxxxx> writes: > >> > >> > On 21/02/11 12:18PM, Toke Høiland-Jørgensen wrote: > >> >> "Brian G. Merrell" <brian.g.merrell@xxxxxxxxx> writes: 8< snip > > OK, what currently happens is we have a separate, centralized Go web > > service exposing an HTTP based API. When the sysadmin calls that API it > > stores the config data in a database. Then, we have another service that > > periodically queries the database and writes the config data to a > > constant database (cdb) and stores that in blob storage. Then, there is > > a service running on each node that periodically pulls down the latest > > cdb. Our orchestration tool running on each node is watching for new > > cdbs using inotify; when the tool sees a new CDB it loads the new > > configuration data--which, for us, literally ends up just being JSON > > data--and does anything that needs done. > > > > I had omitted those details for a couple of reasons: First, it's kind of a > > lot and I didn't know it would be helpful. Second, this is the way it > > works currently because, for expediency, we leveraged the internal > > ecosystem that was already setup. We will likely move away from it, at > > least partially. > > > > So, I think the important part is that our orchestration tool will > > periodically get the config data in JSON format. A path to each BPF > > program is in the config data and the orchestration tool downloads them > > as needed. We may move to just including the BPF program binary in the > > config data--TBD. Obviously, we aren't using libxdp yet, so our config > > data doesn't have "run priority", instead the config data has the order > > the BPF programs need to run, and BPF programs themselves have to do the > > bpf_tail_call (and the orchestration tool does a bunch of complicated > > orchestration to get the chain in the right order). The config data also > > contains a bunch of other information to do the orchestration, e.g., > > interface, ingress or egress, tc or xdp, what userspace code to run and > > any config values for that, etc. > > > > Hopefully that answers your question, and sorry if it was too much > > information :) > > No, that was very helpful, thanks! Just the kind of detail I was after > to understand your deployment scenario :) OK, phew :) > >> > I was planning to set the run order programatically on the XDP program > >> > objects via libxdp calls. It looks like your libxdp implementation > >> > already has ways to do this in the form of xdp_program__set_run_prio() > >> > and xdp_program__chain_call_enabled(). > >> > > >> > Does that make sense? This is still all very theoretical for me at this > >> > point! > >> > >> Yup, totally possible to set this programmatically with libxdp as well > >> today. However, before doing so you still need to communicate the list > >> of BPF programs and their run configuration to each node. And I'm > >> thinking it may be worthwhile to specify how to do this as part of the > >> "protocol" and also teach libxdp about the format, so others won't have > >> to reinvent the same thing later. > > > > It seems like I must be missing something here, but my plan was to do > > this all programmatically by calling libxdp functions from the > > orchestration tool by 1) calling something like xdp_program__open_file() > > to load the XDP program, and then 2) setting the run configuration by > > calling something like xdp_program__set_run_prio() and > > xdp_program__chain_call_enabled(), and 3) adding the programs to the > > dispatcher and loading it. > > > > Correct me if I'm wrong, but I guess you're saying that it might be > > worth creating an abstraction in libxdp where a user can pass in the > > necessary config data and libxdp does the work, that I just summarized > > in the previous paragraph, on the user's behalf. I can see how that > > could be a useful abstraction. > > Yeah, so what I was thinking was whether it would be useful to, for > instance, define a "bundle" format that contains the config data that > libxdp will understand. Could just be a JSON schema containing keys for > priority, chain call actions and a filename, so you can just point > xdp_program__open_file() at that and it will do the rest. > > However, I'm still not quite sure I'm convinced that this will be > generally useful. As you say, you can just as well just set the values > programmatically after loading the file, and I suspect that different > deployments will end up having too much custom stuff around this that > they'll bother using such a facility anyway. WDYT? Yeah, my hunch is that different deployments will have custom config data and then getting the data into libxdp format will just be another data conversion step. I could be wrong. If it is decided to create a data format, then the JSON schema you described is pretty much exactly what I had in mind, too. For whatever that is worth! > > I know in a previous e-mail you mentioned having a config file with > > priority overrides. That's just not a use case that our team would want > > to use. And, my opinion would be that the program using libxdp should be > > the one to implement that sort of policy; it keeps libxdp more simple > > without needing to worry about parsing config files (and handling config > > version changes in the code and the spec). For example, xdp-loader could > > have a config file with priority overrides and people could use that > > code if they wanted to do something similar. > > Yeah, that's totally what would make sense for your deployment case. The > design where libxdp reads a config file comes from my distro > perspective: We want to build a system whereby different applications > can each incorporate XDP functionality and co-exist; and the goal is to > make libxdp the synchronisation point between them. I.e., we can say to > application authors "just use libxdp when writing your application and > it'll work", while at the same time empowering sysadmins to change the > default application ordering. Ah, yes, that perspective helps a lot; I understand much better why you would want a centralized configuration file. My mind is actually kind of melting now thinking about all the use cases. Is the idea that the configuration file would have the final word? For example, if there are multiple applications using libxdp (and possibly even programatically overriding their priorities), could a sysadmin then write a config file to dictate what they actually want? That would made sense to me. If that's the goal, then I guess I would take back my opinion about that policy not belonging in libxdp. I think application writers would be less likely to use the configuration format (for reasons we discussed above), but it does seem like a necessary mechanism for sysadmins to orchestrate multiple applications that don't know about each other. For my team's use case, we have the luxury of being the One Application to rule all BPF programs. > > By having that configuration be part of the library, applications can be > free to use either the command-line loader or include the loading into > their own user-space binary. > > But since you are (notionally) both the application developer and system > owner, that is less of a concern for you as you control the whole stack. > > > Hopefully I'm even making sense, but like I said, I don't have strong > > feelings about the format, as long as we are able to achieve our > > required use case of programmatically setting the run configuration > > values from a libxdp user program. > > Sure, that you can certainly achieve with implementing what libxdp > includes today. I'm just trying to make sure we explore any > opportunities for standardising something useful so others can benefit > from it as well; so I hope you'll forgive my probing :) 8< snip > > We explicitly do not want defaults set by program authors. We want that > > policy to be completely in the hands of the orchestration environment. > > Right, OK. How does the admin configuring the orchestration system > figure out which order to run programs in, BTW? Is this obvious from the > nature of the programs, or do you document it out of band somewhere, or > something like that? We're a pretty huge organization... lots of DCs, public cloud, private cloud, different kernel versions, sister companies, hundreds of applications, etc. We want anyone to be able to write cool BPF programs and userspace applications without needing awareness of what's running before or after or if that order might change in the future. I'm sure the desired order will be more obvious for some programs than others, but we have administrators that can analyze the BPF programs, compose multiple BPF programs together, and order and reorder them. We have a team of people that can work with teams to resolve any interdependencies if necessary. As an example, we've done something similar for HTTP ingress and egress Lua plugins in the past. We have dozens of teams that write Lua code to do custom L7 things with HTTP requests and responses, and then we have a UI where admins/ops folks can literally drag and drog the plugins into the desired order. We wouldn't want teams making assumptions about what order plugins should run in, either. Hopefully that helps! Thanks again, Brian
|