On Tue, Aug 08, 2017 at 01:39:45PM -0400, Ralph Bean wrote: > This is a writeup on a problem we’re facing with modularity, and some ideas on > how to resolve it. > > # The "Problem" > > Imagine I have an **httpd module**. To simplify things, let’s say that this > module has only one stream: **2.4**. Today, in the modulemd for this module, I > declare build and runtime dependencies like this: > > dependencies: > buildrequires: > platform: f26 > requires: > platform: f26 > > This works. > > Now, imagine that Fedora 27 comes along. I want the httpd module to be > installable on f27, so I update those streams to point to f27. But now I can’t > ship updates to f26 anymore! Uh oh. > > ## Just use more streams? > > We could just ask packagers to create more streams to deal with this. httpd > wouldn’t have a 2.4 stream. It would need to have a f26-2.4 stream and a > f27-2.4 stream. > > But, this gets us exactly back into the place we were **before** we had > arbitrary branching! You need to have as many branches for *every component* > as you have releases of the distro (or, as you have "products"). Noooo! > > ## Solution: "Input" Modulemd Syntax Changes > > We’re going to extend the modulemd syntax to allow specifying multiple > dependencies in an "input" modulemd (the one that packagers modify). When > submitted to the build system, the module-build-service (MBS) will *expand* > these values under the hood, and submit **multiple** module builds to > itself based on the expansion. > > Here are some examples of modulemd files (maintained by humans) that might be > submitted to the MBS. > > Build on **all** active streams of the platform module. > dependencies: > buildrequires: > platform: [] > > Build on **only** the f27 and f26 platform streams. > dependencies: > buildrequires: > platform: [ f27, f26 ] > > Build on all active streams of platform **except** for the f26 stream. > dependencies: > buildrequires: > platform: [ -f26 ] Maybe it should be explicitly mentioned that combining the two variants above will be forbidden. > The following syntax, which builds on **only** the f26 stream, will still be supported: > dependencies: > buildrequires: > platform: f26 And this is not only for backwards compatibility -- it's also what the expanded variant will look like. > Take the following example (described above): > dependencies: > buildrequires: > platform: [ f27, f26 ] > > Submitting this modulemd to the MBS would in turn generate **two** module > builds: one of our httpd module against the f27 stream and another against the > f26 stream. Each module build would be associated with its own unique > "flattened" *output* modulemd file that specifies exactly which platform stream > it was built against. > > # New Problems > > Having **multiple** module builds for a single dist-git commit of a modulemd > file poses new implementation problems. > > ## NSV uniqueness > > Today, we uniquely identify a module build in *a variety of systems* with > **<name>-<stream>-<version>** (NSV) where version is derived from the dist-git > commit timestamp. Here we’ll have an httpd-2.4 module built on f26 and an > http-2.4 module built on f27: two different module builds with the same name, > stream, and version. How can we tell them apart? > > We will introduce a new value called the **context** of a built module and > include it in the modulemd metadata that gets carried with the built module. > > * For user facing cases (dnf installation) it will generally be hidden. The > old NSV value will still appear. If a user ever needs to surface the value, > client tooling can find it in the modulemd metadata. The additional > identifier will give users access to explicit pointers to the content that is > installed: > * For cloning a machine. > * For comparing two installed hosts. > * For reporting bugs. > * Some build and compose time systems will have to be modified to use the > context as part of a new unique identifier. **NSVC** will be the > **<name>-<stream>-<version>-<context>**. The systems in question are PDC, > koji/brew, pungi, and bodhi. Perhaps a side note: The exact way of encoding all the module identifiers into a string is still being discussed. I would very much prefer if we used the same format in all systems that have to deal with modules -- not only the infrastructure services listed above but dnf, for instance, as well. Some proposals and comments: https://pagure.io/modularity/pull-request/43 > The context value will be a **hash**, generating as the first step in the build > process (but after expansion). Consider what metadata needs to be hashed: we > think that hashing the whole modulemd is problematic, because the modulemd can > and will be modified after build time. > > Therefore, the *context_hash* value will need to be derived from only the stuff > that uniquely identifies the build and runtime context of the module -- name, > stream, version and crucially, its dependencies > > ## Buildtime/Runtime Dep Correlation > > Another problem. We now have input modulemd files for a single stream that can > expand buildtime dependencies to ‘f27’ and ‘f26’, but what about the > runtime dependencies? > > Consider the following yaml file: > > dependencies: > buildrequires: > platform: [f28, f27, f26] > requires: > platform: [f28, f27, f26] > > With our thinking caps on, this should obviously generate three module builds > (one that buildrequires f28 and run requires f28, a second that buildrequires > f27 and run requires f27, and a third for f26). The naive cross-product of all > streams is not valuable; the MBS needs some logic to **correlate** the build > time requirements with the run-time requirements. That logic will contain > assumptions about how this will be used, and it needs to be well-defined. > > Let’s try to define that. Consider a more complicated yaml file that depends > on both multiple streams of platform and on multiple hypothetical streams of a > "shared-userspace" module: > > dependencies: > buildrequires: > platform: [f28, f27, f26] > shared-userspace: [fancy, nonfancy] > requires: > platform: [f28, f27, f26] > shared-userspace: [fancy, nonfancy] Yes, all this repetition looks ugly. Fortunately, YAML lets you reference and include previously defined blocks, so in practice we could replace the above with: dependencies: buildrequires: &deps platform: [f28, f27, f26] shared-userspace: [fancy, nonfancy] requires: *deps Another point that came up later -- instead of shared-userspace, imagine different streams of your favorite dynamic language. That might make it the reasons for this more obvious. > Here are some rules the MBS should obey: > > * If a build-time dep contains an expansion, that dep **does not** have to also > appear as a run-time dep. > * If a build-time dep contains an expansion, and if that dep *also appears* as > a runtime dep, the runtime expansion **must** match the buildtime expansion > exactly. > > Now that the streams from build-time and runtime *can be mapped one-to-one*, > the cross-product of the deps is taken to produce a set of modules builds. In > our example here, we get: > > 1. One build with platform f28 and shared-userspace fancy. > 2. One build with platform f27 and shared-userspace fancy. > 3. One build with platform f26 and shared-userspace fancy. > 4. One build with platform f28 and shared-userspace nonfancy. > 5. One build with platform f27 and shared-userspace nonfancy. > 6. One build with platform f26 and shared-userspace nonfancy. > > We like this. > > -- > > But, a new** corner case** emerges! What if the *nonfancy* branch of > shared-userspace is only compatible with f27 and f28, but not with f26? > Build #6 will always fail - or, if it doesn’t fail it will be unusable. > > We thought through a number of ways that corner cases like these could be > handled automatically, and are still working through them. At the moment, we > are *leaning towards* suggesting that a failing corner **requires** a new > stream of the module in question to handle the exclusion. I.e., no convenient > syntax to say "all combinations except for this one". > > In this case, you would have two streams of the httpd-2.4 module: > httpd-2.4 and httpd-2.4_f26. The first modulemd file in the > 2.4 stream would be nice, like this: > > dependencies: > buildrequires: > platform: [f28, f27] > shared-userspace: [fancy, nonfancy] > requires: > platform: [f28, f27] > shared-userspace: [fancy, nonfancy] > > The second in the 2.4_f26 stream would exist only to support the desired > corner case: > > dependencies: > buildrequires: > platform: [f26] > shared-userspace: [fancy] > requires: > platform: [f26] > shared-userspace: [fancy] > > -- > > We’re still working through more details, especially on this last part (and how > it may affect upgrades), but figured its best to share now. Any help > appreciated! > > Yours- > -Ralph Thanks for posting this! P
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
