# EPEL-8 Production Layout
## TL; DR:
1. EPEL-8 will have a multi-phase rollout into production.
2. EPEL-8.0 will build using existing grobisplitter in order to use a 'flattened' build system without modules.
3. EPEL-8.1 will start in staging without grobisplitter and using default modules via mock.
4. The staging work will allow for continual development changes in koji, 'ursa-prime', and MBS functionality to work without breaking Fedora 31 or initial EPEL-8.0 builds.
5. EPEL-8.1 is tentatively to be ready by November 2019 after Fedora 31 around the time that RHEL-8.1 may release (if it uses a 6 month cadence.)
## Multi-phase rollout
[As documented elsewhere](https://smoogespace.blogspot.com/2019/06/epel-8.html), EPEL-8 has been slowly rolling out due to the many changes in RHEL and in the Fedora build system since EPEL-7 was initiated in 2014. Trying to roll out an EPEL-8 which was 'final' and thus the way it always will be was too prone to failure as we find we have to constantly change plans to match reality.
We will be rolling out EPEL-8 in a multi-phase release cycle. Each cycle will allow for hopefully greater functionality for developers and consumers. On the flip side, we will find that we have to change expectations of what can and can not be delivered inside of EPEL over that time.
Phases:
1. 8.0 will be a 'minimal viability'. Due to unshipped development libraries and the lack of building replacement modules, not all packages will be able to build. Instead only non-modular RPMs which can rely on only 'default' modules will work. Packages must also only rely on what is shipped in RHEL-8.0 BaseOS/AppStream/CodeReadyBuilder channels versus any 'unshipped -devel' packages.
2. 8.1 will add on 'minimal modularity'. Instead of using a flattened build system, we will look at updating koji to have basic knowledge of modularity, use a tool to tag in packages from modules as needed, and possibly add in the Module Build System (MBS) in order to ship modules.
3. 8.2 will finish adding in the Module Build System and will enable gating and CI into the workflow so that packages can tested faster.
Due to the fact that the phases will change how EPEL is produced, there may be need to be mass rebuilds between each one. There will also be changes in policies about what packages are allowed to be in EPEL and how they would be allowed.
## Problems with koji, modules and mock
If you are wanting to build packages in mock, you can set up a lot of controls in ``/etc/mock/foo.cfg`` which will turn on and off modules as needed so that you can enable the ``javapackages-tools`` or ``virt-devel`` module so that packages like ``libssh2-devel`` or ``javapackages-local`` are available. However koji does not allow this control per channel because it is meant to completely control what packages are brought into a buildroot. Every build records what packages were used to build an artifact and koji will create a special mock config file to pull in those items. This allows for a high level of auditability and confirmation that the package stored is the package built, and that what was built used certain things.
For building an operating system like Fedora or Red Hat Enterprise Linux (RHEL), this works great because you can show how things were done 2-3 years later when trying to debug something else. However when koji does not 'own' the lifecycle of packages this becomes problematic. In building EPEL, the RHEL packages are given to the buildroot via external repositories. This means that koji does not fully know the lifecycle of the packages it 'pulls' in to the buildroot. In a basic mode it will choose packages it has built/knows about first, then packages from the buildroot, and if there is a conflict from external packages will try to choose the one with the highest ``epoch-version-release-timestamp`` so that only the newest version is in. (If the timestamp is the same, it tends to refuse to use both packages).
An improvement to this was adding code to ``mergerepo`` which allows for dnf to make a choice on which packages to use between repositories. This allows for mock's dnf to pull in modules without the repositories having been mangled or 'flattened' as with grobisplitter. However, it is not a complete story. For DNF to know which modules to pull in it needs to set an environment variable for the platform (for fedora releases it is something like f30 and for RHEL it is el8). Koji doesn't know how to do this so the solution would be to set it in the build systems ``/etc/mock/site-defaults.cfg`` but that would affect all builds and would cause problems for building Fedora on the same build system.
## Grobisplitter
A second initiative to deal with building with modules was to try and take modules out of the equation completely. Since a module is a virtual repository embedded in a real one, you should be able to pull them apart and make new ones. [Grobisplitter](https://pagure.io/puiterwijk/grobisplitter) was designed to do this to help get CentOS-8 ready and also allow for EPEL to bootstrap using a minimal buildset. While working on this, we found that we needed also parts of the '--bare' koji work because certain python packages have the same src.rpm name-version but different releases which koji would kick out.
Currently grobisplitter does not put in any information about the module it 'spat' out. This will affect building when dnf starts seeing metadata in individual rpms which says 'this is part of a module and needs to be installed as such'.
## Production plans
We are trying to determine which tool will work better long term in order to make EPEL-8.0 and EPEL-8.1 work.
### EPEL-8.0
| Start Date | End Date | Work Planned | Party Involved |
| ---------- | ---------- | ------------ | -------------- |
| 2019-07-01 | 2019-07-05 | Lessons Learned | Smoogen, Mohan |
| 2019-07-01 | 2019-07-05 | Documentation | Smoogen |
| 2019-07-08 | 2019-07-12 | Release Build work | Mohan, Fenzi |
| 2019-07-08 | 2019-07-12 | Call for packages | Smoogen |
| 2019-07-15 | 2019-07-19 | Initial branching | Mohan, Dawson |
| 2019-07-22 | 2019-07-31 | First branch/test | Dawson, et al |
| 2019-08-01 | 2019-08-01 | EPEL-8.0 GA | EPEL Steering Committee |
| 2019-08-01 | 2019-08-08 | Lessons Learned | Smoogen, et al |
| 2019-08-01 | 2019-08-08 | Revise documentation | Smoogen, et al |
| 2019-09-01 | 2019-09-01 | Bodhi gating turned on | Mohan |
#### EPEL-8.0 Production Breakout
1. **Lessons Learned.** Document the steps and lessons learned from the previous time frame. Because previous EPEL spin-ups have been done multiple years apart, what was known is forgotten and has to be relearned. By capturing it, we hope that EPEL-9 does not take as long.
2. **Documentation.** Write documents on what was done to set up the environment and what is expected in the next section (how to branch to EPEL-8, how to build with EPEL-8, dealing with unshipped packages, updated FAQ)
3. **Call for Packages** This will be going over the steps that packagers need to follow to get packages branched to EPEL-8.
4. **Release Build Work.** This is setting up the builders and environment in production. Most of the steps should be repeats of what was done in staging with additional work done in bodhi to have signing and composes work
5. **Initial Branching.** This where the first set of packages are needed to be branched and built for EPEL-8: epel-release, epel-rpm-macros, fedpkg-minimal, fedpkg (and all the things needed for it).
6. **First Branch** Going over the various tickets for EPEL-8 packages, a reasonable sample will be branched. Work will be done with the packagers on problems they find. This will continue as needed.
7. **EPEL-8.0 GA** Branching can follow normal processes to get done.
8. **Lessons Learned.** Go over problems and feed into other groups backlogs.
9. **Documentation** Update previous documents and add any that were found to be needed.
### EPEL-8.1
| Start Date | End Date | Work Planned | Party Involved |
| ---------- | ---------- | ------------ | -------------- |
| 2019-07-01 | 2019-07-05 | Lessons Learned | Fenzi, Contyk, et al |
| 2019-07 | ???? | Groom Koji changes needed | ??? |
| 2019-07 | ???? | Write/Test Koji changes needed | ??? |
| 2019-07 | ???? | Non-modular RPM in staging | ??? |
| 2019-07 | ???? | MBS in staging | ??? |
| 2019-08? | ???? | Implement Koji changes? | ??? |
| 2019-08? | ???? | Implement bodhi compose in staging? | ??? |
| 2019-09? | ???? | Close off 8.1 beta | ??? |
| 2019-09? | ???? | Lessons learned | ??? |
| 2019-09? | ???? | Begin changes in prod? | ??? |
| 2019-10? | ???? | Open module builds in EPEL | ??? |
| 2019-11? | ???? | EPEL-8.1 GA | EPEL Steering Committee |
| 2019-11? | ???? | Lessons Learned | ??? |
| 2019-11? | ???? | Revise documentation | ??? |
#### EPEL-8.1 Production Breakout
This follows the staging and production of the 8.0 with additional work in order to make working with modules work in builds. Most of these dates and layers need to be filled out in future meetings. The main work will be adding in allowing a program code-named 'Ursa-Prime' to help build non-modular rpms using modules as dependencies. This will allow for grobisplitter to be replaced with a program that has long term maintenance.
--
## TL; DR:
1. EPEL-8 will have a multi-phase rollout into production.
2. EPEL-8.0 will build using existing grobisplitter in order to use a 'flattened' build system without modules.
3. EPEL-8.1 will start in staging without grobisplitter and using default modules via mock.
4. The staging work will allow for continual development changes in koji, 'ursa-prime', and MBS functionality to work without breaking Fedora 31 or initial EPEL-8.0 builds.
5. EPEL-8.1 is tentatively to be ready by November 2019 after Fedora 31 around the time that RHEL-8.1 may release (if it uses a 6 month cadence.)
## Multi-phase rollout
[As documented elsewhere](https://smoogespace.blogspot.com/2019/06/epel-8.html), EPEL-8 has been slowly rolling out due to the many changes in RHEL and in the Fedora build system since EPEL-7 was initiated in 2014. Trying to roll out an EPEL-8 which was 'final' and thus the way it always will be was too prone to failure as we find we have to constantly change plans to match reality.
We will be rolling out EPEL-8 in a multi-phase release cycle. Each cycle will allow for hopefully greater functionality for developers and consumers. On the flip side, we will find that we have to change expectations of what can and can not be delivered inside of EPEL over that time.
Phases:
1. 8.0 will be a 'minimal viability'. Due to unshipped development libraries and the lack of building replacement modules, not all packages will be able to build. Instead only non-modular RPMs which can rely on only 'default' modules will work. Packages must also only rely on what is shipped in RHEL-8.0 BaseOS/AppStream/CodeReadyBuilder channels versus any 'unshipped -devel' packages.
2. 8.1 will add on 'minimal modularity'. Instead of using a flattened build system, we will look at updating koji to have basic knowledge of modularity, use a tool to tag in packages from modules as needed, and possibly add in the Module Build System (MBS) in order to ship modules.
3. 8.2 will finish adding in the Module Build System and will enable gating and CI into the workflow so that packages can tested faster.
Due to the fact that the phases will change how EPEL is produced, there may be need to be mass rebuilds between each one. There will also be changes in policies about what packages are allowed to be in EPEL and how they would be allowed.
## Problems with koji, modules and mock
If you are wanting to build packages in mock, you can set up a lot of controls in ``/etc/mock/foo.cfg`` which will turn on and off modules as needed so that you can enable the ``javapackages-tools`` or ``virt-devel`` module so that packages like ``libssh2-devel`` or ``javapackages-local`` are available. However koji does not allow this control per channel because it is meant to completely control what packages are brought into a buildroot. Every build records what packages were used to build an artifact and koji will create a special mock config file to pull in those items. This allows for a high level of auditability and confirmation that the package stored is the package built, and that what was built used certain things.
For building an operating system like Fedora or Red Hat Enterprise Linux (RHEL), this works great because you can show how things were done 2-3 years later when trying to debug something else. However when koji does not 'own' the lifecycle of packages this becomes problematic. In building EPEL, the RHEL packages are given to the buildroot via external repositories. This means that koji does not fully know the lifecycle of the packages it 'pulls' in to the buildroot. In a basic mode it will choose packages it has built/knows about first, then packages from the buildroot, and if there is a conflict from external packages will try to choose the one with the highest ``epoch-version-release-timestamp`` so that only the newest version is in. (If the timestamp is the same, it tends to refuse to use both packages).
An improvement to this was adding code to ``mergerepo`` which allows for dnf to make a choice on which packages to use between repositories. This allows for mock's dnf to pull in modules without the repositories having been mangled or 'flattened' as with grobisplitter. However, it is not a complete story. For DNF to know which modules to pull in it needs to set an environment variable for the platform (for fedora releases it is something like f30 and for RHEL it is el8). Koji doesn't know how to do this so the solution would be to set it in the build systems ``/etc/mock/site-defaults.cfg`` but that would affect all builds and would cause problems for building Fedora on the same build system.
## Grobisplitter
A second initiative to deal with building with modules was to try and take modules out of the equation completely. Since a module is a virtual repository embedded in a real one, you should be able to pull them apart and make new ones. [Grobisplitter](https://pagure.io/puiterwijk/grobisplitter) was designed to do this to help get CentOS-8 ready and also allow for EPEL to bootstrap using a minimal buildset. While working on this, we found that we needed also parts of the '--bare' koji work because certain python packages have the same src.rpm name-version but different releases which koji would kick out.
Currently grobisplitter does not put in any information about the module it 'spat' out. This will affect building when dnf starts seeing metadata in individual rpms which says 'this is part of a module and needs to be installed as such'.
## Production plans
We are trying to determine which tool will work better long term in order to make EPEL-8.0 and EPEL-8.1 work.
### EPEL-8.0
| Start Date | End Date | Work Planned | Party Involved |
| ---------- | ---------- | ------------ | -------------- |
| 2019-07-01 | 2019-07-05 | Lessons Learned | Smoogen, Mohan |
| 2019-07-01 | 2019-07-05 | Documentation | Smoogen |
| 2019-07-08 | 2019-07-12 | Release Build work | Mohan, Fenzi |
| 2019-07-08 | 2019-07-12 | Call for packages | Smoogen |
| 2019-07-15 | 2019-07-19 | Initial branching | Mohan, Dawson |
| 2019-07-22 | 2019-07-31 | First branch/test | Dawson, et al |
| 2019-08-01 | 2019-08-01 | EPEL-8.0 GA | EPEL Steering Committee |
| 2019-08-01 | 2019-08-08 | Lessons Learned | Smoogen, et al |
| 2019-08-01 | 2019-08-08 | Revise documentation | Smoogen, et al |
| 2019-09-01 | 2019-09-01 | Bodhi gating turned on | Mohan |
#### EPEL-8.0 Production Breakout
1. **Lessons Learned.** Document the steps and lessons learned from the previous time frame. Because previous EPEL spin-ups have been done multiple years apart, what was known is forgotten and has to be relearned. By capturing it, we hope that EPEL-9 does not take as long.
2. **Documentation.** Write documents on what was done to set up the environment and what is expected in the next section (how to branch to EPEL-8, how to build with EPEL-8, dealing with unshipped packages, updated FAQ)
3. **Call for Packages** This will be going over the steps that packagers need to follow to get packages branched to EPEL-8.
4. **Release Build Work.** This is setting up the builders and environment in production. Most of the steps should be repeats of what was done in staging with additional work done in bodhi to have signing and composes work
5. **Initial Branching.** This where the first set of packages are needed to be branched and built for EPEL-8: epel-release, epel-rpm-macros, fedpkg-minimal, fedpkg (and all the things needed for it).
6. **First Branch** Going over the various tickets for EPEL-8 packages, a reasonable sample will be branched. Work will be done with the packagers on problems they find. This will continue as needed.
7. **EPEL-8.0 GA** Branching can follow normal processes to get done.
8. **Lessons Learned.** Go over problems and feed into other groups backlogs.
9. **Documentation** Update previous documents and add any that were found to be needed.
### EPEL-8.1
| Start Date | End Date | Work Planned | Party Involved |
| ---------- | ---------- | ------------ | -------------- |
| 2019-07-01 | 2019-07-05 | Lessons Learned | Fenzi, Contyk, et al |
| 2019-07 | ???? | Groom Koji changes needed | ??? |
| 2019-07 | ???? | Write/Test Koji changes needed | ??? |
| 2019-07 | ???? | Non-modular RPM in staging | ??? |
| 2019-07 | ???? | MBS in staging | ??? |
| 2019-08? | ???? | Implement Koji changes? | ??? |
| 2019-08? | ???? | Implement bodhi compose in staging? | ??? |
| 2019-09? | ???? | Close off 8.1 beta | ??? |
| 2019-09? | ???? | Lessons learned | ??? |
| 2019-09? | ???? | Begin changes in prod? | ??? |
| 2019-10? | ???? | Open module builds in EPEL | ??? |
| 2019-11? | ???? | EPEL-8.1 GA | EPEL Steering Committee |
| 2019-11? | ???? | Lessons Learned | ??? |
| 2019-11? | ???? | Revise documentation | ??? |
#### EPEL-8.1 Production Breakout
This follows the staging and production of the 8.0 with additional work in order to make working with modules work in builds. Most of these dates and layers need to be filled out in future meetings. The main work will be adding in allowing a program code-named 'Ursa-Prime' to help build non-modular rpms using modules as dependencies. This will allow for grobisplitter to be replaced with a program that has long term maintenance.
Stephen J Smoogen.
_______________________________________________ epel-devel mailing list -- epel-devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to epel-devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/epel-devel@xxxxxxxxxxxxxxxxxxxxxxx
