On Tue, Nov 27, 2018 at 7:59 AM Owen Taylor <otaylor@xxxxxxxxxx> wrote: > > A lot of discussion about improving the compose process seem to end up > with a "reality check" - that ideas have already been tried but don't > work because of requirements a) b) c) d). You can't have the pony, but > maybe if a lot of effort is put into it, you can have a faster rocking > horse. > > If want to fundamentally improve the Fedora workflow we need compose > ponies, we can't just have rocking horses! There have several efforts to improve Pungi performance over time. Is there any email list or communication channel where this effort could be coordinated? I work on a project in Ceph that uses Pungi a lot, so I'm really interested in making composes as fast as possible. Our Jenkins system runs Pungi several times a day (every time a build completes in Koji), so that we can deliver composes to QE immediately. I'd like to run it even more frequently (like on every pull request scratch build). Maybe we could write a dedicated page in Pungi's upstream documentation, "performance tips for making Pungi as fast as possible". It could explain the dogpile.cache stuff, hardlinks vs http, etc. > I don't know what the system would look like exactly, but you could > imagine things like: > > * Composed of several micro-composes (micro-compose-services?) to > avoid blocking on everything completing successfully. > > * Able to do speculative composes for CI > > * Either x86_64-only, or with decoupled architectures so that we can > throw x86_64 hardware (or cloud resources) at it, and make it super > fast. > > * No IO /mnt/koji during the compose - having a big network share be > central to the process creates a performance bottleneck, makes it hard > to move to the cloud, and potentially adds a lot of "noise" to > figuring out what is going on where things are slow because of some > other entirely different thing is goin gon. This is a great idea, although it's a little tricky to do everything in a local tmpdir and still take advantage of the speed that NFS hardlinks provide. > Add your own bullet points :-) Another hairbrained idea: reduce or eliminate Pungi's thread model and make it asynchronous, using https://github.com/ktdreyer/txkoji :) _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
