sorry for top-post, but one more thing to consider, if caching would be used, are secondary arches - how it would affect their ability to build the buildroots Dan On Sat, 17 Jan 2015 11:07:53 -0600 Dennis Gilmore <dennis@xxxxxxxx> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Sat, 17 Jan 2015 03:44:10 +0000 > Peter Robinson <pbrobinson@xxxxxxxxx> wrote: > > > >> >>> > > that all being said. koji doesn't use any caching and > > >> >>> > > will not use the lvm plugin. we make every buildroot from > > >> >>> > > scratch using a fully clean environment to help with > > >> >>> > > ensuring reproducability. > > >> >> > > > >> >> > You can cache and still preserve reproducability. What I'm > > >> >> > planning for Copr is to do (every week/month) for chroot in > > >> >> > fedora-20-x86_64 fedora-21_86_64 ... ; do mock --init > > >> >> > $chroot done > > >> >> > take snapshot of that. I plan to do that on VM level. > > >> >> > And when new task come, I will just restore from that > > >> >> > snapshot. And mock will start with already populated cache. > > >> >> > So I will have better caching and yet reproducability. > > >> > you really can't. you would need to make a new cache any time > > >> > one of the packages in the minimal buildroot changes. > > >> > > >> That is why mock run 'yum update' on minimal buildroot before > > >> proceeding further. So if one package in buildroot changes, you > > >> will just download and install just that one package and all > > >> other comes from cache. > > > > > > That invalidates being able to reproduce the build root exactly. > > > as you would need to know and reproduce the package update. This > > > is something that has undergone a lot of thought and > > > consideration and there is very very good reasons things are the > > > way they are. as i showed elsewhere in this thread most repos are > > > used only once or twice at most on any given host. > > > > The only way you could cache a base build root effectively is if you > > track the packages and NVRs and when one of them bump you regenerate > > the buildroot, likely pausing all builds until a new one is there. > > there's likely not that much churn in the packages contained in the > > root buildroot (you'd want to make sure you had everything for > > building .src.rpms too) and in some cases you could likely get days > > on the same image without having to rebuild it, other days you > > might have to do it a few times a time. > > As I showed earlier in the thread most repos are only ever used once > or twice on a host. but we do not know if something in the minimal > buildroot has changed. there is a lot of added complexity needed to > attempt to use caching. which adds many more chances for something to > go wrong. I honestly do not know if its quicker to build the minimal > buildroot from scratch or to unpack the tarball. there is costs both > ways. with the added complexity in tracking things to try and make > caching worthwhile. in the Fedora case we would need to have > somewhere in the order of 6-12 cached buildroots on each builder > depending on sidetags etc in use at different times. but there is > instances of koji where that number would be in the hundreds. we need > to make sure that it works and scales not only for our use cases but > also other users of koji. > > > Whether the effort to implement out weights the effort remains to be > > seen. I think, given the low IOPs on the builds with an underlying > > COP image it might well give a reasonable reduction in building > > especially during mass rebuilds where the builders are very active > > and as it's in a side tag the new build root packages don't end up > > in the build root until tagged. Using fedmsg it's likely not hard > > either to monitor a package subset for changes and regen the image. > > There is a massive effort to figure out if the effort to implement any > caching is worth while. If someone wanted to take the time to figure > that out, making sure to gather the needs of different koji users to > make sure that we did not make things worse for them, then by all > means that person is welcome to do so and I will gladly answer any > and all questions that they had. However I have no interest in going > down that rabbit hole. we can not rely on koji users using fedmsg so > that is something that would likely need to be ruled out of the design > decision. fedmsg also does not guarantee delivery of messages and so > can not be relied on for anything where we have to be 100% sure that > things are done given a trigger. > > Dennis > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2 > > iQIcBAEBAgAGBQJUupbqAAoJEH7ltONmPFDR+iQP/ietNky5y2yhtbUZ/MMGP1er > PDMyHz5MD9v78mstXXjx6WsBrOjc2a51WCufBV2drlPuwyTrstwJShbOhm1XfOld > 1HSI9etIkwsVxBFgRfh6HUxbLGZWSHDZviY34KocxLNuuaRLqc5RHjTTZmzQwKl+ > 4EA4o6HJAYPsVz0Fu2/sJAy9cCclfCPy8Mcpstd1NIHOPhtNTpFiEQBl4zpXRqQs > nlmlNh1MhZepv84k+EvhRZCJlmfn0yLBLlNJfNh6jtdy6zv+lNf2oClrHDR9o5aL > nRCEJhRl/LX+BNf2mV5MULxAliXMME1qoYrm//BgsQk45t3XvVJ2fpPEVs8gcr49 > gZ0yoJ20fcSW/UJCRMhoI9dtp1MYQmF3h3LHfNRROv+Ga9Uumw9OmQyFqFJwbF2o > doLOvDxtpEXWsiXcTUSqcl3CBGyT50AiAbRvv5yrQ5TU1Z8TplcUVUbJ/zifBoid > GexFUETsN3E1P0byvfbEiJJgaAYkWTaho2SwO+gRY/iieHlWyEbK6BwAiw7BsJOM > Tm5P6EjHyqk1kPiycoG9Zz70iiBhrFraMgzWwtrjNid2n8vvV6pdEpXpiaDzOAgS > SdVpHH2R8tu8GjnbONazSt6GpG7oEqnS5Nz2YZT6MMTcNuCvxY5bVoRfAl4DlDTM > X/98RH2w7xXM/fejAZN3 > =Vjxf > -----END PGP SIGNATURE----- > -- > devel mailing list > devel@xxxxxxxxxxxxxxxxxxxxxxx > https://admin.fedoraproject.org/mailman/listinfo/devel > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
