On 06/09/2015 08:44 PM, Sage Weil wrote: > On Tue, 9 Jun 2015, Owen Synge wrote: >> On 06/09/2015 07:22 PM, Sage Weil wrote: >>> On Tue, 9 Jun 2015, Owen Synge wrote: >>>> Sorry to catch this thread late. >>> >>> There were two goals here: >>> >>> 1- make the generated tarball deterministic and independent of >>> configure options. Currently we have horrible hackery to include rocksdb >>> source in the tarball when configure doesn't get the rocksdb arguments, >>> and were hitting similar roadblocks/pain with libxio. This is avoided >>> with the make_dist.sh script (although we're not using it yet because of >>> the specfile generation piece). >> >> I think deterministic is a reasonable goal. >> >> I think independent of configure options, is not really a goal, its a >> solution to a goal that misses a couple of observations. The goal is >> deterministic operation. >> >> The important observations are: >> >> (a) The same source will need to be released multiple times. >> (b) It is nice to release with some differences to each environment. > > This doesn't make sense to me. I guess this is a matter of my poor wording. > There should be *one* upstream .tar.gz (or > whatever) that is the canonical source associated with a release. I agree, git at git hub provides this with a tag. I then think it is presented, on a web site, at this stage the upstream .tar.gz can be post processed if needed. The source is taken by distributions such as redhat, suse, debian, ubuntu, and presented to users both as source and as binaries. Packaging files including ceph.spec being made from ceph.spec.in are part of that post release presentation work that have (for reasons of pragmatism) been placed in the same git repository. While this packaging code/config has to be traceable to a canonical source, they do not have to be the canonical source, and may be generated from templates, provided the values are stored. > That is > entirely independent of which distro you eventually build that source on, > and should (more or less) correspond to a snapshot of the git tree for the > release tag. To clarify; the bits where it does not correspond to a fixed value in git that this discussion is based on. I suggest all such values can be stored together and placed in a release script. > Currently the only reason ceph.spec is generated from autotools is to fill > in the version string. I can see why filling in all the other stuff makes > sense, but that points toward a process like > > tar zxvf ceph-9.2.1.tar.gz # that's the canonical release tarball > cd ceph-9.2.1 > ./autogen.sh > ./configure --with-systemd-libexec-dir=/usr/libexec/ceph \ > --with-rpm-dir=/export/redhat/ > make srpm > > right? right, > Making different release tarball flavors for each distro seems crazy to > me... Thankfully I cant think of anything in ceph that it would needed be needed for apart from the version information. If you had more variables like the version information that was distribution specific then per distribution tar balls might be needed. Should their be a "support contract" variable, this might make the need for different source releases having different values in templated code. I am glad ceph does not have or need such a variable. Owen >> If you combine (a) and (b) you quickly see why the "--configure" step >> exists before: >> >> make dist-bzip2 >> >> as say on release you iterate between distributions to produce each >> setup deterministically. >> >> To illustrate I have put a little of my proposals for a replacement >> model of "make_dist.sh" to illustrate this in my opinion improved process. >> >> But if this step exists imagining a couple of patches later: >> >> $cat make_dist.sh >> #!/bin/sh -e >> # Common code >> autogen.sh >> # Redhat >> ./configure --with-systemd-libexec-dir=/usr/libexec/ceph \ >> --with-rpm-dir=/export/redhat/ >> make srpm >> # Suse >> ./configure --with-systemd-libexec-dir=/usr/lib/ceph/ \ >> --with-rpm-dir=/export/suse/ >> make srpm >> # Debian >> ./configure --with-systemd-libexec-dir=/usr/lib/ceph/ \ >> --with-sdeb-dir=/export/debian/ >> make sdeb >> # Ubuntu >> ./configure --with-systemd-libexec-dir=/usr/libexec/ceph \ >> --with-sdeb-dir=/export/ubuntu >> make sdeb >> >> This will also solve the problems (1) and (2) with ceph.spec.in to >> ceph.spec transformation. And not just solve it once but make it clearer >> for each distribution. >> >> Problems. >> >> These are problems with your planned approach I don't have good answers >> for, and this approach solves now. >> >> (1) having autotools and another thing template ceph.spec.in could be >> complex. >> >> (2) This code need to be documented as its not standard autotools. >> >> (3) How much faster must the template tool be to be acceptably faster >> than ./configure >> >>> 2- avoid multiple passes through ./configure for 'make dist' and the >>> subsequent build. We currently do it for the 'make dist' step (which also >>> generates ceph.spec) and then for the eventual build. Using autotools for >>> generating the spec means we still do it twice. >>> >>> I'm less concerned about build speed and it may be worth consolidating all >>> of the distro logic in configure.ac. Honestly I don't really know if >>> that is better or not (vs conditionals in the specfile)... I have no real >>> love for autotools. >> >> autotools is not some thing I love, but I find it very useful, within >> the scope of building C/C++, I have found it has a solution with _every_ >> use case already catered for. You just need some time to find it, and it >> may look different from what you thought you wanted and how you would >> make it, but I find the pre made autotools solution is a better solution >> than making your own from experience. >> >> I must say though, autotools is much nicer than rpm for macros and >> variables and template expansion. >> >>> But hopefully the tarball step can remain a fast, deterministic step that >>> doesn't require a configure pass? >> >> I really support you desire for deterministic. >> >> I commend your desire that each step remain fast. >> >> I believe you are missing a step if you follow the path of skipping the >> configure pass, and you loose some important functionality. >> >> I point out now what functionality in the hope that we can use it to >> make other work easier, and autoools can be used to template any files >> it needs to, including at least, spec files and deb files by putting >> paths into autotools. >> >> >> Owen >> >> >> >>> sage >>> >>> >>> >>> >>>> >>>> I come here via patch >>>> >>>> https://github.com/ceph/ceph/pull/4911#issuecomment-110422312 >>>> >>>> I think you guys are missing that configure is doing some thing here. >>>> >>>> (1) Configure is generating the spec file. >>>> (2) It could also generate the deb files. >>>> >>>> What no one has done is to: >>>> >>>> (A) Use configure to eliminate the differences between OS's. >>>> >>>> If any one had done (A) you would not be considering this I hope. >>>> >>>> This impact is particularly felt in the ceph.spec.in file where because >>>> of the lack of (A) in the process we are plagued with OS specific >>>> conditionals. >>>> >>>> If the path of say: >>>> >>>> src/ceph-osd-prestart.sh >>>> >>>> is ether: >>>> >>>> /usr/libexec/ceph >>>> >>>> or: >>>> >>>> /usr/lib/ceph >>>> >>>> >>>> Is purely distribution specific. >>>> >>>> Having conditionals in >>>> >>>> ceph.sepc.in >>>> >>>> defeats the point of generating it via autotools. >>>> >>>> we can be remove many hard coded values replaced with variable and that >>>> probably will only grow in number for example >>>> >>>> %if 0%{?rhel} || 0%{?fedora} >>>> --with-systemd-libexec-dir=/usr/libexec/ceph \ >>>> %endif >>>> %if 0%{?opensuse} || 0%{?suse_version} >>>> --with-systemd-libexec-dir=/usr/lib/ceph/ \ >>>> %endif >>>> >>>> some wont need distribution specific locations like: >>>> >>>> --with-systemd-unit-dir=%_unitdir >>>> >>>> In the long term replaced all these path variables could be replaced by >>>> single parameter. >>>> >>>> ./configure --with-distro-defaults=redhat >>>> make rpm >>>> ./configure --with-distro-defaults=suse >>>> make rpm >>>> ./configure --with-distro-defaults=debian >>>> make deb >>>> ./configure --with-distro-defaults=ubuntu >>>> make deb >>>> >>>> Very easily. >>>> >>>> In summary, please reconsider this decision as if you follow this policy >>>> we are left with conditionals all over the ceph.spec.in file. >>>> >>>> >>>> Best regards >>>> >>>> Owen >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 05/05/2015 09:31 PM, Sage Weil wrote: >>>>> On Tue, 5 May 2015, Loic Dachary wrote: >>>>>>> I think the long-term solution to Kefu's issue is that we need to >>>>>>> remove the requirement to run through a full "./configure" invocation >>>>>>> just to get a tarball. All the RPM and Debian packages internally run >>>>>>> ./configure, so running it a second time slows things down. I think it >>>>>>> makes sense to implement the tarball-generation functionality using a >>>>>>> simpler script at the root of the ceph.git tree. The operation should >>>>>>> be about as fast as "git archive". >>>>>> >>>>>> I agree. It's going to be significant work but it's worth it. >>>>> >>>>> Yep! >>>>> >>>>>>> The "ceph.spec.in" -> "ceph.spec" suffers from a similar issue. It >>>>>>> takes a full "./configure" run to get to a point where Make can write >>>>>>> the proper version numbers into that file. Ideally we could skip all >>>>>>> of that and simply do the variable interpolation with sed or something. >>>>> >>>>> Yep! >>>>> >>>>> sage >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>> >>>> -- >>>> SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB >>>> 21284 (AG >>>> Nürnberg) >>>> >>>> Maxfeldstraße 5 >>>> >>>> 90409 Nürnberg >>>> >>>> Germany >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >> >> -- >> SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB >> 21284 (AG >> Nürnberg) >> >> Maxfeldstraße 5 >> >> 90409 Nürnberg >> >> Germany >> -- SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
