On 09/16/2013 08:25 PM, Toshio Kuratomi wrote:
Note: This started as notes and questions from FPC members at the
meeting, progressed to more notes and questions that FPC members
Remi|Fedora and abadger1999 (me) put together after the meeting, and
ended with a thorough rewrite by langdon (one of the SCL Guideline
drafters) after listening to abadger1999 and Remi|Fedora's
explanations of their notes and questions. The final rewrite of these
comments mostly takes the form of things that Should be changed in the
draft but it must be noted that the final form wasn't created by the
FPC so it is quite possible that the FPC will disagree or want to seek
other options about some of the suggestions made here. Sorry for the
confusing parentage.
Comments on the SCL Draft:
https://fedoraproject.org/wiki/User:Bkabrda/SCLGuidelinesDraft
* Can we assume that the implementation can be changed to address
problems, add features, or follow different conventions? Can be done
if backwards compat is possible? Or is implementation set in stone so
we'd have to kludge around any shortcomings?
Yes, we can assume changes. If you have ideas for improvement I'd like
to hear them. Current guidelines are based on usage by wide variety of
developers and users, so there shouldn't be needed many changes.
* One of the goals of SCLs is isolation from the rest of the system,
correct? It seems, though, that once you run the enable script, that
shell would have a different PATH with the possibility of a different
language interpreter. As long as scripts can use /usr/bin/env as a
shebang or scripts are using PATH-based commands these will end up
using programs from the SCL. Should FPC revisit banning these? Is it
okay to run with SCLs even without banning? Just something to
document?
There are mainly two possibilities how to run apps in collections. It
depends what you wish to do with your application. Please do not ban
anything, people have to decide for themselves what is their use-case.
1/ #!/usr/bin/env something is good for developers who want to switch
between versions. I guess on Fedora will be some developers.
2/ #!/opt/vendor/something/root/usr/bin/something it's used as wrapper,
good for deployed apps, because those who deploy app can define in
wrapper, which collections have to be enabled by default
We do not provide wrappers inside collections, projects usually write
them for themselves.
I would be fix it only in documentation.
== Criteria: When is a Collection the Right Choice? ==
* What is the logical limit of the "size" of that collection?
* (toshio) Criteria should state whether collections are to be
"thin" or "thick". Thin collections would consist of only a specific
target (for instance, ruby1.9.3) and the packages needed to implement
it. Thick collections would include the collection, the packages to
implement it, and packages implemented on top of it.
* langdon has the concept of "Ease of use" packages that are often
needed by people developing in a certain environment (for instance,
adding rails to the ruby1.9.3 collection). Including ease of use
packages would be a restricted variety of thick collections.
* langdon and I also discussed thick collections allowing as many
packages being added to it as maintainers wanted. To take a python
example, if people wanted to support django and flask and pyramid and
TurboGears1 and TurboGears2 on python2.6 SCL, they could have packages
for all of those in the python2.6 Collection. langdon thought this
would be preferable to having framework-level SCLs but I have concerns
about how this plays out with a backwards compatibility policy.
* thin collections would depend on inter-scl dependencies to provide
people with things they want for "ease of use". For instance, there
could be a rails3 scl that depends on scl-ruby1.9.3 that provides
rails3 for ruby1.9.3.
* Naming of scls with inter-scl dependencies needs to be worked
out. You might have people who want to package: rails3 + ruby1.9.3,
rails4 + ruby1.9.3, rails3 + ruby2.0, rails4 + ruby2.0 and all of
those need unique names.
We are afraid that giving so much space for combination might lead to
chaos. I do not want to see many combinations of ruby and rails of
different versions with different versions of bundler etc.
I proposed having SCL SIG which would say something like: "We already
have Ruby193 and Rails2.x. If you want to just update rubygem-XY it's
fine with us." or "We have it, updating rubygem-XY will broke it for
sure for those users, create your own collection."
I agree having smaller collections than we have now would be
improvement, but I'd like to give boundaries to all those possible
combinations. In Fedora must be shipped only those collections, which
are known as stable, well tested, with packages working together.
I'm not sure about this whole idea. The freedom might be a good thing,
but we would have to set boundaries for what will go into Fedora. I do
not have any idea how to do it, then give power to those who work on SCL
or testers in bodhi, but than we would need really huge number of +1 or
someone with veto.
* What may the purpose of the Collection be?
* Criteria should indicate that a "language" may be included but
not a framework (Rails) or a complete platform (LAMP). However,
optional, "ease of use" packages may be included but not installed by
the main meta-package. Otherwise, the scls will need to change/fork
too fast
* (toshio): langdon is convinced this is the right path but I'm
not sure. There's a lot of problems with how this interacts with
backwards compat requirements. I think a whole package has to be
worked out with how those shape up.
Collection in Fedora should give developers, who are not ready to move
to latest version, time to develop on older version.
Typically, Ruby and Rails, Python and mod_wsgi, but I do not suggest to
add fast moving frameworks. We have cpan/pip for such packages.
* Databases and other "servers" may be included but should never
"replace" the native versions
I agree with that. We need some system database (or daemon with some
specific functionality).
* Collections and backwards compat:
* A requirement that scls should fit the same size as reasonable
backwards compat defintions from upstream. For instance, if
foo-X.Y.Z and every foo-X release is backwards compatible with other
foo-X releases, the scl should be for fooX and not fooXY.
Fine.
* The "API" of an scl should never "shrink" only "grow." For
example, if Rails3 is included in the Ruby1.9 SCL, and Rails4 wants to
be included it does not replace Rails3. If, hypothetically, Rails4
cannot be installed "next to" Rails3, a new SCL would need to be
created to support Rails4, which requires a "real" change.
Agreed.
* (toshio) What would the new SCL be named?
* (toshio) This is problematic without also specifying a lifetime
for SCLs and replacement of existing SCLs. Some of the knock-on
questions around this:
* What are we going to do about security issues in SCLs? Does
our policy say that backwards compat is more important than security
issues?
No. We do not want to support not supportable (terribly broken) versions.
* Maybe Fedora simply isn't about backwards compat to this
extent and so we have to allow API breaks within a release.
No, it would loose the purpose of scl for projects building above it.
* Maybe the policy for SCLs in The Outer Rings should have a
strict backwards compat section but SCLs in Fedora Commons are more
relaxed (that would make the model: Fedora Commons SCLs are primarily
for enabling different versions of dependencies for software that we
want to ship in Fedora Commons; Outer Ring SCLs are primarily for
enabling developers to target an SCL rather than Fedora as their
platform.)
Rings are still unclear.
* Maybe our policy should only allow "thin" SCLs. ie: a
ruby1.9 SCL would just have ruby1.9 and anything it needed in order to
run. If someone wanted to ship rails3 and rails4 targeted for
ruby1.9, those would be two separate SCLs that depended on the ruby1.9
SCL. This allows individual SCL pieces to be deprecated without
interfering with their parents or neighbors.
* Only partially addressing the situation, could figure out
some way to mark some pieces of an SCL as having guaranteed backwards
compat while others are only implementation detail. If you had an SCL
that shipped ruby1.9 but it needed a newer version of OpenSSL you
might ship that inside the SCL at first. But you wouldn't want people
to rely on that particular version being available in the SCL in the
future (once the base OS was updated to include that version.)
I might misunderstand. Do you suggest collection should "bundle" such
package as openssl? That's security hole, not good for Fedora in circle
1, 2 or 3.
* No matter where it is, need to have some criteria aroud how
to obsolete an SCL, how to remove an SCL altogether, and whether two
separate SCLs can ship largely the same thing.
Collection can be obsoleted same way as other packages, but let's say
that maintainer should say, he do not want to support it for F-22
anymore (before its release). Those still interested could carry on.
* (toshio) Can we make a recommendation on how or when to use
scl-dependencies. For example, in the Rails4 example above, there may
be scenarios that we want to allow this in Fedora. Although, the
recommendation could be that frameworks and the like should be
provided outside of Fedora Commons.
Do you mean dependencies among collections?
* Should a collection be used only for things that cannot be parallel
packaged (e.g. python 3.3 can be packaged natively so it's not a
candidate for an SCL but ruby1.9 cannot and thus it is)
Depends... Some projects would like to work with collections only, but
currently I'm not convinced if it should be a rule or not.
== Naming ==
* problem with the examples (ruby193 and examples from:
http://fedorahosted.org/SoftwareCollections ) For instance, python3.3
would conflict with a python3.3 parallel stack that did not use scls.
Not true. We used python33 as a metapackage without dot.
* Heard about namespace prefixes and that there were thoughts to name
them after vendors (rht-ruby1.9). If we use /opt/$vendor that might
make sense.
* Could still have clashes with things like redhat-release,
redhat-rpm-config, etc... make sure the vendors we use doesn't have
any pre-existing conflicts of this sort.
* scl makes sense as well. mmaslano had said that end-users wouldn't
know what an scl was but remi says you'd need to use a tool called
"scl" to manage which scls are active anyway. So that may not be a
problem?
That's true. We were also thinking about users who are interested in app
and know nothing about scl, so they just install app, run it and are not
aware of scl at all.
* If we have multi-vendor for scl (fedora, rpmfusion, local admin
case again), then vendor may make more sense again.
* Change example naming: ruby193 => ruby1.9.3 Dots are clearer and
thus preferred
Dots and such stuff are evil. I guess dots were omitted on purpose.
* Do we really want minor versions in there? or should this be
ruby1.9 packages? Need to make clear that the version should be about
backwards compatibility. So if ruby doesn't ensure that minor
versions have compatibility, scl-ruby1.9.3 would be appropriate.
OTOH, scl-python2.7 would be appropriate for an upstream that does
have minor version backwards compat.
It depends on software. For example many people believes that Ruby do
bigger changes in minor releases, on the other hand Python is not known
for adding features into minor releases, so it's just python33 and not
python335.
== File Location ==
* If we're going to use /opt/vendor:
* scls should then obey the rest of the standard: /var/opt and
/etc/opt are for host specific state and config. Is that the case?
can it be fixed if not?
Sometimes were used locations in /var/log or /etc because collections
wouldn't work in case of mariadb and other daemons. We had to put init
file somewhere and also logfile. It's documented and we didn't find
better solution yet.
* We'd need to get a fedora vendor string from LANANA
* Might not want it to be "fedora" that might be more likely to be
taken than something else. "fedoraproject"? "fdr"?
* Other locations:
* /usr/scl
* (toshio): "Large software packages must not use a direct
subdirectory under the /usr hierarchy." which could be interpreted as
forbiding this location. (Although others may be more strict about
defining "large sofware packages").
* %{_libdir}/scl
* (toshio): I don't see an FHS problem with this although the same
issue arises about putting the host-specific config in /etc and the
variable state data in /var. I'm also coming around to
/opt/$registered_vendor being the least-wrong place for this.
I support installation into /opt because it will be confusing to have in
some downstreams /opt and in other /usr. Also it's possible to do
symlinks to workaround some problems and these might broke.
* What does the filesystem layout look like inside of this prefix?
/opt/$vendor/scl/rht-ruby1.9.3/{normal filesystem}?
yes, normal filesystem.
* Does this allow local sites and addon repos to make their own
vendor? How? (Thinking of rhel/centos, epel, and local admin here.
Or fedora/rpmfusion, and local admin)
Modify vendor in macros of scl.
* Are scls allowed to install files outside of the scl tree? If so,
what types of files (criteria) and the naming of these? (systemd unit
files, rpm macro files, launchers for applications/language
interpreters in the scl? Should we go with the FHS recommendation
about /opt/vendor containing everything that is necessary for the
package and then making copies(symlinks maybe?) to other needed
locations?
I'll add those exceptions.
* (langdon) personally, i dont think it should touch anything
besides opt and, potentially, symlinks
* (toshio) Well, how do things like the postgres package that
start a daemon work then?
* (toshio) I don't think we can have symlinks into /opt work as
expected but symlinks out of /opt should work.
/etc/opt/scl/[...]/config and /var/opt/scl/[...]/state are both
host-specific while /opt may be a shared network filesystem.
== Spec Files ==
* Why don't we follow the mingw model and have separate spec files
(mingw is both separate spec and separate package) for scl?
Advantages:
* Many fewer conditionals in spec files (greater readability/maintainabilty)
* Primary maintainer doesn't need to know or care about SCLs if
other people do
* SCL could be the only spec that is necessary or desired for a
particular component
* Can a SCL spec be "only" an SCL spec (will not build a non-scl
rpm)? If so, we need some more content describing this scenario.
* Propose moving the "style guide" to before the example (e.g.
Section 3 before Section 2), so that some of the following questions
don't come up for the reader
* Why are there so many macros that mean the same thing? -- %scl,
%pkg_name, %scl_name, %name
I believe we should have branch for scl and other branches might
voluntarily add macros, but it shouldn't be mandatory. For example perl
or httpd don't have nice specfile and adding more condition wouldn't
improve readability.
Having scl macros might improve speed of patching some packages, but not
all.
=== Specific comments on the draft ===
* Move template to the bottom of the draft. Had many questions after
reading the template that were eventually answered in other parts of
the draft.
* Style guide is not to have a period at the end of Summary
* Description in template should probably be more clear that
packagers should fill in a proper description
* Template skips %prep and %build. Are these empty? If so, just
include the empty %prep and %build headers.
* "The -build subpackage should include Requires: scl-utils-build."
Why a should and not a must?
* Content about the enable scriptlet implies that the usual path
macros (_bindir etc) are changed in the scl package. Need to make it
clear what standard macros are being overridden to what. Perhaps
consider "tables" like
http://fedoraproject.org/wiki/Packaging:MinGW#Filesystem_location_macros
but with slightly different columns, e,g, macro | override? | Normally
Modified? | example | description
* "The macros.%{scl}-config in %{scl}-build package contains the %scl
macro definition (name of the SCL). If you need to use some more
macros for this SCL, add them into this file." <= but there's nothing
in the spec file that obviously creates this. Need instructions on
how to create this file for manual use and then how to include the
file in the spec file and not have whatever automated macro normally
creates it not overwrite it.
* "essential for its runtime". "Essential" needs to be explained and
hopefully it is just a link to the earlier part of the doc.
* -runtime package must be arch specific => it contains
%{scl_prefix}%{_libdir} so it's arch dependent. (and this means the
main package must be arch specific too)
* Creating the enable file in %install -- would rather see macros
like: - %scl_add_path %scl_add_ld_library_path %scl_add_manpath than
the HERE-doc style thing that will cut and paste to most specs.
* "set of macros that contain the values of the original RPM macros"
should be expanded on. naming of these could be better => establish a
common prefix for anything that is being overridden instead of
%pkg_name vs %_root_. also pkg_name probably conflicts with some
existing packages. Maybe something like %_prescl_name,
%_prescl_libdir, etc. [can change the recommendation from: packagers
should define %pkg_name to packagers should define a macro that is the
value of pkg_name => %{?scl: %global srcname %pkg_name and %{?!scl:
%global srcname %name}
* Little more guidance needed for Requires. Maybe making the examples
more real-life would clarify things. ("Since ifconfig is a binary
we're going to use the system version of, do not use scl_prefix with
it. rubygem-foo, otoh, has to be in the path of the ruby interpreter
which this scl is to be a part of. Therefore it needs to use
scl_prefix"). Also, an example of a "Versioned Requires" would be
helpful.
* Need more wording on Requires: of non-scl packages as they can act
as conflicts. Something like: Requires: foo = 1.2.3 … where the scl
package is the only thing keeping foo back at that version. So you
can't update "the OS" without updating the scl. wording might be
something like "yes, you can require non-scl stuff … but you have to
be very general in your requirements, and if you need something that
might be updated you have to either scl bundle it or be willing to
rebuild the scl when it updates".
* Should we list somewhere the scl-modified deps scripts? Otherwise
how will someone creating a new ocaml-scl know whether an ocaml-scl
dep script has already been written?
* Please elaborate on the function of %scl_package_override()
* In the dealing with macro files section: I think that the "Now
consider a situation[...]" starts a whole new scenario? Correct? If
so, needs to be better phrased right now it's associating with the
previous examples but it doesn't seem to make sense in that context.
* So If I've understood everything correctly, building in a standard
buildroot will get you a non-scl'd package. Building with the
$scl-build package installed will build the scl version?
* Should say that in the first line of Building Packages
* In Building Packages: Don't need to include scl-utils-build
explicitly in the config_opts line (brought in via dependencies),
correct?
== Design ==
* What directories is an SCL allowed to modify?
* Naming of packages is a problem as currently designed. Liklihood of
conflict with native packages is very high. Propose to include "scl-"
at the begining of names.
Why don't we already prefixed scl before all packages? Because it would
be scl-ruby193-rubygems-blabla-1.1.f21. The prefix must not conflict
with existing packages. It doesn't have to be ruby193, it might be name
which specify what is inside.
* Include "dots" in names, e.g. ruby1.9 rather than ruby19
I prefer without dots, punctuation does everything harder.
== Conversion to SCLs ==
* Parts of this section are more "tips & tricks"? Should it be on a
non-guideline page?
* Probably best to vette the advice anyway and then move what makes
sense to a separate page afterwards.
* Add information about why /usr/bin/env is not a generally good
thing in SCLs. If this is more general than SCLs, does FPC need to
revisit a ban on /usr/bin/env in shebangs?
* sed line should limit to the first line of the file as well as the
shebang pattern.
* how does scl deal with compiled-in paths inside of binaries? Does
it depend on %configure to do the right thing with %{_libdir} etc?
If so we should point out to people that old-school builds which
aren't configure driven may need patches (actually, this applies to
install location as well as compiled-in paths and to scripts that
have file paths embedded as well as binaries)
* How do auto provides and requires for elf libraries work?
== Inter-SCL Dependencies ==
* Example of using this with versioned Requires?
* The %scl_require example just doesn't seem right somehow.... The
ruby193 scl brings in rails-3.2.3? That seems like poor naming (if
the metapackage brings in rails, then it should be named rails. OTOH,
if the package purpose is to provide ruby193 then rails should be
optional not essential.) or poor dependency management (Which would be
part of the higher level questions of what can be an SCL and what can
be inside an SCL).
* Are non-scl packages allowed to dep on scl packages? If so, how do
they do so?
* The filters don't work in EL6 thing.... people seem more excited
about SCLs in EPEL than in fedora so we probably need to document how
to filter the provides and requires for EL6. However, we can point to
the EPEL6 Differences page from here and document the behaviour there.
== Build & Distribution ==
* What are the impacts on the build system?
Until now we used internal instance of koji. I can sum up what we did there.
* How are we preventing build problems from the mainline package
from interfering with the scl package and vice versa?
There are no build interference problems. Package has specific name.
* How are we allowing for a package to be built in more than one scl?
Didn't solve yet. We related build targets (collection) with branch. How
to build for more collections is a good question for our releng. But do
we need it?
* How are we allowing for differences between scls? (ie:
ruby1.9.3 scl with rubygem-foo1.0; ruby2.0 scl with rubygem-foo2.0)
* How are we preventing build problems from one scl from
interfering with a different scl?
* Remi mentions that he thinks it would be done in a separate
branch. Would we need a branch per scl that the package is built in?
Can't happen. Every branch is set to build in some tag. Packages tagged
in one tag are shipped as one collection. You are able to inherit from
one tag to another.
As I said I prefer branch per collection. How else would be tags defined?
* How do SCLs get distributed?
* Is this a "separate repo" or is it just in the normal repo(s)?
Currently separate repo.
* (toshio) Both. Some SCLs can go into Fedora Commons which
would continue to have one repo. SCLs in the outer rings would have
some separate structure.
Agreed.
* Build section would benefit from minor sections, perhaps:
"Overview," "Testing" and "Example"? Sections need a bit more color to
be understandable.
I'll add it.
== General Comments ==
* mingw is a similar set of guidelines (but in separate spec files
rather than the same one.) So issues like what their naming solves
and tracking of fedora packages could be looked at.
http://fedoraproject.org/wiki/Packaging:MinGW. The mingw packaging
guidelines should also be reviewed for "missing content" vs this
proposal.
* No definition of a SCL appears in the document
* Example text: Software Collections (SCLs) give rpm packagers a
means of packaging multiple versions of software for a single
distribution. The version from the software collection does not
interact with the system version. They can provide backwards and
forwards compat versions of a package compared to what's on the
system.
* Include "definitions" at the beginning of the document, some
example things that need definition (and suggestions)
* SCL -> whole software collection? <- originally the term was
"dynamic software collection" for the "whole thing" but it has fallen
in to disuse... we could bring it back
* SCL defining package -> the srpm-level package that defines the scl.
* SCL metapackage -> built from the SCL defining package. defines
the essential packages you get when you install the scl.
* SCL runtime package -> built from the SCL defining package. It's
the setup of the system for this SCL including registering the SCL
with the system and setting up the filesystem.
* SCL build package -> built from the SCL defining package. It
contains macro definitions needed to build an SCL.
* SCL package -> any package which can be built for an SCL.
* Elaborate on what "belongs" in the "-runtime" package
* Extracting one of these from the sample repository seems like
it's a filesystem package directory tree inside of the scl? Anything
else?
* From the SCLinFedora page, why the differentiation between regular
package reviews and scl package reviews?
-Toshio
--
packaging mailing list
packaging@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/packaging
Thanks for your input. I was waiting for something like that so I could
add what is missing or unclear to people who never build a collection.
I'm looking forward to reaction on updated drafts. We will do it with
Slavek tomorrow.
Marcela
--
packaging mailing list
packaging@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/packaging
