On 24 January 2018 at 11:32, Samuel Rakitničan <srakitnican@xxxxxxxxxxxxxxxxx> wrote:
By above want I want to say?
> 1) easier to test upgrades between Fedora versions
>
> As each Fedora major release may have in updates only security and critical
> fixes and ABI (libraries SONAME changes) will be completely locked down it
> will be easier work on a set of test units testing upgrades on top of
> different sets of installed packages.
Doesn't relate to packages which rarely see an update if at all
But *why* asking for troubles and really not keep as REALLY stable what STABLE in distro releases SHOULD be? (even +updates)
I'm really sure that there are much more interesting things touch ABI up to the EOS. Isn't it?
> 2) more people (packagers and regular end users) will be focused on testing
> of latest Fedora version
Do you have some results of a testing or you just assume this would be the case
Yes. Out of my 7 years working and leading PLD more than half I've spend working on two PLD version (const devel and stable).
All necessary changes ni CVS have been kept on ***branches***. Today with git it will be way easier than +10 years ago.
Is it enough?
Minimal overhead on maintaining all packages where in only two flat CVS modules SOURCES and SPECS.
I'm not suggesting to repeat this because current Fedora git model is good enough with additional ACLs control.
Remember that it was only few of us actively working every day on whole distribution.
Even with so imaginable small man/hours resources we been able to punch first 3K (without perl/php packages) source packages barrier when at this time in RH was still IIRC still less than 1k.
When poldek was in first quite usable version (credits to Paweł Gajda) quite quickly was possible to reach high level of rpm dependencies consistency or few types of conflicts checked on top of really wide/broad set of packages without tying to test everything.
Common and strictly guarded spec files indentation/format was FUNDAMENTAL to reach very high level of replace ability on maintaining already CLEAN specs files.
Will divert here (again as I'm quite often doing) to specs format/indentation ..
When only recently I've spotted that core Fedora packages (those which are in RHEL) are maintained by (probably) ONLY RH employees (even as secondary admins), and we are talking about at maybe few hundredths people strong team I'm 100% sure why FPG parts about "common specs format" are only empty wish.
Why? Because communication factor grows with factorial number of people.
In so big team always will be someone who "I didn't know that is is now new rule" or "I don't like this".
Those people are probably quite often are so busy that they are "running with empty barrows on transporting bricks between place where truck unloaded bricks and construction place because they always says that we are so busy that we don't have time to load and unload those bricks" (old communist time joke).
Because many specs are really complicated because it was no proper peer review as result some isolated woks on packages.
This at the end created highly over complicated (by what was dine in the past). This on next level as consequence caused probably as well some number of bad cases on moving packages between people. And finally decision "we cannot loose straight control on this core Fedora distribution packages because only few on this world knows how to maintan each package".
This caused as result limited level of Fedora<->RH trust.
No .. I know that if it is true it is pure bollocks. When I was working PLD I've been controlling ALL packages. Everyone of us in PLD was able to do this using only "sense of broken patterns". Each day review new changes took me only up to 1h. Rest of the free time I've been spending on adding second the same batch (if not more) my own changes.
Yes, I'm good on finding those "broken patterns" and in PLD I was only one and this is why (ONLY) PLD started going down when other people in PLD started thinking on monetize whole project (all without telling me even single word about this).
Yea I was young, naive and still idealist .. who did't?
What I'm trying to say is that IMO I probably know why up to now it was not possible to introduce common indentation in RH and Fedora as well.
(bad news)
I know probably as well why now it will be even harder.
.. because (as result)
a) some people may loose some responsibilities
b) it will be no more "my little precious" spec files
c) some people by this may loose some part of they "uniqueness"
.
.
Choose something ..
In other words on this area way bigger some "human/ego" obstacles than all those technical ones. Am I close?
If not .. try to think why (probably as first but up to now not last) PLD was able to attract attract some people with quite basic technical skills when after few year we reached things which even RH with thousands technical employees is not able after +10 years still is not able to reach. More .. even with few time more Fedora guys still many things are unreachable.
So why in PLD was possible to so high levels? Are we talking about some freaks/geniuses? Of course not ..
We been spending much more time (proportionally) as packagers on improve our METHODOLOGIES ..
Please tell me that I'm not enough close to REAL core ..
(I'm not afraid to say publicly "I was wrong" and apologize all RH hard working employees if I'm wrong)
If I'm really wrong just please tell me why it is like this (without wrapping this into the cotton) .. why it was not possible to standardize so simple/basic thing in Fedora and RH as well?
However if I'm right no one from RH should feel guilty and only recall old Roman sentence "errare humanum est, sed in errare perseverare diabolicum".
Nevertheless, I'm not kind of idiot who is trying to fight and I know that even "Hercules is a wimp when he must fight against heaps of enemies" (Latin original is way shorter "nec Hercules contra plures")
No, I'm only a guy who most of his life spend looking for right "why?" questions.
Guy who knows that only knowledge of correct question is causing almost instantly find right answer.
Yes, I know that I'm exposing myself quite often on lack of understanding but that is only the cost.
Cost which I'm (always) ready to pay .. to have much higher yield/speed/whatever.
OK .."Ab ovo" ..
(I promise .. to the end of this email will be only about technicalities)
Second PLD component was "builder" script which was used on downloading specs and necessary, tagging, retagging, build packages, committing, deleting and send build request to distro build systems .. etc.
Partially it is analog of the fedpkg but it was self constrained, single file pure POSIX SH script which had been using only awk, sed, rpm/rpmbuild and cvs command.
I must say that when I saw first time number of fedpkg dependencies I was really shocked that what I've (mostly) done in few tenths KB SH script in Fedora has been pumped to size which is possible to see.
OK .. fedpkg is doing few more things than PLD builder, but I'm sure that using SH + maybe few new commands still it is possible to write such tool in maybe 2-3 times bigger SH script.
Look on --help output:
Usage: builder [-D|--debug] [-V|--version] [--short-version] [-a|--as_anon] [-b|-ba|--build]
[-bb|--build-binary] [-bs|--build-source] [-bc] [-bi] [-bl] [-u|--try-upgrade]
[{-cf|--cvs-force}] [{-B|--branch} <branch>] [{-d|--cvsroot} <cvsroot>]
[-g|--get] [-h|--help] [--http] [{-l|--logtofile} <logfile>] [-m|--mr-proper]
[-q|--quiet] [--date <yyyy-mm-dd> [-r <cvstag>] [{-T|--tag <cvstag>]
[-Tvs|--tag-version-stable] [-Ts|--tag-stable] [-Tv|--tag-version]
[{-Tp|--tag-prefix} <prefix>] [{-tt|--test-tag}] [--use-greed-sources]
[-nu|--no-urls] [-v|--verbose] [--opts <rpm opts>] [--short-circuit]
[--show-bconds] [--with/--without <feature>] [--define <macro> <value>]
<package>[.spec][:cvstag]
-5, --update-md5 - update md5 comments in spec, implies -nd -ncs
-a5, --add-md5 - add md5 comments to URL sources, implies -nc -nd -ncs
-n5, --no-md5 - ignore md5 comments in spec
-D, --debug - enable builder script debugging mode,
-debug - produce rpm debug package (same as --opts -debug)
-V, --version - output builder version string
--short-version - output builder short version
-a, --as_anon - get files via pserver as cvs@xxxxxxxxxxxxxxxxx,
-b, -ba, --build - get all files from CVS repo or HTTP/FTP and build package
from <package>.spec,
-bb, --build-binary - get all files from CVS repo or HTTP/FTP and build binary
only package from <package>.spec,
-bp, --build-prep - execute the %prep phase of <package>.spec,
-bc - execute the %build phase of <package>.spec,
-bi - execute the %install phase of <package>.spec
-bl - execute the %files phase of <package>.spec
-bs, --build-source - get all files from CVS repo or HTTP/FTP and only pack
them into src.rpm,
--short-circuit - short-circuit build
-B, --branch - add branch
-c, --clean - clean all temporarily created files (in BUILD, SOURCES,
SPECS and $RPM_BUILD_ROOT and CVS/Entries) after rpmbuild commands.
-m, --mr-proper - clean all temporarily created files (in BUILD, SOURCES,
SPECS and $RPM_BUILD_ROOT and CVS/Entries). Doesn't run
any rpm building.
-cf, --cvs-force - use -F when tagging (useful when moving branches)
-d <cvsroot>, --cvsroot <cvsroot>
- setup $CVSROOT,
--define <macro> <value>
- define a macro <macro> with value <value>,
--alt_kernel <kernel>
- same as --define 'alt_kernel <kernel>'
--nodeps - rpm won't check any dependences
-g, --get - get <package>.spec and all related files from CVS repo
or HTTP/FTP,
-h, --help - this message,
--http - use http instead of ftp,
-l <logfile>, --logtofile <logfile>
- log all to file,
-nc, --no-cvs - don't download sources from CVS, if source URL is given,
-ncs, --no-cvs-specs
- don't check specs in CVS
-nd, --no-distfiles - don't download from distfiles
-nm, --no-mirrors - don't download from mirror, if source URL is given,
-nu, --no-urls - don't try to download from FTP/HTTP location,
-ns, --no-srcs - don't download Sources
-ns0, --no-source0 - don't download Source0
-nn, --no-net - don't download anything from the net
-pm, --prefer-mirrors - prefer mirrors (if any) over distfiles for SOURCES
--no-init - don't initialize builder paths (SPECS and SOURCES)
-ske,
--skip-existing-files - skip existing files in get_files
--opts <rpm opts> - additional options for rpm
-q, --quiet - be quiet,
--date yyyy-mm-dd - build package using resources from specified CVS date,
-r <cvstag>, --cvstag <cvstag>
- build package using resources from specified CVS tag,
-A - build package using CVS resources as any sticky tags/date/kopts being reset.
-R, --fetch-build-requires
- fetch what is BuildRequired,
-RB, --remove-build-requires
- remove all you fetched with -R or --fetch-build-requires
remember, this option requires confirmation,
-FRB, --force-remove-build-requires
- remove all you fetched with -R or --fetch-build-requires
remember, this option works without confirmation,
-sd, --source-distfiles - list sources available from distfiles (intended for offline
operations; does not work when Icon field is present
but icon file is absent),
-sc, --source-cvs - list sources available from CVS
-sdp, --source-distfiles-paths - list sources available from distfiles -
paths relative to distfiles directory (intended for offline
operations; does not work when Icon field is present
but icon file is absent),
-sf, --source-files - list sources - bare filenames (intended for offline
operations; does not work when Icon field is present
but icon file is absent),
-sp, --source-paths - list sources - filenames with full local paths (intended for
offline operations; does not work when Icon field is present
but icon file is absent),
-su, --source-urls - list urls - urls to sources and patches
intended for copying urls with spec with lots of macros in urls
-T <cvstag> , --tag <cvstag>
- add cvs tag <cvstag> for files,
-Tvs, --tag-version-stable
- add cvs tags STABLE and NAME-VERSION-RELEASE for files,
-Ts, --tag-stable
- add cvs tag STABLE for files,
-Tv, --tag-version
- add cvs tag NAME-VERSION-RELEASE for files,
-Tp, --tag-prefix <prefix>
- add <prefix> to NAME-VERSION-RELEASE tags,
-tt, --test-tag <prefix>
- fail if tag is already present,
-ir, --integer-release-only
- allow only integer and snapshot releases
-v, --verbose - be verbose,
-u, --try-upgrade - check version, and try to upgrade package
-un, --try-upgrade-with-float-version
- as above, but allow float version
--use-greed-sources
- try download source from tag head if don't find it in
current tag
-U, --update - refetch sources, don't use distfiles, and update md5 comments
-Upi, --update-poldek-indexes
- refresh or make poldek package index files.
-np, --nopatch <patchnumber>
- don't apply <patchnumber>
--show-bconds - show available conditional builds, which can be used
- with --with and/or --without switches.
--show-bcond-args - show active bconds, from ~/.bcondrc. this is used by
./repackage.sh script. in other words, the output is
parseable by scripts.
--with/--without <feature>
- conditional build package depending on %_with_<feature>/
%_without_<feature> macro switch. You may now use
--with feat1 feat2 feat3 --without feat4 feat5 --with feat6
constructions. Set GROUP_BCONDS to yes to make use of it.
--target <platform>, --target=<platform>
- build for platform <platform>.
--init-rpm-dir - initialize ~/rpm directory structure
All -b and -t options are present here because everything started when rpm command and not like today rpmbuild command was in use.
As you see whoever knew already how to use rpm/build almost instantly will be able to use "builder".
As you see typical "don't move, improve" approach was used.
This is why I'm using fedpkg ONLY when must because there is no other possible way :)
This is kind of philosophically different approach.
-----
I want only point on:
-u, --try-upgrade - check version, and try to upgrade package
-un, --try-upgrade-with-float-version
- as above, but allow float version
Very interesting function which have been using Source URLs [1] to detect is available new version tarballs -> inject into current spec new version str -> download new tarball -> rebuild package -> commit change -> (re)label necessary resources in CVS.
[1] we have been spending significant part of the time to keep all those URLs in up-to-date state.
One detail about some context ..
Few days ago I've been upgrading zabbix (3.4.5->3.4.6). With upgrade package I've done it in single line (still I have it in shell history).
$ cd ~/rpmbuild/SPECS; sed -i 's/\(Version:.\( *\|\t*\)\)\(.*\)/\13.4.6/' zabbix.spec; wget -P ../SOURCES $(rpmbuild -bp --define "prep %dump" zabbix.spec 2>&1 | awk '/SOURCEURL0/ {print $3}'); rpmbuild -ba --quiet zabbix.spec; sudo rpm -Fvh ../RPMS/x86_64/zabbix*
Because I was 100% sure that it will be OK I've uses ";" between all commands
So it is nothing more than simple demonstration of some possibilities ..
1) This oneliner is simple shows how to use rpmbuild as specs PARSER. Still not to many people knows that rpmbuild internal parser can be used as external tool
By above want I want to say?
in case fedpkg maybe it is good moment to use similar approach to introduce in Fedora (semi) AUTOMATED UPDATES?
Some packages are so well maintained by source code maintainers that in Ferdora packages there is no even single patch.
Flow of some perl, php or other scripting language packages every day is sometimes quite high and all those packages are very small/simple.
Why waste so much time on update those "better" (from exact point of view) packages <many_big_question_marks>
Definitely in context of such "better" packages would be IMO possible to introduce (semi) automated updates.
I know that even among those "core/RH packages" there a lot of ..
(do you see next "human type obstacle"?
"You <censored>. This can make some of us .. jobless !^@#$% or at least unhappy because we will >>nothing to farm<<"
My answer will be "if you want to farm something it is a lot of game platforms around")
This "semi" automation may mean in many cases that package maintainer will only need to review build log and say "That looks OK .. please finish this" -> (click/tap), and after some time packager receives message "causa finita est".
-----------
Second:
--show-bconds - show available conditional builds, which can be used
- with --with and/or --without switches.
--show-bcond-args - show active bconds, from ~/.bcondrc. this is used by
./repackage.sh script. in other words, the output is
parseable by scripts.
Which should IMO better talk to many people saying/believing that "%bconds are bad!! and I (mighty packager) do this better using my own %ifology (C)"
My answer always will be "No, you did't!!!"
I don't need to even move to prove this .. proof is in (to) maaany Fedora specs.
----------------
And last 3rd:
-5, --update-md5 - update md5 comments in spec, implies -nd -ncs
-a5, --add-md5 - add md5 comments to URL sources, implies -nc -nd -ncs
-n5, --no-md5 - ignore md5 comments in spec
Quite early we started storing in commented lines md5 check sums of source tarballs.
it was used in form of lines:
# Source0-md5: <md5_hash>
Those commented lines and "builder as rpm/rpmbuild" extender allowed use commented lines to implement new features without touch rpm source code.
Maybe it is now good time to add tokens like {Patch,Source}<num>-{md5,sha256} to the official spec syntax (not commented)?
Why to use is probably obvious (?)
If still not ..
it allows easier detect that some public source tarballs file servers have been compromised.
For example Solaris native packages build infrastructure will not allow you even START build process if it will find that that stored in Makefile hash is different than this one which is locally available fs as tar ball when you will bump package version -> download new tarball -> start build.
Do I need to answer on the question why Solaris guys are using such restricted "hedgehogs during the sex" approach?
I think that at least few people during reading above one eyebrow moved up .. just slightly.
> Simple more eyeballs using exact latest Fedora than more likely that after
> hitting sometimes small issue it will be reported to BTS.
> As consequence RH people making own snapshot to start working next major EL
> version may expect that they will have fewer issues to resolve after making
> such snapshot.
>
> 3) fewer packagers will be spending time on backporting some changes
Doesn't relate to packages which rarely see an update
Really? More people looking on each change (because they will be not doing things which is possible to automate may hurt Fedora quality????
hmm .. maybe .. hmm .. if .. "NO, this cannot be truth!"
After sleeping like a baby last nigh today I think that probably know how may look step by step whole backporting process.
Lets assume that someone is doing the change on the master branch ..
1) on top of normal set of operations using fedpkg and final successful build new version/release of the package this tool or even Fedora build infrastructure detects that "aaaa .. this package has el6 and/or el7 labels .. got you :-)"
2) using positive result of above -> sends email (or chat msg or creates ticket) to packager and/or wider audience that it is something to review to backport.
What may happen here next?
2a) ignore signal/close ticket because change has nothing to do with EPEL
2b) try to backport modification which has been committed to the master
Here using git such change can be commuted. If it is conflict it will be necessary to resolve it manually.
This could be done as well by apply whole patch against el6, el7 branches without if resolve of the conflict could be to difficult.
3) build EPEL package
4) clean signal/state that exact package has something to review and backport
5) create every day stats/graphs/list of {fame,shame} ..
Does it may work?
What do you think (yes you anon reader) ..
IMO yes!!!
***********************************************
(AGAIN: try ONLY to think that keeping absolute cleanness, max readability and simplicity of all specs IMO still is worth of even small overheads.
Only this will open Fedora/RHEL few new rooms to which now is not possible to enter).
***********************************************
kloczek
--
Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
