On 23 Oct 2003, seth vidal wrote: > On Thu, 2003-10-23 at 17:37, Jeremy Katz wrote: > > On Thu, 2003-10-23 at 13:02, seth vidal wrote: > > > Quite impossible w/o leaving your rpmdb in a completely un-reliable > > > state during that time. > > > > Your rpmdb is going to be in a completely unreliable state if something > > happens mid-way through the package install anyway. Having the packages > > downloaded in advance doesn't really help with that problem. > > > > but he said > have updates for foo bar and baz > > download foo > install foo > download bar > install bar > download baz > install baz > > that didn't, to me, seem to be in the same transaction, but each as an > independent transaction. > > and if the files have been downloaded it decreases the number of things > that could cause the failure ie: the network. I actually would love this to be implemented as I too have systems with limited /var and would routinely use it to limit the yum resource footprint on all systems as there is no intrinsic advantage to downloading a whole set of rpms (with a few exceptions, where a system is rendered unusable if an installation is aborted in midstream). There is, however, a solution to this problem, one that likely should have been implemented in rpm itself but that can in principle be implemented on top of it. Make yum transaction-state aware, so that it can be invoked in a "recovery" mode to complete an interrupted transaction (set). Much like a journalling filesystem, or like many editors. Create a trace/logfile/lockfile when a yum install or upgrade is begun that writes out basically what its gonna do, then what it's done as it does it. To recover, step through the execution trace, clean up the interrupted step, and complete. If a system goes down in the MIDDLE of writing the rpmdb, well, then it depends on whether rpm manages this with atomic transactions so that it writes to a temp rpmdb, completes the write, cp's the rpmdb to an backup rpmdb, mv's the temp rpmdb to rpmdb, and deletes the backup. If it does this, then it can recover/be recovered even if interrupted in the middle of the mv. One HOPES that rpm is designed robustly, but one doesn't know, at least without doing a lot more study of rpm than I want to ever do:-). There really is no reason for any transaction involving a critical database or file to leave a system in a non-recoverable state; this is a solvable problem and has been solved for many critical systems long ago, for all that lots of software implementations outside of the kernel tend to be immensely sloppy about it. This was for years (and may still be for all I know or care) one of the major problems with the Windows registry, for example -- it was so damn easy to break, and if it broke you were cosmically fubared. SunOS used to have vipw, a wrapper for the editor of your choice which did all editing of the passwd file with near-atomic transactions and a backup. Note that yum can "fix" this problem (as did vipw), if rpm is NOT intrinsically robustly designed by wrapping these steps around its operation -- routinely cp the rpmdb into a backup before beginning a transaction that will modify it, routinely delete the backup (or not) when a transaction (set) is completed and the db is consistent and unbroken again. I would vote for such robustness regardless. One future use that I see for yum is as an install tool. Not all that "future" -- one respondant to the survey (IIRC) said openly that they are developing it AS a primary install tool right now, one that could in principle replace kickstart (although IMO kickstart's real mojo is in the scripting/automagic it performs setting up hardware and base system identity, not package installation per se as that has always been in some sense a separately automated set of transactions). Installation is also a process that requires true robustness and recoverability, as there is an extended window (how extended depending on bandwidth to the installation rpms) in any installation process where an interruption is possible. Right now this is a major flaw in both kickstart and the interactive installation tools that has bitten me repeatedly over years. Get to the very end of an install and system or network goes down (which on a slow DSL connection can easily take hours). Oops! Start over! Who cares that the system only needs to install two more rpm's and run a %post of some sort to render it bootable -- it didn't and isn't. Silly, really. Kickstart and interactive installs could just as easily organize the installation order so that the first handfull of rpm's install the base system and kernel and render the system (re)bootable locally, then set up the list of remaining rpm's for installation in a recoverable loop with state awareness, then work through the loop. If interrupted in midstream, the boot prompt should immediately offer the user a "boot to complete install" option and a "boot into root shell" option, where each does the obvious (completes the state loop). This is the kind of thing that really irritates me. RH is preparing to charge $700,000 to just one University for access to its maintained binary set but hasn't yet made its own primary contributions to linux all that robust or thoughtful in their design. Note that $700,000 is enough to fund (at typical University salaries) some seven full time linux systems programmers PLUS all the hardware they could want, more than enough to take RH's source rpm's, delogo-ify them, and set up a delogo-ified binary distribution that looks precisely like RHAS, RHEL and so forth, likely maintained within 24 hours of the original (certainly within the week, given that most of the download, delogo-ification and rebuilding can be done with scripts and given seven full time people to write and test the scripts). Of course, any small set of entrepreneurs that thought that they could resell 1000 or more delogified feeds at $1000 apiece could do the same thing (allowing for buying the outgoing bandwidth and servers and paying overhead on a space), and would then reap pure profit for the 1001+ feed. How long before some venture group realizes this? How long before some existing company (like Yellow Dog) that currently specializes in e.g. macs realizes that the Intel/AMD space is suddenly wide open with pure marginal profit there for the asking? RH is setting up a cost profile that makes it worthwhile for even e.g. NSF to fund a site that does precisely this for unlimited FREE reuse -- a million a year to clone/rebuild/mirror logo-free RHL for unlimited consumption is cheap indeed compared to spending a million a year per University or government lab that they fund. And who has a better right to do so? NSF (and Universities) probably indirectly funded the development of some 60-80% of the GPL "intellectual property" that RH doesn't own but is preparing to charge out the wazoo for. And yet RH doesn't seem to have to staff to actually ADD VALUE by doing the following: a) Make their own babies, kickstart and interactive install and rpm itself, robust against midstream failures. C'mon, this isn't that hard even in the POST rpm layer e.g. yum. How hard could it be within rpm itself? And who actually wrote the tool (yum) that is FINALLY making rpm's actually usable in a scalable way administratively? Would that be Red Hat? Mmmm, no, that wouldn't, would it. b) Write the software that businesses are missing. There is a list, guys, of mission critical applications that don't exist, are simple to write, and that are very valuable to the end user that any small business could make you aware of. Businesses are willing to pay for this software. There is nothing in the world preventing you from writing this business-specific software and making it closed source/proprietary, real IP, and selling IT instead of trying to resell us our own GPL IP back at absurd prices, and businesses would likely pay you for it up front even if it were GPL. Universities (to the extent that they are also businesses are even willing to pay for this software). It is NOT NEEDED on most of the desks and servers where linux is found today or it would already exist, right, because the community would have written and GPL'd it! c) Sell support services (and I am NOT referring to access to the rpm stream, although they are welcome to sell that at any REASONABLE price that doesn't make it worthwhile to work around it, perhaps a couple of thousand per University, several times that per enterprise) at rates proportional to the consumption of those services to whoever wishes to buy them. d) Get involved supporting gaming on linux. Right now transgaming.com is likely making more money per linux seat than RH is in their own market, because gamers are willing to pay for hot access to games and game support. Clear added value, and what you pay for it is balanced against the hassle of having to check out the source tree from sourceforge to do a local build and install (which is the "free" alternative that more or less automatically makes you a co-developer and morally entitled to free access anyway). e) Use some actual entrepreneurial mojo and figure out, fund, develop, incorporate some actual new software for nextgen systems. C'mon guys, there are only a million or so places to add real value and make money at either the GPL/Open or proprietary level here. Voice recognition, vertical market databases (e.g. medical records, drug databases), artificial intelligence, distributed computing, PDA integration... This is the Old Fashioned way of making money in software. Have a really good and NEW idea, take a risk developing it, and earn the profits that result. Not the Microsoft Way, take software that somebody else takes all the risks for and develops and then clone/resell it yourself, cutting them off entirely. What you are doing is WORSE that the Microsoft Way, as you're trying to resell them their own damn software, not even a clone. Oops. Somehow I keep getting sidetracked into that darned rant, and y'all don't need to hear it; the red hat board needs to hear it. Preferrably before they find that the community that currently is providing THEM with rpm-ready sources splits the rpm tree off into e.g. "gnu package manager" (gpm) and goes its own way. How long would it take before RH become irrelevant? I have nothing against Red Hat. They have been valuable to the community and have been rewarded with slow, steady progress to a state of profitability over the last year or so, with nothing but bright prospects even if they simply pursued their existing business process, and with the absolutely obvious extensions above. I wish them the very best. However, they are trying to charge IP rates for reselling something they don't own, which creates a vast space for the rapid growth of competition. They CANNOT ever hold a monopoly here as they don't own the product that they sell, and they will NEVER make Microsoftian profits out of the software they sell; in fact, the world is evolving past the state where software will ever make this kind of money again. It will be "interesting" watching the market response to their actions, I'm afraid interesting in the sense of the chinese curse for RH itself... rgb > > -sv > > > _______________________________________________ > Yum mailing list > Yum@xxxxxxxxxxxxxxxxxxxx > https://lists.dulug.duke.edu/mailman/listinfo/yum > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@xxxxxxxxxxxx
