On Tue, 2016-09-27 at 19:39 -0500, Bowen Wang wrote: > I saw the 20160927 images in the wesite, but I when I ran the dnf > upgrade --refresh, it says nothing to do, is that normal? Thanks. It's not unusual...there's some other stuff that happens between the compose 'completing' and you actually seeing updated packages... OK, I wrote a really long explanation which is below, but I'll put a summary up here: there's some clever stuff that goes on involving the Fedora mirror system and the repository metadata, a consequence of which is that with a default configuration you *definitely won't* start getting the new packages until an hour or two after the compose 'completes', and you *may* not get them until several hours later. So to break it down, here's what happens (for Rawhide): 1. The compose process itself completes: what that basically means is that in the directory under https://kojipkgs.fedoraproject.org/compose/rawhide/ we have complete repositories, installer images, and all the other bits that get produced by the compose process 2. Most (but not quite all) of the output of the compose process gets synced to the 'master mirror' location, basically meaning it goes to https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/ ;. Some of it instead goes to https://dl.fedoraproject.org/pub/alt/development/rawhide/ ;. 3a. checksums for some of the repository metadata files are generated and stored in the mirrormanager system. You can see these by hitting this URL: https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64 . ; 3b. The public mirrors start syncing the content of the new compose from the primary mirror (mirrors vary in how often they do this, hence how long it takes for them to pick up the new compose) 3a is actually quite an important point. In a standard Fedora system, using the mirrormanager setup for the official repositories, when you ask it to refresh metadata, dnf will get both a list of mirrors and a list of metadata checksums from mirrormanager. It will go to the first mirror on the list and grab the metadata, then it will *check it against the checksums* and if it doesn't match, it will move on to the next mirror on the list and get the metadata again. Once it's got metadata that matches the checksums from mirrormanager, it'll be happy, and it'll then be offering the packages listed in that metadata (when it actually goes to download the packages, it just hits up each mirror in the list in sequence until it finds one which has the file listed in the metadata it's working from; if the expected path 404s, it goes to the next mirror). The primary *reason* for this system is: it's an attempt to make sure you don't get really old metadata from stale mirrors. Before this system was set up, mirrormanager would just send you a list of mirrors, and dnf would grab the metadata from the first mirror on the list, and it would assume it was up to date. If you happened to hit a 'bad' mirror which wasn't syncing regularly enough or was somehow broken, you could get really stale metadata, which would mean you'd get really old packages (if that mirror or some other mirror on the list actually carried the packages matching the metadata) or no packages at all (if the package versions listed in the metadata couldn't be found on any mirror). So this checksumming system is an attempt to avoid that. mirrormanager keeps metadata checksums for (IIRC) the last two or three composes, so if the first mirror you hit has metadata that's older than that, dnf will ignore it and go to another mirror. There's a slight *drawback* to this system, though, which is: with the default repo configuration, you will not get packages from a new compose until mirrormanager has synced the metadata checksums for that new compose and is serving them out. Because obviously, if the new checksums aren't on mirrormanager when you get the metadata from a fast-syncing mirror, dnf will just figure the metadata is *old* (there's no way it can tell it's actually new) and move on to the next mirror. I know about this in quite a bit of detail because it's a problem openQA ran into (and still does, to a degree) all the time :) Because the openQA tests trigger from the fedmsg that's sent when step 1. finishes, the openQA test process is basically racing with steps 2. and particularly with 3a (it doesn't race with 3b because openQA always uses the primary mirror). This was why, until a couple of weeks back, openQA tests would frequently have issues because they were racing with the sync process, and thus getting old packages or even getting stuck at a particularly unfortunate point in the race and not completing the install (or a post-install package transaction) at all. What we did to mitigate this was tweak how the openQA tests work so that *most* of them edit their repository configuration and/or pass anaconda an 'inst.repo' parameter to use the compose directory directly as a repository, instead of using the 'normal' fedora and rawhide repository definitions. The openQA tests run quite fast in production, so if you go run a 'dnf --refresh update' right when you see the check-compose report: 1. 3b) will almost always still be happening (meaning you'll most likely get the metadata from the previous compose, unless you happen to hit a very rapidly-synced mirror) 2. 3a) may well still not have happened yet (meaning even if you do happen to hit a fast-synced mirror and get the metadata from the new compose, dnf will throw it away). At present the process of generating the metadata checksums and getting them into mirrormanager is a) not highly optimized and b) not hooked up with fedmsg, so it is in fact technically impossible to know *exactly* when you may start getting the packages from a new compose if you use the (default) mirrormanager repository setup. (I believe it actually just happens on an hourly cron job or something). You can only be 100% sure you'll *definitely* get the metadata (hence packages) from the new compose after all mirrors currently active in MirrorManager have synced the new metadata, and again, there's no great way of knowing that. You *can* short-circuit this process by editing your repositories so that instead of going through mirrormanager, you just specify a single fast-syncing repository; if you don't use the 'metalink' (i.e. mirrormanager) system in your repo definitions but instead just specify a single 'baseurl', none of this fancy metadata checksum checking stuff happens, and dnf just happily goes ahead and grabs whatever metadata and packages are on that mirror. That way the only thing you have to wait for is for the mirror you pick to sync. But we really strongly discourage people from doing that because it inevitably leads to excessive load on a couple of 'popular' mirrors, which is what we want to avoid. It also does mean that *while the mirror you pick is syncing* you run the risk of 404s, if it doesn't sync the new packages first, *then* sync the metadata files, *then* delete the old packages (I don't know if any mirrors are set up to do that). It's a better idea, if you really want to get some specific update fast, to just go out and grab the particular update you want from Koji or directly from the tree in kojipkgs, if you know exactly what files you need to download. Hope that helps, and wasn't overwhelming! :) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
