On Thu, Mar 11, 2021 at 02:00:44PM -0800, Kevin Fenzi wrote: > On Thu, Mar 11, 2021 at 05:06:30PM +0000, Richard W.M. Jones wrote: > > > > https://download.fedoraproject.org/pub/fedora/linux/releases/33/Cloud/x86_64/images/Fedora-Cloud-Base-33-1.2.x86_64.raw.xz > > > > You probably won't be able to observe this easily, but if you > > repeatedly download this file (or ideally, just do a HEAD request) > > then occasionally you'll hit a mirror that returns a 404. > > Yep. > > > At the time of writing mirror.genesisadaptive.com was offered as a > > redirect, but it does not have the file. > > (https://mirror.genesisadaptive.com/fedora/releases/33/Cloud/x86_64/images/Fedora-Cloud-Base-33-1.2.x86_64.raw.xz) > > Right. > mirrormanager has a list of mirrors that are 'up to date' and redirects > you to one of those. However, it doesn't know if that mirror has every > single file or image on it. :( > > > This is a problem for some cloud software we're writing that would > > like to pull the image on demand when people request a Fedora 33 VM. > > Well, if you are getting the f33 release image, it's never ever going to > change. You could store it locally? AIUI end users are typing in these URLs and we download them on demand. (I only wrote a very low-level component in this huge stack of cloud stuff so I don't have the full picture, but the additional complication I didn't mention is that we're using libcurl to download ranges of the file, and it seems to get redirected to different mirrors on every range, so it can still fail even if the first range succeeded.) > > It breaks unpredictably, and telling the "occasional 404" case from > > the "the URL is wrong" case is hard. > > Yeah. ;( > > > Is there anything that can be done so the mirrors which don't have the > > file are not offered up as redirects? > > Not easily I don't think. That would require mirrormanager to look at > every single file on the entire mirror and store them for every mirror. > ;( > > You could file a RFE on mirrormanager about this if you like. I thought > we had one, but I can't seem to find it. I guess > https://github.com/fedora-infra/mirrormanager2/issues/166 is close. Thanks, will read through it. > We also have a cloudfront cache: > https://d2lzkl7pfhq30w.cloudfront.net/ you could use... > or there is the master mirror at dl.fedoraproject.org, but I don't think > we want everyone to use that. :) I'm not sure I understand the cloudfront URL there. Is that a permanent thing? - It looks temporary. Why does it redirect to dl.fp.org? Rich. > _______________________________________________ > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
