On Fri, Mar 12, 2021 at 09:49:13AM -0800, Kevin Fenzi wrote:
> On Fri, Mar 12, 2021 at 10:21:45AM +0000, Richard W.M. Jones wrote:
> > AIUI end users are typing in these URLs and we download them on
> > demand.
> >
> > (I only wrote a very low-level component in this huge stack of cloud
> > stuff so I don't have the full picture, but the additional
> > complication I didn't mention is that we're using libcurl to download
> > ranges of the file, and it seems to get redirected to different
> > mirrors on every range, so it can still fail even if the first range
> > succeeded.)
>
> Fun. ;(
I think we've settled on "resolving" the location to the mirror before
passing it to nbdkit to do the downloading bit. The code is fairly
simple (see attached) and it gives us the opportunity to catch a 404
cleanly and early, as well as to sanity-check the size.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
/* gcc test.c -o test -lcurl */
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <curl/curl.h>
int
main ()
{
CURL *curl;
CURLcode r;
char *url;
curl_off_t len;
curl = curl_easy_init ();
assert (curl != NULL);
curl_easy_setopt (curl, CURLOPT_URL, "https://download.fedoraproject.org/pub/fedora/linux/releases/33/Cloud/x86_64/images/Fedora-Cloud-Base-33-1.2.x86_64.raw.xz";);
curl_easy_setopt (curl, CURLOPT_NOBODY, 1); /* No Body, not nobody! */
curl_easy_setopt (curl, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt (curl, CURLOPT_FAILONERROR, 1);
r = curl_easy_perform (curl);
if (r != CURLE_OK) {
fprintf (stderr, "error getting HEAD of URL: %d\n", r);
exit (EXIT_FAILURE);
}
/* Resolve effective URL. */
r = curl_easy_getinfo (curl, CURLINFO_EFFECTIVE_URL, &url);
if (r != CURLE_OK) {
fprintf (stderr, "error resolving URL: %d\n", r);
exit (EXIT_FAILURE);
}
printf ("resolved URL: %s\n", url);
/* Get content length. */
r = curl_easy_getinfo (curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD_T, &len);
if (r != CURLE_OK) {
fprintf (stderr, "error getting content length: %d\n", r);
exit (EXIT_FAILURE);
}
printf ("content length: %ld\n", (long) len);
curl_easy_cleanup (curl);
}
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
