During Flock 2016, I had the opportunity to talk with Adam Miller,
Dennis Gilmore, and Pierre-Yves Chibon about the technical challenges
with distributing Docker images with Fedora's extensive mirror network.
These conversations helped me to solidify a proposal for how Fedora
could solve this problem, outlined below.
High level view
===============
In summary, the proposal is to write a patch for the docker client that
will give it the capability to accept metalink responses upon docker
pull operations. We would also need to add support for Docker images to
mirror list and mirror manager. Additionally, we will need a small tool
to pull the content to be mirrored out of a docker registry and write
them to disk in a format that can be mirrored, as well as some Ansible
code to run the tool when there is new content to be mirrored.
Background
==========
The Fedora project wishes to begin distributing new types of content
than it has in the past. One of the types that has been identified as a
goal is the Docker image. Adam Miller has already done the work that
will allow packagers to build Docker images, but we still need a way to
distribute those builds to Fedora's users. Adam Miller's implementation
helpfully drops the builds we want into a Docker registry.
Proposed Changes
================
Mirror List
-----------
Users will be pointing their docker clients at Mirror List when they
docker pull Fedora's Docker images. In order for this to work, we will
need to make two changes to Mirror List so that it can respond to the
docker client properly. The first change is that Mirror List will need
to respond with a special header and a body of "{}" when the docker
client sends a GET request for /v2/. The second change is that it will
need to return a metalink document when the client makes additional
requests so that the clients can be redirected to a list of mirrors
near their locations, just as it does with the dnf client today.
The docker client typically connects to port 5000. We could run a
second instance of Mirror List on port 5000 if we wanted to isolate it
from the current instance. We can also have the docker client pull from
443 as dnf does if we want to keep the deployment simpler.
Mirror Manager
--------------
We will need to make a few changes to Mirror Manager as well. We will
need to provide an interface to allow mirror admins to opt in/out of
mirroring Docker content. We will also need to modify the curler to
detect whether a given mirror is up to date or not. We will need to make
sure that UMDL is updated when content changes.
docker
------
The most significant work required will likely be modifying the docker
client to enable it to properly handle the metalink responses it will be
receiving from Mirror List. When requesting the manifest, it will
receive a metalink document that will give it a priority ordered list of
mirrors. It will need to work through the list in order until it reaches
a mirror that has the correct checksum for the requested manifest. It
will then use that same mirror for the subsequent blob requests.
There is some concern that such a feature would not be accepted by the
upstream docker project. If we were to proceed with this proposal, we
would propose this patch to the upstream Docker project. If upstream
were not willing to accept the feature, we would need to have the Fedora
docker packager carry this patch as a downstream add on.
New Tool
--------
The last piece that is needed is a tool that can create the filesystem
tree that we want to synchronize out to the mirrors. The mirrors only
need to carry manifests and blobs, so the tool needs only to pull these
documents out of the registry that Adam Miller has set up and write them
to disk in a particular structure. For optimization, we could use
hardlinks for blobs that are common across the various images (for
example, the Fedora base blob will be the same in all images) to save
rsync time and mirror disk space.
Additionally, we will need a playbook to run this new tool in response
to fedmsgs. We may be able to use Adam Miller's loopabull project to run
such a playbook at the right times.
Conclusion
==========
Thanks for reading, and please respond with any comments or questions
you have about this proposal. I'm happy to clarify any points further,
and if you have any alternative proposals I'd love to hear those as
well.
_______________________________________________
infrastructure mailing list
infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
https://lists.fedoraproject.org/admin/lists/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
