Wiki - https://fedoraproject.org/wiki/Changes/zstd:chunked Discussion Thread - https://discussion.fedoraproject.org/t/f41-change-proposal-default-podman-created-images-to-zstd-chunked-self-contained/125540 This is a proposed Change for Fedora Linux. This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee. == Summary == Modify `podman push`, `buildah push` and `skopeo copy` to default to creating container images compressed with `zstd:chunked`. Users can modify `containers.conf` to continue to use the legacy `gzip` image format. == Owner == * Name: [[User:dwalsh| Daniel J Walsh]] * Email: dwalsh@xxxxxxxxxx * Name: [[User:giuseppe| Giuseppe Scrivano]] * Email: gscrivan@xxxxxxxxxx == Detailed Description == The goal is to speed up the pulling of container images, via using `zstd:chunked`. `zstd:chunked` format has shown itself to speed up the pulling of images by as much as 90%. This change has been waiting for change for over 3 years, waiting for Docker to support the format. == Feedback == It is not possible to directly push container images with both compression formats. See: * https://github.com/moby/moby/issues/28394#issuecomment-1822741770 * https://github.com/opencontainers/image-spec/issues/803#issuecomment-745335277 Instead, it should be possible to create a manifest list with containers in both formats. See the discussion in https://discussion.fedoraproject.org/t/switch-fedora-container-images-to-support-zstd-chunked-format-by-default/123712 for more details. This change was pushed by mistake to Fedora 41 and Fedora 40 with: * Fedora 41: https://bodhi.fedoraproject.org/updates/FEDORA-2024-4918af16a3 * Fedora 40: https://bodhi.fedoraproject.org/updates/FEDORA-2024-ab42dd0ffb This change has been reverted for Fedora 40 in: * https://bodhi.fedoraproject.org/updates/FEDORA-2024-ebe5c816a1 * https://github.com/coreos/fedora-coreos-tracker/issues/1748 To avoid further disruptions, this change has not been reverted in Fedora 41 yet. It will be reverted if it is rejected. This change also impacts all Bootable Container variants of Fedora. Initial (non-optimized) support for `zstd:chunked` images is available in rpm-ostree since [https://github.com/coreos/rpm-ostree/releases/tag/v2024.6 v2024.6]. == Benefit to Fedora == Fedora would be the first distribution to support building `zstd:chunked` format by default. Container engines (Podman, Buildah, Skopeo, CRI-O) which support the format natively would see a great speed up in pulling images. Other container engines including Docker and Containerd would see some speed up do the the compression algorithms of zstd, but not to the same degree as container engines with full support. == Scope == * Proposal owners: Dan Walsh, Giuseppe Scrivano ** Switch the default to `zstd:chunked` * Other developers: ** Test if their containers still work with the new format. * Release engineering: [https://pagure.io/releng/issues #Releng issue number] (To Do) ** Verify that the Fedora infra can use `zstd:chunked` container images. * Policies and guidelines: N/A (not needed for this Change) * Trademark approval: N/A (not needed for this Change) * Alignment with Community Initiatives: N/A == Upgrade/compatibility impact == The following versions of each package have support for `zstd:chunked`: * podman 5.1 * buildah 1.36 * skopeo 1.15 Moby doesn't support `zstd:chunked` but it handles the container image as a standard `zstd` layer since Moby 23.0. containerd supports `zstd` since 1.5. This change is implemented by updating the `/usr/share/containers/containers.conf` configuration file to set `compression_format = "zstd:chunked"` by default. Users that would like to keep using the existing format can enforce it with the following commands: $ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/ $ echo 'compression_format = "gzip"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf == How To Test == To test this change, you can create the following config file: $ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/ $ echo 'compression_format = "zstd:chunked"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf See: https://github.com/containers/common/blob/main/docs/containers.conf.5.md You can also use the `--compression-format=zstd:chunked` option with `podman push` or the `--dest-compress-format=zstd:chunked` with skopeo. See: https://docs.podman.io/en/latest/markdown/podman-push.1.html --compression-format=gzip | zstd | zstd:chunked Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked. The default is gzip unless overridden in the containers.conf file. --compression-level=level Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted values are in the range 1-20 (inclusive) with a default of 3, while for gzip it is 1-9 (inclusive) and has a default of 5. See: https://github.com/containers/skopeo/blob/main/docs/skopeo-copy.1.md --dest-compress-format format Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked. --dest-compress-level format Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted values are in the range 1-20 (inclusive), while for gzip it is 1-9 (inclusive). Then: * Build and push container images to container registries. * Inspect them to check that their format is `zstd:chunked`. * Pull the image back to the host. * Now make a small change to the image and push it back to the registry. * Remove the changed image leaving the original image in place. * Now pull the updated image again. * You should see a big change in download speed. * Now attempt to pull the same image with Docker/Moby to make sure the image works with those container engines. == User Experience == Users should experience a nice improvement in download speeds of container images that were previously created with `zstd:chunked`. == Dependencies == N/A == Contingency Plan == If we find issues with `zstd:chunked` while turned on in Rawhide, we can easily reverse direction and go back to the original default `gzip` format. All the container images and tags pushed with the `zstd:chunked` format will have to be pushed again with the `gzip` compression. == Documentation == N/A (not a System Wide Change) == Release Notes == Containers are pushed to container registries using the `zstd:chunked` format. -- Aoife Moloney Fedora Operations Architect Fedora Project Matrix: @amoloney:fedora.im IRC: amoloney -- _______________________________________________ devel-announce mailing list -- devel-announce@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-announce-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue -- _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
