On Wednesday, October 9, 2024 10:22:51 AM EDT John Robert Mendoza wrote: > Hello, > > > > I am trying to set up a 3-node Ceph cluster using cephadm and bootstrapping > fails because podman can't pull ceph container images > from quay-quay-quay.apps.os.sepia.ceph.com. Excerpt of the error message I > encountered: > > > ... > > /usr/bin/podman: stderr time="2024-10-09T21:57:22+08:00" level=warning > msg="Failed, retrying in 1s ... (3/3). Error: parsing image configuration: > Get > \"https://quay-quay-quay.apps.os.sepia.ceph.com/_storage_proxy/ZXlKaGJHY2lP > aUpTVXpJMU5pSXNJbXRwWkNJNkltVkJiMFpMTFVvMFFrNUNZeTFpV0ZSU2IxWkdORXBxVFVsQ2Qy > cExRbmQ1UjJnNU5FOURURUpHUzAwaUxDSjBlWEFpT2lKS1YxUWlmUS5leUpwYzNNaU9pSnhkV0Y1 > SWl3aVlYVmtJam9pY1hWaGVTMXhkV0Y1TFhGMVlYa3VZWEJ3Y3k1dmN5NXpaWEJwWVM1alpYQm9M > bU52YlNJc0ltNWlaaUk2TVRjeU9EUTRNakl4TWl3aWFXRjBJam94TnpJNE5EZ3lNakV5TENKbGVI > QWlPakUzTWpnME9ESXlORElzSW5OMVlpSTZJbk4wYjNKaFoyVndjbTk0ZVNJc0ltRmpZMlZ6Y3lJ > Nlczc2lkSGx3WlNJNkluTjBiM0poWjJWd2NtOTRlU0lzSW5WeWFTSTZJbkYxWVhrdlpHRjBZWE4w > YjNKaFoyVXZjbVZuYVhOMGNua3ZjMmhoTWpVMkwySTNMMkkzTTJJMk1XVTVOR1E0TVRBeFpHWmtZ > bUk1WXpJME9XRTROREZtTW1VMk1tRmlNV1E0TWpVMll6RTJNalJrWWprMk5XTTBOVGN3TURFME1H > VmlOREVfUVZkVFFXTmpaWE56UzJWNVNXUTlUamxEVUVvelZqZzNUekJZV1VKTVR6bEhORXNtVTJs > bmJtRjBkWEpsUFU1dlNrZENNblpoTlVGVGIzcFlhalY2ZFdKS2VXa3lkVmwxYXlVelJDWkZlSEJw > Y21WelBURTNNamcwT0RJNE1USWlMQ0pvYjNOMElqb2ljbVZsYzJrd01ESXVabkp2Ym5RdWMyVndh > V0V1WTJWd2FDNWpiMjA2T0RBaUxDSnpZMmhsYldVaU9pSm9kSFJ3SW4xZExDSmpiMjUwWlhoMElq > cDdmWDAubGY2SjNZc1ZSTUY4ZTEtb0Nia1ZONFRlcGFBTFBIbmptTUZZUWYtMWUtUUJ4SzAtTGRl > clpfdC1FSjZ1b2hkMEprcG15Yld3UEVmYkh4NVlseUNuWGwxbGhFMHhpb1dXdGM4bVA3b1hsRGtj > NDhnSUxNaVVucDZuUExiaTU4M3o4c0ZaTXRRUWltWTZfU2RqcFlaU0lIbm1DbnE4Z3p2ZTQtaFpr > akJpOWF5MjktQzNPT1dfWTZ5X0l6UjI4bmtadTl1LTRIeGY0WjhYeDNDRnpfaGFZSUlWbGFEN3Fh > U3BWUWF0VFlWM2J0TnJzWml1eXhGbVlhUUdJU3VlLWp0ZDM4Szh1Q3VkbDFUSW9NVzJNLUs1MjU2 > NzZIdmZ0SzRTRVZ5cGtIVmxST2Y1OVJKbVc0STNiUllFVk11REhBa3ZiczBkdlczSWxoQXFVV1ZF > bGpuNjhB/http/reesi002.front.sepia.ceph.com:80/quay/datastorage/registry/sha > 256/b7/b73b61e94d8101dfdbb9c249a841f2e62ab1d8256c1624db965c45700140eb41?AWSA > ccessKeyId=N9CPJ3V87O0XYBLO9G4K&Signature=NoJGB2va5ASozXj5zubJyi2uYuk%3D&Exp > ires=1728482812\": dial tcp 172.21.0.91:443: i/o timeout" > ... > > > > There must be a mistake with quay-quay-quay.apps.os.sepia.ceph.com DNS > record because it's resolving to a RFC1918 address (172.16/12 prefix). I > tried various online DNS tools (including Google's Admin Toolbox) to help > isolate the issue and they all return the same IP address (172.21.0.91). > Anyone else having this problem? I think everyone who is trying to pull ceph-ci images outside of the sepia lab will be hitting this issue. Others have brought this issue up in the ceph Slack/IRC channels too. FWIW, this issue did not occur prior to the lab outage yesterday. For whatever reason the quay.ceph.io registry now tries to redirect to quay-quay- quay.apps.os.sepia.ceph.com. Prior to this you could access images from either quay.ceph.io outside the lab or quay-quay-quay.apps.os.sepia.ceph.com inside the lab (only inside the lab). One more FWIW: if you are setting up a new cluster and are hitting this error you must be setting up a cluster using dev images, rather than release images. Release images live on `quay.io` (not `quay.ceph.io`). If this is intentional, great. However, you may just want to try and deploy a `squid` cluster for now unless you plan on testing `main` branch changes. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |