Well, actually, we should probibly check in on the thing thats cleaning up the amis? and confirm that it is deleting the snapshots? I think that is this: roles/fedimg/templates/clean-amis.py in ansible. and it does delete the snapshot... so, perhaps indeed all these ones with vol-ffffffff are some mistake or some other amis?
I had time to investigate it a bit:
I deleted one of the ancient snapshot (from 2018) and AWS did not object. So it is not base image for current AMI (otherwise AWS would refuse to delete it).
The snapshots has Description like:
Copied for DestinationAmi ami-052b0ac13b1043c97 from SourceAmi ami-0d9943288750067d3 for SourceSnapshot snap-0a92565926bd815be. Task created on 1,700,729,192,462.
I **think** this description is made when you copy snapshot between regions.
I investigated one of today's such snapshot:
(the description of this snapshot is the one cited above)
and the associated AMI exists. It is
with name
Fedora-Cloud-Base-Rawhide-20231123.n.0.aarch64-hvm-ap-south-1-gp3-0
So this images are really leftover from creating nightly AMIs.
I checked the
roles/fedimg/templates/clean-amis.py
and I think it does not work at all. For two reasons:
1) We have active AMIs that have DeprecationTime se to 2022/08/11 and they are not deleted. So this is likely a date when deleting AMIs stopped working. But the snapshots deleting likely never worked.
2) The code query AMIs with Filters=[{"Name": "tag-key", "Values": ["LaunchPermissionRevoked"]}] but as I see this is not tag, but different attribute.
But anyway the snapshots were not deleted anyway. There is likely a bug I do not see now.
I tried to delete one of the old snapshots that is still used as base for active AMI (F27) and AWS refused with message:
Failed to delete snapshot.
snap-0b271f1b25a3f9b47: The snapshot snap-0b271f1b25a3f9b47 is
currently in use by ami-4ba98e24
Based on this founding I propose:
1) Delete **all** snapshots without FedoraGroup tag older than -
let say - 2021. This way we can actually review if there are some
snapshots other than leftovers form clean-amis that is worth
preserving. But right now I am unable to review manually anything.
If the snapshot will be linked to live AMI then AWS refuse to
delete it and I will ignore such errors. If there will be no
objection I will top post this as separate headsup email.
2) Open ticket that owners of fedimg should fix the tooling to delete the snapshots
3) Open tickets that owners of fedimg should delete cleanup AMIs
with Deprecation time lower than todays date.
-- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys
-- _______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
