On Wed, 22 Feb 2023 at 15:37, Dusty Mabe <dusty@xxxxxxxxxxxxx> wrote:
On 2/22/23 12:51, Kevin Fenzi wrote:
> Greetings.
>
> As some of you may know, our fedora_koji volume is hitting up against
> some limits (namely the netapp 100TB per volume limit). If it hits 100TB
> used, the netapp folks tell me it will go offline and we will need to do
> special things to free up any space and get it working again.
> Obviously, we wish to avoid that.
>
> So, I think we can move /mnt/fedora_koji/koji/compose with minimal
> disruption and give us a bunch of room and actually make things faster.
>
> Here's my tenative plan:
>
> * create ~15-20TB volume on one of our ssd aggregates.
> * rsync all of /mnt/fedora_koji/koji/compose/ to it.
> * Schedule a changeover time/date.
> * Make sure no composes or updates pushes are running.
> (This should be possible after branched/rawhide, but before updates
> and before we are making rc's)
> * Do another sync of content so the new copy is up to date.
> (I am not sure how long a rsync will take, but we can figure it out)
> * move the old directory to compose.old
> * mount the new space on koji01/02, kojipkgs01/02, all compose channel
> builders, compose-x86-01. Nothing else should need it.
> * Wait a short while
> * delete compose.old
>
> This should free up about 13TB or so on the main volume, reduce snapshot
> churn on it, make composes faster because they will be on ssd instead of
> sas drives, and all around be nicer.
>
> I think this can be done during some day without really causing much
> outage. Because the koji space is so tight I would like to do it soon,
> and I think it best to do it before we are too close to release.
> So, later this week or early next week?
>
> Thoughts? +1s? alternative ideas?
I just want to make sure our ostree use cases are considered here. I think we
are already on our own separate volume, so maybe this has no impact, but I do
know at least the mount paths include `compose` in them so I'll list out what
we do and the desire for it to continue to work:
1. pungi composes - composing into compose/ostree/repo
2. coreos-ostree-importer - importing into /mnt/koji/compose/ostree/repo
- https://github.com/coreos/fedora-coreos-releng-automation/blob/main/coreos-ostree-importer/coreos_ostree_importer.py#L50
3. fedora-ostree-pruner - pruning /mnt/koji/compose/ostree/repo
- https://github.com/coreos/fedora-coreos-releng-automation/blob/main/fedora-ostree-pruner/fedora-ostree-pruner#L27
these three look to be on
Filesystem Size Used Avail Use% Mounted on
ntap-iad2-c02-fedora01-nfs01a:/fedora_ostree_content/compose/ostree 5.5T 5.3T 279G 96% /mnt/fedora_koji/koji/compose/ostree
ntap-iad2-c02-fedora01-nfs01a:/fedora_ostree_content/compose/ostree 5.5T 5.3T 279G 96% /mnt/fedora_koji/koji/compose/ostree
4. the `fedora-compose` ostree repo (accessible via clients for testing purposes)
- https://src.fedoraproject.org/rpms/fedora-repos/blob/rawhide/f/fedora-compose.conf
Dusty
_______________________________________________
infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren_______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
