On Wed, 12 Oct 2022 at 07:19, Michal Schorm <mschorm@xxxxxxxxxx> wrote:
Hello,
I'm working on a rebase of the MySQL package (community-mysql).
I'm getting:
'No space left on device'
errors very consistently.
It happens on every single architecture, which is odd as usually such
problems in KOJI are arch-specific.
It happens very early during testsuite execution.
Examples here:
https://koji.fedoraproject.org/koji/taskinfo?taskID=92917358
https://koji.fedoraproject.org/koji/taskinfo?taskID=92919482
I've also tried to re-build the currently released commit, to
eliminate any changes in behaviour during the package rebase.
The errors are there too:
https://koji.fedoraproject.org/koji/taskinfo?taskID=92945427
https://koji.fedoraproject.org/koji/taskinfo?taskID=92945615
Anyone recall any recent changes to KOJI?
No there have been no recent changes to the buildsystem. I think the problem you are running into is the long running one that koji will choose builders for tasks without knowing what ALL the available resources are on that box. So if a lot of builds have been going on boxes, you will have builders with little disk space but koji will still assign tasks to them as their CPU is empty.
Would it be possible to allocate more space to the builders ?
Not without having less builders. Most of the virtual builders split the CPU/memory/disk space to whatever the base hardware has. That means that the disk size per system is different because the lower hardware may have different size. That said the server you ran out of disk space has 100GB of disk space.
/dev/vda2 99G 4.4G 89G 5% /
buildvm-a32-30.iad2.fedoraproject.org
Filesystem Size Used Avail Use% Mounted on
/dev/vda5 136G 4.8G 126G 4% /
Filesystem Size Used Avail Use% Mounted on
/dev/vda5 136G 4.8G 126G 4% /
Other examples to show how disk space can be different
# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/vda3 292G 6.7G 280G 3% /
Filesystem Size Used Avail Use% Mounted on
/dev/vda3 292G 6.7G 280G 3% /
# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/vda2 255G 6.2G 240G 3% /
Filesystem Size Used Avail Use% Mounted on
/dev/vda2 255G 6.2G 240G 3% /
Builders keep old builds for a certain amount of time and then will get cleaned up. This means that if a lot of builds are done from various things, we will run out of disk space on certain builders for a short time.
Michal
--
Michal Schorm
Software Engineer
Core Services - Databases Team
Red Hat
--
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
