On Thu, 2018-06-21 at 08:36 -0700, Kevin Fenzi wrote: > On 06/20/2018 06:15 AM, Martin Kolman wrote: > > Hi, > > I would like to ask if it would be possible to allocate some storage > > space that would be publically available (eq. fedorapeople.org, > > or similar) we could use store logs from our CI so that external > > contributors can access them. > > Sure, it should be. Nice! Thanks in advance! :) > > > Why is this needed ? > > Our CI runs in internal vm hosts and internal open stack, both if which > > is not accessbile from the public internet, has no public IP or domain. > > At the same time we would like the make te test results available, > > so that for example an external contributor can see the results of tests we > > run on their pull request. > > > > What tests does the Anaconda CI run ? > > - unit tests & Pylint we run on all pull requests for Anaconda and related projects > > - periodically run kickstart tests[0], a battery of 130+ integration test, where each test is basically > > a full kickstart-configured installation run > > > > What kind of environment would be needed ? > > Basically just a simple file/static html storage would be needed, where we can SCP the log files to. > > The general idea is that tests results would be uploaded to this space whem the respective > > tests finish & there would a periodic cleanup task that would keep last N logs & remove old logs. > > > > > > What ammount of storage space would be needed for this ? > > While the unit tests & pylint runs we do on Pull requests don't produce that much data, > > the kickstart test suite is in a different class. > > > > Each of the 130+ tests is a full installation run, which produces a sizeable ammount of data > > as we gather a substantial ammount of logs from each test (anaconda logs, dnf logs, lvm & storage logs). > > This data is important to debug all the various issues that can happen in what is essentially a full > > instrumented installation run. > > > > Our measurements have shown that each full kickstart test run (~130 tests) produces ~800 MB of logs. Therefore > > we would like to request 16 GB of space to be available so that we can store ~10 last tests results from kickstart > > test, > > a reasonable number of PR test results & have some room to spare. > > > > Would this be possible ? Requesting 16 GB might seem like a lot but I'm afraid we can't reasonably reduce the space > > requirements for logs from kickstart tests runs without at the same time also making test runs harder to debug. > > > > Looking for your feedback & don't be afraid to ask any questions about this proposal. :) > > Yep. we can do this (at least) two ways: > > 1. We can use a fedorapeople group space. Indeed, there is already an > anaconda space there: https://fedorapeople.org/groups/anaconda/ > > PROs: already there, has ipv6, anyone in fas group can upload. > CONS: not backed up or mirrored, not in phx2, so might be slower to > upload depending on where you are uploading from. > > 2. an alt space under dl.fedoraproject.org/pub/alt > PROs: is on netapp space so it gets snapshotted and mirrored. > CONS: no ipv6, needs setup, more restricted what accounts can upload to it. > > Would you like to just try the fedorapeople space first? It looks like > anyone in the 'gitanaconda' group should have write to the > /project/anaconda/ space there. If you prefer a new group we can set > that up too. Yeah, that looks good: - lack of backup/mirroring should not be a problem (the results are more or less ephemeral anyway & we can easily back them up internally if needed) - upload speed should not be an issue for the PR test logs & we plan to publish daily logs from kickstart test runs, so even if there was a ~hour delay for uploding the logs it should not make much difference for once-a-day run. Using the existing gitanaconda group should be fine as well, thanks. :) BTW, how would you recommend to handle the automated log uploads to this space ? Is there some interface where we can set a SSH public key, so that the test runner can upload the logs there or something similar ? A related cron job running somewhere on our side would would SSH in regularly to cleanup the files to keep only the N remaining results. > > Also it looks like there's a alt anaconda space: > https://dl.fedoraproject.org/pub/alt/anaconda/ > but only bcl is setup to write there. we could add another person or two > there, but wouldn't want to add big groups or automated stuff. > > kevin > _______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx/message/N6WACKC43OZS5HVSY5LWGSTJAFSTRVDI/
