Hi Helen, I appreciate the amount of work you've put into this, to improve testing of the kernel as a whole. I'll need more time to answer, but please see below for a quick comment already. On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: > Dear Kernel Community, > > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a > basic test pipeline triggered by code pushes to a GitLab-CI instance. This > initial version includes static checks (checkpatch and smatch for now) and build > tests across various architectures and configurations. It leverages an > integrated cache for efficient build times and introduces a flexible 'scenarios' > mechanism for subsystem-specific extensions. > > tl;dr: check this video to see a quick demo: https://youtu.be/TWiTjhjOuzg, > but don't forget to check the "Motivation for this work" below. Your feedback, > whether a simple thumbs up or down, is crucial to determine if it is worthwhile > to pursue this initiative. > > GitLab is an Open Source platform that includes integrated CI/CD. The pipeline > provided in this patch is designed to work out-of-the-box with any GitLab > instance, including the gitlab.com Free Tier. If you reach the limits of the > Free Tier, consider using community instances like https://gitlab.freedesktop.org/. > Alternatively, you can set up a local runner for more flexibility. The > bootstrap-gitlab-runner.sh script included with this patch simplifies this > process, enabling you to run tests on your preferred infrastructure, including > your own machine. > > For detailed information, please refer to the documentation included in the > patch, or check the rendered version here: https://koike.pages.collabora.com/-/linux/-/jobs/298498/artifacts/artifacts/Documentation-output/ci/gitlab-ci/gitlab-ci.html . > > > Motivation for this Work > ======================== > > We all know tests are a major topic in the community, so let's mention the > specificities of this approach: > > 1. **Built-in User Interface:** GitLab CI/CD is growing in popularity and has an > user-friendly interface. Our experience with the upstream DRM-CI in the kernel > tree (see this blog post [https://www.collabora.com/news-and-blog/blog/2024/02/08/drm-ci-a-gitlab-ci-pipeline-for-linux-kernel-testing/] ) > has provided insights into how such a system can benefit the wider community. > > 2. **Distributed Infrastructure:** > The proposed GitLab-CI pipeline is designed with a distributed infrastructure > model, being possible to run in any gitlab instance. > > 3. **Reduce regressions:** Fostering a culture where people habitually run > validated tests and post their results can prevent many issues in post-merge > tests. > > 4. **Collaborative Testing Environment:** The kernel community is already > engaged in numerous testing efforts, including various GitLab-CI pipelines such > as DRM-CI, which I maintain, along with other solutions like KernelCI and > BPF-CI. This proposal is designed to further stimulate contributions to the > evolving testing landscape. Our goal is to establish a comprehensive suite of > common tools and files. > > 5. **Ownership of QA:** > Discrepancies between kernel code and outdated tests often lead to misattributed > failures, complicating regression tracking. This issue, often arising from > neglected or deprioritized test updates, creates uncertainty about the source of > failures. Adopting an "always green pipeline" approach, as detailed in this > patch's documentation, encourages timely maintenance and validation of tests. > This ensures that testing accurately reflects the current state of the kernel, > thereby improving the effectiveness of our QA processes. > > Additionally, if we discover that this method isn't working for us, we can > easily remove it from the codebase, as it is primarily contained within the ci/ > folder. > > > Future Work > =========== > > **Expanding Static Checks:** > We have the opportunity to integrate a variety of static analysis tools, > including: > - dtbs_checks > - sparse > - yamllint > - dt-doc-validate > - coccicheck > > **Adding Userspace Tests on VMs:** > To further our testing, we can implement userspace tests that run on virtual > machines (VMs), such as: > - kselftests > - kunit tests > - Subsystem-specific tests, customizable in the scenarios. > > **Leveraging External Test Labs:** > We can extend our testing to external labs, similar to what DRM-CI currently > does. This includes: > - Lava labs > - Bare metal labs > - Using KernelCI-provided labs > > **Other integrations** > - Submit results to KCIDB > > **Lightweight Implementation for All Developers:** > We aim to design these tests to be lightweight, ensuring developers with limited > computing resources can still run essential tests. Resource-intensive tests can > be set to trigger manually, rather than automatically, to accommodate diverse > development environments. > > > Chat Discussions > ================ > > For those interested in further discussions: > > **Join Our Slack Channel:** > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > Feel free to join and contribute to the conversation. The KernelCI team has > weekly calls where we also discuss the GitLab-CI pipeline. Could we communicate using free software please ? Furthermore, it's not possible to create an account on that slack instance unless you have an e-mail address affiliated with a small number of companies (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go for me. > **Acknowledgments:** > A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - > and KernelCI community for their valuable feedback and support in this proposal. > > > I eagerly await your thoughts and suggestions on this initiative. > > Also, if you want to see this initiave move faster, we are happy to discuss > funding options. > > Best regards, > Helen Koike > > Helen Koike (3): > kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing > kci-gitlab: Add documentation > kci-gitlab: docs: Add images > > .gitlab-ci.yml | 2 + > Documentation/ci/gitlab-ci/gitlab-ci.rst | 404 ++++++++++++++++++ > .../ci/gitlab-ci/images/job-matrix.png | Bin 0 -> 159752 bytes > .../gitlab-ci/images/new-project-runner.png | Bin 0 -> 607737 bytes > .../ci/gitlab-ci/images/pipelines-on-push.png | Bin 0 -> 532143 bytes > .../ci/gitlab-ci/images/the-pipeline.png | Bin 0 -> 91675 bytes > .../ci/gitlab-ci/images/variables.png | Bin 0 -> 277518 bytes > Documentation/index.rst | 7 + > MAINTAINERS | 9 + > ci/gitlab-ci/bootstrap-gitlab-runner.sh | 55 +++ > ci/gitlab-ci/ci-scripts/build-docs.sh | 35 ++ > ci/gitlab-ci/ci-scripts/build-kernel.sh | 35 ++ > ci/gitlab-ci/ci-scripts/ici-functions.sh | 104 +++++ > ci/gitlab-ci/ci-scripts/install-smatch.sh | 13 + > .../ci-scripts/parse_commit_message.sh | 27 ++ > ci/gitlab-ci/ci-scripts/run-checkpatch.sh | 19 + > ci/gitlab-ci/ci-scripts/run-smatch.sh | 45 ++ > ci/gitlab-ci/docker-compose.yaml | 18 + > ci/gitlab-ci/linux.code-workspace | 11 + > ci/gitlab-ci/yml/build.yml | 43 ++ > ci/gitlab-ci/yml/cache.yml | 26 ++ > ci/gitlab-ci/yml/container.yml | 36 ++ > ci/gitlab-ci/yml/gitlab-ci.yml | 71 +++ > ci/gitlab-ci/yml/kernel-combinations.yml | 18 + > ci/gitlab-ci/yml/scenarios.yml | 12 + > ci/gitlab-ci/yml/scenarios/file-systems.yml | 21 + > ci/gitlab-ci/yml/scenarios/media.yml | 21 + > ci/gitlab-ci/yml/scenarios/network.yml | 21 + > ci/gitlab-ci/yml/static-checks.yml | 21 + > 29 files changed, 1074 insertions(+) > create mode 100644 .gitlab-ci.yml > create mode 100644 Documentation/ci/gitlab-ci/gitlab-ci.rst > create mode 100644 Documentation/ci/gitlab-ci/images/job-matrix.png > create mode 100644 Documentation/ci/gitlab-ci/images/new-project-runner.png > create mode 100644 Documentation/ci/gitlab-ci/images/pipelines-on-push.png > create mode 100644 Documentation/ci/gitlab-ci/images/the-pipeline.png > create mode 100644 Documentation/ci/gitlab-ci/images/variables.png > create mode 100755 ci/gitlab-ci/bootstrap-gitlab-runner.sh > create mode 100755 ci/gitlab-ci/ci-scripts/build-docs.sh > create mode 100755 ci/gitlab-ci/ci-scripts/build-kernel.sh > create mode 100644 ci/gitlab-ci/ci-scripts/ici-functions.sh > create mode 100755 ci/gitlab-ci/ci-scripts/install-smatch.sh > create mode 100755 ci/gitlab-ci/ci-scripts/parse_commit_message.sh > create mode 100755 ci/gitlab-ci/ci-scripts/run-checkpatch.sh > create mode 100755 ci/gitlab-ci/ci-scripts/run-smatch.sh > create mode 100644 ci/gitlab-ci/docker-compose.yaml > create mode 100644 ci/gitlab-ci/linux.code-workspace > create mode 100644 ci/gitlab-ci/yml/build.yml > create mode 100644 ci/gitlab-ci/yml/cache.yml > create mode 100644 ci/gitlab-ci/yml/container.yml > create mode 100644 ci/gitlab-ci/yml/gitlab-ci.yml > create mode 100644 ci/gitlab-ci/yml/kernel-combinations.yml > create mode 100644 ci/gitlab-ci/yml/scenarios.yml > create mode 100644 ci/gitlab-ci/yml/scenarios/file-systems.yml > create mode 100644 ci/gitlab-ci/yml/scenarios/media.yml > create mode 100644 ci/gitlab-ci/yml/scenarios/network.yml > create mode 100644 ci/gitlab-ci/yml/static-checks.yml -- Regards, Laurent Pinchart
