[PATCH rdma-core] build: Set up CI with Azure Pipelines

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Azure Pipelines is intended to replace travis as the CI for rdma-core.

Build times are much faster at around 5 mins (vs 23 mins for travis) and
because it is built on pre-built containers it should not suffer from the
same flakiness that travis historically has. We can now also scale up the
number of distros we are testing.

This commit replicates all the functionality of travis, except for the
automatic .tar.gz upload on release. That requires a more careful
security analysis first.

However AZP is setup to use a bionic based container, not the ancient
xenial stuff. This container uses the built in bionic arm64 and 32 bit
x86 cross compiler, including all the support libraries. This means it
can now cross build everything except for pyverbs.

The GitHub experience is similar with two notable differences:

1) PRs from non-team outside contributors will not run automatically.
   Someone on the github team has to trigger them with a '/azp run'
   comment. See azure-pipelines.md for why

2) Checkpatch warnings/errors now result in a green checkmark. Clicking
   through the check will show a count of errors/warnings and Azure's GUI
   will show an orange checkmark.

Signed-off-by: Jason Gunthorpe <jgg@xxxxxxxxxxxx>
---
 buildlib/azp-checkpatch      |  72 +++++++++++++++
 buildlib/azure-pipelines.yml | 167 +++++++++++++++++++++++++++++++++++
 buildlib/cbuild              | 141 ++++++++++++++++++++++++++++-
 3 files changed, 379 insertions(+), 1 deletion(-)
 create mode 100755 buildlib/azp-checkpatch
 create mode 100644 buildlib/azure-pipelines.yml

I've been threatening to do this for some time, as travis's
limitations are finally too much to cope with

The UCF Consortium has donated their Azure Tenant and is funding the
Azure Container Registry instance to make this work. Like travis,
Azure Pipelines provides free build time to open source, but Azure has
more parallel VMs and I think the VMs are faster too.

I've set it up to run faster, so it is not quite 100% the same as
travis, but I feel the result is overall an improvement. I
particularly like the GUI log viewer and how it breaks things down and
actually points out where errors are.

I have this linked to my testing repo, if there are no objections I'll
set it up up on the main linux-rdma github sometime after the next
release comes out and we can start to try it out. It would run in
parallel with travis until we decide to turn off travis.

To see the experience here, is a sample PR:

https://github.com/jgunthorpe/rdma-plumbing/pull/2

And the resulting build log/report (click checks then 'View more
details on Azure Pipelines' to get here):

https://dev.azure.com/ucfconsort/rdma-core%20test/_build/results?buildId=48

After this is going I will revise the containers to again use latest
compilers, with gcc 9 and clang 8, and probably add FC30 and Centos6
to the matrix since that won't actually increase build time now..

Jason

diff --git a/buildlib/azp-checkpatch b/buildlib/azp-checkpatch
new file mode 100755
index 00000000000000..d7149a59af02e7
--- /dev/null
+++ b/buildlib/azp-checkpatch
@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+import subprocess
+import urllib.request
+import os
+import re
+import tempfile
+import collections
+import sys
+
+base = os.environ["SYSTEM_PULLREQUEST_TARGETBRANCH"]
+if not re.match("^[0-9a-fA-F]{40}$", base):
+    base = "refs/remotes/origin/" + base
+
+with tempfile.TemporaryDirectory() as dfn:
+    patches = subprocess.check_output(
+        [
+            "git", "format-patch",
+            "--output-directory", dfn,
+            os.environ["SYSTEM_PULLREQUEST_SOURCECOMMITID"], "^" + base
+        ],
+        universal_newlines=True).splitlines()
+    if len(patches) == 0:
+        sys.exit(0)
+
+    ckp = os.path.join(dfn, "checkpatch.pl")
+    urllib.request.urlretrieve(
+        "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/scripts/checkpatch.pl";,
+        ckp)
+    urllib.request.urlretrieve(
+        "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/scripts/spelling.txt";,
+        os.path.join(dfn, "spelling.txt"))
+    os.symlink(
+        os.path.join(os.getcwd(), "buildlib/const_structs.checkpatch"),
+        os.path.join(dfn, "const_structs.checkpatch"))
+    checkpatch = [
+        "perl", ckp, "--no-tree", "--ignore",
+        "PREFER_KERNEL_TYPES,FILE_PATH_CHANGES,EXECUTE_PERMISSIONS,USE_NEGATIVE_ERRNO,CONST_STRUCT",
+        "--emacs", "--mailback", "--quiet", "--no-summary"
+    ]
+
+    failed = False
+    for fn in patches:
+        proc = subprocess.run(
+            checkpatch + [os.path.basename(fn)],
+            cwd=dfn,
+            stdout=subprocess.PIPE,
+            universal_newlines=True,
+            stderr=subprocess.STDOUT)
+        if proc.returncode == 0:
+            assert (not proc.stdout)
+            continue
+        sys.stdout.write(proc.stdout)
+
+        failed = True
+        for g in re.finditer(
+                r"^\d+-.*:\d+: (\S+): (.*)(?:\n#(\d+): (?:FILE: (.*):(\d+):)?)?$",
+                proc.stdout,
+                flags=re.MULTILINE):
+            itms = {}
+            if g.group(1) == "WARNING":
+                itms["type"] = "warning"
+            else:
+                itms["type"] = "error"
+            if g.group(4):
+                itms["sourcepath"] = g.group(4)
+                itms["linenumber"] = g.group(5)
+            print("##vso[task.logissue %s]%s" % (";".join(
+                "%s=%s" % (k, v)
+                for k, v in sorted(itms.items())), g.group(2)))
+
+if failed:
+    print("##vso[task.complete result=SucceededWithIssues]]azp-checkpatch")
diff --git a/buildlib/azure-pipelines.yml b/buildlib/azure-pipelines.yml
new file mode 100644
index 00000000000000..eb371b76c80129
--- /dev/null
+++ b/buildlib/azure-pipelines.yml
@@ -0,0 +1,167 @@
+# See https://aka.ms/yaml
+
+trigger:
+  - master
+pr:
+  - master
+
+resources:
+  # Using ucfconsort_registry2 until this is activated in the agent:
+  #   https://github.com/microsoft/azure-pipelines-agent/commit/70b69d6303e18f07e28ead393faa5f1be7655e6f#diff-2615f6b0f8a85aa4eeca0740b5bac69f
+  #   https://developercommunity.visualstudio.com/content/problem/543727/using-containerized-services-in-build-pipeline-wit.html
+  containers:
+    - container: azp
+      image: ucfconsort.azurecr.io/rdma-core/azure_pipelines:latest
+      endpoint: ucfconsort_registry2
+    - container: centos7
+      image: ucfconsort.azurecr.io/rdma-core/centos7:latest
+      endpoint: ucfconsort_registry2
+    - container: leap
+      image: ucfconsort.azurecr.io/rdma-core/opensuse-15.0:latest
+      endpoint: ucfconsort_registry2
+
+stages:
+  - stage: Build
+    jobs:
+      - job: Compile
+        displayName: Compile Tests
+        pool:
+          vmImage: 'Ubuntu-16.04'
+        container: azp
+        steps:
+          - task: PythonScript@0
+            displayName: checkpatch
+            condition: eq(variables['Build.Reason'], 'PullRequest')
+            inputs:
+              scriptPath: buildlib/azp-checkpatch
+              pythonInterpreter: /usr/bin/python3
+
+          - bash: |
+              set -e
+              mkdir build-gcc8
+              cd build-gcc8
+              CC=gcc-8 CFLAGS="-Werror" cmake -GNinja .. -DIOCTL_MODE=both -DENABLE_STATIC=1
+              ninja
+            displayName: gcc 8.3 Compile
+
+          - bash: |
+              set -e
+              cd build-gcc8
+              python2.7 ../buildlib/check-build --src .. --cc gcc-8
+            displayName: Check Build Script
+
+          # Run sparse on the subdirectories which are sparse clean
+          - bash: |
+              set -e
+              mkdir build-sparse
+              mv CMakeLists.txt CMakeLists-orig.txt
+              grep -v "# NO SPARSE" CMakeLists-orig.txt > CMakeLists.txt
+              cd build-sparse
+              CC=cgcc CFLAGS="-Werror" cmake -GNinja .. -DIOCTL_MODE=both -DNO_PYVERBS=1
+              ninja | grep -v '^\[' | tee out
+              # sparse does not fail gcc on messages
+              if [ -s out ]; then
+                 false
+              fi
+              mv ../CMakeLists-orig.txt ../CMakeLists.txt
+            displayName: sparse Analysis
+
+          - bash: |
+              set -e
+              mkdir build-clang
+              cd build-clang
+              CC=clang-7 CFLAGS="-Werror -m32 -msse3" cmake -GNinja .. -DIOCTL_MODE=both -DNO_PYVERBS=1
+              ninja
+            displayName: clang 7.0 32-bit Compile
+
+          - bash: |
+              set -e
+              mv util/udma_barrier.h util/udma_barrier.h.old
+              echo "#error Fail" >> util/udma_barrier.h
+              cd build-gcc8
+              rm CMakeCache.txt
+              CC=gcc-8 CFLAGS="-Werror" cmake -GNinja .. -DIOCTL_MODE=both
+              ninja
+              mv ../util/udma_barrier.h.old ../util/udma_barrier.h
+            displayName: Simulate non-coherent DMA Platform Compile
+
+          - bash: |
+              set -e
+              mkdir build-arm64
+              cd build-arm64
+              CC=aarch64-linux-gnu-gcc-8 CFLAGS="-Werror" cmake -GNinja .. -DIOCTL_MODE=both -DNO_PYVERBS=1
+              ninja
+            displayName: gcc 8.3 ARM64 Compile
+
+          # When running cmake through debian/rules it is hard to set -Werror,
+          # instead force it on by changing the CMakeLists.txt
+          - bash: |
+              set -e
+              echo 'set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Werror")' >> buildlib/RDMA_EnableCStd.cmake
+              sed -i -e 's/-DCMAKE_BUILD_TYPE=Release/-DCMAKE_BUILD_TYPE=Debug/g' debian/rules
+              sed -i -e 's/ninja \(.*\)-v/ninja \1/g' debian/rules
+              debian/rules CC=clang-7 build
+            displayName: clang 7.0 Bionic Build
+          - bash: |
+              set -e
+              fakeroot debian/rules binary
+            displayName: clang 7.0 Bionic .deb Build
+
+      - job: SrcPrep
+        displayName: Build Source Tar
+        pool:
+          vmImage: 'Ubuntu-16.04'
+        container: azp
+        steps:
+          - checkout: self
+            fetchDepth: 1
+
+          - bash: |
+              set -e
+              mkdir build-pandoc artifacts
+              cd build-pandoc
+              CC=gcc-8 cmake -GNinja ..
+              ninja docs
+              cd ../artifacts
+              # FIXME: Check Build.SourceBranch for tag consistency
+              python2.7 ../buildlib/cbuild make-dist-tar ../build-pandoc
+            displayName: Prebuild Documentation
+
+          - task: PublishPipelineArtifact@0
+            inputs:
+              # Contains a rdma-core-XX.tar.gz file
+              artifactName: source_tar
+              targetPath: artifacts
+
+      - job: Distros
+        displayName: Test Build RPMs for
+        dependsOn: SrcPrep
+        pool:
+          vmImage: 'Ubuntu-16.04'
+        strategy:
+          matrix:
+            centos7:
+              CONTAINER: centos7
+              SPEC: redhat/rdma-core.spec
+              RPMBUILD_OPTS:
+            leap:
+              CONTAINER: leap
+              SPEC: suse/rdma-core.spec
+              RPMBUILD_OPTS: --without=curlmini
+        container: $[ variables['CONTAINER'] ]
+        steps:
+          - checkout: none
+
+          - task: DownloadPipelineArtifact@0
+            inputs:
+              artifactName: source_tar
+              targetPath: .
+
+          - bash: |
+              set -e
+              mkdir SOURCES tmp
+              tar --wildcards -xzf rdma-core*.tar.gz  */$(SPEC) --strip-components=2
+              RPM_SRC=$(rpmspec -P rdma-core.spec | awk '/^Source:/{split($0,a,"[ \t]+");print(a[2])}')
+              (cd SOURCES && ln -sf ../rdma-core*.tar.gz "$RPM_SRC")
+              rpmbuild --define '_tmppath '$(pwd)'/tmp' --define '_topdir '$(pwd) -bb rdma-core.spec $(RPMBUILD_OPTS)
+            displayName: Perform Package Build
diff --git a/buildlib/cbuild b/buildlib/cbuild
index 7b0b9a9de0ed44..ccf45e68962de9 100755
--- a/buildlib/cbuild
+++ b/buildlib/cbuild
@@ -79,7 +79,11 @@ class Environment(object):
     proxy = True;
     build_pyverbs = True;
 
+    to_azp = False;
+
     def image_name(self):
+        if self.to_azp:
+            return "ucfconsort.azurecr.io/%s/%s"%(project, self.name);
         return "build-%s/%s"%(project,self.name);
 
 # -------------------------------------------------------------------------
@@ -122,6 +126,7 @@ class centos7(YumEnvironment):
     build_pyverbs = False;
     specfile = "redhat/rdma-core.spec";
     python_cmd = "python";
+    to_azp = True;
 
 class centos7_epel(centos7):
     pkgs = (centos7.pkgs - {"cmake","make"}) | {
@@ -168,7 +173,7 @@ class APTEnvironment(Environment):
     build_python = True;
     def get_docker_file(self):
         res = DockerFile(self.docker_parent);
-        res.lines.append("RUN apt-get update && apt-get install -y --no-install-recommends %s && apt-get clean"%(
+        res.lines.append("RUN apt-get update && apt-get install -y --no-install-recommends %s && apt-get clean && rm -rf /usr/share/doc/ /usr/lib/debug"%(
             " ".join(sorted(self.pkgs))));
         return res;
 
@@ -356,6 +361,7 @@ class leap(ZypperEnvironment):
         'python3-devel',
     };
     rpmbuild_options = [ "--without=curlmini" ];
+    to_azp = True;
     name = "opensuse-15.0";
     aliases = {"leap"};
 
@@ -368,6 +374,51 @@ class tumbleweed(ZypperEnvironment):
 
 # -------------------------------------------------------------------------
 
+class azure_pipelines(bionic):
+    pkgs = bionic.pkgs | {
+        "abi-compliance-checker",
+        "abi-dumper",
+        "ca-certificates",
+        "clang-7",
+        "fakeroot",
+        "gcc-8",
+        "git",
+        "python2.7",
+        "sparse",
+    } | {
+        # 32 bit build support
+        "libgcc-8-dev:i386",
+        "libc6-dev:i386",
+        "libnl-3-dev:i386",
+        "libnl-route-3-dev:i386",
+        "libsystemd-dev:i386",
+        "libudev-dev:i386",
+    } | {
+        # ARM 64 cross compiler
+        "gcc-8-aarch64-linux-gnu",
+        "libgcc-8-dev:arm64",
+        "libc6-dev:arm64",
+        "libnl-3-dev:arm64",
+        "libnl-route-3-dev:arm64",
+        "libsystemd-dev:arm64",
+        "libudev-dev:arm64",
+    }
+    to_azp = True;
+    name = "azure_pipelines";
+    aliases = {"azp"}
+
+    def get_docker_file(self):
+        res = bionic.get_docker_file(self);
+        res.lines.insert(1,"RUN dpkg --add-architecture i386 &&"
+                         "dpkg --add-architecture arm64 &&"
+                         "sed -i -e 's/^deb /deb [arch=amd64,i386] /g' /etc/apt/sources.list &&"
+                         "echo 'deb [arch=arm64] http://ports.ubuntu.com/ bionic main universe\\n"
+                               "deb [arch=arm64] http://ports.ubuntu.com/ bionic-security main universe\\n"
+                               "deb [arch=arm64] http://ports.ubuntu.com/ bionic-updates main universe' > /etc/apt/sources.list.d/arm64.list");
+        return res;
+
+# -------------------------------------------------------------------------
+
 environments = [centos6(),
                 centos7(),
                 centos7_epel(),
@@ -380,6 +431,7 @@ environments = [centos6(),
                 leap(),
                 tumbleweed(),
                 debian_experimental(),
+                azure_pipelines(),
 ];
 
 class ToEnvActionPkg(argparse.Action):
@@ -751,6 +803,91 @@ def run_travis_build(args,env):
             raise;
         copy_abi_files(os.path.join(tmpdir, "src/ABI"));
 
+def run_azp_build(args,env):
+    import yaml
+    # Load the commands from the pipelines file
+    with open("buildlib/azure-pipelines.yml") as F:
+        azp = yaml.load(F);
+    for bst in azp["stages"]:
+        if bst["stage"] == "Build":
+            break;
+    else:
+        raise ValueError("No Build stage found");
+    for job in bst["jobs"]:
+        if job["job"] == "Compile":
+            break;
+    else:
+        raise ValueError("No Compile job found");
+
+    script = ["#!/bin/bash"]
+    workdir = "/__w/1"
+    srcdir = os.path.join(workdir,"s");
+    for I in job["steps"]:
+        script.append("echo ===================================");
+        script.append("echo %s"%(I["displayName"]));
+        script.append("cd %s"%(srcdir));
+        if "bash" in I:
+            script.append(I["bash"]);
+        elif I.get("task") == "PythonScript@0":
+            script.append("set -e");
+            script.append("%s %s"%(I["inputs"]["pythonInterpreter"],
+                                   I["inputs"]["scriptPath"]));
+        else:
+            raise ValueError("Unknown stanza %r"%(I));
+
+    with private_tmp(args) as tmpdir:
+        os.mkdir(os.path.join(tmpdir,"s"));
+        os.mkdir(os.path.join(tmpdir,"tmp"));
+
+        opwd = os.getcwd();
+        with inDirectory(os.path.join(tmpdir,"s")):
+            subprocess.check_call(["git",
+                                   "--git-dir",os.path.join(opwd,".git"),
+                                   "reset","--hard","HEAD"]);
+            subprocess.check_call(["git",
+                                   "--git-dir",os.path.join(opwd,".git"),
+                                   "fetch",
+                                   "--no-tags",
+                                   "https://github.com/linux-rdma/rdma-core.git","HEAD";,
+                                   "master"]);
+            base = subprocess.check_output(["git",
+                                            "--git-dir",os.path.join(opwd,".git"),
+                                            "merge-base",
+                                            "HEAD","FETCH_HEAD"]).strip();
+
+        opts = [
+            "run",
+            "--read-only",
+            "--rm=true",
+            "-v","%s:%s"%(tmpdir, workdir),
+            "-w",srcdir,
+            "-u",str(os.getuid()),
+            "-e","SYSTEM_PULLREQUEST_SOURCECOMMITID=HEAD",
+            # azp puts the branch name 'master' here, we need to put a commit ID..
+            "-e","SYSTEM_PULLREQUEST_TARGETBRANCH=%s"%(base),
+            "-e","HOME=%s"%(workdir),
+            "-e","TMPDIR=%s"%(os.path.join(workdir,"tmp")),
+        ] + map_git_args(opwd,srcdir);
+
+        if args.run_shell:
+            opts.append("-ti");
+        opts.append(env.image_name());
+
+        with open(os.path.join(tmpdir,"go.sh"),"w") as F:
+            F.write("\n".join(script))
+
+        if args.run_shell:
+            opts.append("/bin/bash");
+        else:
+            opts.extend(["/bin/bash",os.path.join(workdir,"go.sh")]);
+
+        try:
+            docker_cmd(args,*opts);
+        except subprocess.CalledProcessError, e:
+            copy_abi_files(os.path.join(tmpdir, "s/ABI"));
+            raise;
+        copy_abi_files(os.path.join(tmpdir, "s/ABI"));
+
 def args_pkg(parser):
     parser.add_argument("ENV",action=ToEnvActionPkg,choices=env_choices_pkg());
     parser.add_argument("--run-shell",default=False,action="store_true",
@@ -766,6 +903,8 @@ def cmd_pkg(args):
     for env in args.ENV:
         if env.name == "travis":
             run_travis_build(args,env);
+        elif env.name == "azure_pipelines":
+            run_azp_build(args,env);
         elif getattr(env,"is_deb",False):
             run_deb_build(args,env);
         elif getattr(env,"is_rpm",False):
-- 
2.21.0




[Index of Archives]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Photo]     [Yosemite News]     [Yosemite Photos]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux