Hello,
I'm a long time user and only an occasional contributor, but I do have
experience on the Go side, so perhaps my input is helpful. I keep
forgetting that I can't reply to arch-dev-public, so I'm replying to the
poster and arch-general.
Overall I agree with simplifying and modernizing the Go guidelines.
Practically every Go project uses modules now, not using Go's own linker
will come as a surprise to most Arch users, and I've always found all
those env vars and flags on top of "go build" to be a surprising setup.
Your suggestion of just three flags in GOFLAGS seems far better.
That said, I think your suggestion to just use "go build" everywhere is
unfortunately not realistic, at least not yet given my experience out in
the wild. That's because runtime/debug.ReadBuildInfo isn't actually
quite enough.
It does embed a module semver version when building via GOPROXY, e.g.
"go install cuelang.org/go/cmd/cue@latest", and it also embeds VCS
information when doing a local build inside a git clone, e.g. "git
checkout master && go install ./cmd/cue". Crucially, however, in the
latter case the main module lacks a module semver version, and the
stamped VCS information is just a commit hash and a timestamp, without
any notion about what the current or last semver tag was.
This is a known limitation, and tracked at
https://github.com/golang/go/issues/50603. Until this is fixed, some
projects still support the ldflags hack, or hard-code a version string
to manually change at every release, or simply live with "devel" or
"commit X" versions when built locally. Upstream accepted the proposal,
but it may be another couple of years before it's shipped and widespread
enough that the ldflags hack is gone for good.
I also see that you suggest adding -buildvcs=false to GOFLAGS; I'm not
sure if that is the right choice or not for Arch, but consider the
limitation of the BuildInfo version string I explain above. In the Go
programs that I write, I make the "version" command or flag show the VCS
information, precisely because otherwise there's no way to tell what the
real version might actually be:
$ git rev-parse HEAD # in https://github.com/cue-lang/cue
b648cf45d5adc6592633206cc126670906881729
$ git describe --tags # note how I'm a few commits after the
v0.9.0-0.dev tag
v0.9.0-0.dev-4-gb648cf45$ go install ./cmd/cue
$ cue version # the version at the top is a hard-coded string named
"fallbackVersion"
cue version v0.9.0-0.dev
go version devel go1.23-0a6f05e30f 2024-03-17 15:57:54 +0000
-buildmode exe
-compiler gc
DefaultGODEBUG
asynctimerchan=1,httplaxcontentlength=1,httpmuxgo121=1,tls10server=1,tlsrsakex=1,tlsunsafeekm=1,winreadlinkvolume=0,winsymlink=0
CGO_ENABLED 1
GOARCH amd64
GOOS linux
GOAMD64 v3
vcs git
vcs.revision b648cf45d5adc6592633206cc126670906881729
vcs.time 2024-03-18T07:55:11Z
vcs.modified false
TL;DR: sounds good to me, just don't assume that every project can
simply "go build" now, and consider not disabling -buildvcs, at least
for the time being.
Cheers.
On 18/03/2024 23.17, kpcyrd wrote:
hello,
the Arch Linux Go packaging guidelines[0] currently suggest this for
building software written in Go:
> export CGO_CPPFLAGS="${CPPFLAGS}"
> export CGO_CFLAGS="${CFLAGS}"
> export CGO_CXXFLAGS="${CXXFLAGS}"
> export CGO_LDFLAGS="${LDFLAGS}"
> export GOFLAGS="-buildmode=pie -trimpath -ldflags=-linkmode=external
-mod=readonly -modcacherw"
>
> # or alternatively you can define some of these flags from the CLI
options
> go build \
> -trimpath \
> -buildmode=pie \
> -mod=readonly \
> -modcacherw \
> -ldflags "-linkmode external -extldflags \"${LDFLAGS}\"" \
> .
[0]: https://wiki.archlinux.org/title/Go_package_guidelines
There's also section "2.2.2 Supporting debug packages" which
contradicts the previous section.
## Suggestion
I'd like to propose changing this to something like:
> prepare() {
> cd ${pkgname}-${pkgver}
> go mod download
> }
>
> build() {
> cd ${pkgname}-${pkgver}
> go build .
> }
>
> package() {
> cd ${pkgname}-${pkgver}
> install -Dm0755 -t "${pkgdir}/usr/bin/" "${pkgname}"
> }
Plus any extra flags (for example for debug packages to work, if any
are needed at all - there's a note about this towards the end of this
essay). Instead of suggesting two (three) different approaches, the
packaging guidelines should endorse argv over environment variables
(how to set them with environment variables can still be documented in
a later section of the guideline). The reason I'm suggesting this:
## Passing flags over argv
Setting these options through environment variables is for cases when
`go` isn't executed directly by our PKGBUILD, but instead by some
indirection like a Makefile. Makefiles are more popular in Go projects
than they are in e.g. Rust projects, but presumably because for many
years (10+) there was no analogous to `env!("CARGO_PKG_VERSION")` to
embed a version string into the binary for --version output.
Since almost exactly two years ago, in Go 1.18 released March 2022,
there's now `runtime/debug.ReadBuildInfo` which gives you a reference
to an embedded struct that contains a version string[1].
[1]: https://pkg.go.dev/runtime/debug#ReadBuildInfo
Hopefully more Go projects are going to favor this in the future over
custom build scripts and passing `-X main.Version=...` with output
from `git describe --tags`. If this catches on, people might think of
`go build` more like they currently think of `cargo build` and I think
the Go package guideline should encourage this.
Not just for Go but for software in general, using `git describe`
should be considered an anti-pattern. In the way the Arch Linux build
system currently works, only the checked out files are authenticated
and pinned by sha256sums= and friends, but `git describe` explicitly
works on unauthenticated git objects in `.git/`. I've published a
writeup on how to take advantage of this on the reproducible builds
list in September 2023[2].
[2]:
https://lists.reproducible-builds.org/pipermail/rb-general/2023-September/003075.html
## Using the Go native Linker
Binaries built by the Go compiler have been reproducible for a long
time, however Go binaries built by Arch Linux usually aren't. This has
led to some confusion, with bug reports to upstreams of Go software
because it wasn't obvious that the package not being reproducible in
this case isn't a problem in upstream's source code, but the way Arch
Linux compiles it.
Arch Linux generally prefers binaries with hardening flags enabled[3]
and therefore explicitly opts into a non-default linker (Cgo), but it
seems to be the unpopular choice. Cgo seems to have less support than
normal Go on Linux (evidently, since Cgo has been known to have
reproducible build issues for years now). Other Linux distributions
seem to stick to normal Go too, Nix has Cgo off by default[4] and
searching through nixpkgs for "buildGoModule" yields about 2k results,
searching for "CGO_ENABLED" has 105 results, with more than half being
"CGO_ENABLED=0" though. Alpine Linux only sets GOFLAGS globally[5]
(which Arch Linux currently doesn't), searching for `rg -l --multiline
'\tgo.*build'` in aports matches 356 files, `rg -l CGO_ENABLED=0`
matches 23 files and `rg -l CGO_ENABLED=1` matches 18 files.
[3]: https://github.com/jvoisin/compiler-flags-distro
[4]:
https://github.com/NixOS/nixpkgs/blob/b081342f1c16e4cbe4f40f139bbdda1475ea306a/pkgs/build-support/go/module.nix
[5]: https://git.alpinelinux.org/abuild/tree/default.conf?h=3.12.0#n5
The global GOFLAGS as specified in Alpine Linux are:
> export GOFLAGS="-buildmode=pie -modcacherw -trimpath -buildvcs=false"
Which we could consider setting globally too, like we do with RUSTFLAGS.
Figuring out what Debian does was somewhat challenging, their
integration with the go build system is called dh-golang[6], they seem
to set CGO_ENABLED=1 only when cross-compiling, with go >= 1.13
(released 2019) they seem to build with `go install -trimpath ...` and
no further GOFLAGS environment variable.
[6]:
https://salsa.debian.org/go-team/packages/dh-golang/-/blob/debian/sid/lib/Debian/Debhelper/Buildsystem/golang.pm?ref_type=heads
I understand the concern of Go possibly being more prone to ROP-style
exploits when 1) built with the native linker and 2) used with e.g. a
vulnerable version of libgit2, however, as of 2024, there have been
barely any memory-corruption based exploits for Go software.
## Motivation
- Most of our Go software is currently not reproducible due to Cgo,
including core/libcap, which is the last unreproducible package in
docker.io/library/archlinux
- The barrier for packaging Go in Arch Linux is currently somewhat
high (compared to e.g. packaging Rust), the guideline requires too
much interpretation and could be improved
- Quirks that are only needed for old Go projects (like 2.1.1) should
be listed towards the end instead of being the first code block in the
guideline
---
cheers,
kpcyrd
