Re: [libgpiod][WIP PATCH 0/2] Convert the build from autotools to meson

["Andrew Jeffery" <andrew@xxxxxxxx>]

On Fri, 9 Dec 2022, at 05:18, Bartosz Golaszewski wrote:
> On Thu, Dec 8, 2022 at 12:11 PM Andrew Jeffery <andrew@xxxxxxxx> wrote:
>>
>>
>>
>> On Thu, 8 Dec 2022, at 19:57, Bartosz Golaszewski wrote:
>> > On Thu, Dec 8, 2022 at 5:23 AM Andrew Jeffery <andrew@xxxxxxxx> wrote:
>> >>
>> >>
>> >>
>> >> On Tue, 6 Dec 2022, at 05:25, Bartosz Golaszewski wrote:
>> >> > On Mon, Dec 5, 2022 at 2:22 PM Andrew Jeffery <andrew@xxxxxxxx> wrote:
>> >> >>
>> >> >> Hello,
>> >> >>
>> >> >> Based on a recent poke [1] and in-between meetings I've put together a
>> >> >> WIP series that converts libgpiod's build from autotools to meson. As
>> >> >> far as I'm aware the meson build supports all the significant options to
>> >> >> enable or disable features exposed by the autotools build:
>> >> >>
>> >> >> * Tests
>> >> >> * Tools
>> >> >>   * Interactive gpioset
>> >> >> * Bindings
>> >> >>   * C++
>> >> >>   * Python
>> >> >>   * Rust
>> >> >> * Documentation
>> >> >>   * Manpages
>> >> >>   * Doxygen
>> >> >>
>> >> >> [1] https://lore.kernel.org/all/CAMRc=Mda8UnyH+_GxeX_4MyKd+DPN0BVH5K+J+VWnMJNC1vwTQ@xxxxxxxxxxxxxx/
>> >> >>
>> >> >> Meson has pretty good support for handling python and so the patch does
>> >> >> away with setup.py entirely.
>> >> >
>> >> > Eek! No, please do keep setup.py. Autotools too is capable of building
>> >> > python C extensions on its own and it's what we use in v1 but I want
>> >> > the python code to be built the standard python way. I actually plan
>> >> > to post libgpiod v2 on pypi and split out building python bindings
>> >> > into a separate bitbake recipe in meta-openembedded using the
>> >> > setuptools3 class.
>> >> >
>> >> > So let's keep setup.py and just call it from meson.
>> >>
>> >> I've poked at this for a little while and it's not a great experience.
>> >> Meson's design pushes back against calling out in this way, and I don't
>> >> really have the motivation to carry on fighting it to make it do what
>> >> you request. Unless someone else has that motivation, I think there are
>> >> two options if meson is still desired:
>> >>
>> >> 1. Use the meson python support as posted in this series
>> >> 2. Split out the python (and probably rust) bindings, keeping the
>> >>    dependency relationships pointing in one direction and using the
>> >>    language's own package management tooling.
>> >>
>> >> Given there's nothing to do in the install phase for rust we don't have
>> >> as big of an issue there, but it is problematic for python.
>> >>
>> >> Let me know which way you want to go, including if you want to abandon
>> >> meson :)
>> >>
>> >
>> > No, I don't want to abandon it. What is the problem exactly? Is meson
>> > unable to simply add external commands to its ninja output?
>>
>> Not as far as I'm aware. I think it's best covered by this policy
>> description:
>>
>> https://mesonbuild.com/Mixing-build-systems.html
>>
>> There are some things that might make it sound feasible but aren't
>> actually appropriate:
>>
>> 1. run_command(): https://mesonbuild.com/Reference-manual_functions.html#run_command
>> 2. run_target(): https://mesonbuild.com/Reference-manual_functions.html#run_target
>> 3. custom_target(): https://mesonbuild.com/Reference-manual_functions.html#custom_target
>>
>> run_command() isn't appropriate as it executes in the `meson setup`
>> phase. run_target() isn't appropriate as it disregards any output
>> artifacts and so has no impact in the `meson install` phase.
>>
>> custom_target() is probably closest to what is required, but there's a
>> lot of pain in trying to get the artifacts to line up for correct
>> deployment in the `meson install` phase. This is exacerbated by the
>> requirement that setup.py be run from its containing directory in the
>> source tree. Further, I couldn't get all the options to line up such
>> that setuptools would relocate its output into meson's own build tree
>> (and out of the source tree). Here's a not entirely working attempt at
>> abusing custom_target() to that end:
>>
>> ```
>> diff --git a/bindings/python/meson.build b/bindings/python/meson.build
>> index 26f7ff13e0dd..136d10824345 100644
>> --- a/bindings/python/meson.build
>> +++ b/bindings/python/meson.build
>> @@ -3,14 +3,31 @@
>>
>>  python = import('python')
>>  python3 = python.find_installation('python3')
>> -python3_dep = python3.dependency()
>>
>> -subdir('gpiod')
>> +python_build_dir = 'python-build'
>> +python_install_dir = 'python-install'
>> +python_include_dirs = '../../include:../../tests/gpiosim'
>> +python_lib_dirs = '@0@/lib:@0@/tests/gpiosim'.format(meson.project_build_root())
>> +python_install_cmd = [ python3.full_path(), '@INPUT@', '--no-user-cfg',
>> +                      'build_ext', '--include-dirs', python_include_dirs, '--library-dirs', python_lib_dirs,
>> +                      'install', '--root', python_build_dir, '--prefix', get_option('prefix')]
>>
>> -if get_option('examples')
>> -    subdir('examples')
>> -endif
>> +python_env = environment()
>> +python_env.set('GPIOD_WITH_TESTS', get_option('tests').to_string())
>>
>> -if get_option('tests')
>> -    subdir('tests')
>> -endif
>> +python_setuptools = custom_target('python-setuptools',
>> +                                 input: 'setup.py',
>> +                                 output: python_build_dir,
>> +                                 depends: [gpiod, gpiosim],
>> +                                 env: python_env,
>> +                                 command: python_install_cmd)
>> +
>> +cp = find_program('cp')
>> +
>> +custom_target('python-install',
>> +             input: 'setup.py',
>> +             output: python_install_dir,
>> +             depends: python_setuptools,
>> +             command: [ cp, '-r', meson.current_source_dir() / python_build_dir, meson.current_build_dir() / python_install_dir ],
>> +             install: true,
>> +             install_dir: get_option('prefix'))
>> diff --git a/bindings/python/setup.py b/bindings/python/setup.py
>> index ec8f99d4013d..9eddae7466a1 100644
>> --- a/bindings/python/setup.py
>> +++ b/bindings/python/setup.py
>> @@ -1,9 +1,14 @@
>>  # SPDX-License-Identifier: GPL-2.0-or-later
>>  # SPDX-FileCopyrightText: 2022 Bartosz Golaszewski <brgl@xxxxxxxx>
>>
>> -from os import environ
>> +import os
>> +import sys
>> +
>> +from os import environ, path
>>  from setuptools import setup, Extension, find_packages
>>
>> +os.chdir(path.dirname(sys.argv[0]) or '.')
>> +
>>  gpiod_ext = Extension(
>>      "gpiod._ext",
>>      sources=[
>> ```
>>
>> This commits a bunch of crimes:
>>
>> 1. Assumes the structure of the meson build directory via the paths in
>>    python_lib_dirs
>> 2. Adds a chdir() in setup.py to relocate the process out of the meson
>>    build directory back into the source tree
>> 3. Assumes the chdir() operation in the setup of python_include_dirs
>>    rather than relying on meson's built-in dependency tracking as the
>>    inc objects are opaque[1]
>> 4. Hacks the setuptools output back into the meson build directory using
>>    a crufty target invoking `cp` so meson can locate the artifacts in the
>>    `meson install` phase
>> 5. Still doesn't correctly install the artifacts in the end due to
>>    restrictions on path mangling (can't strip off the parent directory)
>>    and the fact that we're trying to install an entire tree rather than
>>    specific files.
>>
>> [1] https://mesonbuild.com/Reference-manual_returned_inc.html
>>
>> It might feel like install_data() or install_subdir() could be used
>> here, but from experiment their behaviour also seems unfit to be used
>> in this context.
>>
>> At least, that's what I've experimented with. Maybe others can see the
>> way through here, but it really is fighting against the policy linked
>> earlier.
>>
>> Andrew
>
> I see. I understand that meson doesn't like dealing with other build-systems.
>
> The thing I like about the current autotools setup is that with the
> following one-liner:
>
> ./autogen.sh --prefix=/tmp/gpio/inst --enable-bindings-cxx
> --enable-examples --enable-tests --enable-tools
> --enable-gpioset-interactive --enable-bindings-python
> --enable-bindings-rust && make -j16 && sudo ./tests/gpiod-test && sudo
> ./bindings/cxx/tests/gpiod-cxx-test && sudo
> PYTHONPATH=./bindings/python
> LD_LIBRARY_PATH=./lib/.libs/:./tests/gpiosim/.libs/:bindings/python/
> python -B -m tests && cd bindings/rust/; sudo
> CARGO_TARGET_DIR=/tmp/libgpiod-rust PATH=/home/brgl/.cargo/bin/:$PATH
> /home/brgl/.cargo/bin/cargo test; cd ../.. && sudo
> ./tools/gpio-tools-test
>
> I can configure, build and test the entire code base while also using
> the language specific build tools for python and rust.

Right; I agree it's desirable to retain that capability.

>
> I will try to play with your patches and maybe figure it out or even a
> close approximation of the current functionality but then again: I'm
> not well versed with meson yet. Between it and rust and dayjob my cup
> runneth over...

I recognise that feeling :)

Happy to bounce ideas around if you experiment with it.

Andrew