Re: [libgpiod][WIP PATCH 0/2] Convert the build from autotools to meson

["Andrew Jeffery" <andrew@xxxxxxxx>]

On Thu, 8 Dec 2022, at 19:57, Bartosz Golaszewski wrote:
> On Thu, Dec 8, 2022 at 5:23 AM Andrew Jeffery <andrew@xxxxxxxx> wrote:
>>
>>
>>
>> On Tue, 6 Dec 2022, at 05:25, Bartosz Golaszewski wrote:
>> > On Mon, Dec 5, 2022 at 2:22 PM Andrew Jeffery <andrew@xxxxxxxx> wrote:
>> >>
>> >> Hello,
>> >>
>> >> Based on a recent poke [1] and in-between meetings I've put together a
>> >> WIP series that converts libgpiod's build from autotools to meson. As
>> >> far as I'm aware the meson build supports all the significant options to
>> >> enable or disable features exposed by the autotools build:
>> >>
>> >> * Tests
>> >> * Tools
>> >>   * Interactive gpioset
>> >> * Bindings
>> >>   * C++
>> >>   * Python
>> >>   * Rust
>> >> * Documentation
>> >>   * Manpages
>> >>   * Doxygen
>> >>
>> >> [1] https://lore.kernel.org/all/CAMRc=Mda8UnyH+_GxeX_4MyKd+DPN0BVH5K+J+VWnMJNC1vwTQ@xxxxxxxxxxxxxx/
>> >>
>> >> Meson has pretty good support for handling python and so the patch does
>> >> away with setup.py entirely.
>> >
>> > Eek! No, please do keep setup.py. Autotools too is capable of building
>> > python C extensions on its own and it's what we use in v1 but I want
>> > the python code to be built the standard python way. I actually plan
>> > to post libgpiod v2 on pypi and split out building python bindings
>> > into a separate bitbake recipe in meta-openembedded using the
>> > setuptools3 class.
>> >
>> > So let's keep setup.py and just call it from meson.
>>
>> I've poked at this for a little while and it's not a great experience.
>> Meson's design pushes back against calling out in this way, and I don't
>> really have the motivation to carry on fighting it to make it do what
>> you request. Unless someone else has that motivation, I think there are
>> two options if meson is still desired:
>>
>> 1. Use the meson python support as posted in this series
>> 2. Split out the python (and probably rust) bindings, keeping the
>>    dependency relationships pointing in one direction and using the
>>    language's own package management tooling.
>>
>> Given there's nothing to do in the install phase for rust we don't have
>> as big of an issue there, but it is problematic for python.
>>
>> Let me know which way you want to go, including if you want to abandon
>> meson :)
>>
>
> No, I don't want to abandon it. What is the problem exactly? Is meson
> unable to simply add external commands to its ninja output?

Not as far as I'm aware. I think it's best covered by this policy 
description:

https://mesonbuild.com/Mixing-build-systems.html

There are some things that might make it sound feasible but aren't 
actually appropriate:

1. run_command(): https://mesonbuild.com/Reference-manual_functions.html#run_command
2. run_target(): https://mesonbuild.com/Reference-manual_functions.html#run_target
3. custom_target(): https://mesonbuild.com/Reference-manual_functions.html#custom_target

run_command() isn't appropriate as it executes in the `meson setup`
phase. run_target() isn't appropriate as it disregards any output
artifacts and so has no impact in the `meson install` phase.

custom_target() is probably closest to what is required, but there's a
lot of pain in trying to get the artifacts to line up for correct
deployment in the `meson install` phase. This is exacerbated by the
requirement that setup.py be run from its containing directory in the
source tree. Further, I couldn't get all the options to line up such
that setuptools would relocate its output into meson's own build tree
(and out of the source tree). Here's a not entirely working attempt at
abusing custom_target() to that end:

```
diff --git a/bindings/python/meson.build b/bindings/python/meson.build
index 26f7ff13e0dd..136d10824345 100644
--- a/bindings/python/meson.build
+++ b/bindings/python/meson.build
@@ -3,14 +3,31 @@
 
 python = import('python')
 python3 = python.find_installation('python3')
-python3_dep = python3.dependency()
 
-subdir('gpiod')
+python_build_dir = 'python-build'
+python_install_dir = 'python-install'
+python_include_dirs = '../../include:../../tests/gpiosim'
+python_lib_dirs = '@0@/lib:@0@/tests/gpiosim'.format(meson.project_build_root())
+python_install_cmd = [ python3.full_path(), '@INPUT@', '--no-user-cfg',
+                      'build_ext', '--include-dirs', python_include_dirs, '--library-dirs', python_lib_dirs,
+                      'install', '--root', python_build_dir, '--prefix', get_option('prefix')]
 
-if get_option('examples')
-    subdir('examples')
-endif
+python_env = environment()
+python_env.set('GPIOD_WITH_TESTS', get_option('tests').to_string())
 
-if get_option('tests')
-    subdir('tests')
-endif
+python_setuptools = custom_target('python-setuptools',
+                                 input: 'setup.py',
+                                 output: python_build_dir,
+                                 depends: [gpiod, gpiosim],
+                                 env: python_env,
+                                 command: python_install_cmd)
+
+cp = find_program('cp')
+
+custom_target('python-install',
+             input: 'setup.py',
+             output: python_install_dir,
+             depends: python_setuptools,
+             command: [ cp, '-r', meson.current_source_dir() / python_build_dir, meson.current_build_dir() / python_install_dir ],
+             install: true,
+             install_dir: get_option('prefix'))
diff --git a/bindings/python/setup.py b/bindings/python/setup.py
index ec8f99d4013d..9eddae7466a1 100644
--- a/bindings/python/setup.py
+++ b/bindings/python/setup.py
@@ -1,9 +1,14 @@
 # SPDX-License-Identifier: GPL-2.0-or-later
 # SPDX-FileCopyrightText: 2022 Bartosz Golaszewski <brgl@xxxxxxxx>
 
-from os import environ
+import os
+import sys
+
+from os import environ, path
 from setuptools import setup, Extension, find_packages
 
+os.chdir(path.dirname(sys.argv[0]) or '.')
+
 gpiod_ext = Extension(
     "gpiod._ext",
     sources=[
```

This commits a bunch of crimes:

1. Assumes the structure of the meson build directory via the paths in
   python_lib_dirs
2. Adds a chdir() in setup.py to relocate the process out of the meson
   build directory back into the source tree
3. Assumes the chdir() operation in the setup of python_include_dirs
   rather than relying on meson's built-in dependency tracking as the
   inc objects are opaque[1]
4. Hacks the setuptools output back into the meson build directory using
   a crufty target invoking `cp` so meson can locate the artifacts in the
   `meson install` phase 
5. Still doesn't correctly install the artifacts in the end due to
   restrictions on path mangling (can't strip off the parent directory)
   and the fact that we're trying to install an entire tree rather than
   specific files.

[1] https://mesonbuild.com/Reference-manual_returned_inc.html

It might feel like install_data() or install_subdir() could be used
here, but from experiment their behaviour also seems unfit to be used
in this context.

At least, that's what I've experimented with. Maybe others can see the 
way through here, but it really is fighting against the policy linked 
earlier.

Andrew