Thanks for writing all of this. I'm writing from the perspective of a long-term user of the autotools. A discussion like the one you have started will likely attract many opinions. Some will be contradictory. However, somebody in the end will have to decide. The challenge seems to be to evolve the system with regard to backward compatibility. You mention the use of many languages, like m4. This brings me to my first suggestion: make automake work without autoconf. This would potentially allow configuration scripts (if there is to be such a thing) to be written in a language that doesn't use m4. The purpose of autoconf, and of configure scripts, could be clarified. The manual says, "Autoconf is a tool for producing shell scripts that automatically configure software source code packages to adapt to many kinds of Posix-like systems." Posix-like systems, and thus portability, is implied to be a major issue. Many would not care about portability, especially when starting a small personal project. (This was my reaction when I first started using the autotools.) I see the purpose of a configure script to be two-fold: * Allowing a user to make choices about what features are built, and how the package is built (e.g. installation paths) * Adapting to operating system variants The first will always be relevant, even if you don't care about the second. I think configuration scripts on GNU-like systems would always have a heavy element of shell script in them, even if m4 or autoconf macros were abandoned. There would be no need to move so far away from this. If automake could generate its own configure script without autoconf, then the interface of the GNU Build System could continue unchanged for people building software. It would also simplify matters for people learning to use the autotools, that they could just create a single Makefile.am to describe their project (like a CMakefile for cmake or .pro file for qmake), rather than having to create Makefile.am and configure.ac. It could be made easier to get started with automake. For example, "AUTOMAKE_OPTIONS=foreign" should be the default; otherwise automake refuses to finish if required files aren't found in the project. There's a political problem here to persuade people of this. The documentation could also be improved. I remember somebody was complaining about this page: https://www.gnu.org/software/automake/manual/html_node/Program-and-Library-Variables.html and asking what "maude" meant - it turned out it was the name of the dog or something of the person who first wrote the program, who didn't want it to change. So think getting consensus is impossible for changes that are needed to make the program simpler, easier and better. There's a vacuum of responsibility where nobody wants to step on others' feet. Forking the project would go nowhere as nobody would use the fork. Your idea about adding support for new programming languages is good and if someone did this along with other changes, it could stimulate adoption of the other changes. It would be nice if all the code required to support a new language could occur as a self-contained "module" within automake. On Wed, Jan 20, 2021 at 10:15 PM Zack Weinberg <zackw@xxxxxxxxx> wrote: > > Now we've all had a while to recover from the long-awaited Autoconf > 2.70 release, I'd like to start a conversation about where the > Autotools in general might be going in the future. Clearly any future > development depends on finding people who will do the work, but before > we worry about that I think we might want to figure out what work we > _want_ to do. > > As a starting point, I wrote up a "strengths, weaknesses, > opportunities, and threats" analysis for Autotools -- this is a > standard project management technique, if you're not familiar with it, > there's a nice writeup in the draft of the book my friend and > colleague Sumana Harihareswara is writing [ > https://changeset.nyc/resources/getting-unstuck-sampler-offer.html ]. > I'm going to paste the full text of this analysis below, please reply > inline. You can also read it on my blog at > https://www.owlfolio.org/development/autoconf-swot/ . > > zw > ---- > > I’ve been a contributor to GNU projects for many years, notably both > GCC and GNU libc, and recently I led the effort to make the first > release of Autoconf since 2012 (release announcement for Autoconf > 2.70). For background and context, see the LWN article my colleague > Sumana Harihareswara of Changeset Consulting wrote. > > Autoconf not having made a release in eight years is a symptom of a > deeper problem. Many GNU projects, including all of the other > components of the Autotools (Automake, Libtool, Gnulib, etc.) and the > software they depend upon (GNU M4, GNU Make, etc.) have seen a steady > decline in both contributor enthusiasm and user base over the past > decade. I include myself in the group of declining enthusiasts; I > would not have done the work leading up to the Autoconf 2.70 release > if I had not been paid to do it. (I would like to say thank you to the > project funders: Bloomberg, Keith Bostic, and the GNU Toolchain Fund > of the FSF.) > > The Autotools are in particularly bad shape due to the decline in > contributor enthusiasm. Preparation for the Autoconf 2.70 release took > almost twice as long as anticipated; I made five beta releases between > July and December 2020, and merged 157 patches, most of them bugfixes. > On more than one occasion I was asked why I was going to the > trouble—isn’t Autoconf (and the rest of the tools by implication) > thoroughly obsolete? Why doesn’t everyone switch to something newer, > like CMake or Meson? (See the comments on Sumana’s LWN article for > examples.) > > I personally don’t think that the Autotools are obsolete, or even all > that much more difficult to work with than some of the alternatives, > but it is a fair question. Should development of the Autotools > continue? If they are to continue, we need to find people who have the > time and the inclination (and perhaps also the funding) to maintain > them steadily, rather than in six-month release sprints every eight > years. We also need a proper roadmap for where further development > should take these projects. As a starting point for the conversation > about whether the projects should continue, and what the roadmap > should be, I was inspired by Sumana’s book in progress on open source > project management (sample chapters are available from her website) to > write up a strengths, weaknesses, opportunities, and threats analysis > of Autotools. > > This inventory can help us figure out how to build on new > opportunities, using the Autotools’ substantial strengths, and where > to invest to guard against threats and shore up current weaknesses. > > Followup discussion should go to the Autoconf mailing list. > > Strengths > > In summary: as the category leader for decades, the Autotools benefit > from their architectural approach, interoperability, edge case > coverage, standards adherence, user trust, and existing install base. > > Autoconf’s feature-based approach to compiled-code portability scales > better than lists of system quirks. > The Autotools carry 30+ years’ worth of embedded knowledge about > portability traps for C programs and shell-based build scripting on > Unix (and to a lesser extent Windows and others), including variants > of Unix that no other comparable configuration tool supports. > Autoconf and Automake support cross-compilation better than competing > build systems. > Autoconf and Automake support software written in multiple languages > better than some competing build systems (but see below). > Autoconf is very extensible, and there are lots of third-party macros available. > Tarball releases produced by Autotools have fewer build dependencies > than tarball releases produced by competing tools. > Tarball releases produced by Autotools have a predictable, > standardized (literally; it’s a key aspect of the GNU Coding > Standards) interface for setting build-time options, building them, > testing them, and installing them. > Automake tries very hard to generate Makefiles that will work with any > Make implementation, not just GNU make, and not even just (GNU or BSD) > make. > The Autotools have excellent reference-level documentation (better > than CMake and Meson’s). > As they are GNU projects, users can have confidence that Autotools are > and will always remain Free Software. > Relatedly, users can trust that architectural decisions are not driven > by the needs of particular large corporations. > There is a large installed base, and switching to a competing build > system is a lot of work. > > Weaknesses > > In summary: Autoconf’s core function is to solve a problem that > software developers, working primarily in C, had in the 1990s/early > 2000s (during the Unix wars). System programming interfaces have > become much more standardized since then, and the shell environment, > much less buggy. Developers of new code, today, looking at existing > configure scripts and documentation, cannot easily determine which of > the portability traps Autoconf knows about are still relevant to them. > Similarly, maintainers of older programs have a hard time knowing > which of their existing portability checks are still necessary. And > weak coordination with other Autotools compounds the issue. > > Autoconf > > Autoconf (and the rest of the Autotools) are written in a combination > of four old and difficult programming languages: Bourne shell, the > portable subset of Make, Perl, and M4. Competing build systems tend to > use newer, more ergonomic languages, which both makes it easier for > them to get things done, and makes it easier for them to attract new > developers. > All the supported languages except C and C++ are second-class citizens. > The set of languages that are supported has no particular rationale. > Several new and increasingly popular compiled-code languages (e.g. > Swift and Rust) are not supported, while oddities like Erlang are. > Much of that 30 years’ worth of embedded knowledge about portability > traps is obsolete. There’s no systematic policy for deciding when some > problem is too obsolete to worry about anymore. > Support for newer platforms, C standard editions, etc. is weaker than > support for older things. > Autoconf’s extensibility is unsystematic; many of those third-party > macros reach into its guts, and do things that create awkward > compatibility constraints on core development. Same for existing > configure.acs. > The code quality of third-party macros varies widely; bad third-party > macros reflect poorly on Autoconf proper. > Some of the ancillary tools distributed with Autoconf don’t work well; > most importantly, autoupdate (which is supposed to patch a > configure.ac to bring it in line with current Autoconf’s > recommendations) is so limited and unreliable that it might be better > not to have it at all. > Feature gaps in GNU M4 hold back development of Autoconf. > > The Autotools as a whole > > There are few active developers and no continuing funders. > GNU project status discourages new contributors because of the > paperwork requirements and the perceived lack of executive-level > leadership. > There is no continuous integration and no culture of code review. Test > suites exist but are not comprehensive enough (and at the same time > they’re very slow). > Bugs, feature requests, and submitted patches are not tracked > systematically. (This is partially dependent on FSF/GNU infrastructure > improvements which are indefinitely delayed.) > There’s a history of releases breaking compatibility, and thus people > are hesitant to upgrade. At the same time, Linux distributions > actively want to force-upgrade everything they ship to ensure > architecture support, leading to upstream/downstream friction. > Guide-level documentation is superficial and outdated. > Building an Autotools-based project directly from its VCS checkout is > often significantly harder than building it from a tarball release, > and may involve tracking down and installing any number of unusual > tools. > The Autotools depend on other GNU software that is not actively > maintained, most importantly GNU M4, and to a lesser extent GNU Make. > Coordination among the Autotools is weak, even though the tools are > tightly coupled to each other. There are portions of codebases that > exist solely for interoperability with other tools in the toolchain, > which leads to overlapping maintainer and reviewer responsibility, > slow code review and inconvenient copyright assignment processes > multiplying, and causing confusion and dropped balls. For instance, > there is code shared among Autoconf, Automake, and/or Gnulib by > copying files between source repositories; changes to these files are > extra inconvenient. The lack of coordination also makes it harder for > tool maintainers to deprecate old functionality, or to decouple > interfaces to make things more extensible; maintainers do not > negotiate policies with each other to help. For instance, Autoconf has > trouble knowing when it is safe to remove internal kludges that old > versions of Automake depend on, and certain shell commands (e.g. > aclocal) are distributed with one package but abstractly belong to > another. > Division of labor among the Autotools, and the sources of third-party > macros, is ad-hoc and unclear. (Which macros should be part of > Autoconf proper? Which should be part of Gnulib? Which should be part > of the Autoconf Macro Archive? Which should be shipped with Automake? > Which tools should autoreconf know how to run? Etc.) > Automake and Libtool are not nearly as extensible as Autoconf is. > Unlike several competitors, Automake only works with Make, not with > newer build drivers (e.g. Ninja). > Because Automake tries to generate Makefiles that will work with any > Make implementation, the Makefiles it generates are much more > complicated and slow than they would be if they took advantage of GNU > and/or BSD extensions. > Libtool is notoriously slow, brittle, and difficult to modify (even > worse than Autoconf proper). This is partially due to technical debt > and partially due to maintaining support for completely obsolete > platforms (e.g. old versions of AIX). > Libtool has opinions about the proper way to manage shared libraries > that Linux distributions actively disagree with, forcing them to > kludge around its code during package builds. > Alternatives to Libtool have all failed to gain traction, largely > because Automake only supports building shared libraries using Libtool > or an exact drop-in replacement. > > Opportunities > > Because of its extensible architecture, install base, and wellspring > of user trust, Autotools can react to these industry changes and thus > spur increases in usage, investment, and developer contribution. > > Renewed interest in Autotools due to the Autoconf 2.70 release. > Renewed interest in systems programming due to the new generation of > systems programming languages (Go, Rust, D, Swift(?), Dart(?), etc. > may create an opportunity for a build system that handles them well > particularly if it handles polyglot projects well (see below). > Cross-compilation is experiencing new appeal because of the increasing > popularity of ARM and RISC-V CPUs, and of small devices (too small to > compile their own code) based on these chips. > The Free software ecosystem as a whole would benefit from a > reconciliation between the traditional model of software distribution > (compiled code with stable interfaces, released as tarballs at regular > intervals, installed once on any given computer and depended on as > shared libraries and/or binaries) and the newer depend directly on VCS > checkouts and bundle everything model described below. Autotools > contributors have the experience and knowledge to lead this effort. > Funding may be available for projects targeting the weaknesses listed above. > > Threats > > These threats may lead to a further decrease in Autotools developer > contribution, funding, and momentum. > > Increasing mindshare of competing projects (CMake, Meson, Generate-Ninja, …). > Increasing mindshare of programming languages that come with a build > system that works out of the box, as long as you only use that one > language in your project. (These systems typically cannot handle a > polyglot project at all, hence the above opportunity for a third-party > system that handles polyglot projects well.) > Increasing preference for building software from VCS checkouts > (perhaps at a specific tag, perhaps not) rather than via tarballs. > Increasing mindshare of the software distribution model originated by > Node.js, Ruby, etc. where each application bundles all of its > dependencies. While this is considered a profoundly bad idea by Linux > distribution maintainers in particular (because it makes it much > harder to find and patch a buggy dependency) and makes it harder for > end-users to modify the software (because out-of-date dependencies may > be very different from what their own documentation—describing the > latest version—says), it is significantly more convenient for upstream > developers. Competing build systems handle this model much better than > Autoconf does. >
