And, oh by the way, underneath? Shells have some of the absolutely bonkersly dumb parsers and interpreters; absolutely embarrassingly dumb stuff like aliases that can override keywords and change the parsing of shell script. The fact that some (most) shells interpret a shell script one line at a time, and that "built-ins" like the syntax for conditions in ifs looks like syntax but might be little binaries underneath (look up how "[" and "]" is handled in some shells--it might not be!).

What a wild irony that all the ickiest parts of UNIX--shells and scripts and autoconf and all that stringly-typed pipe stuff, ended up becoming (IMHO) the most reusable, pluggable programming environment yet devised...

Unix settled for string as a data structure and it's little too low level. All Lisp programmers know that s-expressions would have been the right choice. JSON works also.

Even smaller companies I’ve seen end up with an “interfaces” or “types” repo that contains definitions of every external object.

Narrow Waists Can Be Interior or Exterior: PyObject vs. Unix Files - https://www.oilshell.org/blog/2023/06/narrow-waist.html

The appendix links to: Why is Python the language of machine learning?

https://news.ycombinator.com/item?id=29171519

In particular, I highly recommend this bit of history from NumPy creator Travis Oliphant:

https://www.youtube.com/watch?v=gFEE3w7F0ww

https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#co...

Files had structure on pre-Unix OSes, but they don't on Unix, because it doesn't compose.

The "Uniform Interface Constraint" of REST is the same thing, and resembles a file system -- everything is GET / POST on URLs.

---

So the reason that shell/Unix and HTTP are so common is a MATHEMATICAL property of software growth.

How do you make a Zig program talk to a Mojo program? Probably with a byte stream.

What about a Clojure program and a Common Lisp program? Probably a byte stream. (Ironically, S-expressions have no commonly used "exterior" interchange format)

Every time a new language is introduced, I think "well there's another reason you're going to need a shell script".

---

The larger the system, the more heterogeneous it is. And software is larger now, which is why MORE GLUE is needed.

This is why shell was the #6 fastest growing language on Github in 2022: https://octoverse.github.com/2022/top-programming-languages

And the #1 fastest growing language is HCL, which is a very closely related form of glue. It's basically "shell for distributed systems"

---

https://www.oilshell.org/ has JSON as of a few months ago, and is now pure native code (no more Python)

All the normal shell stuff works:

    osh$ ls */*.py | wc -l; whoami
    327
    andy

    osh$ var d = {name: 42}

    osh$ json write (d)
    {
      "name": 42
    }

    shopt --set ysh:upgrade

I think of the data languages as layered -- i.e. an entire json message literally is a string, and it can also contain strings:

https://www.oilshell.org/blog/2023/06/ysh-design.html#exteri...

Probably because the details of S-expressions differ underneath the hood - a simple (foo ()) can mean a different thing on CL (where the second element is the symbol NIL) than on Scheme (where it is an empty list and not a symbol); also because Clojure introduces additional bracketing patterns, such as [] for arrays and {} for maps, which will confuse non-Clojures. And let's not talk about sexp variants of Lisp-Flavored Erlang.

Also, if you squint hard enough, https://en.wikipedia.org/wiki/Canonical_S-expressions works.

Also I have no idea if anyone ever cared enough to implement cross-language software transactional memory that would work across a process boundary, to close the gap between a Clojure process and a Lisp process. It could work in theory, but someone would need to do it - either get paid to do it, or have enough mojo of their own to write and bulletprove such a thing.

And, since everyone serializes to JSON anyway, serializing to JSON is a decent enough choice for communicating across Lisps as well.

Narrow Waists Can Be Interior or Exterior: PyObject vs. Unix Files

https://www.oilshell.org/blog/2023/06/narrow-waist.html

However the "exterior" extension seems obvious to me -- why hasn't anyone produced a distributed Lisp?

Well I guess Clojure/EDN is that, but nobody has produced a POLYGLOT system involving distributed Lisp. Common Lisp users probably don't use EDN very much, I gather.

---

I found this POSE exterior s-expression format, but it seems to have little interest from the Lisp community:

https://lobste.rs/s/lwf4jv/pose_portable_s_expressions_pose_...

It seems like JSON will have an "impedance mismatch" with Lisp data structures, but yeah that's the cost of interoperability

Common Lisp can read and write s-expressions from files. It can also compile source files and load files (both source and compiled).

The syntax for Common Lisp is standardized. Thus a lot of data may be shipped either in text or compiled files. Several implementations can also create snapshots of the runtime memory. A few domains use CL s-expressions (or a subset of that) as language syntax or as a data syntax.

Distributed Lisp applications exist, but may not use text to exchange data. For example there is a defined CORBA mapping for Common Lisp, which enabled Common Lisp software to take part in distributed software using a standard, which allows mixed language software.

How about Lisp Flavored Erlang?

I've noticed Erlang/Elixir also favor interior composition! Parsing is not idiomatic there -- they're actually BAD languages for parsing text. (Though they do have good affordances for parsing bytes)

They prefer to pass around Erlang terms, which are copied inside the VM.

I get that -- it's more convenient. But it's also a reason why "the rest of the world" has more code reuse -- e.g. Go talks to JavaScript talks to Python.

Most projects have multiple languages. Monoglot projects become polyglot projects when they grow bigger.

---

So I say

- Lisp prefers its own (interior) narrow waist

- Erlang prefers its own (interior) narrow waist -- even the Lisp one is second class!

- The rest of the world uses the (exterior) byte stream narrow waist, but they complain about it a lot :)

I mean, I love the look of it, but how can it help me from day dot?

    1. you already have a bash script (e.g. in your CI)
    2. you want to use an array or dict

You can upgrade to YSH, and start using our Python/JS like arrays and dicts instead. And then you can write them to JSON and upload them somewhere.

---

Another situation is if you use ShellCheck, but you want even more strictness at RUNTIME, then you can do

    shopt --set strict:all

https://www.oilshell.org/blog/2021/01/why-a-new-shell.html

Oils fixes all this, it's a from-scratch reimplementation of shell with an upgrade path from OSH (compatible) to YSH (new).

---

aliases that can override keywords and change the parsing of shell script

You can control this with shopt -u expand_aliases in bash, and YSH turns it off by default.

Oils is statically parsed: How To Parse Shell Like a Programming Language - https://www.oilshell.org/blog/2019/02/07.html

---

look up how "[" and "]" is handled in some shells--it might not be!

OSH runs POSIX [ as-is, for compatibility:

    osh$ [ x = x ] && echo yes
    yes

    ysh$ [ x = x ] && echo yes
      [ x = x ] && echo yes
      ^
    [ interactive ]:2: '[' should be invoked as 'test' (simple_test_builtin)

    ysh$ if ('x' === 'x') { echo yes }
    yes

A Tour of YSH - https://www.oilshell.org/release/latest/doc/ysh-tour.html

Why Create a New Shell? - https://www.oilshell.org/blog/2021/01/why-a-new-shell.html

(OSH is very stable and compatible, but YSH isn't stable yet. Getting there)

As a heavy user of bash I'm fully aware of how unaware I am of everything that can go wrong.

    shopt -s strict:all

On the other hand, YSH has strict:all and then a bunch of NEW features.

There's been good feedback recently, which has led to many concrete changes. So your experience can definitely influence the language! https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback

And back in 2005-ish I thought for sure that we'd have standardized on perl as a replacement for bash scripting by now.

https://www.oilshell.org/blog/2021/01/why-a-new-shell.html

https://www.oilshell.org/blog/2023/03/faq.html

OSH should be trivial to use on Solaris/AIX -- all you need is a C++ compiler and /bin/sh, no make tool even.

https://www.oilshell.org/blog/2023/12/screencasts.html#appen...

But if everyone on Linux and BSD uses bash/OSH, and Solaris AIX use /bin/sh, it will be a success. The latter platforms do not influence the rest of computing very much -- they lag behind.

---

The busybox shell and the FreeBSD shell have been gaining bash features lately -- which means they are gaining OSH features.

Because OSH is the most bash-compatible shell, by a mile. There's no other project like it.

---

I agree with the logic -- it is very hard to change languages.

It's like changing English to Esperanto. The inertia is incredible.

So you need to provide an upgrade path, and that's what Oils does. OSH is compatible, and YSH is a new design.

At any rate, you need to actually accomplish everyone on Linux and BSD having that available and having used it for the past 10 years first. And get the core distros switching over to using it so that the glue code in the distro is all YSH plus probably python and some legacy sh. Actually show me that.

And yeah, I think the analogy of trying to change English to Esperanto is about right. I learned about Esperanto in the mid-80s...

configure is one use case. And autoconf in particular will probably be around forever, regardless of the existence of Solaris/AIX -- for the simple reason that many old and important packages like coreutils use autoconf.

Probably nobody is going to rewrite the coreutils build system soon. (coreutils might be rewritten/obsolete before that happens!)

The good news is that OSH can run configure scripts as-is :) It's actually easy to run them, because they are meant to run on many different shells.

---

It's also not a binary yes/no thing. Cobol and Fortran aren't dead, but people choose to write new projects in different languages now, and that's good! I had an entire career without seeing any Cobol at all. Zero.

Likewise OSH can run POSIX configure scripts forever (and those same configure scripts can be run by /bin/sh and bash)

But I think lots of people will choose to write brand new YSH scripts as well.

The computing world is only getting more heterogeneous (in both time and space). Shell accomodates that, and Oils is designed to accomodate that.

At least on cloud and mobile space, we have largely moved beyond that, in the age of serverless, managed containers and programming languages.

Smalltalk and Lisp based REPLs are just as programmable, even more so given the whole OS exposure, and is no wonder why Notebooks got adopted by the science community and are inspired by them.

Just throw POSIX into the trash where it belongs. Then we can finally evolve past it.

Also, shellcheck is a wonderful tool. The script isn't done until shellcheck stops complaining.

I increasingly believe any shell script over ~128 “symbols” should be re-written in a “real” programming language.

I’ll gladly take a slightly longer Python, Go, or Rust “script” over Bash hell every day of the week.

I mean yes, but also only because the rest of computing is stuck in the same era. Meanwhile I'm over here popping nix-shells having my nice exact same nushell on every machine I could possibly want it on.

IMO it's all symptoms of the same problems. Related, I don't know how people start new projects and use autotools. But then again I'd say the same about cmake. But then again I just yesterday read a nightmare Lobsters comment about the realities of meson at the edges. And then I just sigh, thankful that I'm a Rust hipster (and then deal with cmake at the edges all the same). Computers are fun!

At least on cloud and mobile space, we have largely moved beyond that, in the age of serverless, managed containers and programming languages.

Autoconf also has a config.cache[2], which it uses to store answers to reuse across multiple of the same checks and reruns of the configure script.

[1]: https://www.gnu.org/software/autoconf/manual/autoconf-2.63/h...

[2]: https://www.gnu.org/software/autoconf/manual/autoconf-2.65/h...

Autoconf in a nutshell

Worse, automake and autoconf have a cyclic dependency, so if you're a developer (as opposed to a user building from source on a system) you wind up running configure all the time. This is because, most probably, your Makefile.am lists every source file individually. If you add a new source file, you must rerun autoreconf to regenerate your Makefile — and that also generates configure anew. Then the Makefile sees that configure is newer than config.h (or some other output, I don't want to look up the gory details) and, blammo!, when you run make, first configure runs.

Of course with git, this happens not just when you add a new source file to your project... it happens whenever you switch branches, where there's variance in the source files in the project. configure configure configure configure.

As much as I hate it, autoconf isn't the worst part of autotools. Automake is... pointless? Is it really less effort to write Makefile.am than to write a straight Makefile? Wouldn't that be simpler?

And then there's libtool, which should be drug out into a back alley and executed. A 2000 LoC shell script, whose whole purpose in life is to figure out, over and over and over again, stateless in the same way as TFA complains about with autoconf, precisely which compiler flags should be used to build a shared library on a system amongst all the old compilers, even though today everyone just uses gcc and clang, which are flag-compatible, and determining which flags to use is not really that hard. Please, someone, anyone, please kill libtool and banish it from this earth.

I tried Meson — ran into a problem, read the docs, asked a question on the mailing list, never got an answer, never got past it. Bazel's obviously a terrible Google project and won't work if you're not sufficiently Googly. Cmake, a disaster. I liked tup, but it seems to have died (and it didn't solve the autoconf problem anyway).

Which new build system will deliver us from the 18th circle of hell? (probably cargo, as the Rust fanbois work to get C/C++ deprecated...)

No you don't: automake generates rules for `Makefile.am -> Makefile.in` (`automake`) and `Makefile.in -> Makefile` (`config.status`). You should just be able to edit `Makefile.am` and run `make`.

> Is it really less effort to write Makefile.am than to write a straight Makefile? Wouldn't that be simpler?

I don't think so. Not if you want to get DESTDIR support right, and proper dependency tracking (dep info as a side-effect of compilation, so you get all the headers and don't slow down one-off builds), and support for all the standard Makefile targets, etc.

As for your "which build tool to actually use?" question, I wrote a longer post about that in the context of bootstrappable software: http://jackkelly.name/blog/archives/2024/04/01/which_build_t...

It really does seem like autotools remain one of the least bad options, but maybe muon (C99!meson) and samurai (C99!ninja) might be a workable option.

With any make, yes. With modern GNU make you can embed sh code inside a makefile with the define hack, so no, ugly, but no.

* determining availability of different syscalls (yay, printf)

* determining compiler

* verifying dependencies are installed and available (e.g., openssl)

This last one is made much easier with pkg-config and the pkg-config macros in autoconf. There's an argument to be made that pkg-config is bad because it will fail the build if you're just missing the wrong .pc file somewhere, even if your system satisfies the needs of the program - true enough. But pkg-config _does_ make checking for dependencies in autoconf much easier, and it does something essential for correctness, which is gathering up the necessary compilation and linker flags of your dependencies to pass along to the compiler so that you can use them correctly and match ABIs.

A good configure script will guide you through getting dependencies installed, which is much nicer for users than confronting them with compilation/linker errors.

Just write a little program that tries calling the API with two arguments, and see if it compiles. If it does, you could automatically define THAT_API_HAS_TWO_ARGS and your code will do the right thing. If it fails to compile, then define something else.

So you start writing a shell script that will run through all of these tests and dump out a header containing all of these configuration choices. Call the script `configure` and the header `config.h` and you are good.

Except that actually writing a shell script that is really actually POSIX correct is pretty hard. And the code is pretty repetitive. Mistakes start to creep in, bashism break things when it’s actually run under `sh` and so on. And it turns out that all the little tools we rely on, like find and grep and sed and awk, are all different on different platforms too! Turns out that nice option you were passing to grep is just an extension, only present on your specific flavor of unix.

So the next clever idea was to write a program that generates the configure script. The repetitive bits can mostly be solved by text substitution, so someone had the bright idea to use M4. Thus autoconf was born.

Autoconf solved a lot of problems. It ensured that the configure script was actually POSIX correct. It did this even though the generated script was even more complex than ever, with extra logging and tracing and debug output that most people never look at, but which is actually super useful for debugging. The configure script you had written by hand didn’t have any of that!

And autoconf came with baked–in knowledge about all the things you might need to test for. Any time someone discovered some new difference between Unices, new tests were added to autoconf for it. Autoconf users hardly had to do any more work than to look up the name of the macro that it would define for you.

Of course, M4 causes actual brain damage in regular practitioners, and nobody cares to write software for any Unix except Linux these days. Even the BSDs are an afterthought. Technically OpenSolaris is still out there, with superior features like ZFS and Zones and whatnot, but that’s a lot of extra work. So autoconf is a dinosaur, solving problems that nobody really has any more.

Rachel is not wrong that it’s crazy to rerun the same configure script over and over, but really only developers have to do that. Most people just run the thing once when they install your software, and never see it again. And most people don't even do that any more, because they download binaries from their package manager most of the time. It’s just the distro packagers that actually run your configure script.

And she’s right that the answers configure gets should be cached system–wide. Autoconf does support that, but it’s harder to do correctly than she remembers. And by the time it was possible usage was already waning in practice.

Despite the description above of the genesis of autoconf, at the time when there were dozens of nearly compatible unices, it was a key enabling technology for free software to be distributed and shouldn’t be discounted just because it’s a little hard to understand.

Gnulib is effectively the GNU Portability Library. Your fancy, new, auto generated configure script can find all the differences, but someone still needs to account for them and write alternative code to support the various platforms. This is where gnulib comes into play. It reads all the configure checks and plugs in replacements / stubs for whatever is different. This allows you, the developer to simply target GNU/Linux in your code while gnulib handles everything else (and pretty automatically) making is portable across all the unices and even non UNIX systems.

Out of curiosity, what does your C++ development workflow look like? Do you stick to using a text editor rather than an IDE?

That said, as a user I prefer autoconf. The semantics of --with-foo or --enable-bar are simple and well documented. And ./configure --help shows all the possibilities. Faffing around with -DCMAKE_STUFF is horrible. Ccmake should be nice but in practice never works properly.

As a developer, both autoconf and cmake are horrifying and a huge time sink to get them to behave.

The closest I have come to ideal is a custom ninja build file generator, which is pretty simple. It's been easier for other people to maintain than make or even rake.

The problem with Make, is that it works just well-enough to allow people to sorta get it to work without requiring them to learn how to actually use the thing. Then they don't read the manual, and complain that Make doesn't, in fact, work.

Make is full of this stuff and as a distro packager having to deal with each's individual quirks is prone to drive one insane. Please consider using meson, at least.

I prefer to just run "make clean" after I delete, rename or move source files.

With a properly written "Makefile" nothing else needs to be done and there will never be a broken build. With a properly written "Makefile" nothing has to be done when adding source files.

And as much as autotools blows, every time I try to build some project that uses cmake I inevitably go through some rigmarole making a build directory (not that out-of-source tree builds aren't the right thing, they are, but why make me create one?) and finding some obscure set of -D options to get the sucker to build on my system. configure && make still has cmake beat from a user's perspective.

So you end up running 'make install' when you didn’t actually want to install or trying to remember how to find the actual ELF files. And you then wonder why 'make install' appears to be building things instead of just installing them, and you curse at libtool.

In its slight defense, the magic incantation to convince gcc (or, yikes, pre-gcc cc or ld or…) to competently produce a DSO is both nontrivial and platform dependent. Last time I tried to get all the options right off the top of my head, sensible defaults were not a thing.

Contra your slight defense, is it really so hard to produce a good DSO these days? -fPIC is half the battle and a couple other weird flags for macOS/darwin will do you. Cross-compiling for Windows, sure, a few more flags — but you look them up once on stackoverflow, you have three options, and you're done.

In the early 2000s, I ripped libtool out of a product that built on Linux, Solaris, FreeBSD, Aix, Tru64, Mac OSX and Windows.

I did say “slight” for a reason :)

My team abandoned Bazel for Gradle, and it has been a tremendous quality-of-life improvement.

It sounds like you've mainly dealt with Java because it's not a pain in C/C++ - if you wish you can link dynamically with system libs or vendor .so binaries. And at least for Maven I used rules_jvm_external which worked fine.

I’ll grant that container rules were a mess for a long time but new oci rules are much improved.

> My team abandoned Bazel for Gradle, and it has been a tremendous quality-of-life improvement.

I hear as much if not more gripes about gradle of the same variety - "hurr durr it sucks bc i don't like it". Personally I'm thinking this is more appropriate critique for an art piece not a build system.

The reason Gradle is a quality-of-life improvement is that we had boxed ourselves into a corner. rules_jvm_external, rules_docker, rules_proto, and a set of others all relied on internal details of other rules (accidentally, because pre-MODULE.bazel everything gets loaded into one giant global namespace and collisions happen). We basically got ourselves locked in, unable to move off of rules_docker, upgrade our JVM beyond 17, or even upgrade Spring, without breaking and having to upgrade everything else.

Gradle, which has first-party support for dependency management, and which is the primary system supported by large OSS vendors (Spring, etc), doesn't have these pitfalls. Poorly-written Gradle plugins can sink you as surely as Bazel packages can, but there is less opportunity for you to discover them.

I'm well aware that we were Doing Bazel Wrong, and I've certainly had my share of pain with Gradle, but never in my life have I seen so much time wasted on a build system. For plain old JVM stuff, Bazel didn't work well for us.

The author misses that the buildtime magic for the xz exploit is not in the m4 file but in an obfuscated, compressed, encrypted, binary disguised as a test file that alters the build process at multiple stages (configure and build)

A better argument can be made that the act of compiling a binary / obfuscating / minifying code instead of interpreting code directly is the fault.

That means it's totally normal to ship all kinds of really crazy looking stuff, and so when someone noticed that and decided to use that as their mechanism for extracting some badness from a so-called "test file" that was actually laden with their binary code, is it so surprising that it happened? To me, it seems inevitable.

Yeah, no. The author is well aware of how and why autotools are not awesome but also with the background of why they exist.

A better argument can be made that the act of compiling a binary / obfuscating / minifying code instead of interpreting code directly is the fault.

I can’t decide if you’ve never worked in systems software or trying to be hyperbolic. Given that it’s HN I’ll assume the best. But who do you think would do the interpreting? The priests at Delphi?

https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2...

[1] And even then it's more hatred of M4 than it is anything in autoconf per se.

To be fair, M4 is rather…special. And when you add that in a Makefile template with shell thrown into the mix it gets _very_ hairy.

But the boring truth is that configuration management is just a hard problem and realistically you'd hate it anyway.

[1] Not so much automake/libtool, which are configured by straightforward idioms and mostly harmless.

[2] Though if you've got some C preprocessor fu, it doesn't look quite so alien.

[3] Though it also needs to be pointed out that 90% of the problem autoconf was originally intended to solve (gratuitously incompatible Unix variants) is basically gone now. You either write to straight ISO-C/POSIX if you're doing simple stuff, work through well-established abstraction libraries with fixed APIs, or you include well-known platform headers (Linux or otherwise) for the complicated bits. No more nonsense testing like "Hm... what header do I need for strncat()?"

Typically, GNU project programs assume that anyone working on the program (cloning the repo, looking for bugs, making changes, ...) has the Autotools installed. Not just any Autotools but the exact version the program wants.

Only the "release tarball" if a GNU program contains the generated ./configure script that can run on a machine that doesn't have Autotools installed. You have to do some "make boostrap".

So to make a source code release, they can't just tag a code baseline and serve that; they have to generate the release source!

See the stupidity? The source code that is being released has to be compiled from the real source source.

Now this makes life hell if you have to, say, git bisect to find a commit that introduces a problem. To be sure you're building each bisect step correctly, you have to disclean everything and re-run the autoconf bootstrapping, then ./configure and so on. The old code you're checking out may call for different Autotools versions from the trunk head.

Working with some GNU programs is basically hell.

If you're actually running the steps to regenerate the Makefile.in from Makefile.am, and configure from configure.ac, and all that, then you run into the version or version range checks.

For instance, I have working repo of GNU Make. In the generate Makefile there are lines like:

  AUTOMAKE = ${SHELL} /home/kaz/make/build-aux/missing automake-1.16

  # Build prerequisites
  buildreq="\
  autoconf   2.69
  automake   1.16.1
  "

  AC_PREREQ([2.69])
  # [ ... ]
  AM_INIT_AUTOMAKE([1.16.1 foreign -Werror -Wall])

Needless to say, these lines change in the git history.

In all likelihood there's no process to do such an audit? Let's assume such a process did exist, forcing attackers to take a different strategy. Could the rain have hidden in plain sight in the source repo anyways?

Die GNU Autotools https://a.co/d/bQyvEYm

And classic tweet: https://x.com/timmartin2/status/23365017839599616?s=46 ... "I saw a book entitled "Die GNU Autotools" and I thought "My feelings exactly". Turns out the book was in German."

Getting this to work is a major pain, we cannot expect the users to figure it out themselves, and we cannot list all the possible combinations in a Makefile.

We tried several fancy build systems, and every single one breaks in more or less subtle ways.

That stopped making sense several decades ago, when FOSS began to be heavily used for embedded systems, where you are crosscompiling. Autoconf supports cross-compiling, but very badly. Programs not prepared with crosscompiling in mind will do stupid things at configure time that require a compiled program to be executed. For that reason, distros ended up using Qemu for cross builds.

> What I don't understand is why we kept running those stupid little shell snippets and little bits of C code over and over. It's like, okay, we established that this particular system does <library function foobar> with two args, not three. So why the hell are we constantly testing for it over and over?

Autoconf does have a cache system for this. I think it's only used within one package, not globally. There may be some knobs and levers to share autom4te.cache stuff between projects.

Even when the configuration is not cached for the purposes of the test itself, obviously the information is cached in the generated config.h and config.make type files; once the program is configured, each time you recompile it, it's not testing for whether foobar has two args, not three; it's just referring to HAVE_TWO_ARG_FOO or whatever.

There could be a global cache for autoconf tests, or at least the stanadard ones built into autoconf. Each test would have to have a unique ID, which changes whenever it changes between autoconf versions.

The cache would have to distinguish targets. Packages building for the build machine itself must not share configuration results with packages built for the target.

Even if it were done right in these regards, there is a risk of introducing bugs due to package build interaction through such a shared cache. A clean build of any package is not really clean if it refers to material from a previous build, possibly of another package. I can see distro builds opting out of something like this, and they would be the primary audience, since it is distro builds that are hurt by hundreds of packages executing essentially the same tests over again.

But it's 2024, people. If you port to Linux, macOS, and Windows, that's cross-platform enough to capture 99% of your userbase, and everything after that is diminishing returns. Still running DEC Ultrix on Alpha, a CPU architecture that's been out of manufacture for literal decades? Maintain the support yourself if it's so important.

In conclusion, use cmake or Meson. Let autotools die. Kill them if you have to.

Building a project shouldn’t be complicated. There should be extremely minimal branching and if checks.

My blog post where I argue this in more detail: https://www.forrestthewoods.com/blog/dependencies-belong-in-...

"I need common.lib, it gets linked into my application at compile-time"

"why does every binary include this common.lib code? Let's pull that out to a shared module so we only have one version of it! See? that saved 35739475749Kb of storage! Everything is smaller and sleeker now!"

"which version of common.lib does this application need? Hmm, but that other application needs an older version. I'll have to hack some way of having multiple versions of the common.lib available"

"ahh, this application needs version 5.2.1 but that dependency needs version 4.11.6! Two incompatible versions in the same application! Who designed this thing?"

"right, let's compile whatever versions of common.lib we need into our binary. It'll make life easier and to be honest there are so many versions of common.lib around that we might as well have a separate version for every application"

"Ah, small problem, I have to compile a binary for my application for each variant of each platform because common.lib actually varies per platform... hmm, wonder if I can link to the system version of that lib at run-time..."

<repeat until heat death of the universe>

Somewhat. Although it's agnostic to static vs dynamic linking. The philosophy goes a bit deeper than that.

> See? that saved 35739475749Kb of storage!

Storage capacity hasn't been relevant for over a decade.

Besides, we foolishly replaced "use shared modules to save storage space" with "build multi-gigabyte docker images because it's the only way to reliably launch a simple program without crashing on startup".

The Linux installs that run your Pi, router, toaster, security camera and toothbrush begs to differ.

> Besides, we foolishly replaced "use shared modules to save storage space" with "build multi-gigabyte docker images because it's the only way to reliably launch a simple program without crashing on startup".

But... you just said it still is :) By two orders of magnitude now.

Bullshit. It's especially visible with games. I cannot install more than 3-4 big games on my 512GB drive because apparently all game developers believe that "storage capacity isn't relevant".

AAA video games have massive storage requirements primarily due to textures, and sometimes audio. Which is a totally different conversation. It's not a bunch of redundant static libraries causing your pain.

$100 will get you an 8Tb spinning drive or a 1Tb SSD or possibly 2TB. You can also get an external SSD for the same price. Large game sizes is a real and annoying issue. But even 4 year old consoles launched with a 1Tb SSD.

Insane? It’s what all of BigTech and every game dev does. It’s tried and true.

> vcpkg, conan, and Fetch_Content fix all the issues

Not really, no.

> Have you ever had a build fail because of a network error on some third-party server? Commit your dependencies and that will never happen.

author's "ideal" solution:

> First, the user clones a random GenAI repo. This is near instantaneous as files are not prefetched. The user then invokes the build script. As files are accessed they're downloaded. The very first build may download a few hundred megabytes of data.

Seems like contradiction to me? Do you want your builds to be offline or online?

I have a feeling that the real problem is that (1) author never worked with build systems which supply their own compiler, like buildroot or hermetic bazel and (2) author only downloaded dependencies from third-party servers.

You can solve all the problems in the blog post by (1) changing build system to download compilers and libraries and (2) adopting some sort of "binary storage" (something as simple as private ftp server + small text file checked into repo with path and checksum + few scripts to upload and download). And it would not require any new major code investment, nor new tooling, nor new VCS.

The key idea is that all dependencies should always come from your local binary server, never from the third-party server. This is a bit of PITA as many modern tools default to downloading stuff from third-party servers, and it could be hard to convince them to look at your local server instead. But once you do it, you get reliability + reproducibility + sustainability, without having to reinvent the world.

Admittedly, my feels are that we socially tend to overly emphasize the brutalist shades of Darwinianism, of natural selection as a competitive pressure.

Evolution though is a much broader game of adaption, of finding niches. It's about capturing energy. And having diverse & creative means to adapt & succeed, less one finds oneself trapped only able to survive in a overly narrow environment. Evolutions greater story is more about over specialization being doom than it is about refining the exact perfect match; the perfect matches are amazing & elegant but rarely able to stick around.

I don't really know my stance here. But it does feel like a call for authority. Why are we still diverse? Why still have complexity? Can't someone free us of the diversity of options and crown a champion, decree how it will be? That sort of resembles Darwinianism on the small, but in the big, we get change & evolution from having non-static environments, from having many ways to shape and be shaped, all about.

That shift from dynamic to static systems, when evolution peaks, is usually not far from when that stasis turns to ossification & rot, decay & the fall. We probably should cleanup some & not support a million options, for our own sake, but this notion the past had of supporting a deeply rich & diverse technical ecosystem I find enormously beautiful.

Anyhow, I forget how often autoconf needs to be run. I think often many projects would work fine for aomg time using the same generated conf across many many builds? There's focus here on the silliness of re-running config again and again and again, but I'm not sure when technically it's required; it seems more like habit & ritual than a need?

much like history, you don't actually notice it till after its done :-P

Do QEMU, tcc, and OCaml run on BSD and Mac? Pretty sure they do

https://news.ycombinator.com/item?id=39925033

https://github.com/lucasaiu/ocaml/blob/master/configure

They have hand-written configure scripts, and don't use autoconf.

---

Also, https://www.oilshell.org/ runs on BSD and Mac, and it doesn't use autoconf. The configure script is a few hundred lines of "boring" shell.

At a minimum you need to have /etc, /usr/local/etc and ~/etc databases of "union mounted" layered configuration information, along with the ability to add and subtract arbitrary other locations where software might get installed. And it all needs to get updated with every new change to packages in those locations.

As someone else pointed out in a comment, this is sort of the very unrealized promise of pkg-config.

You can't do with just a static file/directory in /etc that defines the distro, that isn't even half the problem once you start installing stuff in /usr/local that you want to link against.

This is a pretty good example of a Chesterton's Fence. You have to first fully understand why the fence is there before you knock it down.

> They were a reflection of the way the kernel, C libraries, APIs and userspace happened to work. Short of that changing, the results wouldn't change either.

These things do change, especially as toolchain and OS vendors realize their quirks that cause portability friction.

// and of course 'should' is more like "it would be great if someone would rewrite it in Rust"

https://news.ycombinator.com/item?id=37205647

Without that process the options are committing generated files (boo) and burdening those that would build from a git clone instead of a distribution tarball with the demands of details from the development environment. That's intentionally a little vague because of the variety of things that can ultimately affect the autotools output.

All of this is kind of the design of the autotools build process and could very well be a reason to take a harder look at it and the tradeoffs incumbent on its use.

In this particular case xz-utils actually did support CMake for building liblzma, so that's what you use if you don't care about the command line tools.

It's also worth appreciating that in the case of the xz project, the CMake flow is secondary other than for Windows. While I have no evidence one way or another, I wonder if the period in the CMake test that disabled Linux Landlock was actually intentional. Since it disabled it for the CMake build and not the autotools build, it seems like it could have been accidental, unless some significant liblzma consumer builds with CMake and would have otherwise had Landlock enabled.

The sendmail configuration file is one of those files that looks like someone beat their head on the keyboard. After working with it... I can see why! -- Harry Skelton

https://github.com/oilshell/oil/blob/master/configure

https://github.com/oilshell/oil/blob/master/build/detect-pwe...

Not an unholy mix of m4, shell, and C, all in the same file.

---

These are the same style as a the configure scripts that Fabrice Bellard wrote for tcc and QEMU.

They are plain ass shell scripts, because he actually understands the code he writes.

https://github.com/qemu/qemu/blob/master/configure

https://github.com/TinyCC/tinycc/blob/mob/configure

OCaml’s configure script is also “normal”.

You don’t have to copy and paste thousands of lines of GNU stuff that you don’t understand.

(copy of lobste.rs comment)

Then standardize those scripts and share them so others can benefit from code reuse.

Then build a configuration tool to make it easy for everyone to define collections of tests applicable to their software.

Then create a meta-level scripting system so all of the above can be integrated into the software build process.

Then complain about how confusing and complicated the whole thing has become and start over from the beginning again.

I believe m4 was used because some Unix shells at the time did not have functions. All shells have functions now -- it's in POSIX.

If you have shell functions, then you can have modularity without concatenating blocks of code. You can make a library of functions, and then allow people to reuse them.

It could just be

    source lib_configure.sh
    
    detect_foo
    detect_bar

POSIX shell can do everything that m4 can, and it's MORE deployed, not less. (Related: some people missed that m4 never runs on the end user machine which compiles the source code -- only on the machine that makes the tarball from the repo.)

They don't copy and paste stuff they don't understand

(The DJB scripts were almost certainly plain sh and not bash, since I think this machines were BSD. At that time bash wasn't so ubiquitous -- there was more diversity in shells. I did read a bunch of his scripts many years ago.)

If that’s this OCaml, it has a configure.ac file in the root directory, which looks suspicious for an Autotools-free package: https://github.com/ocaml/ocaml

glibc is a special evil. zig, which can compile C/C++, solves that conundrum. Hopefully someday the core glibc project fixes itself.

Food for thought: cross-compile should be a first class feature of any build system. If you’re depending on a pile of arbitrary system state you’re doing it wrong.

zig can compile for any target from any target. It’s beautiful. And not nearly as difficult as people seem to think.

C and C++ aren't the only languages for which this is true, by the way. It is de rigueur these days to have versioned releases for the purposes of feature gating, but there's half a century of code out there written in languages without that, and explicit feature testing will be with us as long as that code is.

Anyway, native cross-compilation is also not a language feature. Plan 9's C compilers can target any supported architecture from any other supported architecture with no special configuration.

Autotools, CMake, and "compile and see" are all artifacts of a highly diverse unix ecosystem. Tools like it are the price we pay for heterogeneity. Flexing to accomodate so many subtly different platforms is a big part of what brought things like GNU and llvm to promenence, so it's not a good idea to dismiss it as a bad thing.

Yes, it is the norm (in certain circles, but not others). No, it doesn't have to be that way. CMake and Autotools are both genuinely awful. The world would be a better place if Bazel/Buck2/similar were the norm.

I find it extraordinarily bizarre how defensive Linux people get when I say "the status quo is actually bad, but it doesn't have to be!".

> there's half a century of code out there written in languages without that, and explicit feature testing will be with us as long as that code is.

I think you radically overestimate how difficult of a problem this is. It doesn't have to be this way. I promise you!

> native cross-compilation is also not a language feature.

Correct. That's why I called it a feature of a build system. C/C++ standards famously do not even attempt to describe a build system. This has resulted in many elements that really ought to be language features to actually be implementation defined.

> so it's not a good idea to dismiss it as a bad thing.

"because of" vs "despite of"

The thing you're missing is that Bazel, Buck2, Zig, and all these things that do not require the compile-testing approach is they are not portable. They support Windows, MacOS, and Linux, by special-casing each platform, and even the Linux support is restricted to a few mainstream distros. This is a perfectly valid technical decision, and one I support, but it was not possible for decades.

Now people take Linux for granted, but until relatively recently you had dozens of almost-compatible Unix clones from dozens of vendors, of which Linux was but one. All of them had to be, essentially, written from scratch or based on a BSD release, with the differentiating features implemented on top by a given vendor. Since different developers had different interpretations and priorities for "Unix compatibility," this led to a combinatorial explosion of possible software configurations even just at the libc level, much less kernel API. The Single Unix Specification and POSIX standards were assembled to try to correct this, but there was no way to go back and un-differentiate all these Unix clones. So developers targeted SUS or POSIX, and mostly got what they needed, but something had to account for all the little differences that remained. This was the purpose of Autotools, and Cmake was written because Autotools sucked.

So, these things you're praising which avoid that whole scene, are able to do so because, for better or worse, Linux won. The other Unix clones have either completely died out (SCO Unix etc), are dying out quickly (Solaris etc), or have stabilized into their own distinct platforms (FreeBSD, MacOS, etc).

It has nothing to do with the build system. It has everything to do with platform consistency. That's why Plan 9 can do it without any build system beyond mk (a weird clone of Make), and why the FAANG companies can crank out build systems as get-me-promoted projects -- they're targeting three stable operating systems, instead of an uncountable number of weird almost-compatible monsters.

But again, from this terrible Unix scenery emerged a remarkably robust and portable set of tools (gcc, the gnu coreutils, etc) which became the de facto standard across all the Unix clones. People liked them because they worked the same on every Unix, and because (thanks to Autotools) they could build on every Unix. Because they had this versatile build system, when Linux came around the GNU tools were the easiest to port. Because they were the main tools running on Linux, and people had experience with them from other Unix clones, Linux adoption was quick and easy. And so it grew in popularity...

So yes, I mean "because of." It's easy to hate on complicated old rickety shit like Autotools, and I will be happy when I never have to deal with it again, but pretending it has no value and typing in "ewww" is needlessly dismissive and denies the actual importance this code had in getting us to a place where we can live without it.

> they are not portable. it was not possible for decades.

I could be missing something, but I don't think this is true at all. The definition of "portable" can be quite fuzzy.

Supporting many different platforms is easy. Coming from gamedev the target platform list looks something like: Windows, macOS, SteamOS, Linux, Android, iOS, PS4, PS5, XSX, Switch. A given codebase may have supported another 10 to 20 consoles depending on how old it. Supporting all the different consoles is quite a bit more work than supporting all the niche nix flavors.

Building anything "new" probably requires building it at least three times. You make mistakes, hit edge causes, learn what's important, and eventually build something that's pretty good. It's a process. One perspective is "the approach exists for a reason". Another perspective is "damn we made a ton of mistakes, I really wish we knew then what we know now!".

> I said that approach exists for a reason

My core thesis here is that the reason is a bad one. It was a mistake. Folks didn't know better back then. That's ok. No shade on the individuals. But it doesn't mean it's a good design. In a parallel universe maybe the better design would have been made before a bad one. Such is life.

> They support Windows, MacOS, and Linux

Publicly. Internally Buck2 supports dozens of weird hardware platforms and OS variants. Adding support for a new hardware or OS platform doesn't have to be hard!

I swear compiling computer programs doesn't have to be hard! Build systems don't have to try to compile and see if it fails or not!

As one example, Windows handles linking a dynamic library much, much better than GCC/Clang on Linux. All you need is a header and a thin import static lib that only contains function stubs. GCC made a mistake by expecting a fully compiled .so to be available. The libc headers across all the different weird Unix variants are 99% identical. Supporting a new variant or version should be as simple as grabbing headers and import lib. Zig jumps through hoops to pre-produce exactly that. It'd be even better if libc headers were amalgamated into a single header. Two files per variant. Easy peasy.

"The libc headers across all the different weird Unix variants are 99% identical."

This is both true and not applicable. The headers are nearly identical. The behaviors of the systems are not. Functions with the same name do different things, functions have different names but have slightly-incompatible shims with the name you expect, the headers have the function primitives but those functions are stubbed to NOPs in the actual library, and a million other things. These differences, furthermore, are not documented. All of the platforms you listed are meticulously documented and many of them have SDKs available. This, again, is not something that existed for many decades.

If libc headers being close were so meaningful, widevine binaries would work on musl libc. They do not. Modern build systems work around these incompatibilities by declaring incompatible platforms unsupported and ignoring them. That's a perfectly valid business decision that makes everyone's lives easier! But it's not the only approach, and other approachs are valid too.

Meanwhile, if you want complex software to accurately and correctly support a broad array of platforms, you use Autotools to compile-and-see what behavior you're dealing with, and then something like libtool to polyfill the differences. Doing this is not the result of ignorance, but the result -- borne of hard experience -- of trying to get code working on divergent undocumented platforms.

I don't really understand your point in the last paragraph. Neither GCC nor Clang handle linking. GNU ld or gold or some linker is invoked to do that.

> if you want complex software to accurately and correctly support a broad array of platforms, you use Autotools to compile-and-see what behavior you're dealing with, and then something like libtool to polyfill the differences. Doing this is not the result of ignorance, but the result -- borne of hard experience -- of trying to get code working on divergent undocumented platforms.

Yes, when you have differences in behavior you need to create an abstraction layer that behaves in a single, unified way.

"The Right Thing" is to do that once ever for a given target. Write a bootstrap tool if you want. Or let one person spend one week testing and implementing shims for a brand new platform. Which is what you need to do anyways for embedded platforms that can't host the compiler. In any case don't force every user to run overly complex and brittle scripts when the result never changes for a given target.

Oh hey, this is exactly what the source article argues! At least I'm not alone.

> Neither GCC nor Clang handle linking. GNU ld or gold or some linker is invoked to do that.

Bleh. ld is part of the GNU toolset. Pretend I said "standard Linux linking behavior". As you said, Linux's popularity stems from a collection of tools that work together nicely. The point I tried and failed to make is "compiling and linking doesn't have to be hard". Implementing an abstraction layer for a new platform isn't particularly hard either. Probably best to forget I said it.

> this will be my last try:

Thanks for the chat!

How else do you determine which one to use other than by compiling against the chosen library and seeing what fails?

[1] just an example, musl mighy actually have same amount of parameters as glibc for all I know. But newlib doesn't.

The way C projects from the 90s work is you locally run .configure and it generates a whole bunch of headers, if not .c files, with a whole bunch of crap based on local system state. Then you run a build script. "The Right Thing" is for the repo files to contain a superset and for the build system pass the tiny handful of defines necessary to select the target platform. For example glibc, musl, PlayStation4, NintendoSwitch, etc.

It's very, very easy to do. musl does things right. glibc is the only bad citizen stuck in the 90s. The difference between the libc headers across the kajillion target platforms is exceedingly minimal.

Autoconf is most useful when the build isn't relying on current system state! That's it's most useful characteristic. If the goal was to build against the current system, autoconf probably would never have existed. Building against the current system is a side-effect of doing proper cross-compilation builds.

> "The Right Thing" is for the repo files to contain a superset and for the build system pass the tiny handful of defines necessary to select the target platform. For example glibc, musl, PlayStation4, NintendoSwitch, etc.

For common platforms, sure... that works well and that is sort of how my Makefiles are written.[1]

But to cover the domain that autoconf/configure.sh covers ... well that is not feasible. Glibc might have 3 parameters when compiling against Linux, but 2 when compiling against Solaris. Linux itself might have some `#define`s when compiling on x86_64, and a different set when compiling on RISC-V. The target distribution might have different parameters when compiled for Debian vs when compiled for Alpine.

It's why gcc uses hyphenated targets: `ISA`-`kernel`-`platform`. So you see things like `x86_64-w64-mingw32` and `sparc-unknown-linux-gnu`. The combinatorial explosion of ISA + kernel + ABI + distro + user libs is much much harder to manage for a portable project that has many dependencies than simply putting up with autoconf.

I mean, I don't even like autoconf (which is why I avoid it), but I gotta say, autoconf actually does solve those problems especially for cross-compiling, because then the entire chain of libraries getting linked in will be guaranteed to be the correct ones, for the correct ABI, on the correct target ISA, on the correct target kernel, on the correct target distribution ... and if something doesn't match, you'll know before Make is even run. This is especially important with C++, which needs another value in the tuple for seh/sjlj, and you cannot detect which of the two you need until you run the resulting program and it generates an exception.

It's just possible that, in 2024, we don't actually need the level of portability that autoconf brings to cross-compilation anymore - build it against Windows/Linux/BSD on x86_64/ARM and you cover about 50% of useful scenarios.

Those that need the remaining 50% probably already know what they are doing and can tweak the Makefiles and/or source code to build for their target.

[1] I do minimal detection and expect the user to set certain variables if they don't want the defaults.

No, I’m not switching build systems because everything people use as dependencies for Real Work™ relies on you using the incumbents.

They don’t provide those templates either, anyway. Mind boggling.