improved install-time glibc detection (or how we help you be confident your app works for your users)

cargo-dist wants to make the hard parts of distribution easy, and one of the big things we handle for you is installing your software. Installing software is never easy, and more than that installing software reliably is never easy. The worst thing an installer can do is fail to install software, but the second-worst thing is to successfully install software that doesn't work. cargo-dist has several protections to ensure the software to be installed will actually work for the particular user installing it, and error out if it can't. Starting in cargo-dist 0.21.0, we've taken one of those types of features and made it even better.

Why build on such old Linuces? (in which we find out Compatibility Is Hard)

A question we're often asked is, why do we run your Linux builds on an old distro by default? GitHub provides three different Ubuntu releases today (24.04, 22.04 and 20.04), and cargo-dist uses the very oldest one.

We do this to try to keep your software running on as many systems as we can. Even a simple program that doesn't link against any third party libraries will still usually use the system C library, a standard library shipped with the OS which provides many core OS functions. Most Linux distros use the GNU libc library (glibc). Software built against newer glibc versions usually fails to run on systems with an older glibc, but the reverse isn't true: you can usually run glibc-based software built on older systems on newer ones. (A similar situation is true on macOS: software built for older OSs generally runs on newer OSs, but not vice versa.) For compatibility, the answer is obvious: pick the oldest system you can practically build for and use that to generate your binaries so they run across as many systems as you can¹.

When actually getting ready to install your software, cargo-dist's shell installer checks the actual glibc version on the user's system and compares it to the version it was built against. If the user's system seems too old, we don't try to continue the probably-doomed install; we either fall back to a compatible binary, if one's available, or we error out so that we can provide the user with a better error message at install time. We've provided this feature since 0.4.0, and it's served us well so far.

...until, that is, we introduced custom runners in a later release. Because we were using a fixed set of runners at the time, we were able to hardcode the glibc version we'd check for. While most of our users still use our default runners, it's now possible to configure Linux builds to run on a different OS release than the default, whether that be an older or newer version². Clearly, this wasn't going to be ideal going forward: we never want to install binaries the user can't run, and we don't want to filter out binaries that actually would run because they're built for an older glibc than we thought they were.

runtime conditions (or, how we check what runs on what before the install even starts)

Entirely separate from glibc version checking, cargo-dist has had an idea of "runtime conditions" in our installer generation code. It answers similar kinds of questions: what binaries can we run, where? It also has fallback compatibility checking, letting us know that certain binaries can run even if they're not "native" to a platform — for example, x86_64 macOS binaries can run on Apple silicon Macs under Rosetta, and static-musl Linux binaries (which don't link against any system libaries) can run on glibc-based Linux distributions.

These requirements are calculated in advance, before we generate your installer, so that we can precalculate as much of this logic we can. We can do this because the runtime condition feature exclusively covered static compatibility cases: cases where we can be sure something will or won't run without actually running any logic on the end user's computer at install-time.

Instead of injecting complex fallback logic into the installers, we use this information to calculate the set of installable artifacts for each installation target in much the same way we prepare the list of artifacts that we built in the first place. For example, knowing that x86_64 binaries runs on arm64 lets us "substitute" binaries in the installation list; the list of target triples we inline into your installer can include aarch64-apple-darwin, even if we didn't actually build an aarch64-apple-darwin target, and it simply points to the x86_64 binaries. This keeps the logic inside the installer itself lightweight to reduce the complexity of what we have to do at runtime.

towards better runtime conditions

The initial version of this feature was based on static information about different platforms, however, like the knowledge that x86_64 binaries run under Rosetta; it wasn't able to take into account information specific to your app like, say, what glibc version was used.

The obvious first step was just to start OS-specific tracking information about the build environment on any given system that builds your software. A few things we prepped to start tracking include:

• glibc version (on Linux)
• musl version (on Linux)
• macOS (SDK) version (on Mac)

Tracking this information is fairly straightforward: every time we spin up an instance of cargo-dist to perform a local build, we can capture the build environment before the build itself begins not too differently from how we check which libraries your software linked against.

Actually implementing this was done via a revamped version of the runtime conditions API. We kept all of the information we were tracking before, but with a new structure that lets us track more complex dependency information as well. Our strategy for the existing cases hasn't changed — we still calculate compatibility information statically as often as we can (such as the Apple silicon-to-x86_64 fallback). But this new structure lets us pass more dynamic compatibility information into the installers, allowing us to be smarter about compatibility checks we can only perform at install-time on the user's machine — such as glibc. Starting in cargo-dist 0.21.0, the hardcoded glibc version is now replaced with the real glibc version we determined that your app was linked against, giving you even more confidence that your binary will run on your users' machines.

The current implementation, featured in 0.21.1, is fine-grained and handles glibc versions per platform. This ensures that even if you use different OS versions on your Linux runners, we handle glibc checks per architecture — for example, if your x86_64 Linux runner has an older glibc than your aarch64 Linux runner, we'll allow your x86_64 binaries to be installed on older versions of Linux than the aarch64 binaries. This replaces the initial version of this feature, released in 0.21.0, which featured a slightly simplified implementation: we would record the glibc versions encountered on every one of your builders, but the installer would still only embedd a single version like in the previous hardcoded version of the installer. We specifically chose the newest/least compatible version of glibc to use as the minimum glibc for all of your binaries.

While this kept the structure of the installers a bit closer to the previous revision, it also meant that we were overly cautious — with mismatched Linux runner versions, the installer could end up rejecting older OS versions on the one platform where it's actually safe. (Fun fact, this actually affected cargo-dist's own installer: due to runner availability, we build aarch64 Linux binaries on Ubuntu 22.04 instead of the older 20.04 runner we use for x86_64.)

While the current version of the feature we've shipped only makes use of your builder's glibc version, we're already making plans to use other relevant data that we're now collecting. Keep an eye out for a future feature which will use the minimum macOS version!