You have a cool idea for a Rust application, quickly hack up a Cargo project for it, and hey, it works! You pull out that old laptop from university -- the one that barely works because you convinced it to dual boot some version of Gentoo and Windows 7 -- and hey, it works on there too! Dang, isn't it nice for Rust projects to be so reliably portable between systems?

Happily convinced of your coding prowess, you mash cargo publish into your terminal and tell your friends to cargo install axolotlsay. It's glorious:

axolotlsay "blub blub blub"         +----------------+         | blub blub blub |         +----------------+        /≽(◕ ᴗ ◕)≼

Then your friend who uses Ubuntu says

@tux4life: cargo install fails because you're using unstable features?

Huh??? You definitely aren't using unstable features!

Oh. Oh no. They did apt install rustc. Their compiler is SO OLD and you just HAD to use new features from the latest release! Clippy complained that you weren't!!

Then your other friend who uses Windows chimes in

@dosboxer: what's cargo?

Oh right, you haven't infected them with your obsession with Rust yet.

Maybe cargo install isn't that great? Maybe you should make prebuilt binaries! Yeah! You've definitely seen some of the bigger Rust projects do that. Like uh, ripgrep! They've got Github Releases with prebuilt binaries! Looks like they have some Github CI that generates it!

Hmm, that looks kinda reasonable, although they do a bunch of stuff that doesn't make sense for your project, so I guess you've gotta comment out a bunch of these shell scripts and, oh damn you made a typo and the CI job failed 20 minutes in. And again. And again...

Congrats kid you're A Release Engineer now and your life is hell. Enjoy debugging basic typos on a remote machine with 20 minute latency because you sure can't run those Github CI bash-scripts-in-yaml files locally! Couldn't be me! I definitely don't have several repos with dozens of prerelease git tags from me trying to debug the dang release process!

But anyway you get it working and link your friends the Github Release and then they just come back with

@tux4life: i'm getting weird glibc errors when I try to run it...

@dosboxer: windows just "protected my PC" and prevented it from running...

@bigmacfan: how do I get this on my PATH?

@nixuserdottxt: it just says "file not found"???

There's Gotta Be A Better Way!™️

What Problems Do We Have?

As you can see, you're going to run into annoying end-user issues regardless of how you choose to publish a Rust application.

This section is maybe just an excuse for me to rant about all these random problems with Release Engineering... you can skip to the next section if you just want to learn more about cargo-dist, I GUESS.

If you're going to put no effort into it, cargo publish and cargo install are... surprisingly effective! Say what you will about Cargo, but it's really dang reliable when it comes to building a random project on crates.io on a random machine! This makes it a pretty solid option for tools-for-other-rust-developers. Just a few keystrokes and a few minutes of waiting and I've reliably got a development machine setup with cargo-release, cargo-binstall, cargo-fuzz, cargo-insta, and so on! Heck it will probably even work on systems the application was never tested on, because Rust builds tend to Just Work.

But there's definitely some tradeoffs:

• You need to have a compatible version of Rust: You need to hold back on using The Absolute Latest Rust Features if you want to support people with older Rust toolchains (something that's more reasonable for libraries, but needlessly restrictive for applications). Also on rare occasion a newer compiler could lose the ability to compile the program, but that's not a huge concern with Rust (Rust devs try hard to avoid this, but nothing's perfect).
• Not everyone has a Rust toolchain: whether you're making a CLI devtool or The Next Big Videogame, you're making an application for someone who probably doesn't care about the language it's written in and just wants to download and run it! Won't anyone think of the Gamers!?
• Some things can't be published to crates.io: Most obviously you might have proprietary source code or secrets-that-are-needed-at-build-time that can't be published. But even if you don't care about that you might just run into size limits on crates.io if your application has too many static assets like images.
• Building rust packages takes a lot of time and resources: Look we're not talking Chromium build times, but if you have a task that takes 5 minutes and 4 of those are building the Rust application that does the work... that adds up. Heck, if the target hardware is too weak or obscure it might be literally impossible to run rustc on it!

If any of those are too big of a problem for you, then you probably want to build official binaries/installers for your applications and have users download and run them directly. Unfortunately, when you try to do this you quickly learn that cargo install was solving a lot of annoying problems for you:

• System Dependencies: Rust applications don't have many system requirements (equivalent to C), but they're unavoidable. Any time your application needs a system API you're saying "I won't run on machines that don't have this API". When you build your Rust applications, do you take the time to say what Linux glibc versions you support? What macOS SDKs you support? If not, the compiler has no idea what to choose other than "whatever's supported by the current machine". That can very easily result in a binary which runs great on the machine it was built on but can't run at all on other systems. This is one of many reasons folks justifiably love musl for prebuilt Linux binaries, but now you need to get musl-tools and learn about cross-compiling and... 😭
• Building For Everything: Speaking of cross-compilation... how many computers do you have? Windows? macOS? Linux? Oh great there's ARM64 macbooks now? Oh no they've got Linux running on those too now..? Wait, what's RISC-V? Tools like cargo-cross definitely help with reducing the number of machines you need, but also most Rust devs I know run away screaming as soon as you say "Docker", to say nothing of the fact that Apple... Really Does Not Want You Doing That (and weirdly NixOS? (unless you're making a Flatpak?? (but those don't work for CLIs??? (aAaAAAaaAAAA????))))
• Building Locally: Of course the Easy Solution for most of your cross-compilation woes is that you use something like Github CI to spin up a bunch of native machines for the targets you care about (and for the platforms you do need to cross-compile for you're hopefully doing a smaller leap, like x64 macOS -> ARM64 macOS). taiki-e/crate-gh-release-action is absolutely wonderful for streamlining a lot of this! But now your build system is... checks notes... shell scripts embedded in an executable yaml file that has its execution distributed across 7 different machines over the network? I really hope you don't need to debug that locally (sobs in Windows user).
• Hosting The Artifacts: Nice binaries you've got there, where are you putting them? How are people going to find them? How are they going to know which one to use? Ah just uploaded some .tar.gz bundles to a Github Release your users need to find? And they need to know they want aarch64-apple-darwin.tar.gz for an arm macbook?
• Downloading, Installing, Updating: It sure was handy that cargo install handled fetching over the network and unpacking the contents of the package! And that it put it on your PATH! And registered the application such that cargo uninstall and updates work! Now your users need to fumble around with unpacking it manually, find the actual application, and put it in a custom bin directory they've added to their PATH!
• Scary Warnings: Another thing cargo install was doing for you was making the binary implicitly trusted by the system. With prebuilt binaries on macOS and Windows, your user is probably going to get some scary warnings about untrusted applications they need to bypass with a secret handshake! Fun! (Did you know the warning you get on Windows isn't about code signing, but is actually just a special flag Windows' builtin unzipping tool sets on all executables it extracts?)

And regardless of which approach you're opting for, you're probably dropping a lot of useful and important things on the ground!

• Debuginfo: By default cargo install and cargo build --release disable most debuginfo, which makes sense, debuginfo is Very Big... but this is a false dichotomy! This is why we have split-debuginfo which lets us have a nice and small optimized binary with the big debuginfo on a symbol server somewhere for when a debugger or crashreporter needs it! Even if you don't have a use for them today, once you've thrown out the build directory those things are gone forever! Are you archiving pdbs, dSYMs, and dwps for all your executables? Why not!?
• Frame Pointers: I AM ASKING YOU AGAIN TO ENABLE FRAME POINTERS IN YOUR BUILDS, YOUR LOCAL PERFORMANCE ENGINEER WILL THANK YOU FOR THE GIFT.
• Link-Time Optimization: Did you know that install and cargo build --release actually leave some performance on the table? For the sake of compile times (and resource usage), they don't use the heaviest form of LTO. This is honestly probably the right call for the way most people use those commands, but if you're building A Published Release there's a decent argument to crank the knob back towards "hellish build times" for those last few percent!

There's so many things to care about!!!

Definitely Literally A Panacea: cargo-dist

Alright so here's my pitch on how to deal with those problems: cargo-dist, cargo build but For Building Final Distributable Artifacts! No more copying around hard-to-run shell scripts, just run cargo dist and it will handle everything for you -- on any machine!

This includes (to various levels of "implemented"):

• Building all your binaries with Production Flags
• Copying the binaries and static assets into zips/tars (or more!)
• Gathering split-debuginfo files for all the binaries
• Generating installers (scope TBD, but at least curl-sh scripts)

But of course that doesn't really solve all of those problems. I haven't solved the treachery of CI scripting -- OH BUT WAIT I HAVE. Because cargo-dist fundamentally understands what it's supposed to do for any given platform/target, we can have my favourite command: cargo dist generate-ci! This command generates the ci scripts to invoke itself!

For Github this means it will:

• Wait for a git tag to be pushed that looks like a version
• Create a draft Github Release™️ for that version
• Spin up machines to build all the targets
• Download a prebuilt version of cargo-dist
• Invoke cargo-dist as needed
• Upload the resulting artifacts to the Github Release™️
• Upload a machine-readable manifest describing all the artifacts
• On success of all tasks, mark the Github Release™️ as a non-draft

The easiest way to get that started is with another great command that already exists, cargo-release. So a way-too-quick start for this is just:

# install toolscargo install cargo-distcargo install cargo-release# one-time setup of ci scripts and build flagscargo dist init --ci=githubgit add .git commit -am "wow shiny new cargo-dist CI!"# cut a releasecargo release 0.1.0

After that you can just kick back and relax, because in a few minutes and you'll have a Github Release with everything built and uploaded!

At this point it's worth noting that you're not locked into Github's ecosystem, because you haven't invested a drop of effort into it. All you did was tell cargo-dist "hey I'm using github" and it took the wheel. In principle (once implemented), switching to Gitlab or whatever else should be as easy as cargo dist generate-ci gitlab. Or if you've got something more bespoke that we don't support, well, the cargo dist command was already doing most of the heavy lifting, so hand-writing CI orchestration for it should hopefully be a lot easier!

Or heck, go really bespoke and run cargo dist locally, because again, you can! Really savor the experience of artisanally uploading the artifacts to your favourite FTP server.

Oh, and did I mention that this all happens to work well with another community tool, cargo-binstall? It tries to fetch prebuilt binaries for projects published on crates.io, and as it turns out if you make sure your Cargo.toml has the repository field properly set to point at your Github repo then binstall will perfectly understand the Github Releases™️ cargo-dist creates, so it can fetch and install those binaries automagically!

Does It Work?

Alright so I've given you The Pitch of how this should work, but how does it actually work today? Well, yes, I wouldn't announce this if not -- but there's a lot of future work to do! Actually no I take that back, it's definitely perfect. Those familiar with my broader oeuvre know this will absolutely be a perfectly crafted demo that avoids all the sharp corners of cargo-dist!

Let's do some trivial updates to one of my personal projects, minidump-debugger, and try to publish a new version with cargo-dist!

I had already put in some work to setup release CI and it's... yep, some scripts I copied from ripgrep and edited to work for my project!

The commit history for the file is uh, about what you'd expect:

But I'm pretty sure it works? At very least it to produced the v0.3.1 Github Release with prebuilt zips for the 3 major x64 platforms. Anyway time to burn that all to the ground and replace it with cargo-dist!

First let's install cargo-dist's own prebuilt binaries with cargo-binstall. This should:

• look for cargo-dist on crates.io
• see 0.0.2 is the latest release
• set that 0.0.2 specifies repository="https://github.com/axodotdev/cargo-dist/" in its Cargo.toml
• look for a github release for that version
• find the right zip for my current platform
• download it, unpack it, and copy it to my ~/.cargo/bin dir just like cargo install would

PS C:\Users\gankra\dev\minidump-debugger> cargo binstall cargo-dist --no-symlinks INFO resolve: Resolving package: 'cargo-dist' WARN The package cargo-dist v0.0.2 will be downloaded from github.com INFO This will install the following binaries: INFO   - cargo-dist (cargo-dist.exe -> C:\Users\gankra\.cargo\bin\cargo-dist.exe)Do you wish to continue? yes/[no]? yes INFO Installing binaries... INFO Done in 17.9008318sPS C:\Users\gankra\dev\minidump-debugger> cargo dist -Vcargo-dist 0.0.2

Hooray! At least cargo-dist seems to be packaging itself up properly!

Now lets try it out:

PS C:\Users\gankra\dev\minidump-debugger> cargo distbundling minidump-debugger-v0.3.0-x86_64-pc-windows-msvc.pdbbundling minidump-debugger-v0.3.0-x86_64-pc-windows-msvc.zipbuilding cargo target (x86_64-pc-windows-msvc/dist)error: profile `dist` is not definedERROR  × failed to find bin minidump-debugger for minidump-debugger 0.3.0 (path+file:///C:/Users/gankra/dev/minidump-debugger)

...Oops! I really need to produce a better diagnostic when you forget to run cargo dist init.

Let's properly setup cargo dist first (with Github CI enabled):

PS C:\Users\gankra\dev\minidump-debugger> cargo dist init --ci=githubPS C:\Users\gankra\dev\minidump-debugger> git statusOn branch mainYour branch is ahead of 'origin/main' by 1 commit.  (use "git push" to publish your local commits)Changes not staged for commit:  (use "git add <file>..." to update what will be committed)  (use "git restore <file>..." to discard changes in working directory)        modified:   .github/workflows/release.yml        modified:   Cargo.toml

Cool, it did something! Let's see what it did to my Cargo.toml:

diff --git a/Cargo.toml b/Cargo.tomlindex aa8a491..53c2797 100644--- a/Cargo.toml+++ b/Cargo.toml@@ -32,3 +32,10 @@ tracing = { version = "0.1.34", features = ["log"] } tracing-subscriber = "0.3.14" linked-hash-map = "0.5.6" clap = { version = "3.2.15", features = ["derive"] }++# generated by 'cargo dist init'+[profile.dist]+inherits = "release"+debug = true+split-debuginfo = "packed"+

Ok so now my project has a custom "dist" profile that's tuned to produce split debuginfo files like pdbs (in the future this will also probably turn on things like full LTO).

The CI changes are... a lot bigger, so I'm just going to link them here. Because I gave cargo-dist the same naming convention for Github CI files that I normally use, it straight-up blindly overwrote my old release.yml file. This is super convenient for me in this precise moment, but in the future I should probably ask permission before doing that (although you can just not commit the changes if you don't like them).

Anyway the CI scripts are... a bit messy. Although I actually generate comments describing how it works, so you get gems like this:

- name: Run cargo-dist  # This logic is a bit janky because it's trying to be a polyglot between  # powershell and bash since this will run on windows, macos, and linux!  # The two platforms don't agree on how to talk about env vars but they  # do agree on 'cat' and '$()' so we use that to marshal values between commmands.  run: |    cargo dist --target=${{ matrix.target }} --output-format=json > dist-manifest.json

Is good or bad design? All I know for sure is that's Very Funny and I can fix it in one place for all my future projects (by improving cargo-dist's CI generation).

The crux of the scripts is just "run cargo dist, parse the json output, and upload the results to github". That means I should just be able to run cargo-dist locally to see what it produces on windows:

PS C:\Users\gankra\dev\minidump-debugger> cargo distbundling minidump-debugger-v0.3.1-x86_64-pc-windows-msvc.pdbbundling minidump-debugger-v0.3.1-x86_64-pc-windows-msvc.zipbuilding cargo target (x86_64-pc-windows-msvc/dist)   Compiling autocfg v1.1.0   ...   Compiling minidump-debugger v0.3.1 (C:\Users\gankra\dev\minidump-debugger)    Finished dist [optimized + debuginfo] target(s) in 1m 04sbundled C:\Users\gankra\dev\minidump-debugger\target\distrib\minidump-debugger-v0.3.1-x86_64-pc-windows-msvc.pdbbundled C:\Users\gankra\dev\minidump-debugger\target\distrib\minidump-debugger-v0.3.1-x86_64-pc-windows-msvc.zip

Hey, that worked! It found my project, built it with the dist profile, collected up a zip and pdb (Microsoft's debuginfo format), and told me where to find them! No more debugging the build in CI for me!

Alright, I'm happy with this, time to commit it and publish 0.3.2 (using cargo-release of course):

PS C:\Users\gankra\dev\minidump-debugger> git commit -am "replace release ci with cargo-dist"[main 3ad6beb] replace release ci with cargo-dist 2 files changed, 131 insertions(+), 161 deletions(-) rewrite .github/workflows/release.yml (95%)PS C:\Users\gankra\dev\minidump-debugger> cargo release 0.3.2   Upgrading minidump-debugger from 0.3.1 to 0.3.2  Publishing minidump-debugger    Updating crates.io index   Packaging minidump-debugger v0.3.1 (C:\Users\gankra\dev\minidump-debugger)   Verifying minidump-debugger v0.3.1 (C:\Users\gankra\dev\minidump-debugger)   Compiling cfg-if v1.0.0   ...   Compiling minidump-debugger v0.3.1 (C:\Users\gankra\dev\minidump-debugger\target\package\minidump-debugger-0.3.1)    Finished dev [unoptimized + debuginfo] target(s) in 37.50s    Packaged 14 files, 189.7KiB (43.0KiB compressed)   Uploading minidump-debugger v0.3.1 (C:\Users\gankra\dev\minidump-debugger)warning: aborting upload due to dry run     Pushing Pushing main, v0.3.2 to originwarning: aborting release due to dry run; re-run with `--execute`

Oh right, cargo-release rules and requires me to add --execute to do it For Real:

PS C:\Users\gankra\dev\minidump-debugger> cargo release 0.3.2 --executeRelease minidump-debugger 0.3.2? [y/N]y   Upgrading minidump-debugger from 0.3.1 to 0.3.2[main 5a471ae] chore: Release minidump-debugger version 0.3.2 2 files changed, 2 insertions(+), 2 deletions(-)  Publishing minidump-debugger  ...  Compiling minidump-debugger v0.3.2 (C:\Users\gankra\dev\minidump-debugger\target\package\minidump-debugger-0.3.2)    Finished dev [unoptimized + debuginfo] target(s) in 37.07s    Packaged 15 files, 189.8KiB (43.1KiB compressed)   Uploading minidump-debugger v0.3.2 (C:\Users\gankra\dev\minidump-debugger)    Updating crates.io index     Waiting on `minidump-debugger` to propagate to crates.io index (ctrl-c to wait asynchronously)    Updating crates.io index     Pushing Pushing main, v0.3.2 to originEnumerating objects: 30, done.Counting objects: 100% (30/30), done.Delta compression using up to 32 threadsCompressing objects: 100% (18/18), done.Writing objects: 100% (19/19), 10.47 KiB | 714.00 KiB/s, done.Total 19 (delta 9), reused 0 (delta 0), pack-reused 0remote: Resolving deltas: 100% (9/9), completed with 7 local objects.remote: This repository moved. Please use the new location:remote:   git@github.com:rust-minidump/minidump-debugger.gitTo github.com:Gankra/minidump-debugger.git   02cf5ca..5a471ae  main -> main * [new tag]         v0.3.2 -> v0.3.2

Ok published to crates.io and pushed to Github, let's see if the CI is working..!

...FUCK! What the hell broke?

A Brief Intermission: Unfucking The Repository

  `"pkg-config" "--libs" "--cflags" "gdk-3.0" "gdk-3.0 >= 3.18"` did not exit successfully: exit status: 1  error: could not find system library 'gdk-3.0' required by the 'gdk-sys' crate

Ohhhhh riiiight...

So minidump-debugger is based on egui, and for whatever reason egui builds on linux require you to install a ton of dev packages (on Windows and macOS it Just Works so I always forget this). This is why the old release CI I blindly blew away had this part:

- name: Install packages (Linux)  if: runner.os == 'Linux'  run: sudo apt-get update && sudo apt-get install libxcb-render0-dev libxcb-shape0-dev libxcb-xfixes0-dev libspeechd-dev libxkbcommon-dev libssl-dev libgtk-3-dev

Welp, automating and understanding those kinds of build deps are another feature for me to add to cargo-dist one day, but for now I'll rely on the fact that the scripts are checked in so I can make bespoke hand-edits to the scripts if need be (being able to do this is an intentional feature, although obviously suboptimal because now we're getting back some dependency on the CI's precise behaviour).

Time to enter Release Engineer Mode and start doing panicked edits to CI and the Git history. If I wasn't made of hubris I might have run cargo release with --no-publish to save the publish to crates.io until everything else succeeds (release transactionality across multiple disjoint platforms is Hard to say the least. More things for me to work on.).

I'll settle for deleting the tag and the github release and pushing a fresh one manually, because cargo-release is just a convenience, and the only thing that triggers the CI is a git tag (and the commit it points to) being pushed up.

Bask in the wonders of me unfucking my git repo (not pictured: me deleting the Draft Github Release in Github's web UI, so hooray for the Draft machinery working, at least!):

PS C:\Users\gankra\dev\minidump-debugger> git tag -d v0.3.2Deleted tag 'v0.3.2' (was f5e27a7)PS C:\Users\gankra\dev\minidump-debugger> git push --delete origin v0.3.2remote: This repository moved. Please use the new location:remote:   git@github.com:rust-minidump/minidump-debugger.gitTo github.com:Gankra/minidump-debugger.git - [deleted]         v0.3.2PS C:\Users\gankra\dev\minidump-debugger> git diffdiff --git a/.github/workflows/release.yml b/.github/workflows/release.ymlindex 04cd439..e41251a 100644--- a/.github/workflows/release.yml+++ b/.github/workflows/release.yml@@ -75,6 +75,9 @@ jobs:         run: rustup update stable && rustup default stable       - name: Install cargo-dist         run: ${{ matrix.install-dist }}+      - name: Install packages (Linux)+        if: runner.os == 'Linux'+        run: sudo apt-get update && sudo apt-get install libxcb-render0-dev libxcb-shape0-dev libxcb-xfixes0-dev libspeechd-dev libxkbcommon-dev libssl-dev libgtk-3-dev       - name: Run cargo-dist         # This logic is a bit janky because it's trying to be a polyglot between         # powershell and bash since this will run on windows, macos, and linux!PS C:\Users\gankra\dev\minidump-debugger> git commit -am "Add back linux install stuff"[main 24208a6] Add back linux install stuff 1 file changed, 3 insertions(+)PS C:\Users\gankra\dev\minidump-debugger> git tag v0.3.2PS C:\Users\gankra\dev\minidump-debugger> git push origin --tagEnumerating objects: 9, done.Counting objects: 100% (9/9), done.Delta compression using up to 32 threadsCompressing objects: 100% (4/4), done.Writing objects: 100% (5/5), 588 bytes | 294.00 KiB/s, done.Total 5 (delta 2), reused 0 (delta 0), pack-reused 0remote: Resolving deltas: 100% (2/2), completed with 2 local objects.remote: This repository moved. Please use the new location:remote:   git@github.com:rust-minidump/minidump-debugger.gitTo github.com:Gankra/minidump-debugger.git * [new tag]         v0.2.1 -> v0.2.1 * [new tag]         v0.3.2 -> v0.3.2PS C:\Users\gankra\dev\minidump-debugger> git push originTotal 0 (delta 0), reused 0 (delta 0), pack-reused 0remote: This repository moved. Please use the new location:remote:   git@github.com:rust-minidump/minidump-debugger.gitTo github.com:Gankra/minidump-debugger.git   5a471ae..24208a6  main -> main

Back To Our Regularly Scheduled Flawless Demo

please work please work please work please work

Yeah!

YEAAAH!!!!!

PS C:\Users\gankra\dev\minidump-debugger> cargo binstall minidump-debugger --no-symlinks INFO resolve: Resolving package: 'minidump-debugger' WARN The package minidump-debugger v0.3.2 will be downloaded from github.com INFO This will install the following binaries: INFO   - minidump-debugger (minidump-debugger.exe -> C:\Users\gankra\.cargo\bin\minidump-debugger.exe)Do you wish to continue? yes/[no]? yes INFO Installing binaries... INFO Done in 38.2950944s

YEEAAAHHHH!!!!!!!!!!!

PS C:\Users\gankra\dev\minidump-debugger> minidump-debugger

YEEEAAAHHHHHHH!!!!!!!!!!!

Oh how sweet it is for the things you make to Work. Is it perfect? Not in the slightest! But it's a start, a step in the right direction, and it's my dang job to keep on taking those steps until cargo-dist is the slam dunk obvious choice for packaging up your Rust projects!

Post-Credits Foreshadowing

All of that sounds pretty nice, but why stop there? After all, our Github Release™️ has that nice machine-readable manifest, so maybe some other tools can take this even further! Hmm, that sure sounds convenient, I wonder if anyone is working on that kind of thing...