axo Releases: IT'S ALL ABOUT THE URLS

We recently launched the closed beta of axo Releases, our alternative to GitHub Releases. Because cargo-dist handles the differences for you, it should largely be a drop-in replacement and you don't need to worry about the differences.

But there sure ARE some differences, and to quote myself derailing approximately every company meeting for the last few months: IT'S ALL ABOUT THE URLS. Specifically, it's about the URLs artifacts should be fetched from. As it turns out, our model comes with more complexity than GitHub's and it's all because of The URLs. Or rather, because of the awesome features we can unlock by really caring about the URLs.

To understand how we got here, let's step back and design our dream release process, and then look at how GitHub Releases breaks this BECAUSE URLs.

My Dreamy Perfect Release Process

At a baseline, every release process needs to do these things:

• plan the release
• build and package our project
• publish our project to relevant package managers
• announce the release

As a project gets bigger and tries to reach more users, all of these steps get more complicated. Sure cargo publish and cargo install works great for a Rust tool made for Rust programmers, but what if it's a Rust tool made for everyone? Well pretty soon you're dealing with prebuilt binaries, installers, platform support, other package managers, changelogs, docs...

Your chill 30 second release process quickly becomes a stressful full day gauntlet! So with all our experience in this area, our dream release process looks like this:

me: "Hello! I want to publish v1.0.0!"

computer: builds absolutely everything it can with no side-effects

computer: "ok here it is for you to inspect, and here's the tentative changelog for you to edit"

me: installs the release preview to mess around with it, or shares it with a tester/user

me: does some last minute edits to the changelog

me: "Ok! looks good to me, ship it!"

computer: publishes to package managers and announces on social media

And on top of all that, I want this entire process to be reversible up until the last possible moment. We can't have perfect transactions because publishing to package managers tends to be irreversible and immediate, but, all of that should be done as late as possible with as many dry-run checks as possible before-hand, so that once we start doing irreversible things we're as confident as possible it will all work and we won't have the worst evening ever unbreaking our release.

"Sounds great, let's do it!"

Woah woah woah, how did you read that and not immediately enter a panic about THE URLS? What do you mean that has nothing to do with URLs? Oh you haven't been working on this exact problem all day every day? Wow, that must be weird. Ok let's walk through why THE URLS are the crux of this issue.

THE URLS

Ok so you're doing All The Release Processes so you've got:

• prebuilt binaries that you shove in some tarballs
• various installers and packages that want to fetch those binaries
• webpages that link those (direct downloads, curl | sh expressions, ...)
• codesigning / tagging / immutability-of-releases

All of the things we've said so far put together lead to a few interesting conclusions!

First off, wherever you're shoving all the prebuilt binaries and installers is going to have to be self-referential: when you download and run installer.sh, it needs to know how to download and unpack tarballs from that very same location. Which is to say, installer.sh needs to contain its own URL!

Between codesigning, hashing, and just "I want to test the actual thing that will actually ship", we really don't want to edit our self-referential artifacts once we've made them, so we need to know that URL before building anything.

If we want any kind of "release preview" functionality where we can stop and share the installers, we need those exact URLs to work before and after the release. They can't just be blah/v1.0.0/installer.sh, because we don't want that URL to be live until we agree this is the actual release.

This suggests some kind of random UUID-ish URL, but, we really don't want to tell end-users to curl https://myhost.com/my-app/a8fdse28dsaserfg8/installer.sh | sh. We'd really like to say curl https://myhost.com/my-app/1.0.0/installer.sh | sh!

Also, we don't want the user to have to update their install docs every time they cut a release, so it'd be great if they could have a stable "latest" URL so they could just slap in their docs and forget forever.

Release artifact URLs are a whole thing.

GitHub Releases: Too Simple To Work

Let's see how GitHub Releases fares here:

• ✅ pretty release URLs (.../releases/download/:tag/...)
• ✅ stable latest URLs (.../releases/latest/download/...)
• ✅ can predict both of these URLs ahead of time
• ✅ can "draft" a release with a UUID-ish URL
• ❌ can use the same URL before and after releasing

nice, nice... OH NO. Yes indeed, GitHub Releases as an artifact host does not allow you upload your artifacts to a staging/draft location without changing the URL after release. The UUID-ish URL that drafts get is disabled as soon as the draft is turned into a Real Release!

This scuppers basically the whole premise of working release previews. Sure I can hand you a draft GitHub Release with all your artifacts and release notes, but if you try to use an installer hanging off of it, the installer will just error out because it's trying to fetch from the pretty release URL that doesn't exist yet! 😭

This also muddies the reversibility of our entire release process: if we want to publish npm packages that fetch from the GitHub Release, we basically need to do a race: either we announce the release on GitHub before we're sure things like npm publish worked, or we npm publish before the hosting for the files it fetches is actually live! This is sadness.

axo Releases: Complex Enough To Work

So this is why axo Releases has a more complex model than GitHub (that again, cargo-dist deals with for you): we want the dang URLs to work right and we don't want to have any random races.

To that end, the process of creating an axo Release is broken up into stages:

1. Plan your release, creating an Artifact Set (draft) with a random Set URL
2. Build your artifacts (these bake in the Set URL)
3. Host your builds on the Artifact Set
4. Preview the Artifact Set, creating a Release Preview (optional, not yet impl'd)
5. Release the Artifact Set, making pretty Release URLs live
6. Publish packages that might want to use either the Set URL or Release URL
7. Announce the Release, making pretty Latest URLs live

The fact that we split out "Release" and "Announcement" is another interesting complexity. This exists to prevent anything asking for "the list of releases" or "the latest release" from actually seeing your release until ABSOLUTELY EVERYTHING has succeeded.

That way no one will ever fetch a messed up release if they ask for "the latest release" — even if it turns out there was something subtly wrong with the release that caused your npm publish to fail after you made the axo Release (which is absolutely going to be miserable and take a while to deal with no matter what we do).

Staging the release process like this also lets us incrementally increase the immutability of a release: the Artifact Set is initially freely mutable, the Release is immutable but can be deleted, and the Announcement is immutable and permanent (with the usual caveats of "laws and special cases exist", but these are the defaults).

Also the entire concept of a "preview" need not be restricted to something you intend to release. We could potentially allow you to enable previews for every pull request, so everyone can easily try out the changes a PR makes to an application!

Really there's a lot of awesome stuff that we're unlocking by doing our own design with its own opinionated APIs and semantics. It's going to be a very exciting year for us now that we've got things working!

In the next post we'll dig into the more technical details of how this design works!