Proposal: Smoother releases: 3 parts to have mostly-automated releases

[huonw]

There's a lot of enthusiasm for making releases simpler. I've been listening to the discussion and it seems like various people (@thejcannon, @benjyw and @stuhood come up a lot) have lots of information and plans in their heads and there's a few issues lying around too, but I'm not sure of an explicit document capturing it all, so I'm not sure we're all driving towards the same vision.

I've tried to pull one together, here.

⚠️ It has the caveat that I've never run a release! ⚠️ I am likely to be making wild assumptions or missing key facts. ⚠️ Huge warning! ⚠️ I cannot understate this limitation of me writing this document! ⚠️ Hopefully it's at least a useful starting point for discussion, let me know of mistakes/improvements.

Summary: it seems like automated releases are well within reach, with 3 pieces of automation, mostly without changing the basic process and output artifacts. I think this means we can get to more automatic releases with:

• less effort (by augmenting existing scripts)
• without needing to be blocked behind agreeing on new behaviour (e.g. changelog generation) (we can still adopt new behaviours after the current process is automated)
• waiting for vast new features (e.g. Sphinx docs)

Where we are now

The release procedure https://github.com/pantsbuild/pants/blob/e0ef5b83820ce5d269516a3ce5c04f5fc354a9ab/docs/markdown/Contributions/releases/release-process.md seems to involve a whole range of tasks, including ones a human is definitely best placed to do, down to ones a robot can easily manage.

There's been a flurry of great automation improvements recently:

• automatic cherrypicking (Add an auto-cherry-pick GitHub Action #19021 and follow-ups, @thejcannon)
• wheel publishing (Do not publish wheels for all SHAs, and publish to PyPI from Github Actions. #19028, @stuhood)
• the automation account @WorkerPants existing to make future automations easier, too
• recently, easier ability to test actions before they run for real (Add act to Pants (all in BUILD) and smoke test our workflows #19278, @thejcannon)

Proposal for getting to smoother releases

The proposal

Based on the step-by-step analysis below, it seems like we could get to smoother releases with 3 (augmented) pieces of automation:

1. Start release:
   • Manually triggered, providing a version number.
   • Updates the changelog, contributors and version as required (replaces parts existing "Step 1") and opens a PR for a human to validate and merge.
   • Replaces most of step 1
2. Tag release:
   • Runs when "start release" PR merges.
   • Cherrypicks and bumps version in milestone branch (if required), adds a git tag, and pushes.
   • Replaces steps 1 substep 5b, step 3
3. Run release:
   • Augmenting existing release workflow.
   • Add updating of docs site, publishing of PEX.
   • Replaces steps 2, 4

Of these, 3 is potentially highest value, then 1, then 2. (Or maybe 1, 3, 2?)

(Each of these have details to work out and fine-tune, of course.)

Discussion

If we do this, I think the releases will be mostly automatic, steps 1 (partially), 2, 3 and 4 become automatic, and humans will only have to:

1. manage deprecations and any non-auto cherry-picks (Step 1)
2. approve and merge the "start release" PR, having made any updates required (Step 1)
3. smoke test/validate the automation ran successfully (Step 5)
4. announce the release (Step 6)
5. Some extra tasks that seem like they're lower value to automate:
   • once per major version: create branches, doc site versions, release note files
   • delete dev versions from PyPI to manage size (Manage pantsbuild.pants release size on PyPI #11614)

A downside to have all this automation is that it can be hard to manage and debug if things go wrong. There's various ways to alleviate this risk:

• log lots
• test
• allow workflows be run manually through GHA, if automatic triggers fail (like how @thejcannon has set up the cherry-picker to support manual runs by specifying the PR number explicitly)
• keep automation idempotent as much as possible, so it can be retried freely, if there's seemingly spurious failures

Next steps, after core proposal

There's some additional improvements that could happen once this is in place:

• automate the once-per-major-version tasks
• tweaking how the changelog is managed (Proposal: Shift changelog content generation away from releaser onto PR author #19247)
• publishing SCIEs (or similar) to Github releases (Pants should be distributed on Github Releases #12397) to sidestep the PyPI release size issues
• switching off Readme.com

Release process, step by step

I've gone through the current (2.18) release process https://github.com/pantsbuild/pants/blob/e0ef5b83820ce5d269516a3ce5c04f5fc354a9ab/docs/markdown/Contributions/releases/release-process.md and looked at each step along four dimensions:

• Human effort now: my estimate of how much effort it takes for the person doing a release (Huge caveat: I've never run release!)
• Human in the loop required: whether the task seems like it fundamentally requires a human to be making decisions
• Short term automation: a proposal for how it might be automated in the short term, to most alleviate load for doing releases
• Long term automation: some further options for automation once the short-term automation is in place

I'll post each step as separate comments to make for easier specific discussion about any details in them.

[huonw]

Existing process step-by-step: Step 1: Prepare the release

Summary:

1. Several steps (0a, 5a) that happen once per major version that are probably lower value to automate
2. Steps 1-4 could potentially be packaged into a script (potentially run in GHA) that takes a version number and does the various mutations etc.
3. Step 5b could similarly be a follow-on script/automation after that
4. Steps 0b, 0c require humans to resolve merge conflicts etc., so don't seem (further) automatable

Sub-step	Human effort now	Human in the loop required	Short term automation	Long term automation
0a dev0 - set up the release series	Low (rare and fast)	Partially: specifying the new version number	Doesn't seem worth it?	Sure
0b dev - check for any deprecations	Depends on specific deprecations	Yes, arbitrary edits may be required	Not possible to usefully automate
0c Release candidates - cherry-pick relevant changes	Depends on specific merge conflicts/failures	Yes, arbitrary edits may be required	Done 🎉	Continue iterating on auto-cherry-picker
1 Prepare the changelog	Moderate?	Yes, for typo review/fix-ups	Have build-support/bin/changelog.py write to the file	Options like #19247
2 Update CONTRIBUTORS.md	Low?	No (a human will refer to this for composing announcements, though)	Have the changelog automation include updating CONTRIBUTORS.md?	?
3 dev and a0 - Bump the VERSION	Low?	Partially: specifying the new version number	Logic to update src/python/pants/VERSION	?
4 Post the prep to GitHub	Low	No	Commit and opening a PR automatically	Merge the PR automatically once reviews are approved?
5a. a0 - create a new Git branch	Low (rare and fast)	Partially: specifying the new version number	Doesn't seem worth it?	Sure
5b. release candidates - cherry-pick and bump the VERSION	Low?	Partially: specifying the new version number	Once the step 4 PR merges, do the cherry picking, version update and push

[huonw]

Existing process step-by-step: Step 2: Update this docs site

Summary:

1. The release GHA could potentially run the sync steps (2, 3)
2. the Readme.com CLI suggest it can generate GHA config: https://www.npmjs.com/package/rdme#github-actions-configuration
3. Several steps (1, 4) that happen once per major version that are probably lower value to automate3.

Sub-step	Human effort now	Human in the loop required	Short term automation	Long term automation
[1] dev0 - set up the new version	Looks low? and rare	No?	Doesn't seem worth it?	Moving off Readme.com to Sphinx or similar?
[2] Sync the docs/ content	Moderate?	No? Except for 2FA?	Run the right command in GHA	as above
[3] Regenerate the references	Moderate?	No? Except for 2FA?	Run the right command in GHA	as above
[4] stable releases - Update the default docsite	Looks low? and rare	No?	Doesn't seem worth it?	as above

[huonw]

Existing process step-by-step: Step 3: Tag the release to build wheels

Summary:

1. recently mostly automated (Do not publish wheels for all SHAs, and publish to PyPI from Github Actions. #19028) 🎉
2. issues with size of wheels on PyPI (Manage pantsbuild.pants release size on PyPI #11614)
3. there's been talk of switching to publishing artefacts to GH releases, that scie-pants consumes (e.g. scies) (Pants should be distributed on Github Releases #12397)

Sub-step	Human effort now	Human in the loop required	Short term automation	Long term automation
only one sub-step	low?	no	If step 1 sub-step 5b is automated, push a tag as part of that process	Publish artefacts (SCIEs?) to GH releases

[thejcannon]

OK long-term automation is done here: #19431

[thejcannon]

Nevermind. The discussion is here with #19431 likely split into several PRs to make that change.

Same outcome.

[thejcannon]

OK tagging kicks off a release build. That release build is fully-automated now with uploads to GH for local PEXs and wheels, and triggers a rebuild of the cheeseshop.

[huonw]

Existing process step-by-step: Step 4: Release a Pants PEX

Summary:

1. Seems like CI could build and push this
2. There's been talk of not publishing a universal PEX at all (https://pantsbuild.slack.com/archives/C0D7TNJHL/p1685645383249239?thread_ts=1685644290.953559&cid=C0D7TNJHL)

Sub-step	Human effort now	Human in the loop required	Short term automation	Long term automation
only one sub-step	moderate?	no	release GHA builds PEX and attaches to GH release	stop publishing PEX? (see step 3)

[thejcannon]

I actually think we might be able to avoid the pex (or bring down some reasons it might exist) by making pants have 0 deps.

It was something I was going to look into, as it's a good idea outside of this issue/workflow.

[huonw]

The goal here is to get to automated releases ASAP, without requiring negotiation about making changes in approach, like getting agreement on "pants should vendor/reimplement any functionality it needs from PyPI". In addition, I am under the impression third party deps are only part of the story with the PEX: it's universal/not-platform-specific by including all the different pants wheels, and so can be copied and run on any supported machine.

That said, there has been discussion of not publishing the PEX at all, right now, based on it apparently not getting much use.

If we're looking to automate releases soon, my assessment suggested to me energy is best spent on the 3 pieces of automation in the original document. I don't think having the release workflow build and attach the PEX will be major effort given the scripts already exist, but I'm not 100% sure.

That said, teasing out and discussing these areas of focus was the point of writing this post, so please continue to discuss, but maybe let's be explicit about focusing on whether something is important for getting to acceptably automatic in the short term, or is something that could happen longer-term, as iteration on that base.

[thejcannon]

This step is about to go away: #19431

[thejcannon]

Slight change. that PR will likely be closed in favor of smaller dedicated PRs

[thejcannon]

OK, this is officially done. See https://github.com/pantsbuild/pants/releases/tag/release_2.18.0.dev5

[huonw]

Existing process step-by-step: Step 5: Test the release

Summary:

1. Unsure what's involved!
2. Probably makes sense for a human to, e.g., confirm that the release artefacts seem to be in the right place to catch complete automation failures

Sub-step	Human effort now	Human in the loop required	Short term automation	Long term automation
only one sub-step	I don't know?	I don't know? Probably	?	?

[huonw]

Existing process step-by-step: Step 6: Announce the change

Summary:

1. Basic announcements to email and slack are probably automatable... but probably nicer to have a human touch (e.g. a human acknowledging a first time contributor is nicer than a robot doing it)?
2. Writing/publishing "what's news" and blog posts... isn't

Sub-step	Human effort now	Human in the loop required	Short term automation	Long term automation
Dev release	?	Human to find right email therad to reply to?	?	?
Alpha release	?	as above	?	?
Release candiate	?	as above	?	?
Stable release	?	yes, blog posts etc	?	?

[benjyw]

I'd actually support automated emails and Slack announcements. Humans can follow up for the human touch (congratulating the first time contributors etc.) But those emails and slack posts are actually surprisingly annoying to author and send, because of the links...

[huonw]

Ah, okay! 👍

I don't know what it takes to automate emails (I know slack is easy, though).

Just brainstorming some options:

1. Fully automatic announcements
2. Automate slack and a human can copy into an email (if emails are annoying to automate, in the short term)
3. Format the basic message with links etc, and just print it out for a human to copy-paste and adjust
4. Write the message into the GH release as a place to copy-paste from (potentially nice to do with any of the above, but might be better considered "long term automation", unless it helps in the short term)

[thejcannon]

One thing to point out is once we have branch protection rules on release branches (something we've agreed upon and only haven't turned on because @stuhood would love easier releases before that) that the merging of PRs is automatic through enabling "auto-merge" (sans CI flakiness or legitimate failures).

[thejcannon]

... you lost me. I don't follow (at least not with the context of my comments)

[huonw]

Ah, okay. I might've misinterpreted. I'd assumed you were suggesting cherry-picking to the release branch could be done as a PR, but maybe you're suggesting that we automate the cherry-pick as a normal push still, but do it in a way that lets us turn branch protection rules on?

[thejcannon]

Let me try again. 😅

One part of the human in the loop is just clicking the green merge button when CI is done (this could be at the same time as approval, or possibly later). Not a big deal, but IS annoying that's it's an extra click.

We could easily eliminate that if we enabled auto merge, but that requires branch protection rules. The reason we don't have those today is because the version bump pushes directly to git. It does so simply to avoid the churn of a PR on the release branch.

So, working backwards, if the PR that updated the release notes on main for a new version is being cherry picked, we could have the cherry pick script also bump the version in the release branch so those two steps are now one (also removing the need for the human to bump the version and push).

Then, we can enable branch protection and auto-merge.

We get safety (branch protection) and 2 automation items (version bumping and auto merge)

[benjyw]

I think the version bump being a direct push is simply to save the releaser some hassle. But doing it automatically is even less hassle, so...

[thejcannon]

Exactly! We get our cake and eat it too! 😋

[huonw]

First steps towards automated "start release" in #19303.

[huonw]

Recent changes related to this effort:

• Update changelog/contributors/version bump in one script #19379
• A script to generate a release announcement message. #19361
• Move more release code to src/python/pants_release. #19300
• Create a top-level pants_release package. #19297

Potential additional release process step to be automated:

• Add workflow for testing against public repos #19161

[huonw]

We've well and truly implemented enough here to call this done, mostly by Josh (nice work!). Releases are much easier now.

#20438 is another simplification well into the "next steps"/"beyond MVP" phase.

[benjyw]

This has been a journey, and the end result is excellent! Thanks to all who contributed.