Add tag_and_release script

[BenGalewsky]

Problem

Deployment is cumbersome and error-prone

#Approach
Added a new tag_and_release script that:

1. Updates the version.py string to match the version
2. Creates a branch named after the version
3. Commits and pushes to trigger CI job

[shortcut-integration]

This pull request has been linked to Clubhouse Story #9770: Automate deployment of Forwarder Version.

[sirosen]

If you are going to have automation which does a git-commit, you should probably check git status --porcelain or something to ensure you're not committing other changes.
However, I think this would be much simpler -- and you'd dodge other issues like having to satisfy BSD sed's lack of -i -- if you drove the process in a different direction.

Instead of making the version an input, have people change the version number prior to running a release. You can then parse it out of version data.

We have been doing it that way for a few years now in globus-sdk and globus-cli without issue, and I've used a similar process in other products and open source libs successfully. The current SDK make release target tags, pushes the tag, and runs a tox target that covers a build + twine upload. Even before it was done with tox we had something very similar.

If you boil it down to that point, it comes out as something like

fxf_version="$(grep '^__version__' funcx_forwarder/version.py | cut -d '"' -f2)"
git branch "v${fxf_version}"  # I think this should be a signed tag, but let's not deal with that today...
git push origin "v${fxf_version}"

[BenGalewsky]

@sirosen - thank you this seems wiser.
Just to clear up my understanding about why I suggest a branch for a release instead of a tag is that I want the versions of dependencies to float in main so we can always hope to use the latest version of everything. If something breaks the CI then we can fix it... in a release, I would want the dependencies to be locked for security of building a new point release for bug fixes.
Any thoughts on this motivation? I don't see how I could do that without checking in a commit with a lock file into the release branch

[sirosen]

I know we disagree about the dependency management in these projects, so at some point we need to talk through what we're going to do. We might need to meet, but I'll lay out my perspective on this as briefly as I can (brevity is my weakest point).

Pulling in a git dependency from a branch makes the build process less stable. If A depends on B:main and the build for A:main breaks, it could be a change in A or B. You need to evaluate all changes which happened in either project since the last successful build. This is particularly problematic when PRs against A fail. The failing build no longer sends a clear signal that the PR is bad and needs changes -- it's possible that B was temporarily broken at the time of the PR build, and that it gets fixed later.
The problem does not manifest only in CI: when a developer runs make test (or equivalent) locally, they are exposed to the latest branch heads.

It's better that any branch in A, including main, specifies a version of B under which the build is known to pass. When a PR is opened on A, a build failure should always mean that the PR is bad and needs changes.

This begs a question: why did we want A to depend on B:main? Most probably, it gives us confidence in our testing process. We don't want to release a version of B which catastrophically breaks A.

The testing of B should guarantee that it fulfills its contracts, so that breakage is very rare. This is not very useful to us: we can't require more extensive automated testing before anyone does any more work. But we very much agree that we want much more testing, and confidence in our ability to release upstream code is a place where it helps a lot. Strictly defining interfaces -- the contracts that B needs to fulfill -- is another part of this.

A more immediate solution: a daily build (with a manual trigger) can run the testsuite for A against B:main. Because the daily build is "unpinned", it makes sure that the two projects haven't drifted too far apart. It installs the pinned dependency versions of things, then replaces B with a VCS install from B:main before running tests.

Putting a manual trigger on the unpinned build lets people manually kick it off right before doing a release of B off of main. I have a few builds in GitHub Actions that do daily + manual runs. Here's an example of the trigger part:

on:
  workflow_dispatch:
  # build every weekday at 15:00 UTC
  schedule:
    - cron: '0 15 * * 1-5'

workflow_dispatch allows you to start the build with a few clicks in the GitHub Actions UI or a curl with a github token. The gh CLI probably supports it too, but I haven't looked.