Optimize GitHub Actions workflows with path filters and caching

[Copilot]

CI workflows were running on every push to any branch and taking up to 90 minutes due to full conda environment reinstalls. Optimized workflows with appropriate path filters, caching, and targeted triggers.

Changes

• Renamed build-test-conda.yml → run-tests.yml

  • Trigger: push to master AND pull_request to master
  • Added conda package caching via actions/cache@v4
  • Enabled mamba for faster dependency resolution
  • Added explicit permissions: contents: read

• Created check-conda-env.yml

  • Triggers only when SQANTI3.conda_env.yml changes on PRs to master
  • Validates environment installs correctly with import tests for core deps

• Updated generate-docker-image.yml

  • Removed push trigger, PR only
  • Added path filter: Dockerfile, build_docker.sh, .dockerignore

• Updated push-to-dockerhub-on-release.yml

  • Added edited event type to rebuild and push Docker when a release is updated

Trigger Summary

Workflow	Trigger	Path Filter
run-tests.yml	push to master, PR to master	none
check-conda-env.yml	PR to master	SQANTI3.conda_env.yml
generate-docker-image.yml	PR to master	Docker files
push-to-dockerhub-on-release.yml	release published or edited	none

Original prompt

Problem Statement

The current GitHub Actions workflows are running too frequently and taking too long (up to 90 minutes), causing unnecessary resource consumption and slow feedback cycles. The workflows need to be optimized to:

1. Only run when relevant files change
2. Avoid re-running checks on master after a PR with passing checks is merged
3. Separate environment validation from test execution

Current Issues

1. build-test-conda.yml: Installs the full conda environment (very slow) and runs tests on every push to any branch and every PR
2. generate-docker-image.yml: Builds Docker image on every push/PR to master, even when Docker files haven't changed
3. push-to-dockerhub-on-release.yml: Already optimized (runs only on releases) - keep as is

Required Changes

1. Update build-test-conda.yml (Rename to run-tests.yml)

• Trigger: Only on pull requests to master (not on pushes to master after merge)
• Purpose: Run test suite on every PR commit
• Optimization: Use cached conda environment instead of reinstalling from scratch every time
• Add caching for conda environment to speed up subsequent runs
• Keep the test execution but optimize environment setup

2. Create new check-conda-env.yml

• Trigger: Only when SQANTI3.conda_env.yml file changes (on pull requests)
• Purpose: Validate that the conda environment file is correct and can be installed
• Actions:
  • Install conda environment from scratch
  • Verify all dependencies resolve correctly
  • Run basic import tests

3. Update generate-docker-image.yml

• Trigger: Only on pull requests to master AND only when Docker-related files change:
  • Dockerfile
  • build_docker.sh
  • .dockerignore (if exists)
• Purpose: Validate Docker builds when Docker configuration changes
• Keep existing disk space cleanup and build steps

4. Keep push-to-dockerhub-on-release.yml unchanged

• Already optimized - only runs on release publication
• Builds and pushes Docker image to DockerHub with proper tags

Implementation Details

Option 2 Strategy (As requested)

• Run test and validation workflows only on pull_request events, not on push to master
• Once a PR's checks pass and it's merged, don't re-run the same checks on master
• Exception: Release workflow still runs on releases as expected
• Docker validation runs on PRs when Docker files change

Benefits

• Reduces CI/CD time from 90 minutes to a few minutes for most commits
• Conda environment only rebuilt when configuration changes (rare)
• Docker only rebuilt when Docker files change (rare)
• No duplicate work after merging PRs with passing checks
• Faster feedback cycles for developers
• Reduced resource consumption

Acceptance Criteria

• Tests run on every PR commit but NOT on pushes to master after merge
• Conda environment installation only happens when SQANTI3.conda_env.yml changes
• Docker builds only happen when Docker-related files change
• Release workflow continues to work as before
• All workflows use appropriate caching to speed up runs
• Workflow files are properly configured with path filters where needed

This pull request was created as a result of the following prompt from Copilot chat.

Problem Statement

The current GitHub Actions workflows are running too frequently and taking too long (up to 90 minutes), causing unnecessary resource consumption and slow feedback cycles. The workflows need to be optimized to:

1. Only run when relevant files change
2. Avoid re-running checks on master after a PR with passing checks is merged
3. Separate environment validation from test execution

Current Issues

1. build-test-conda.yml: Installs the full conda environment (very slow) and runs tests on every push to any branch and every PR
2. generate-docker-image.yml: Builds Docker image on every push/PR to master, even when Docker files haven't changed
3. push-to-dockerhub-on-release.yml: Already optimized (runs only on releases) - keep as is

Required Changes

1. Update build-test-conda.yml (Rename to run-tests.yml)

• Trigger: Only on pull requests to master (not on pushes to master after merge)
• Purpose: Run test suite on every PR commit
• Optimization: Use cached conda environment instead of reinstalling from scratch every time
• Add caching for conda environment to speed up subsequent runs
• Keep the test execution but optimize environment setup

2. Create new check-conda-env.yml

• Trigger: Only when SQANTI3.conda_env.yml file changes (on pull requests)
• Purpose: Validate that the conda environment file is correct and can be installed
• Actions:
  • Install conda environment from scratch
  • Verify all dependencies resolve correctly
  • Run basic import tests

3. Update generate-docker-image.yml

• Trigger: Only on pull requests to master AND only when Docker-related files change:
  • Dockerfile
  • build_docker.sh
  • .dockerignore (if exists)
• Purpose: Validate Docker builds when Docker configuration changes
• Keep existing disk space cleanup and build steps

4. Keep push-to-dockerhub-on-release.yml unchanged

• Already optimized - only runs on release publication
• Builds and pushes Docker image to DockerHub with proper tags

Implementation Details

Option 2 Strategy (As requested)

• Run test and validation workflows only on pull_request events, not on push to master
• Once a PR's checks pass and it's merged, don't re-run the same checks on master
• Exception: Release workflow still runs on releases as expected
• Docker validation runs on PRs when Docker files change

Benefits

• Reduces CI/CD time from 90 minutes to a few minutes for most commits
• Conda environment only rebuilt when configuration changes (rare)
• Docker only rebuilt when Docker files change (rare)
• No duplicate work after merging PRs with passing checks
• Faster feedback cycles for developers
• Reduced resource consumption

Acceptance Criteria

• Tests run on every PR commit but NOT on pushes to master after merge
• Conda environment installation only happens when SQANTI3.conda_env.yml changes
• Docker builds only happen when Docker-related files change
• Release workflow continues to work as before
• All workflows use appropriate caching to speed up runs
• Workflow files are properly configured with path filters where needed

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[pabloati]

@copilot Include a new condition on the release workflow. If the release is updated, rebuild and push the docker

[Copilot]

@copilot Include a new condition on the release workflow. If the release is updated, rebuild and push the docker

Added the edited event type to the release workflow trigger in commit bfb0272. The workflow will now run when a release is published or updated.

[pabloati]

@copilot I want to know how can I test this actions before doing the pull-request

[Copilot]

@copilot I want to know how can I test this actions before doing the pull-request

You can test GitHub Actions workflows before merging using these approaches:

1. Use act locally: Install act to run workflows locally:

   act pull_request -W .github/workflows/run-tests.yml

2. Use workflow_dispatch: Temporarily add a manual trigger to test:

   on:
     workflow_dispatch:  # Allows manual trigger from Actions tab
     pull_request:
       branches: ['master']

   Then trigger it from the "Actions" tab in GitHub.

3. Create a draft PR: Open this PR as a draft - it will trigger the workflows without marking it ready for review.

4. Push to a test branch: Since run-tests.yml triggers on PRs to master, you can create a test PR from any branch to master to see the workflows run.

The workflows in this PR will automatically run when you open/update the PR to master, so you can verify them directly from the PR's "Checks" tab.

[pabloati]

@copilot I saw that the conda environment installation and the test check are only done for pull-request to master. Would it make sense to expand them to be done on any push? I think that at least the tests should be like that, to be done on every push, in case something gets broken in production, to detect it asap, right?

[Copilot]

@copilot I saw that the conda environment installation and the test check are only done for pull-request to master. Would it make sense to expand them to be done on any push? I think that at least the...

Good point! Added push trigger to master for run-tests.yml in commit 3563766. Tests will now run on:

• Every push to master (catches production issues)
• Every PR to master (validates changes before merge)

The conda environment check (check-conda-env.yml) remains PR-only with path filter since it's specifically for validating environment file changes, which is rare and doesn't need to run on every push.

[pabloati]

This file is just a split of the previous action where both the conda environment and the tests were tested. However, now it will only be run if the conda environment file is modified