Skip to content

Develop #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
matthewpeterkort wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Develop #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
eed4b78
modifies multipart upload to call single part when file is <5GB
matthewpeterkort Dec 18, 2025
c6be76f
split client out further into more digestable parts (#13)
matthewpeterkort Dec 29, 2025
1130f52
update client
matthewpeterkort Dec 29, 2025
7289156
add back bug fixes
matthewpeterkort Dec 29, 2025
a95f442
bugfix make single part / file uploader work
matthewpeterkort Dec 30, 2025
0aaac8b
Preserve error chain in upload error wrapping (#16)
Copilot Jan 13, 2026
91b365c
rework retry to be fully integrated with refresh
matthewpeterkort Jan 15, 2026
d42f12d
fix logic
matthewpeterkort Jan 15, 2026
2b7b0e8
fix fatalf bug
matthewpeterkort Jan 16, 2026
958f166
add retry wait bounds
matthewpeterkort Jan 16, 2026
b5432fe
adds OptimalChunkSize, ProgressCallback
bwalsh Jan 27, 2026
6fd53ed
Refactor/calypr clients (#26)
matthewpeterkort Feb 3, 2026
16fb1ee
Refactor/progress callback (#29)
bwalsh Feb 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
214 changes: 0 additions & 214 deletions .github/workflows/build_and_test_push.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,218 +27,4 @@ jobs:
- name: Run Tests
run: go test -v github.com/uc-cdis/gen3-client/tests

build:
env:
goarch: amd64
needs: test
runs-on: ubuntu-latest
strategy:
matrix:
include:
- goos: linux
goarch: amd64
zipfile: dataclient_linux.zip
- goos: darwin
goarch: amd64
zipfile: dataclient_osx.zip
- goos: windows
goarch: amd64
zipfile: dataclient_win64.zip
steps:
- name: Checkout Code
uses: actions/checkout@v4

- name: Setup Go 1.17
uses: actions/setup-go@v4
with:
go-version: '1.17'

- name: Run Setup Script
run: |
bash .github/scripts/before_install.sh
env:
GITHUB_BRANCH: ${{ github.ref_name }}
ACCESS_KEY: ${{ secrets.AWS_S3_ACCESS_KEY_ID }}
SECRET_ACCESS_KEY: ${{ secrets.AWS_S3_SECRET_ACCESS_KEY }}


- name: Run Build Script
run: |
bash .github/scripts/build.sh
env:
GOOS: ${{ matrix.goos }}
GOARCH: ${{ env.goarch }}
GITHUB_BRANCH: ${{ github.ref_name }}
GITHUB_PULL_REQUEST: ${{ github.event_name == 'pull_request' }}

- name: Upload Artifacts
uses: actions/upload-artifact@v4
with:
name: build-artifact-${{ matrix.goos }}
path: ~/shared/${{ matrix.zipfile }}
retention-days: 3


sign:
needs: build
runs-on: macos-latest
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Download OSX Artifact
uses: actions/download-artifact@v4
with:
name: build-artifact-darwin
path: ./dist
- name: Unzip OSX Artifact and remove zip file
run: |
cd ./dist
ls
unzip dataclient_osx.zip
rm dataclient_osx.zip



- name: Build executable
shell: bash
env:
APPLE_CERT_PASSWORD: ${{ secrets.APPLE_CERT_PASSWORD }}
APPLE_NOTARY_UUID: ${{ secrets.APPLE_NOTARY_UUID }}
APPLE_NOTARY_KEY: ${{ secrets.APPLE_NOTARY_KEY}}
APPLE_NOTARY_DATA: ${{ secrets.APPLE_NOTARY_DATA }}
APPLE_CERT_DATA: ${{ secrets.APPLE_CERT_DATA }}
APPLICATION_CERT_PASSWORD: ${{ secrets.APPLICATION_CERT_PASSWORD }}
APPLICATION_CERT_DATA: ${{ secrets.APPLICATION_CERT_DATA }}
APPLE_TEAM_ID: WYQ7U7YUC9

run: |
# Setup
SIGNFILE="$(pwd)/dist/gen3-client"

# Export certs
echo "$APPLE_CERT_DATA" | base64 --decode > /tmp/certs.p12
echo "$APPLE_NOTARY_DATA" | base64 --decode > /tmp/notary.p8
echo "$APPLICATION_CERT_DATA" | base64 --decode > /tmp/app_certs.p12

# Create keychain
security create-keychain -p actions macos-build.keychain
security default-keychain -s macos-build.keychain
security unlock-keychain -p actions macos-build.keychain
security set-keychain-settings -t 3600 -u macos-build.keychain

# Import certs to keychain
security import /tmp/certs.p12 -k ~/Library/Keychains/macos-build.keychain -P "$APPLE_CERT_PASSWORD" -T /usr/bin/codesign -T /usr/bin/productsign
security import /tmp/app_certs.p12 -k ~/Library/Keychains/macos-build.keychain -P "$APPLICATION_CERT_PASSWORD" -T /usr/bin/codesign -T /usr/bin/productsign

# Key signing
security set-key-partition-list -S apple-tool:,apple: -s -k actions macos-build.keychain

# Verify keychain things
security find-identity -v macos-build.keychain | grep "$APPLE_TEAM_ID" | grep "Developer ID Application"
security find-identity -v macos-build.keychain | grep "$APPLE_TEAM_ID" | grep "Developer ID Installer"

# Force the codesignature
codesign --force --options=runtime --keychain "/Users/runner/Library/Keychains/macos-build.keychain-db" -s "$APPLE_TEAM_ID" "$SIGNFILE"

# Verify the code signature
codesign -v "$SIGNFILE" --verbose

mkdir -p ./dist/pkg
cp ./dist/gen3-client ./dist/pkg/gen3-client
pkgbuild --identifier "org.uc-cdis.gen3-client.pkg" --timestamp --install-location /Applications --root ./dist/pkg installer.pkg
pwd
ls
productbuild --resources ./resources --distribution ./distribution.xml gen3-client.pkg
productsign --sign "$APPLE_TEAM_ID" --timestamp gen3-client.pkg gen3-client_signed.pkg

xcrun notarytool store-credentials "notarytool-profile" --issuer $APPLE_NOTARY_UUID --key-id $APPLE_NOTARY_KEY --key /tmp/notary.p8
xcrun notarytool submit gen3-client_signed.pkg --keychain-profile "notarytool-profile" --wait
xcrun stapler staple gen3-client_signed.pkg
mv gen3-client_signed.pkg dataclient_osx.pkg

- name: Upload signed artifact
uses: actions/upload-artifact@v4
with:
name: build-artifact-darwin-signed
path: dataclient_osx.pkg

sync_signed_to_aws:
runs-on: ubuntu-latest
needs: sign

steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Run Setup Script
run: |
bash ./.github/scripts/before_install.sh
env:
GITHUB_BRANCH: ${{ github.ref_name }}
ACCESS_KEY: ${{ secrets.AWS_S3_ACCESS_KEY_ID }}
SECRET_ACCESS_KEY: ${{ secrets.AWS_S3_SECRET_ACCESS_KEY }}

- name: Download OSX Artifact
uses: actions/download-artifact@v4
with:
name: build-artifact-darwin-signed

- name: Sync to AWS
env:
GITHUB_BRANCH: ${{ github.ref_name }}
run: |
rm ~/shared/dataclient_osx.zip
zip dataclient_osx_signed.zip dataclient_osx.pkg
mv dataclient_osx_signed.zip ~/shared/
aws s3 sync ~/shared s3://cdis-dc-builds/$GITHUB_BRANCH


get_tagged_branch:
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubuntu-latest
needs: [build,sign]
outputs:
branch: ${{ steps.check_step.outputs.branch }}
steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Get current branch
id: check_step
# 1. Get the list of branches ref where this tag exists
# 2. Remove 'origin/' from that result
# 3. Put that string in output
# => We can now use function 'contains(list, item)''
run: |
raw=$(git branch -r --contains ${{ github.ref }})
branch="$(echo ${raw//origin\//} | tr -d '\n')"
echo "{name}=branch" >> $GITHUB_OUTPUT
echo "Branches where this tag exists : $branch."


deploy:
needs: get_tagged_branch
if: startsWith(github.ref, 'refs/tags/') && contains(${{ needs.get_tagged_branch.outputs.branch }}, 'master')
runs-on: ubuntu-latest
steps:
- name: Download Linux Artifact
uses: actions/download-artifact@v4
with:
name: build-artifact-linux

- name: Download OSX Artifact
uses: actions/download-artifact@v4
with:
name: build-artifact-darwin-signed

- name: Download Windows Artifact
uses: actions/download-artifact@v4
with:
name: build-artifact-windows

- name: Create Release gh cli
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_TAG: ${{ github.ref_name }}
run: gh release create "$GH_TAG" dataclient_linux.zip dataclient_osx.pkg dataclient_win64.zip --repo="$GITHUB_REPOSITORY"
103 changes: 103 additions & 0 deletions .github/workflows/coverage.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
name: "Test Coverage Check"

on:
pull_request:
branches:
- master
push:
branches:
- master

jobs:
coverage:
name: Test Coverage
runs-on: ubuntu-latest

steps:
- name: Checkout Code
uses: actions/checkout@v4

- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: '1.24.2'

- name: Run Tests with Coverage
run: |
go test -coverprofile=coverage.out -covermode=atomic ./...
continue-on-error: true

- name: Generate Coverage Report
id: coverage
run: |
# Get overall coverage
OVERALL=$(go tool cover -func=coverage.out | grep total | awk '{print $3}' | sed 's/%//')
echo "overall=$OVERALL" >> $GITHUB_OUTPUT

# Generate detailed report
echo "## Test Coverage Report" > coverage-report.md
echo "" >> coverage-report.md
echo "**Overall Coverage:** ${OVERALL}%" >> coverage-report.md
echo "" >> coverage-report.md
echo "### Package Coverage" >> coverage-report.md
echo "" >> coverage-report.md
echo "| Package | Coverage |" >> coverage-report.md
echo "|---------|----------|" >> coverage-report.md

# Extract package coverage
go test -coverprofile=/dev/null -covermode=atomic ./... 2>&1 | \
grep "coverage:" | \
grep -v "setup failed" | \
awk '{
pkg=$1;
cov=$4;
gsub(/github.com\/calypr\/data-client\//, "", pkg);
if (cov ~ /statements/) {
print "| " pkg " | " cov " |"
} else {
print "| " pkg " | " cov " |"
}
}' >> coverage-report.md

cat coverage-report.md

- name: Check Coverage Thresholds
run: |
chmod +x ./scripts/check-coverage.sh
./scripts/check-coverage.sh 30 20

- name: Upload Coverage to Codecov (Optional)
uses: codecov/codecov-action@v4
if: always()
with:
files: ./coverage.out
flags: unittests
name: codecov-umbrella
fail_ci_if_error: false
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

- name: Comment PR with Coverage
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const coverage = fs.readFileSync('coverage-report.md', 'utf8');

github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: coverage
});

- name: Upload Coverage Artifact
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-report
path: |
coverage.out
coverage-report.md
retention-days: 30
34 changes: 34 additions & 0 deletions .github/workflows/release.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Release

on:
push:
tags:
- '*'
workflow_dispatch:

permissions:
contents: write

jobs:
goreleaser:
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout@v4
with:
submodules: recursive
-
name: Set up Go
uses: actions/setup-go@v5
-
name: Run GoReleaser
uses: goreleaser/goreleaser-action@v6
with:
# either 'goreleaser' (default) or 'goreleaser-pro'
distribution: goreleaser
# 'latest', 'nightly', or a semver
version: 'latest'
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,5 @@

# Build artifacts
/build/
/bin/
checksums.txt
Loading
Loading