feat(test reports): Object storage integration and test-report key

[timhuynh94]

Overview

1. Introduce a pluggable object storage subsystem that Vela can use for test artifacts.
2. Add first‑class test report and test attachment models at the database and API layers.
3. Expose HTTP endpoints for creating, listing, and updating test reports and attachments, plus admin endpoints for managing storage (buckets, presigned URLs).
4. Wire storage + test reports into the router, middleware, CLI flags, and local dev environment so the feature can be exercised end‑to‑end.
5. Proposal feat(proposal): native test report integration community#1022

---

High‑Level Architecture

1. Storage subsystem

Packages & files (high level)

• storage/
  • context.go, setup.go, service.go, storage.go (+ tests)
  • flags.go (+ tests) for CLI/env configuration
• storage/minio/
  • Bucket operations: create_bucket.go, bucket_exists.go, list_bucket.go, get_bucket.go
  • Object operations: upload.go, list_objects.go, stat_object.go, presigned_get_object.go
  • Driver config / enablement: opts.go, storage_enable.go, minio.go
  • Test data: test_data/create_bucket.json, test_data/test.xml
• constants/driver.go, constants/filetypes.go, constants/table.go

Design

• A new storage service interface is introduced (see storage/service.go / storage/storage.go), following the same Service + Setup + Context pattern as other Vela subsystems.
• The initial implementation is a MinIO/S3‑compatible driver, isolated under storage/minio/.
• Storage enablement is driven by configuration + feature flag:
  • The driver and settings are wired via storage/flags.go and storage/setup.go.
  • Middleware injects a storage-enable flag and a storage client into the Gin context.
• Common constants for:
  • Drivers (e.g. MinIO),
  • File types for test attachments,
  • Table names for the new report/attachment tables.

2. Test reports & attachments data model

Database layer

• New database/reports/testreport/ package:
  • CRUD, list, count, and index operations:
    • create.go, get.go, list.go, list_by_build.go, list_by_repo.go, count.go, count_build.go, count_repo.go, update.go, delete.go, index.go, opts.go, table.go
  • Full test suite mirroring the above (*_test.go).
• New database/reports/testattachment/ package:
  • Similar structure for attachments: create.go, get.go, get_build.go, list.go, count.go, update.go, delete.go, index.go, opts.go, table.go + tests.
• New persistence models under database/types/:
  • test_report.go
  • test_attachment.go
• database/database.go, database/interface.go, and database/resource.go updated to:
  • Register the new report / attachment services with the main database interface.
  • Expose them via the existing resource wiring.
• database/testutils/api_resources.go updated to provide helpers for the new resources.

Conceptually

• Test reports represent high‑level test result records associated with repos/builds.
• Test attachments represent individual artifacts (e.g. XML files) stored in object storage and linked back to test reports/builds via IDs, file type, and object metadata.

3. API + routing surface

API packages

• Storage admin + info
  • api/admin/storage.go: admin handlers for bucket creation and presigned URLs for objects.
  • api/storage/doc.go + api/storage/storage.go: public storage handlers (e.g., storage info / health).
• Test reports
  • api/testreport/create.go
  • api/testreport/update.go
• Test attachments
  • api/testattachment/create.go
  • api/testattachment/update.go
• API types
  • api/types/test_report.go
  • api/types/test_attachment.go
  • api/types/storage.go
  • api/types/storage_info.go
  • Minor updates in api/types/pipeline.go / tests to incorporate the test‑report related shape.

Router & middleware

• New or updated router files:
  • router/storage.go — routes for storage info and/or admin integration.
  • router/testreport.go — routes for test report CRUD & listing.
  • router/testattachment.go — routes for test attachment CRUD & listing.
  • router/admin.go — wires the new admin storage operations.
  • router/build.go — extended to add build‑scoped report/attachment routes.
• Middleware:
  • router/middleware/storage.go (+ tests): constructs the storage service, sets storage-enable, and injects it into Gin.
  • router/middleware/testreport/context.go, testreport/testreport.go: helpers for binding test report resources to requests.
  • router/middleware/testattachment/context.go, testattachment/testattachment.go: same pattern for attachments.

Swagger / HTTP endpoints

• Storage admin:
  • PUT /api/v1/admin/storage/bucket — create a bucket.
  • GET /api/v1/admin/storage/presign — generate a presigned URL for a given bucket + object.
• Test reports & attachments:
  • New repo/build‑scoped CRUD + list endpoints under testreports and testattachments (see router + API handlers).
  • These endpoints are backed by the new database packages and, for attachments, the storage service.

4. Compiler & pipeline types

• compiler/types/pipeline/test_report.go (+ tests) and compiler/types/yaml/test_report.go:
  • Introduce a test report concept into the compiler/pipeline types.
• compiler/types/pipeline/container.go, compiler/types/yaml/step.go, compiler/types/yaml/step_test.go:
  • Allow steps to declare test‑report related configuration.
• compiler/native/validate.go (+ tests):
  • Validation extended/adjusted to understand the new test report key usage in the pipeline YAML.

This sets up the server side so that pipelines can emit test report metadata that is persisted and linked to stored artifacts.

5. Configuration & local dev

• cmd/vela-server/main.go, metadata.go, server.go:
  • Wire storage flags and setup into server startup.
• storage/flags.go, storage/setup.go:
  • Define CLI/env flags for storage configuration (driver, connection details, etc.).
  • Build the storage.Service and register it with the router via middleware.
• docker-compose.yml and nginx.conf:
  • Add MinIO to the local dev stack.
  • Configure NGINX / hostnames so minio is reachable from the server and UI.
• DOCS.md:
  • Document the MinIO dev setup (e.g. adding minio to /etc/hosts) and reference storage in the setup flow.

---

Testing

• New unit tests added for:
  • Storage driver operations (storage/minio/*_test.go).
  • Storage wiring (storage/context_test.go, storage/setup_test.go, storage/storage_test.go, storage/flags_test.go).
  • Database report/attachment CRUD, list, count, and indexing (database/reports/testreport/*_test.go, database/reports/testattachment/*_test.go).
  • New database resource wiring (database/resource_test.go).
  • Router + middleware behavior for storage, test reports, and attachments.
• Mocks:
  • mock/server/testreport.go added to support API tests.

Example

• Standalone step

  - name: test-results
    image: golang:1.20
    ruleset:
      status: [failure, success]
    test_report:
      results: ["test-results/*.xml"]
      attachments: [ "cypress/screenshots/**/*.png", "cypress/videos/**/*.mp4"]

• Part of existing step

  - name: cypress_tests
    image: cypress/browsers:node-20.16.0-chrome-127.0.6533.119-1-ff-129.0.1-edge-127.0.2651.98-1
    ruleset:
      event: push
    commands:
      - npm install
      - npm run cy:run
    test_report:
      results: ["test-results/*.xml"]
      attachments: [ "cypress/screenshots/**/*.png", "cypress/videos/**/*.mp4"]

[github-actions]

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

golangci

---

🚫 [golangci] _{reported by reviewdog 🐶}
missing whitespace above this line (invalid statement above assign) (wsl_v5)

server/mock/server/testreport.go

Line 20 in ba647b1

_ = json.Unmarshal(data, &body)

---

🚫 [golangci] _{reported by reviewdog 🐶}
unnecessary whitespace (trailing-whitespace) (wsl_v5)

server/router/router.go

Line 161 in ba647b1

---

🚫 [golangci] _{reported by reviewdog 🐶}
unnecessary whitespace (trailing-whitespace) (wsl_v5)

server/router/testattachment.go

Line 20 in ba647b1

---

🚫 [golangci] _{reported by reviewdog 🐶}
unnecessary whitespace (trailing-whitespace) (wsl_v5)

server/router/testreport.go

Line 20 in ba647b1

---

🚫 [golangci] _{reported by reviewdog 🐶}
missing whitespace above this line (no shared variables above if) (wsl_v5)

server/storage/minio/bucket_exists_test.go

Line 48 in ba647b1

if resp.Code != http.StatusOK {

---

🚫 [golangci] _{reported by reviewdog 🐶}
missing whitespace above this line (no shared variables above if) (wsl_v5)

server/storage/minio/bucket_exists_test.go

Line 81 in ba647b1

if resp.Code != http.StatusOK {

---

🚫 [golangci] _{reported by reviewdog 🐶}
missing whitespace above this line (no shared variables above if) (wsl_v5)

server/storage/minio/create_bucket_test.go

Line 38 in ba647b1

if resp.Code != http.StatusOK {

---

🚫 [golangci] _{reported by reviewdog 🐶}
missing whitespace above this line (too many statements above defer) (wsl_v5)

server/storage/minio/list_bucket_test.go

Line 36 in ba647b1

fake := httptest.NewServer(engine)

---

🚫 [golangci] _{reported by reviewdog 🐶}
missing whitespace above this line (too many statements above defer) (wsl_v5)

server/storage/minio/list_bucket_test.go

Line 71 in ba647b1

fake := httptest.NewServer(engine)

---

🚫 [golangci] _{reported by reviewdog 🐶}
missing whitespace above this line (no shared variables above expr) (wsl_v5)

server/storage/minio/list_objects_test.go

Line 66 in ba647b1

c.Stream(func(w io.Writer) bool {

---

🚫 [golangci] _{reported by reviewdog 🐶}
missing whitespace above this line (invalid statement above expr) (wsl_v5)

server/storage/minio/list_objects_test.go

Line 71 in ba647b1

c.XML(http.StatusOK, objects)

---

🚫 [golangci] _{reported by reviewdog 🐶}
missing whitespace above this line (too many statements above defer) (wsl_v5)

server/storage/minio/list_objects_test.go

Line 76 in ba647b1

fake := httptest.NewServer(engine)

---

🚫 [golangci] _{reported by reviewdog 🐶}
missing whitespace above this line (no shared variables above range) (wsl_v5)

server/storage/minio/list_objects_test.go

Line 234 in ba647b1

for _, name := range results {

[wass3rw3rk]

i have more of a fundamental piece of feedback and maybe something to open up for discussion (maybe even on the proposal itself). the current proposal didn't really lay out what the integration into a step would look like. it showed one option for implementing this through a custom step/plugin which we didn't pursue here in the end.

as you called out in the description the proposed structure with this PR is:

  - name: test-results
    image: golang:1.20
    test_report:
      results: ["test-results/*.xml"]
      attachments: [ "cypress/screenshots/**/*.png", "cypress/videos/**/*.mp4"]

with test_report and it's associated sub-fields being the main addition here. i am not a fan of the top-level test_report naming for one, partly because it contains multiple words. in addition, it really strongly ties all this to "testing" although the base functionality is more basic - uploading an artifact.

GitLab's implementation, for example, looks something like the following:

job:
  script: echo "build xyz project"
  artifacts:
    paths:
      - path/*xyz/*

here, artifacts is the top key. a fuller example that includes tests is:

job:
  artifacts:
    when: always
    paths:
      - rspec.xml
    reports:
      junit: rspec.xml

so, paths are the items to upload, and then you can further specify test-specific information through reports.

that matches more with my mental model although i am not saying that our implementation should copy this.

BuildKite has similar functionality also where steps can be defined as follows:

steps:
  - label: "🔨 Tests"
    command:
      - "npm install"
      - "tests.sh"
    artifact_paths:
      - "logs/**/*"
      - "coverage/**/*"

they chose artifact_paths as their top-level name - also not great, in my opinion but focused on "artifacts". and for their testing support, it looks like you use a custom token and post the test file to a specific BuildKite URL in different ways depending on what language you're using in your project (Go example). seems a little involved, our general approach seems more user friendly.

i just picked two semi-random platforms here, but the point is that i feel like we can improve the setup/naming. to be honest, i'm quite fond of the GitLab approach here. artifacts as a top-level key makes a lot of sense to me, and reports with an object that is <test format>: <path> does give some flexibility there even if we just support one format to start with.

curious if you, or anyone else, have feedback on this part?

[KellyMerrick]

I've been using attachments or test attachments. Artifacts also works. The database is currently coded as testattachments and testreports. Easily can be simplified to just attachments or artifacts, and reports.

Agree, the naming has been difficult and am completely open to a rename or at least alignment for consistency across the codebase.

Test reports is not quite right to me, as Reports, can be more than just test "attachments", such as a test results summary, code coverage, or anything pertinent to a single build.

Rough idea, with just attachments for now, but could have other options in the future: