🧪 Fix SDK E2E tests to work with vizzly tdd run #183

Robdel12 · 2026-01-17T07:28:36Z

Summary

This PR fixes SDK E2E tests to work properly with both vizzly tdd run (local TDD mode) and vizzly run (cloud mode), and reorganizes CI workflows for better maintainability.

Changes

CI Workflow Reorganization

Split monolithic ci.yml into focused workflow files:
- sdk-e2e.yml - E2E tests for all SDKs (runs on every PR)
- sdk-unit.yml - Unit tests with matrix testing (runs when SDK files change)
- reporter.yml - Reporter UI tests
- tui.yml - TUI tests
E2E tests now run in both TDD and cloud modes for each SDK

Ruby SDK Fixes

Add cloud_mode? helper to detect when running with VIZZLY_TOKEN
Add assert_screenshot_success helper for mode-aware assertions
TDD mode: asserts status is 'new' or 'match'
Cloud mode: asserts success is true (no local comparison happens)
Fixed in both integration_test.rb and e2e_test.rb

Vitest SDK Fixes

Refactored to use shared test-site for consistency with other SDKs
Test the actual SDK functionality (toMatchScreenshot matcher)
Load test-site CSS for consistent styling
Removed custom commands that were for full-page navigation, not SDK testing

All SDKs

Detect VIZZLY_SERVER_URL env var to skip starting own TDD server when running under vizzly tdd run or vizzly run
Fix assertions to handle both TDD mode (returns status field) and cloud mode (returns success: true)

Test plan

All SDK E2E tests pass in TDD mode (vizzly tdd run)
All SDK E2E tests pass in cloud mode (vizzly run)
Ruby lint passes (bundle exec rubocop)
CI workflow YAML is valid

vizzly-testing · 2026-01-17T07:29:46Z

🐻 Vizzly — 7 changes need review

Status	Count
Passed	12
Changed	7
Auto-approved	12

Visual differences (7)

fullscreen-viewer · Firefox · 1920×1080 · 1.5% diff

search-no-results · Firefox · 1920×1080 · 0.1% diff

search-no-results · Firefox · 375×978 · 3.3% diff

bulk-accept-dialog · Firefox · 1920×1080 · 5.8% diff

viewer-overlay-mode · Firefox · 1920×1080 · 1.6% diff

viewer-zoomed-100 · Firefox · 1920×1080 · 0.3% diff

...and 1 more in Vizzly.

Review changes →

_{fix-sdk-e2e-tests · 8369a5a3}

vizzly-testing · 2026-01-17T07:30:24Z

🐻 Vizzly — 1 change needs review

Status	Count
Changed	1
Auto-approved	4

Visual differences (5)

vizzly-help · 1202×1430 · 4.9% diff

vizzly-invalid-command · 1202×578 · 12.0% diff

vizzly-doctor · 1202×578 · 12.0% diff

vizzly-run-help · 1202×722 · 9.6% diff

vizzly-tdd-help · 1202×578 · 12.0% diff

Review changes →

_{fix-sdk-e2e-tests · 8369a5a3}

claude · 2026-01-17T07:31:42Z

Code Review - PR #183

This PR makes significant improvements to SDK E2E tests, ensuring they work correctly when run via vizzly tdd run. The changes fix API misuse, improve test structure, and add proper server detection.

Strengths

Excellent Server Detection Pattern - The VIZZLY_SERVER_URL detection is clean and consistent
Correct API Fixes - Fixed Vitest to use Testing Library, Static-Site to use URL paths, Ember assertions, Storybook patterns
Comprehensive Test Coverage - All SDKs now test element/full screenshots, properties, thresholds, multiple pages
Proper Config Structure - Fixed pageDiscovery nesting

Issues Found

CRITICAL

Command Injection in Ruby (clients/ruby/test/e2e_test.rb)
Uses string interpolation in shell command. While port is a number, this pattern is risky. Use spawn array syntax instead.

WARNINGS

Hardcoded Timeouts - May fail on slow CI. Use env-based timeouts (CI ? 5000 : 2000)
Missing Error Handling - Server spawn needs consistent error handling before promise resolves
Double CI Execution - Tests run twice (TDD + Cloud). Consider making conditional to save CI time

SUGGESTIONS

Test Site Coupling - All SDKs depend on ../../../test-site. Document this, add CI check
Code Duplication - Extract shared E2E setup logic to test-utils
Documentation - Add SDK README sections for E2E requirements and troubleshooting

Performance

CI time doubles with TDD+Cloud runs - monitor for bottlenecks
Browser instances could be reused within SDKs
Consider parallel test execution within each SDK

Verdict

Approve with minor fixes

Priority before merge:

Fix Ruby command injection (low risk but best practice)
Ensure consistent error handling in server spawns

Nice to have: shared E2E helpers, documentation, conditional CI execution

Great work improving test infrastructure!

vizzly-testing · 2026-01-17T07:43:59Z

🐻 Vizzly — 5 changes need review

Status	Count
Passed	14
Changed	5
Auto-approved	14

Visual differences (5)

fullscreen-viewer · Firefox · 1920×1080 · 1.5% diff

viewer-overlay-mode · Firefox · 1920×1080 · 1.6% diff

fullscreen-viewer · Firefox · 375×667 · 0.9% diff

search-no-results · Firefox · 375×978 · 3.5% diff

viewer-zoomed-100 · Firefox · 375×667 · 198.1% diff

Review changes →

_{fix-sdk-e2e-tests · 31d6df25}

vizzly-testing · 2026-01-17T07:44:34Z

🐻 Vizzly — 1 change needs review

Status	Count
Changed	1
Auto-approved	4

Visual differences (5)

vizzly-help · 1202×1430 · 4.9% diff

vizzly-doctor · 1202×578 · 12.0% diff

vizzly-tdd-help · 1202×578 · 12.0% diff

vizzly-invalid-command · 1202×578 · 12.0% diff

vizzly-run-help · 1202×722 · 9.6% diff

Review changes →

_{fix-sdk-e2e-tests · 31d6df25}

vizzly-testing · 2026-01-17T07:49:57Z

🐻 Vizzly — Approved ✓

19 comparisons approved.

Status	Count
Passed	11
Changed	8
Auto-approved	11

Visual differences (8)

bulk-accept-dialog · Firefox · 1920×1080 · 5.1% diff

viewer-overlay-mode · Firefox · 1920×1080 · 1.6% diff

search-no-results · Firefox · 1920×1080 · 0.1% diff

fullscreen-viewer · Firefox · 1920×1080 · 1.5% diff

viewer-zoomed-100 · Firefox · 1920×1080 · 0.3% diff

search-no-results · Firefox · 375×978 · 3.5% diff

...and 2 more in Vizzly.

View build →

_{fix-sdk-e2e-tests · 890d7460}

vizzly-testing · 2026-01-17T07:50:28Z

🐻 Vizzly — Approved ✓

5 comparisons approved.

Status	Count
Changed	1
Auto-approved	4

Visual differences (5)

vizzly-help · 1202×1430 · 4.9% diff

vizzly-tdd-help · 1202×578 · 12.0% diff

vizzly-run-help · 1202×722 · 9.6% diff

vizzly-doctor · 1202×578 · 12.0% diff

vizzly-invalid-command · 1202×578 · 12.0% diff

View build →

_{fix-sdk-e2e-tests · 890d7460}

- Detect VIZZLY_SERVER_URL env var to skip starting own TDD server - Fix Vitest tests to use correct browser mode API (getByRole, getByText) - Fix Static-Site tests to use URL-style paths (/, /features) not filenames - Fix Storybook pattern matching to use story.id format (*button*) - Fix Ember assertions to check result.status instead of result.success - Add proper config structure for page discovery (pageDiscovery.useSitemap) All SDK E2E tests now pass when run via `vizzly tdd run`: - Storybook: 13/13 - Static-Site: 13/13 - Ember: 9/9 - Vitest: 24/24 - Ruby: 10/10

- Add custom `loadPage` command to fetch and inject HTML content - Start static server for test-site with dynamic port allocation - Use real FluffyCloud test-site instead of inline HTML - Screenshots now match other SDKs (Storybook, Static-Site, Ember, Ruby) The `loadPage` command works around Vitest browser mode's limitation of not supporting `page.goto()` by fetching HTML and using `frame.setContent()` to inject it into the test iframe.

- Split monolithic ci.yml (900+ lines) into focused workflows: - ci.yml: Core CLI lint, tests, build (78 lines) - reporter.yml: Reporter visual tests (Vizzly cloud) - tui.yml: TUI visual tests (Vizzly cloud) - sdk-e2e.yml: All SDK E2E tests (TDD + Cloud mode) - sdk-unit.yml: SDK unit tests (conditional on file changes) - Add cloud mode for Ruby, Storybook, and Static-Site SDKs - Each workflow has its own status check job for branch protection - SDK E2E always runs to catch CLI breaking changes - SDK unit tests only run when that SDK's files change New secrets needed: - VIZZLY_RUBY_CLIENT_TOKEN - VIZZLY_STORYBOOK_CLIENT_TOKEN - VIZZLY_STATIC_SITE_CLIENT_TOKEN

- Test the actual SDK functionality (toMatchScreenshot matcher) - Load test-site CSS for consistent styling with other SDKs - Remove custom commands (loadPage was for full-page navigation, not SDK testing) - Test page screenshots, element screenshots, properties, and thresholds

- Add cloud_mode? helper to detect when running with VIZZLY_TOKEN - Add assert_screenshot_success helper for mode-aware assertions - TDD mode: asserts status is 'new' or 'match' - Cloud mode: asserts success is true (no local comparison)

Switch from blacksmith to ubuntu-latest for all CI workflows since this is an open source project with free GitHub Actions minutes. Keeps blacksmith only for release workflows (internal).

vizzly-testing · 2026-01-19T04:15:54Z

🐻 Vizzly — Approved ✓

5 comparisons approved.

Status	Count
Changed	1
Auto-approved	4

View build →

_{CLI TUI · fix-sdk-e2e-tests · 29e55456}

vizzly-testing · 2026-01-19T04:16:06Z

🐻 Vizzly — Approved ✓

19 comparisons approved.

Status	Count
Passed	6
Changed	13
Auto-approved	6

View build →

_{CLI Reporter · fix-sdk-e2e-tests · 29e55456}

Robdel12 force-pushed the fix-sdk-e2e-tests branch from 8369a5a to 31d6df2 Compare January 17, 2026 07:42

Robdel12 force-pushed the fix-sdk-e2e-tests branch from 31d6df2 to 890d746 Compare January 17, 2026 07:48

Robdel12 force-pushed the fix-sdk-e2e-tests branch from 890d746 to 242af43 Compare January 17, 2026 07:59

This comment was marked as outdated.

Sign in to view

This comment has been minimized.

Sign in to view

Robdel12 force-pushed the fix-sdk-e2e-tests branch from defd76a to e30998d Compare January 19, 2026 03:25

This comment has been minimized.

Sign in to view

Robdel12 force-pushed the fix-sdk-e2e-tests branch from 75ebbf4 to c8b8462 Compare January 19, 2026 04:07

This comment has been minimized.

Sign in to view

Robdel12 force-pushed the fix-sdk-e2e-tests branch from c8b8462 to 4bde262 Compare January 19, 2026 04:09

This comment has been minimized.

Sign in to view

♻️ Use GitHub-hosted runners for CI workflows

29e5545

Switch from blacksmith to ubuntu-latest for all CI workflows since this is an open source project with free GitHub Actions minutes. Keeps blacksmith only for release workflows (internal).

Robdel12 enabled auto-merge (squash) January 19, 2026 04:16

Robdel12 merged commit c203bfa into main Jan 19, 2026
32 of 33 checks passed

Robdel12 deleted the fix-sdk-e2e-tests branch January 19, 2026 04:24

🧪 Fix SDK E2E tests to work with vizzly tdd run #183

🧪 Fix SDK E2E tests to work with vizzly tdd run #183

Conversation

Robdel12 commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

CI Workflow Reorganization

Ruby SDK Fixes

Vitest SDK Fixes

All SDKs

Test plan

Uh oh!

vizzly-testing bot commented Jan 17, 2026

🐻 Vizzly — 7 changes need review

Uh oh!

vizzly-testing bot commented Jan 17, 2026

🐻 Vizzly — 1 change needs review

Uh oh!

claude bot commented Jan 17, 2026

Code Review - PR #183

Strengths

Issues Found

CRITICAL

WARNINGS

SUGGESTIONS

Performance

Verdict

Uh oh!

vizzly-testing bot commented Jan 17, 2026

🐻 Vizzly — 5 changes need review

Uh oh!

vizzly-testing bot commented Jan 17, 2026

🐻 Vizzly — 1 change needs review

Uh oh!

vizzly-testing bot commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🐻 Vizzly — Approved ✓

Uh oh!

vizzly-testing bot commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🐻 Vizzly — Approved ✓

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

vizzly-testing bot commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🐻 Vizzly — Approved ✓

Uh oh!

vizzly-testing bot commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🐻 Vizzly — Approved ✓

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Robdel12 commented Jan 17, 2026 •

edited

Loading

vizzly-testing bot commented Jan 17, 2026 •

edited

Loading

vizzly-testing bot commented Jan 17, 2026 •

edited

Loading

vizzly-testing bot commented Jan 19, 2026 •

edited

Loading

vizzly-testing bot commented Jan 19, 2026 •

edited

Loading