fix: validate scope checks ran and report overall test status #80
+99
−7
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add FAILURE checks to scope scenarios when authorization flow doesn't complete (ScopeFromWwwAuthenticateScenario, ScopeFromScopesSupportedScenario, ScopeOmittedWhenUndefinedScenario) - Report client timeout and exit errors in test results - Add OVERALL: PASSED/FAILED summary at end of test output - Exit with code 1 on any failure (check failures, client timeout, or exit error) Previously, if the OAuth authorization flow didn't complete, the scope checks were never triggered and the test would incorrectly report success. Now these scenarios emit a FAILURE if the expected scope check wasn't recorded. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
commit: |
Warnings now contribute to the overall failure status, causing the test to exit with code 1 when any warnings are present. Also displays warning checks in the output summary, similar to how failures are displayed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Update the suite summary to show ✗ for scenarios with warnings, and exit with code 1 if any warnings are present across the suite. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
felixweinberger
approved these changes
Dec 16, 2025
| (c) => c.id === 'scope-from-www-authenticate' | ||
| ); | ||
| if (!hasScopeCheck) { | ||
| this.checks.push({ |
Collaborator
There was a problem hiding this comment.
nit: mutations in a getSomething might be slightly weird, but I deduplication above makes it idempotent at least.
| const { overallFailure } = printClientResults( | ||
| result.checks, | ||
| verbose, | ||
| result.clientOutput |
Collaborator
There was a problem hiding this comment.
more of a suggestion: IIUC we're only showing these output summaries for single scenarios, could consider doing that for test suites as well.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Problem
Previously, if the OAuth authorization flow didn't complete (e.g., client timed out or exited with an error before the authorization request reached the server), the scope checks were never triggered and the test would incorrectly report success like "Passed: 6/6, 0 failed".
Solution
The following scope scenarios now emit a FAILURE if their expected scope check wasn't recorded:
ScopeFromWwwAuthenticateScenarioScopeFromScopesSupportedScenarioScopeOmittedWhenUndefinedScenarioAdditionally,
printClientResultsnow:clientOutputto check for timeout/exit errorsoverallFailureboolean for proper exit codeExample output (failure case)
🤖 Generated with Claude Code