Skip to content

Commit ad0e693

Browse files
Add Unique Test Instances feature to analytics documentation
- Introduced the **Show Unique Instances** toggle in both Build Insights and Test Insights documentation to enhance clarity on how retry runs are consolidated for more accurate test results. - Updated explanations on the functionality and processing time considerations for the new feature, ensuring users understand its impact on reporting and metrics.
1 parent 881f414 commit ad0e693

File tree

2 files changed

+36
-13
lines changed

2 files changed

+36
-13
lines changed

docs/analytics-build-insights.md

Lines changed: 18 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,24 @@ Use filters to narrow analysis to exactly the slice you care about (for example,
136136
- Click the share icon next to the filters to share the current view with your team
137137
- Generate shareable links to the build details page (note: filter settings are not preserved in shared links)
138138

139+
### Unique Test Instances
140+
141+
The **Show Unique Instances** toggle consolidates retry runs to give you a cleaner, more accurate view of your test results. This feature applies to both the **Insights** and **Tests** tabs.
142+
143+
**How it works:**
144+
145+
- When **ON**: Within the current build, tests are grouped by **test name + environment** (browser + OS + device + resolution) as a single instance. Only the **final run** of each instance is considered in reporting, eliminating noise from intermediate retry attempts. This affects:
146+
- **Insights tab**: Key metrics, charts, and Smart Tags reflect deduplicated counts based on final run results
147+
- **Tests tab**: The test list shows only the final execution per unique test-environment combination
148+
149+
- When **OFF**: All individual test executions are shown, including every retry attempt.
150+
151+
The grouping is scoped to the individual build ID, meaning each build's metrics reflect only the deduplicated results within that specific build run.
152+
153+
:::note Processing Time
154+
Retry run consolidation requires a small amount of processing time after test execution completes. If you've just finished a build, wait a moment before toggling on Unique Instances to ensure all data is consolidated.
155+
:::
156+
139157
## Tab 1: Insights
140158

141159
Use the **Insights** tab to understand the overall health and performance of the selected build before you dive into individual tests.
@@ -200,19 +218,6 @@ Each metric points you directly to tests that need attention (for example, focus
200218

201219
Use the **Tests** tab when you are ready to debug at the individual test level.
202220

203-
### Show Unique Instances Toggle
204-
205-
The **Show Unique Instances** toggle consolidates retry runs to give you a cleaner view of your test results.
206-
207-
**How it works:**
208-
209-
- When **ON**: Tests are grouped by **test name + environment** (browser + OS + device + resolution) as a single instance. Only the **final run** of each instance is considered in reporting, eliminating noise from intermediate retry attempts.
210-
- When **OFF**: All individual test executions are shown, including every retry attempt.
211-
212-
:::note Processing Time
213-
Retry run consolidation requires a small amount of processing time after test execution completes. If you've just finished a build, wait a moment before toggling on Unique Instances to ensure all data is consolidated.
214-
:::
215-
216221
:::tip Build Comparison
217222
Want to compare two builds side by side? Use the **Compare** tab to identify new failures, fixed tests, and stability changes between any two builds. This is especially useful for release validation and regression detection. Learn more in the [Build Comparison](/support/docs/analytics-build-comparison/) documentation.
218223
:::

docs/analytics-test-insights.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,24 @@ To get started with Test Insights, follow these steps:
6868
- **Actionable Analytics**: Gain actionable insights from your test data to make informed decisions and improve the overall quality of your software.
6969
- **Historical Data Comparison**: Compare your current test runs with historical data to identify regressions and improvements over time.
7070

71+
## Unique Test Instances
72+
73+
The **Show Unique Instances** toggle consolidates retry runs to give you a cleaner, more accurate view of your test results.
74+
75+
**How it works:**
76+
77+
- When **ON**: Within a single build run, tests are grouped by **test name + environment** (browser + OS + device + resolution) as a single instance. Only the **final run** of each instance is considered in reporting, eliminating noise from intermediate retry attempts.
78+
79+
- When **OFF**: All individual test executions are shown, including every retry attempt.
80+
81+
The grouping is scoped to individual build IDs, meaning each build's metrics reflect only the deduplicated results within that specific build run.
82+
83+
This feature is especially useful when your test pipeline uses automatic retries, as it shows the true pass/fail state of each test-environment combination rather than inflating counts with intermediate failures.
84+
85+
:::note Processing Time
86+
Retry run consolidation requires a small amount of processing time after test execution completes. If you've just finished running tests, wait a moment before toggling on Unique Instances to ensure all data is consolidated.
87+
:::
88+
7189
## Use Cases for Test Insights
7290

7391
- **Test Optimization**: Use Test Insights to identify slow-running tests and optimize them for better performance.

0 commit comments

Comments
 (0)