Skip to content

Conversation

@jfet97
Copy link
Collaborator

@jfet97 jfet97 commented Dec 8, 2025

What

Adds a new validateSample() method to the Repository interface that validates a random sample of repository items by applying jitM transformation and schema decode. This enables testing that migrations and schema changes work correctly on existing data without validating the entire dataset.

Why

When performing migrations or schema updates, there's a need to verify that the transformation logic (jitM) and new schema constraints properly handle existing data. Rather than validating all items (which could be expensive), sampling allows quick validation of data integrity while keeping runtime costs low.

How

  • New ValidationError and ValidationResult schema classes in validation.ts capture detailed error context
  • validateSample method accepts optional percentage (default 10%) and maxItems parameters to control sample size
  • For each sampled item:
    1. Retrieves raw data from store without triggering main schema decode
    2. Applies jitM transformation
    3. Attempts schema decode
    4. Records failures with full context: id, rawData, jitMResult, and ParseError
  • Returns ValidationResult with total count, sampled count, valid count, and detailed errors list
  • Comprehensive test coverage including success cases, jitM corruption scenarios, maxItems capping, and error context validation

@jfet97 jfet97 changed the title feat(infra): add validateSample method to Repository for schema valid… feat(infra): add validateSample method to Repository Dec 8, 2025
@jfet97 jfet97 merged commit 5777431 into main Dec 8, 2025
3 checks passed
@jfet97 jfet97 deleted the f/test-jitM-samples branch December 8, 2025 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants