Skip to content

Conversation

@philippta
Copy link

@philippta philippta commented Jan 6, 2026

This PR introduces a way to gradually roll out the new certificate storage format. Instead of switching every domain to the new system at once, we can now move them over in small groups to ensure everything is working correctly.

Percentage-Based Rollout

  • The new setting, CERTMAGIC_STORAGE_MODE_ROLLOUT_PERCENT allows us to control the amount of domains which enter "transition" mode.
  • CertMagic looks at each domain name and assigns it to a "bucket" from 0 to 100 using a hash function.
  • So when can set the rollout to e.g. 10, only 10% of all domains fall into the "rollout buckets" and enter transition mode.
  • Because this is based on the domain name itself, the same domain will always land in the same bucket.

Why this is helpful

  • We can start with a 1% rollout to monitor for any storage errors before committing to 100%.
  • Since the decision is deterministic (based on the name), a domain won't accidentally flip-flop between formats.
  • If we don't set a percentage, it defaults to 0. This means no behavior changes for our existing setup until we are ready to start the migration.

Notes

  • We should only ever increase the rollout percentage, never decrease, to avoid inconsistencies bundle formats.

Proof of work

ROLLOUT_PERCENT = 40

  1. Domains in buckets < 40 go into transition mode
  2. Domains in buckets >= 50 stay in legacy mode
image

ROLLOUT_PERCENT = 0

  1. All domains stay in legacy mode
image

ROLLOUT_PERCENT = 100

  1. All domains go into transition mode
image

@philippta philippta force-pushed the feature/granular-rollout branch from fce1213 to 9dee912 Compare January 6, 2026 09:26
@philippta
Copy link
Author

@cursor review

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment @cursor review or bugbot run to trigger another review on this PR

@philippta
Copy link
Author

@cursor review

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment @cursor review or bugbot run to trigger another review on this PR

@philippta philippta force-pushed the feature/granular-rollout branch from 851944a to e600a41 Compare January 6, 2026 10:01
@philippta
Copy link
Author

@cursor review

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment @cursor review or bugbot run to trigger another review on this PR

@philippta
Copy link
Author

@cursor review

@philippta philippta marked this pull request as ready for review January 6, 2026 15:53
Log domain, storage_mode, and rollout_bucket at each storage mode
decision point to help observe the gradual rollout behavior.
@philippta
Copy link
Author

@cursor review

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no bugs!


Comment @cursor review or bugbot run to trigger another review on this PR

Replace brute-force test approach with targeted test cases that verify
behavior at rollout boundaries (0%, 1%, 50%, 99%, 100%). Remove helper
functions and large domain list that were only used for test generation.
@philippta philippta merged commit 90e6b2c into master Jan 8, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants