-
Notifications
You must be signed in to change notification settings - Fork 70
PTC Cache Implementation #488
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: epbs
Are you sure you want to change the base?
PTC Cache Implementation #488
Conversation
* Add Gloas helper, state transition functions * Add DataColumnSidecar as Enum * Add Gloas attestation gossipsub rules * Extract slot bytes for data column sidecar, deal with zero bid correctly * apply suggestions
52fb2d5 to
f36e5d4
Compare
* Add Gloas PTC assignment and payload attestation flow * Extend `TickKind` to support `PayloadAttestation` for Gloas * Payload attestation validation * Add payload attestation aggregation pool * Prepare payload attestations in block producer
3448f84 to
3dfd9d9
Compare
Adds PTC caching for Gloas fork. Stores entire epoch. Lazy init via OnceCell.
- stores all occurrences for validators with multiple ptc slots. Changes ptc_positions to Vec<Vec<(Slot, usize)>>. - higher memory usage.
- Hybrid approach: Vec for single occurrences, HashMap for multiples. - build with pre-counting. - Better memory efficiency.
- Balance-weighted selection depends on effective_balance which changes every epoch. unlike other existing cache object(s). - Only cache Current epoch, clear on transition. - Reduces memory usage. - committee cache keeps multiple epochs (balance dosent affect).
3dfd9d9 to
a61e3f5
Compare
a61e3f5 to
f5783b6
Compare
|
@hangleang can you have a look at this. i am planning to get these merged to be used in validator duty calls |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you simply explain what it does? what technique you use here to manage the cache? I can't fully wrap my head around
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hey @hangleang so the goal is to reduce the calls and do the actual calculation once per epoch.
So it operates like a regular cache. But the added element is that the balance_weighted_selection. So that's on top of seed and validator indices.
Call Flow
` get_ptc(state, slot)
↓
state.ptc_cache().get_or_init() ← OnceCell: thread-safe lazy init
↓
build_ptc_cache(state, epoch)
↓
for each slot in epoch:
compute_ptc_for_slot_internal() ← Calls balance_weighted_selection
↓
Store in ptc_shuffling: [slot0: 512 validators, slot1: 512, ..., slot31: 512]
↓
Build reverse indexes for validator→slots lookup `
so in first commit i made 2 mistakes 1. keeping a cache for 3 epochs but thats wrong as balance can change per epoch. 2. didnt handle multiple occurance of single validator for reverse lookup ptc_positions(used in get_validator_ptc_slots which answers which slots is this validator assigned).
in 2nd commits. instead of keeping the track of validators position in a vec i kept in a vec<vec<>> to give multiple positons per validator.
in 3rd commit , instead of creating vec vec for every validator, i changed to maintaining 2 lookups oneptc_positions for single occurance and other multiple_positions for validators with multiple occurances. thus reducing the chances of 2 vector overhead for every single validator index. manged by a hash map with vec of position of slots as value(validator index as key)
in 4th commit, i fixed my mistake of keeping/precalculating ptc for multiple epochs assuming that balance would remain same across multiple epochs.
the whole constraint of build_ptc_cache will be need once per epoch is managed by the cache clearing logic by calling clear_ptc_cache in process_epoch.
- Forward index (get_ptc): slot → validators
- Uses ptc_shuffling (flat Vec)
- "answers who are the ptc members for slot x" - Reverse index (get_validator_ptc_slots): validator → slots
- Uses ptc_positions + multiple_positions
- "validator v is assigned to which slots"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hey @Subhasish-Behera, I like the idea to cache PTC members for an epoch which can be used in validate_payload_attestation. currently, we do have own_ptc_members.rs which keep track of assigned slots for validators attached to the beacon node (we don't need to keep track other validators assigned slots, which is non of the beacon node business), so I think we don't need the reverse lookup per validator in this cache, this way we can make the cache less complex. I also don't think keep ptc cache within state is the right place, the best place might be in fork_choice_store
968d5ac to
2a96e60
Compare
Implements caching for Payload Timeliness Committee (PTC) to avoid recomputing balance-weighted selection multiple times per epoch.
Problem: PTC uses expensive balance-weighted selection. Without caching, the same committee is recomputed for every
validator duty check and every attestation verification.
Solution: Cache the entire epoch's PTC with constant time validator lookup.
New Functions:
get_validator_ptc_slots(validator_index)- Returns all (slot, position) pairs where validator appears.will be used in validator duty calls to know which slots to attest.
get_ptc(state, slot)- Returns full PTC committee for a slot. Used for gossip verification, block processing.