Refactors the randomness service to be able to process pending commitments

[GabrielMartinezRodriguez]

Linked Issues

• closes https://linear.app/happychain/issue/HAPPY-255/process-all-pending-commitments-to-be-revealed-not-just-the-one-from

Description

Refactors the randomness service to be able to process pending commitments instead of only the last commitment

• Introduces a Randomness entity with clear state transitions and business logic
• Replaces CommitmentManager with RandomnessRepository for better data management
• Adds status tracking for both commitment and reveal transactions
• Implements proper error handling for failed transactions
• Adds transaction status change hooks to update randomness states
• Updates database schema to track both commitment and reveal transaction states
• Adds new Makefile commands for starting the service and running migrations

Toggle Checklist

Checklist

Basics

• B1. I have applied the proper label & proper branch name
• B2. This PR is not so big that it should be split & addresses only one concern
• B3. The PR targets the lowest branch it can (ideally master)

Correctness

• C1. Builds and passes tests

• C2. The code is properly parameterized & compatible with different environments

• C3. I have manually tested my changes & connected features

• Local environment

• Tested commitment and reveal flows, including error scenarios and state transitions

• C4. I have performed a thorough self-review of my code

Architecture & Documentation

• D1. I made it easy to reason locally about the code
• D2. All public-facing APIs & meaningful internal APIs are properly documented
• D3. If appropriate, the general architecture of the code is documented

[cloudflare-workers-and-pages]

Deploying happychain with    Cloudflare Pages

Latest commit:	1dfe4da
Status:	✅  Deploy successful!
Preview URL:	https://32dfccb3.happychain.pages.dev
Branch Preview URL:	https://gabriel-pending-commitments.happychain.pages.dev

View logs

[linear]

HAPPY-257 Create OnNewBlock hook

This hook will allow users to execute code on every new block without having to place that code inside the transaction collector or use another web3 client.

In addition, after implementing this hook, we will have to move the randomness-service prune database function to this new hook and mark as expired the randomness for which the commitment has been submitted but not revealed, and it has already expired

[GabrielMartinezRodriguez]

• Deploy randomness service #416 : 2 dependent PRs (#434 , #441 )
• Add config contract #404
• Randomness service: Improve performance and reliability #372
• Add transaction submission failure hook and error handling #366
• Increase default RPC timeout to 2000ms #365
• Add configurable HTTP polling interval for block monitoring #362
• Force initialization before using the TXM #361
• Script to monitor randomness service #325
• Script to easily launch the randomness service locally #324
• Submit drand numbers #318
• Random oracle contracts #307
• Only prune finalized randomness entries #305
• Add NewBlock hook #304
• Refactors the randomness service to be able to process pending commitments  #303 👈 (View in Graphite)
• Add SQLite persistence to randomness service #300
• Txm no longer receives Viem objects #298 : 1 other dependent PR (#299 )
• Add flush mechanism #289 : 1 other dependent PR (#297 )
• Allow run multiple txm simultaneously within the same service #285
• Added docs for txm #272
• Mutex in tx monitor #264
• Purge transactions #232
• Txm hooks #229
• Use kysely instead of mikro-orm #250
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[linear]

HAPPY-255 Process all pending commitments to be revealed, not just the one from the last block

Commitments should have a field indicating the status of the submitted commitment. For example:

• Commitment Submitted
• Commitment Successfully Executed
  ... etc.

[aodhgan]

if the db is large, wont this load a huge amount of values into memory? or do i misunderstand

[GabrielMartinezRodriguez]

Yes, it would happen. However, this database is going to be small because we only save the randomness of the last two minutes, so it will contain approximately 60 rows

[not-reed]

for a long running process, i see the database is pruned, but not this map yet, correct?

[norswap]

This will actually grow pretty large — we need to register commitments 12 hours in advance to make sure the sequencer is unable to reorg the chain before posting the data to DA. We might be able to lower that number.

But even so, for 14 hours worth of commitments, with 3 commitments per second (assuming the faster future block times) and a memory footprint of 500 bytes per Randomness object (probably too high) we have 12*60*60*3*500 bytes = ~64 megabytes, so I think this should be quite manageable. It's ~130k rows, which is also well within manageable boundaries I think.

I wonder about the performance of pruning however, since it needs to run the predicate on all randomnesses (both in memory and in the DB).

I don't think the DB will be smart enough to do this, because it would need to maintain a sorted index of the transactions, then find the cutoff point based on the condition. Note that we might also need to maintain a sort list of timestamps (this will need to be block numbers in the future btw, as there could be multiple blocks per timestamp) in-memory to do this.

The alternative is to get a set of timestamps (block numbers) and delete these specifically, or even prune blocks one by one on expiry. Pruning would not happen when we are offline, but we can prune once when we load the DB.

Let's add issues to:

• change the primary identification of a randomness to be the block number instead of the timestamp
• handle the performance challenges of pruning when there is a large number of randomness

[GabrielMartinezRodriguez]

I've already updated this to work with blocks instead of timestamps in this PR: #318.

Regarding the performance issues, I created this issue: https://linear.app/happychain/issue/HAPPY-305/handle-the-performance-challenges-of-pruning-when-there-is-a-large

[not-reed]

nice and clear, i like it

[norswap]

Only thing left here is discuss the DB scenario, I think we figure it out and open an issue to handle it later. Let's merge once that is done.