fixes exec queue add and pop #7591

AdoAdoAdo · 2026-01-05T14:04:22Z

Reasoning behind the pull request

Proposed changes

Testing procedure

Pre-requisites

Based on the Contributing Guidelines the PR author and the reviewers must check the following requirements are met:

was the PR targeted to the correct branch?
if this is a larger feature that probably needs more than one PR, is there a feat branch created?
if this is a feat branch merging, do all satellite projects have a proper tag inside go.mod?

Copilot

Pull request overview

This PR fixes issues with the execution queue's Add and Pop operations, addressing concurrency problems and improving error handling in the async execution system.

Key Changes

Fixed the Pop() method condition from > 1 to >= 1 to correctly handle single-item queues
Updated AddOrReplace notification logic to only notify when the first item is added to an empty queue
Added ValidateQueueIntegrity() method for periodic queue health checks
Improved error handling with proper error propagation and retry logic with exponential backoff
Enhanced error handling in baseSync.go using errors.Is() instead of direct comparison

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
process/asyncExecution/queue/blocksQueue.go	Fixed Pop() condition bug, restructured notification logic, added ValidateQueueIntegrity() method, and enhanced logging
process/asyncExecution/headersExecutor.go	Added retry mechanism with max attempts and exponential backoff, periodic validation ticker, nil checks for headers/bodies, and fixed error propagation
process/sync/baseSync.go	Updated error comparison to use errors.Is() and ensured deferred error handling works correctly
process/asyncExecution/queue/errors.go	Removed unused ErrMissingHeaderNonce and added ErrQueueIntegrityViolation
process/asyncExecution/errors.go	Added new error types for nil handlers and execution results
process/interface.go	Added ValidateQueueIntegrity() to BlocksQueue interface
process/asyncExecution/interface.go	Added ValidateQueueIntegrity() to BlocksQueue interface
testscommon/processMocks/blocksQueueMock.go	Added ValidateQueueIntegrityCalled field and method to mock
process/asyncExecution/queue/blocksQueue_test.go	Added comprehensive test coverage for ValidateQueueIntegrity() and concurrent operations
process/asyncExecution/headersExecutor_test.go	Fixed test expectations to properly validate error propagation

Comments suppressed due to low confidence (1)

process/asyncExecution/headersExecutor.go:143

This select statement with a default case creates a busy-wait loop when the validation ticker doesn't fire. The default case will be executed continuously, causing high CPU usage even when the queue is empty and Pop() is blocking. Consider removing the default case or adding a small sleep in the default branch to avoid busy-waiting. The ticker and Pop() blocking should provide sufficient responsiveness without the default case.

		select {
		case <-ctx.Done():
			return
		case <-validationTicker.C:
			// Periodic queue validation
			err := he.blocksQueue.ValidateQueueIntegrity()
			if err != nil {
				log.Error("headersExecutor.start: queue integrity validation failed", "err", err)
			}
		default:
			he.mutPaused.RLock()
			isPaused := he.isPaused
			he.mutPaused.RUnlock()

			if isPaused {
				time.Sleep(timeToSleep)
				continue
			}

			// blocking operation
			headerBodyPair, ok := he.blocksQueue.Pop()
			if !ok {
				log.Debug("headersExecutor.start: not ok fetching from queue")
				// close event
				return
			}

			if check.IfNil(headerBodyPair.Header) || check.IfNil(headerBodyPair.Body) {
				log.Debug("headersExecutor.start: popped nil header or body, continuing...")
				continue
			}

			err := he.process(headerBodyPair)
			if err != nil {
				he.handleProcessError(ctx, headerBodyPair)
			}
		}

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

process/asyncExecution/headersExecutor.go

process/asyncExecution/queue/blocksQueue_test.go

process/asyncExecution/queue/blocksQueue.go

process/asyncExecution/headersExecutor.go

sstanculeanu · 2026-01-05T14:32:08Z

process/asyncExecution/headersExecutor.go

 	if err != nil {
 		log.Warn("headersExecutor.process process block failed",
 			"nonce", pair.Header.GetNonce(),
+			"hash", pair.Header.GetPrevHash(),


Suggested change

"hash", pair.Header.GetPrevHash(),

"prevHash", pair.Header.GetPrevHash(),

ssd04 · 2026-01-05T15:06:49Z

process/asyncExecution/headersExecutor.go

+	// Validate input parameters
+	if check.IfNil(pair.Header) {


don't think its really needed, there are already some nil checks before process call

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-08T08:53:57Z

process/asyncExecution/headersExecutor.go

+	if check.IfNil(executionResult) {
+		log.Warn("headersExecutor.process - nil execution result received",
+			"nonce", pair.Header.GetNonce())
+		return ErrNilExecutionResult
+	}


The new nil execution result check added in the process method (lines 203-207) lacks test coverage. There is no test case that verifies the behavior when ProcessBlockProposal returns a nil execution result with no error, which would trigger the ErrNilExecutionResult error path.

Copilot · 2026-01-08T08:53:57Z

process/asyncExecution/headersExecutor.go

+	retryCount := 0
+	backoffTime := timeToSleepOnError
+
+	for retryCount < maxRetryAttempts {
 		pairFromQueue, ok := he.blocksQueue.Peek()
 		if ok && pairFromQueue.Header.GetNonce() == pair.Header.GetNonce() {
 			// continue the processing (pop the next header from queue)
 			return
 		}
+
 		select {
 		case <-ctx.Done():
 			return
 		default:
 			// retry with the same pair
 			err := he.process(pair)
 			if err == nil {
+				log.Debug("headersExecutor.handleProcessError - retry succeeded",
+					"nonce", pair.Header.GetNonce(),
+					"retry_count", retryCount)
 				return
 			}
-			time.Sleep(timeToSleepOnError)
+			retryCount++
+			log.Warn("headersExecutor.handleProcessError - retry failed",
+				"nonce", pair.Header.GetNonce(),
+				"retry_count", retryCount,
+				"max_retries", maxRetryAttempts,
+				"err", err)
+
+			// Exponential backoff with maximum limit
+			time.Sleep(backoffTime)
+			backoffTime = backoffTime * 2
+			if backoffTime > maxBackoffTime {
+				backoffTime = maxBackoffTime
+			}
 		}
 	}
+
+	log.Error("headersExecutor.handleProcessError - max retries exceeded, skipping block",
+		"nonce", pair.Header.GetNonce(),
+		"max_retries", maxRetryAttempts)


The new retry logic with exponential backoff and max retry limits in handleProcessError lacks comprehensive test coverage. There are no tests verifying that the retry count increments correctly, exponential backoff works as expected, or that max retries are enforced and properly logged.

Copilot · 2026-01-08T08:53:58Z

process/asyncExecution/errors.go

+// ErrNilHeaderHandler signals that a nil header handler has been provided
+var ErrNilHeaderHandler = errors.New("nil header handler")
+
+// ErrNilBlockBody signals that a nil block body has been provided
+var ErrNilBlockBody = errors.New("nil block body")
+


The error variables ErrNilHeaderHandler and ErrNilBlockBody are defined but never used in the codebase. The code uses common.ErrNilHeaderHandler and data.ErrNilBlockBody instead (as seen in blocksQueue.go lines 42 and 45). These unused error definitions should be removed to avoid confusion.

Suggested change

// ErrNilHeaderHandler signals that a nil header handler has been provided

var ErrNilHeaderHandler = errors.New("nil header handler")

// ErrNilBlockBody signals that a nil block body has been provided

var ErrNilBlockBody = errors.New("nil block body")

Copilot · 2026-01-08T08:53:58Z

process/asyncExecution/queue/blocksQueue.go

+		// Clear any stale notification from the channel when we take the fast path
+		select {
+		case <-bq.notifyCh:
+		default:
+		}


Clearing the notification channel on the fast path (lines 146-149) could lead to a lost wakeup problem. If AddOrReplace sends a notification while Pop is processing (after checking the length but before clearing the channel), that notification will be consumed and discarded, potentially causing another waiting Pop call to miss its wakeup. Consider only clearing the channel when the queue becomes empty after popping.

Copilot · 2026-01-08T08:53:58Z

process/asyncExecution/headersExecutor.go

+	validationTicker := time.NewTicker(validationInterval)
+	defer validationTicker.Stop()
+
 	for {
 		select {
 		case <-ctx.Done():
 			return
+		case <-validationTicker.C:
+			// Periodic queue validation
+			err := he.blocksQueue.ValidateQueueIntegrity()
+			if err != nil {
+				log.Error("headersExecutor.start: queue integrity validation failed", "err", err)
+			}


The periodic validation in the select statement may not execute as expected. The default case will be taken immediately when no other channels are ready, causing the loop to continuously process without giving the validation ticker a chance to fire. Consider removing the default case and using a non-blocking approach or restructuring the select to prioritize validation checks.

process/asyncExecution/headersExecutor.go

process/sync/baseSync.go

The base branch was changed.

codecov · 2026-01-08T10:27:48Z

Codecov Report

❌ Patch coverage is 82.05128% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.69%. Comparing base (dbae8d7) to head (4e40327).

Files with missing lines	Patch %	Lines
process/asyncExecution/headersExecutor.go	60.60%	11 Missing and 2 partials ⚠️
process/asyncExecution/queue/blocksQueue.go	97.50%	1 Missing ⚠️

Additional details and impacted files

@@                    Coverage Diff                     @@
##           feat/supernova-async-exec    #7591   +/-   ##
==========================================================
  Coverage                      77.69%   77.69%           
==========================================================
  Files                            876      876           
  Lines                         121012   121077   +65     
==========================================================
+ Hits                           94015    94066   +51     
- Misses                         20790    20805   +15     
+ Partials                        6207     6206    -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

fixes exec queue add and pop

4a3db54

AdoAdoAdo requested a review from Copilot January 5, 2026 14:07

Copilot started reviewing on behalf of AdoAdoAdo January 5, 2026 14:07 View session

Copilot AI reviewed Jan 5, 2026

View reviewed changes

sstanculeanu approved these changes Jan 5, 2026

View reviewed changes

ssd04 approved these changes Jan 5, 2026

View reviewed changes

fixes

60655ce

AdoAdoAdo requested a review from Copilot January 8, 2026 08:47

Copilot started reviewing on behalf of AdoAdoAdo January 8, 2026 08:47 View session

ssd04 previously approved these changes Jan 8, 2026

View reviewed changes

Copilot AI reviewed Jan 8, 2026

View reviewed changes

sstanculeanu previously approved these changes Jan 8, 2026

View reviewed changes

clean channel only when queue becomes empty

207937d

Base automatically changed from fix-tx-pool-restore-on-bootstrap to feat/supernova-async-exec January 8, 2026 10:14

An error occurred while trying to automatically change base from fix-tx-pool-restore-on-bootstrap to feat/supernova-async-exec January 8, 2026 10:14

Merge branch 'feat/supernova-async-exec' into fix-race-execution

82892b5

sstanculeanu approved these changes Jan 8, 2026

View reviewed changes

AdoAdoAdo added 2 commits January 8, 2026 15:19

Merge branch 'feat/supernova-async-exec' into fix-race-execution

09ba7f3

Merge branch 'feat/supernova-async-exec' into fix-race-execution

4e40327

	"hash", pair.Header.GetPrevHash(),
	"prevHash", pair.Header.GetPrevHash(),

fixes exec queue add and pop #7591

Are you sure you want to change the base?

fixes exec queue add and pop #7591

Uh oh!

Conversation

AdoAdoAdo commented Jan 5, 2026

Reasoning behind the pull request

Proposed changes

Testing procedure

Pre-requisites

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sstanculeanu Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

AdoAdoAdo Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

ssd04 Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AdoAdoAdo Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

AdoAdoAdo Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ssd04 Jan 5, 2026 •

edited

Loading

codecov bot commented Jan 8, 2026 •

edited

Loading