Reindex logging and perf enhancements #5214
Open
+403
−72
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added Performance logging in ReindexProcessingJob Added parallel processing of resources in ReindexProcessingJob increasing runtimes 2x per job Added IgnoreSearchParamHash to queryHintParameterNames collection in SearchOptionsFactory to prevent error during reindexing queries
jestradaMS
added
Enhancement-Optimization
PTaladay
previously approved these changes
Oct 30, 2025
… counters Fixing logging in ReindexProcessingJob
SergeyGaluzo
reviewed
Nov 1, 2025
|
|
||
| private async Task<SearchResult> GetResourcesToReindexAsync(SearchResultReindex searchResultReindex, CancellationToken cancellationToken) | ||
| { | ||
| var stopwatch = Stopwatch.StartNew(); |
There was a problem hiding this comment.
My problem with detailed time logging is thar it makes a lot of clutter and makes main code not readable.
SergeyGaluzo
reviewed
Nov 1, 2025
| entry.Resource.SearchParameterHash = searchParamHash; | ||
| _resourceWrapperFactory.Update(entry.Resource); | ||
| updateSearchIndices.Add(entry.Resource); | ||
| // Parallel processing of search parameter extraction |
There was a problem hiding this comment.
As discussed offline, we should not do this.
SergeyGaluzo
requested changes
Nov 1, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This pull request introduces significant improvements to logging, error handling, and performance instrumentation in the reindex job orchestration and processing workflow. The changes provide detailed timing and progress information for each stage of the reindex process, enhance cancellation and exception handling, and optimize resource processing through parallelization. These updates will make it easier to monitor, debug, and optimize reindex jobs.
Logging and Performance Instrumentation:
ReindexProcessingJob.cs. Each major step now records elapsed time and throughput metrics for better observability. [1] [2] [3] [4] [5] [6] [7]Error and Cancellation Handling:
OperationCanceledExceptionin both orchestrator and processing jobs, ensuring logs accurately reflect whether cancellation was due to a caller request or a timeout, and added additional error logging for unexpected exceptions. [1] [2] [3] [4]Performance Optimization:
ProcessSearchResultsAsyncto use parallel processing withConcurrentBag, leveraging all available CPU cores for faster search parameter extraction and batch preparation. Added detailed logging for extraction and batch update timings. [1] [2]Minor Improvements:
KnownQueryParameterNames.IgnoreSearchParamHashto the set of query hint parameter names inSearchOptionsFactory.cs, potentially enabling new query behaviors.Related issues
Addresses AB#175355.
Testing
<style> </style>FHIR Team Checklist
Semver Change (docs)
Patch|Skip|Feature|Breaking (reason)