Feature: Add Redis search module indexes metrics collection #1046

nantiferov · 2025-10-07T05:40:27Z

Add Redis search module indexes metrics collection
Add tests for Redis search module indexes metrics collection
Add valkey-bundle to tests search indexes collection

nantiferov · 2025-10-07T05:41:53Z

So far in draft mode as unfinished.

I believe I hit limitations of ScanStruct() gomodule/redigo#691

codecov · 2025-10-07T05:45:26Z

Codecov Report

❌ Patch coverage is 80.51948% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.93%. Comparing base (5659898) to head (bc95bbd).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
exporter/search_indexes.go	76.08%	8 Missing and 3 partials ⚠️
main.go	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1046      +/-   ##
==========================================
- Coverage   81.96%   81.93%   -0.04%     
==========================================
  Files          19       20       +1     
  Lines        3017     3094      +77     
==========================================
+ Hits         2473     2535      +62     
- Misses        431      443      +12     
- Partials      113      116       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

coveralls · 2025-10-07T05:45:37Z

Pull Request Test Coverage Report for Build 18565399734

Details

65 of 77 (84.42%) changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.03%) to 85.723%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
main.go	0	4	0.0%
exporter/search_indexes.go	38	46	82.61%

Totals
Change from base Build 18485080311:	-0.03%
Covered Lines:	2660
Relevant Lines:	3103

💛 - Coveralls

nantiferov · 2025-10-08T19:43:52Z

Implemented metrics collection, descriptions are taken from FT.INFO page, example:

redis_search_index_bytes_per_record_avg{index_name="idx:bicycle"} 7.244767189025879
redis_search_index_cleaning{index_name="idx:bicycle"} 0
redis_search_index_doc_table_size_bytes{index_name="idx:bicycle"} 47417
redis_search_index_geoshapes_size_bytes{index_name="idx:bicycle"} 0
redis_search_index_hash_indexing_failures{index_name="idx:bicycle"} 0
redis_search_index_indexing{index_name="idx:bicycle"} 0
redis_search_index_inverted_size_bytes{index_name="idx:bicycle"} 47417
redis_search_index_key_table_size_bytes{index_name="idx:bicycle"} 5858
redis_search_index_max_doc_id{index_name="idx:bicycle"} 187
redis_search_index_num_docs{index_name="idx:bicycle"} 187
redis_search_index_num_records{index_name="idx:bicycle"} 6545
redis_search_index_num_terms{index_name="idx:bicycle"} 33
redis_search_index_number_of_uses{index_name="idx:bicycle"} 74
redis_search_index_offset_bits_per_record_avg{index_name="idx:bicycle"} 8
redis_search_index_offset_vectors_size_bytes{index_name="idx:bicycle"} 6545
redis_search_index_offsets_per_term_avg{index_name="idx:bicycle"} 1
redis_search_index_percent_indexed{index_name="idx:bicycle"} 1
redis_search_index_records_per_doc_avg{index_name="idx:bicycle"} 35
redis_search_index_sortable_values_size_bytes{index_name="idx:bicycle"} 0
redis_search_index_tag_overhead_size_bytes{index_name="idx:bicycle"} 29
redis_search_index_text_overhead_size_bytes{index_name="idx:bicycle"} 1155
redis_search_index_total_index_memory_size_bytes{index_name="idx:bicycle"} 91647
redis_search_index_total_inverted_index_blocks{index_name="idx:bicycle"} 69
redis_search_index_vector_index_size_bytes{index_name="idx:bicycle"} 0

exporter/exporter.go

nantiferov · 2025-10-08T19:49:24Z

exporter/exporter.go

+		"search_index_total_index_memory_size_bytes":         {txt: "Total memory used by search index", lbls: []string{"index_name"}},
+		"search_index_geoshapes_size_bytes":                  {txt: "Memory used by GEO-related fields", lbls: []string{"index_name"}},
+		"search_index_records_per_doc_avg":                   {txt: "Average number of records (including deletions) per document", lbls: []string{"index_name"}},
+		"search_index_bytes_per_record_avg":                  {txt: "Average size of each record in bytes", lbls: []string{"index_name"}},


I converted other metrics from megabytes, _mb to _bytes and keep _bytes in the end according to Prometheus metrics naming schema.

However, for this metric I think it's worth to keep naming as is, since it's pre-aggregated value from redis and there're 4 metrics in total with similar naming, like search_index_records_per_doc_avg

I could go either way about the pre-aggregated ones. in theory they can be calculated on the prometheus side (plus avg is only so useful) - if you want to just drop them then I
'm fine too

one thing re naming is, unit should go to the end so sohuld be something like search_index_avg_per_record_bytes

I think it's worth to keep these avg metrics just in case as I'm not sure atm about their usefulness. But it's only 4 metrics per index.

Regarding renaming. This will break naming schema for these 4 metric - now they all end with _avg. Though doesn't look like a big deal to me.

Actually I think it's worth to rename all 4 of them a little bit, like

search_index_records_per_doc_avg -> search_index_avg_per_doc_records

search_index_bytes_per_record_avg -> search_index_avg_per_record_bytes

search_index_offsets_per_term_avg -> search_index_avg_per_term_offsets

search_index_offset_bits_per_record_avg -> search_index_avg_per_record_offset_bits

exporter/exporter.go

nantiferov · 2025-10-08T19:52:26Z

exporter/search_indexes.go

+	log "github.com/sirupsen/logrus"
+)
+
+// All fields of the streamInfo struct must be exported


I added this header similar to exporter/streams.go file, I guess limitation is the same, since I'm using redis.ScanStruct here too.

super nit: this should be searchIndexInfo and not streamInfo

Thank you for noticing, fixed now.

go.mod

exporter/search_indexes_test.go

nantiferov · 2025-10-08T20:59:32Z

exporter/search_indexes_test.go

+	log "github.com/sirupsen/logrus"
+)
+
+func setupSearchIndex(t *testing.T, addr string) error {


I added helper function, similar to setupKeys() in exporter/exporter_test.go to create index in empty redis.

Otherwise there's nothing and this test scenario fails because no search_index metrics collected: {addr: os.Getenv("TEST_REDIS8_URI"), inclSearchIndexesMetrics: true, wantSearchIndexesMetrics: true}

nantiferov · 2025-10-08T21:03:40Z

@oliver006 PTAL when you have time, I think PR is ready for review.

* Add Redis search module indexes metrics collection * Add tests for Redis search module indexes metrics collection * Bump redigo to v1.9.3 due to gomodule/redigo#691 Fix: oliver006#1043

exporter/search_indexes.go

oliver006 · 2025-10-10T01:26:46Z

Thanks for the PR - I'll try to get to reviewing it later this week

…etrics-collection

oliver006 · 2025-10-14T04:07:41Z

PR looks good - only a few minor questions or comments.
Thanks for submitting this!

oliver006

Looks good, thank you!

exporter/modules_test.go

oliver006 · 2025-10-16T14:52:23Z

RFM?

oliver006

👍

nantiferov force-pushed the Feature--Add-Redis-search-module-indexes-metrics-collection branch from 3c143bb to 6fc64d7 Compare October 7, 2025 05:40

nantiferov force-pushed the Feature--Add-Redis-search-module-indexes-metrics-collection branch 2 times, most recently from b9e61f4 to 24fd8c5 Compare October 8, 2025 18:21

nantiferov force-pushed the Feature--Add-Redis-search-module-indexes-metrics-collection branch from 24fd8c5 to fdcd4ec Compare October 8, 2025 19:44