Ability to expose simulated kubestate-metrics endpoint for added church and cardinality

[hartfordfive]

Many Victoriametrics users are likely monitoring k8s resources and kubestate-metrics is often notorious for high cardinality and churn. In some cases, users may have specific use cases where they need to map additional k8s labels and/or annotations as metric labels which can lead to substantial memory consumption. It would be great to additionally to have the ability to expose an endpoint similar to kubestate-metrics which simulates a configurable level of churn for resources of type deployment, job, and statefulset. The first two are most relevant as environments where a large amount of jobs are scheduled throughout the day (e.g.: Prefect 2), this can lead to a massive amounts of churn in a 24h period.

The ability to combine this along with the current simulated node_exporter metrics would give an even better idea as to what to expect from VM with regards to performance and resource usage.

[hagen1778]

Thanks for the feature request! So far this benchmark doesn't have endpoint for adjusting churn rate by resource type.

But it has settings to configure the churn rate of the scraped targets - see

prometheus-benchmark/chart/values.yaml

Lines 23 to 38 in f6a6905

	# scrapeConfigUpdatePercent is the percent of nodeexporter targets
	# which are updated with unique label on every scrape config update
	# (see scrapeConfigUpdateInterval).
	# This option allows tuning time series churn rate.
	# For example, if scrapeConfigUpdatePercent is set to 1 for targetsCount=100,
	# then a single target gets updated labels on every scrape config update.
	# This generates around 500 new time series every scrapeConfigUpdateInterval.
	scrapeConfigUpdatePercent: 1
	# scrapeConfigUpdateInterval specifies how frequently to update labels
	# across scrapeConfigUpdatePercent nodeexporter targets.
	# This option allows tuning time series churn rate.
	# For example, if scrapeConfigUpdateInterval is set to 10m for targetsCount=100
	# and scrapeConfigUpdatePercent=1, then a single target gets updated labels every 10 minutes.
	# This generates around 500 new time series every 10 minutes.
	scrapeConfigUpdateInterval: 10m