Skip to content

Conversation

@dateutli
Copy link
Contributor

@dateutli dateutli commented Dec 1, 2025

Description

This PR introduces the --generate-config flag for the rebalance command.

The flag would update the main process as outlined below:

  • From the inventoried config files we create a set to keep track of the existing config files
  • Bootstrap missing topic configs
  • Re-inventory config files to take into account the newly generated ones
  • Continue the process as usual (if a topic config file already existed it'll take precedence and the generated one will be dismissed)
  • Clean up bootstrapped topic configs

This PR also adds a wrapper function around the traversal of topic configs for reusability purposes.

Notes

  • This is effectively calling the main functionality of the bootstrap command in a limited capacity. We could potentially add support for all of the bootstrap arguments but that's outside the scope of this feature

Testing

  1. Picked topic check-topic--JNKfqh for testing
  2. $ topicctl get balance 'check-topic--JNKfqh' --zk-addr=localhost:2181

Broker replicas:

ID RACK POSITION 1 POSITION 2 TOTAL
1 zone1 2 1 3
2 zone1 1 2 3
3 zone2 1 2 3
4 zone2 2 1 3
5 zone3 1 2 3
6 zone3 2 1 3

Broker rack replicas:

RACK POSITION 1 POSITION 2 TOTAL
zone1 3 3 6
zone2 3 3 6
zone3 3 3 6
  1. Manually bootstrapped check-topic--JNKfqh in order to remove one broker

  2. $ topicctl apply 'examples/local-cluster/topics/check-topic--JNKfqh.yaml' --rebalance --to-remove 1

  3. $ topicctl get balance 'check-topic--JNKfqh' --zk-addr=localhost:2181

Broker replicas:

ID RACK POSITION 1 POSITION 2 TOTAL
1 zone1 0 0 0
2 zone1 2 2 4
3 zone2 1 2 3
4 zone2 2 2 4
5 zone3 2 2 4
6 zone3 2 1 3

Broker rack replicas:

RACK POSITION 1 POSITION 2 TOTAL
zone1 2 2 4
zone2 3 4 7
zone3 4 3 7
  1. Deleted manually bootstrapped topic config

  2. $ topicctl rebalance --cluster-config='examples/local-cluster/cluster.yaml' --path-prefix='examples/local-cluster/topics' --bootstrap-missing-configs

Broker replicas:

ID RACK POSITION 1 POSITION 2 TOTAL
1 zone1 2 1 3
2 zone1 1 2 3
3 zone2 1 2 3
4 zone2 1 2 3
5 zone3 2 1 3
6 zone3 2 1 3

Broker rack replicas:

RACK POSITION 1 POSITION 2 TOTAL
zone1 3 3 6
zone2 2 4 6
zone3 4 2 6

@dateutli dateutli requested a review from a team as a code owner December 1, 2025 18:57
@pazhang-25
Copy link
Contributor

Sorry if you were already planning on doing this, but could you perhaps add a Description section on your PR and also a Testing section to show how you tested the code? :)

@dateutli
Copy link
Contributor Author

dateutli commented Dec 1, 2025

Sorry if you were already planning on doing this, but could you perhaps add a Description section on your PR and also a Testing section to show how you tested the code? :)

My bad, didn't mean to send it out for review just yet.

@dateutli dateutli marked this pull request as draft December 1, 2025 19:48
David Teutli added 2 commits December 1, 2025 18:49
…existing generated files; wrap topic file and config traversal in reusable function
@dateutli dateutli requested a review from hhahn-tw December 2, 2025 18:33
@dateutli dateutli marked this pull request as ready for review December 2, 2025 18:34
Copy link
Contributor

@petedannemann petedannemann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good, left two minor comments

@hhahn-tw
Copy link
Contributor

hhahn-tw commented Dec 3, 2025

Please also provide update the README outlining this new feature

@dateutli dateutli changed the title [TUBE-3309] Rebalance of topics with no config files Rebalance of topics with no config files Dec 3, 2025
David Teutli added 2 commits December 4, 2025 10:07
@dateutli dateutli requested a review from hhahn-tw December 4, 2025 17:23
Co-authored-by: Peter Dannemann <[email protected]>
Copy link
Contributor

@hhahn-tw hhahn-tw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good!

@dateutli dateutli merged commit 1a410cd into master Dec 8, 2025
9 checks passed
@dateutli dateutli deleted the TUBE-3309/rebalaing-topcis-wo-config-files branch December 8, 2025 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants