Rate limit gemini based on different metric #383
Draft
+1,409
−30
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Added configurable rate limiting system with RPM (requests/min), TPM (tokens/min), and RPD (requests/day) limits - Created external YAML configuration for rate limits with conservative defaults for Gemini models - Enhanced GlobalRateLimiter with new MultiRateLimiter class using sliding windows for precise limit tracking - Integrated rate limiting into GeminiBackend with automatic token usage tracking - Added comprehensive documentation
- Moved rate limits to centralized configuration file instead of hardcoding in orchestrator - Added mandatory cooldown periods after agent startup to prevent API burst calls - Implemented _load_rate_limits_from_config() to dynamically load and validate rate limits - Added conservative rate limiting for low-RPM models (<=2 RPM) - Added detailed logging for rate limit configuration and cooldown periods - Added fallback defaults if config loading fails
- Removed three deprecated markdown files related to rate limiting documentation: - CHANGELOG_RATE_LIMITING.md - QUICK_START_RATE_LIMITING.md - RATE_LIMITING_COMPLETE.md - These files contained old implementation details that have since been superseded by newer documentation in the main docs directory
- Added separate rate limits for Gemini Flash (9 RPM) and Pro (2 RPM) models - Removed shared 7 RPM limit to prevent rate limit errors - Updated orchestrator startup timing to handle model-specific limits - Added documentation explaining rate limit changes and performance implications - Included recommendations for optimizing agent configurations based on rate limits - Added test instructions and example logs for verification
- Added enable_rate_limit flag to control agent startup rate limiting behavior - Modified Orchestrator to skip rate limit checks when feature is disabled - Updated CLI to pass rate limit configuration through to Orchestrator - Added documentation for new enable_rate_limit parameter The changes introduce an opt-in rate limiting system for agent startup, allowing users to control whether cooldown delays are enforced between agent initializations. This helps
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Feat: Rate limit for Gemini models
Valid types:
fix:- Bug fixesfeat:- New featuresbreaking:- Breaking changesdocs:- Documentation updatesrefactor:- Code refactoringtest:- Test additions/modificationschore:- Maintenance tasksperf:- Performance improvementsstyle:- Code style changesci:- CI/CD configuration changesExamples:
fix: resolve memory leak in data processingfeat: add export to CSV functionalitybreaking: change API response formatdocs: update installation guideDescription
Brief description of the changes in this PR
Type of change
fix:) - Non-breaking change which fixes an issuefeat:) - Non-breaking change which adds functionalitybreaking:) - Fix or feature that would cause existing functionality to not work as expecteddocs:) - Documentation updatesrefactor:) - Code changes that neither fix a bug nor add a featuretest:) - Adding missing tests or correcting existing testschore:) - Maintenance tasks, dependency updates, etc.perf:) - Code changes that improve performancestyle:) - Changes that do not affect the meaning of the code (formatting, missing semi-colons, etc.)ci:) - Changes to CI/CD configuration files and scriptsChecklist
Pre-commit status
How to Test
Add test method for this PR.
Test CLI Command
Write down the test bash command. If there is pre-requests, please emphasize.
massgen --config massgen/configs/basic/multi/seven_gemini_agents.yaml --rate-limit "France has had a lot of capital in past. Which one is the most popular and why?"Expected Results
Additional context
The goal is to implement rate limiter logic for any backend model, so it doesn’t make endless requests and spam the API.