Skip to content

add background task queue #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ardzz merged 4 commits into from
Oct 30, 2025
Merged

add background task queue #1

ardzz merged 4 commits into from
Oct 30, 2025

Conversation

@ardzz
Copy link
Owner

@ardzz ardzz commented Oct 30, 2025

No description provided.

claude added 3 commits October 30, 2025 06:53
@claude
feat: add background task queue system similar to Laravel's queue:work
3db60fa 
Implement a comprehensive background task queue system for processing jobs
asynchronously, keeping MQTT handlers fast and responsive.

Features:
- Job base class with automatic serialization and retry logic
- Redis queue driver (fast, in-memory) and Database queue driver (persistent)
- Queue manager for dispatching jobs with delayed execution support
- Queue worker CLI command (--queue-work) with configurable options
- Failed job tracking and custom failure handlers
- Multiple queue support for different priorities
- Graceful shutdown handling

Components added:
- core/job.py: Base Job class for background tasks
- core/queue/: Queue infrastructure (manager, worker, drivers)
- app/models/queue_job.py: Database model for pending jobs
- app/models/queue_failed_job.py: Database model for failed jobs
- app/jobs/: Example jobs (email, data processing, reports)

CLI usage:
  python main.py --queue-work [options]

Options:
  --queue: Queue name to process (default: default)
  --connection: Queue driver (redis/database)
  --max-jobs: Maximum jobs to process before stopping
  --max-time: Maximum runtime in seconds
  --sleep: Sleep duration when no jobs available
  --timeout: Job timeout in seconds

Configuration (.env):
  QUEUE_CONNECTION=redis  # or 'database'
  ENABLE_REDIS=true      # for Redis queue
  ENABLE_MYSQL=true      # for Database queue

Documentation:
- docs/queue-system.md: Comprehensive guide with examples
- QUEUE_FEATURE.md: Feature summary and usage

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@claude
feat: add Docker configuration for queue workers and production deplo…
55ddb83 
…yment

Update Docker Compose setup to support the new background task queue system
with Redis, MySQL, and multiple queue workers.

Changes:
- Add Redis service for fast queue backend
- Add MySQL service for persistent storage
- Add 3 queue worker services (default, high-priority, emails)
- Update main app service with queue configuration
- Add health checks for all services
- Configure resource limits and dependencies

Development support:
- Add docker-compose.dev.yml for local development
- Add .env.docker with recommended Docker settings
- Add Makefile with convenient Docker commands

Documentation:
- Add comprehensive Docker deployment guide
- Update README with Docker and queue examples
- Add project structure showing queue components

Services included:
- routemq: Main MQTT application
- queue-worker-default: Default queue processor
- queue-worker-high: High-priority queue processor
- queue-worker-emails: Email queue processor
- redis: Queue backend (fast, in-memory)
- mysql: Database backend (persistent)

Quick start:
  docker compose up -d       # Start all services
  make up                    # Alternative with Makefile
  make logs                  # View logs
  make scale-default         # Scale workers

See docs/docker-deployment.md for complete guide.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@claude
docs: reorganize queue documentation into dedicated folder structure
c09aa09 
Restructure queue documentation from a single file into an organized
folder with dedicated pages for each topic.

Changes:
- Remove QUEUE_FEATURE.md (temporary feature summary)
- Remove docs/queue-system.md (single large file)
- Create docs/queue/ folder with organized structure:
  - README.md - Overview and quick start
  - getting-started.md - Installation and configuration
  - creating-jobs.md - Job creation guide
  - dispatching-jobs.md - Job dispatching methods
  - running-workers.md - Worker management
  - drivers.md - Redis vs Database queue drivers
  - failed-jobs.md - Failed job handling
  - best-practices.md - Best practices and patterns

Documentation improvements:
- Better navigation with focused pages
- Easier to find specific information
- Consistent with other feature docs (redis/, routing/, etc.)
- Added more examples and use cases
- Expanded troubleshooting sections
- More detailed code samples

Updates:
- docs/SUMMARY.md - Add queue section with all pages
- README.md - Update queue docs link
- docs/docker-deployment.md - Fix queue docs reference

Benefits:
- ✅ Easier to navigate
- ✅ Better searchability
- ✅ Consistent with GitBook structure
- ✅ More maintainable
- ✅ Clearer organization

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@ardzz ardzz requested a review from Copilot October 30, 2025 07:26
@github-actions
Copy link
Contributor

github-actions bot commented Oct 30, 2025

📚 Documentation Summary Update

The docs/SUMMARY.md file has been automatically updated to reflect the changes in this PR. Please review the updated table of contents.

Copilot AI reviewed Oct 30, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements a comprehensive background task queue system for RouteMQ, similar to Laravel's queue functionality. The system enables asynchronous job processing with support for both Redis and MySQL backends, includes complete documentation, and provides production-ready Docker deployment configurations.

  • Adds a queue system with Redis and Database drivers for job processing
  • Includes CLI commands for running queue workers with various configuration options
  • Provides comprehensive documentation covering jobs, dispatching, workers, drivers, and best practices

Reviewed Changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
core/job.py Base job class with serialization/deserialization support
core/queue/*.py Queue manager, drivers (Redis/Database), and worker implementation
app/models/queue_*.py Database models for queue jobs and failed jobs
app/jobs/example_*.py Example job implementations for emails, reports, and data processing
main.py CLI command for queue worker with configuration options
test_queue.py Test script for queue system verification
docker-compose.yml Production Docker setup with Redis, MySQL, and multiple queue workers
docker-compose.dev.yml Development Docker setup with minimal services
Makefile Convenience commands for Docker and queue management
docs/queue/*.md Complete documentation for the queue system
docs/docker-deployment.md Docker deployment guide
README.md Updated with queue system and Docker information

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docs/queue/drivers.md
- ✅ **Delayed jobs** - Efficient sorted sets for scheduling
- ✅ **Scalable** - Easy to cluster Redis for more capacity

###Requirements
Copy link

Copilot AI Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing space between '###' and 'Requirements'. Should be '### Requirements' to follow Markdown heading syntax.

Suggested change
###Requirements
### Requirements

Copilot uses AI. Check for mistakes.
README.md
python main.py --queue-work --queue emails
```

See [Queue System Documentation](./docs/queue-system.md) for complete guide.
Copy link

Copilot AI Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The link points to ./docs/queue-system.md but the actual documentation file is ./docs/queue/README.md. This broken link should be updated to point to the correct file.

Suggested change
See [Queue System Documentation](./docs/queue-system.md) for complete guide.
See [Queue System Documentation](./docs/queue/README.md) for complete guide.

Copilot uses AI. Check for mistakes.
core/queue/queue_driver.py
Comment on lines +3 to +5
from datetime import datetime


Copy link

Copilot AI Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import of 'datetime' is not used.

Suggested change
from datetime import datetime

Copilot uses AI. Check for mistakes.
test_queue.py

from dotenv import load_dotenv
from core.job import Job
from core.queue.queue_manager import dispatch, queue
Copy link

Copilot AI Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import of 'dispatch' is not used.
Import of 'queue' is not used.

Copilot uses AI. Check for mistakes.
@github-actions
Copy link
Contributor

github-actions bot commented Oct 30, 2025

📚 Documentation Summary Update

The docs/SUMMARY.md file has been automatically updated to reflect the changes in this PR. Please review the updated table of contents.

@ardzz ardzz merged commit c9194b8 into master Oct 30, 2025
4 checks passed
@ardzz ardzz deleted the claude/add-background-task-queue-011CUcvsPKaeBhohjpwZJsDy branch October 30, 2025 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ardzz @claude