add deploy architecture and resource requirements #271
Conversation
Summary of ChangesHello @wingwing2005, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! 本次拉取请求旨在通过新增详细的部署架构文档和更新资源规划指南，提升TDengine IDMP的部署和运维指导的全面性与实用性。新的文档清晰地阐述了不同规模和可用性需求下的部署策略，并提供了具体的资源配置建议，帮助用户更好地规划和实施TDengine IDMP系统，确保其在各种生产环境下的稳定高效运行。 Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds comprehensive documentation on deployment architectures and resource requirements for TDengine IDMP, including both Chinese and English versions. The new documents are well-structured and provide valuable guidance for different deployment scenarios. My review includes several suggestions to improve the accuracy and clarity of the documentation, such as correcting typos in component names (e.g., 'Consul', 'Elasticsearch'), fixing an incomplete sentence in the English documentation, and updating an external link to point to the correct version of the TDengine documentation. These minor changes will help ensure the documentation is professional and easy to understand.
| - Web Browser →（穿越 Firewall）→ **API Gateway** → **TDengine IDMP（多实例）** | ||
| - taosX agent →（穿越 Firewall）→ **TDengine TSDB（6055）** | ||
| - **TDengine IDMP → TDengine TSDB（6041）** | ||
| - IDMP 外部依赖（完整展示）：**Redis、MySQL、Distributed File System、Elastic Logdash Search、Kafka/RabbitMQ、Consule/Apollo** 等 |
There was a problem hiding this comment.
There are a couple of typos on this line:
Elastic Logdash Searchseems incorrect.Logdashis likely a typo forLogstash, butElastic Logstash Searchis not a standard component name. Given the component description on line 107 refers to search and indexing, you probably meanElasticsearch.Consuleis a typo forConsul.
Please correct these for accuracy.
| - IDMP 外部依赖（完整展示）：**Redis、MySQL、Distributed File System、Elastic Logdash Search、Kafka/RabbitMQ、Consule/Apollo** 等 | |
| - IDMP 外部依赖（完整展示）：**Redis、MySQL、Distributed File System、Elasticsearch、Kafka/RabbitMQ、Consul/Apollo** 等 |
| - **Redis**：缓存与短期状态（也常用于分布式锁、计数器等）。 | ||
| - **MySQL**：关系型元数据（用户与权限、配置、任务定义、业务元信息等）。 | ||
| - **Distributed File System**：文件/对象持久化（元数据、上传图元、导入导出文件、上传文件、设备文档等）。 | ||
| - **Elastic Search**：集中索引管理和搜索。 |
There was a problem hiding this comment.
Elastic Search is usually written as a single word: Elasticsearch. Please update for consistency.
| - **Elastic Search**：集中索引管理和搜索。 | |
| - **Elasticsearch**：集中索引管理和搜索。 |
| - **Distributed File System**：文件/对象持久化（元数据、上传图元、导入导出文件、上传文件、设备文档等）。 | ||
| - **Elastic Search**：集中索引管理和搜索。 | ||
| - **Kafka/RabbitMQ**：异步消息与事件总线（解耦、削峰、任务编排、通知）。 | ||
| - **Consule/Apollo**：服务治理/配置中心（服务发现、动态配置、配置版本管理）。 |
There was a problem hiding this comment.
There is a typo here. Consule should be Consul.
| - **Consule/Apollo**：服务治理/配置中心（服务发现、动态配置、配置版本管理）。 | |
| - **Consul/Apollo**：服务治理/配置中心（服务发现、动态配置、配置版本管理）。 |
docs/07-operation/01-planning.md
Outdated
| 4. **外部依赖高可用**：生产环境建议 Redis/MySQL 配置主从或集群以保证高可用。 | ||
| 5. **预留扩展空间**：建议按预期峰值测点数的 1.5 倍规划资源，为业务增长预留空间。 | ||
|
|
||
| > **说明**：以上配置为参考建议，实际资源需求还受采集频率、查询并发、数据保留周期等因素影响，请根据实际业务场景进行调整。关于 TDengine TSDB 的资源规划，请参考 [TDengine 容量规划](https://docs.taosdata.com/operation/planning/)。 |
There was a problem hiding this comment.
The link https://docs.taosdata.com/operation/planning/ points to the documentation for TDengine 2.x. However, this document mentions a dependency on TDengine TSDB-Enterprise: 3.3.7.0 or later. To ensure consistency and provide the correct information, please update the link to point to the capacity planning documentation for TDengine 3.x, such as https://docs.tdengine.com/cn/v3.0/reference/planning for the Chinese version.
| > **说明**：以上配置为参考建议，实际资源需求还受采集频率、查询并发、数据保留周期等因素影响，请根据实际业务场景进行调整。关于 TDengine TSDB 的资源规划，请参考 [TDengine 容量规划](https://docs.taosdata.com/operation/planning/)。 | |
| > **说明**：以上配置为参考建议，实际资源需求还受采集频率、查询并发、数据保留周期等因素影响，请根据实际业务场景进行调整。关于 TDengine TSDB 的资源规划，请参考 [TDengine 容量规划](https://docs.tdengine.com/cn/v3.0/reference/planning/)。 |
| - Web Browser → (Through Firewall) → **API Gateway** → **TDengine IDMP (Multi-instance)** | ||
| - taosX agent → (Through Firewall) → **TDengine TSDB (6055)** | ||
| - **TDengine IDMP → TDengine TSDB (6041)** | ||
| - IDMP External Dependencies (Full): **Redis, MySQL, Distributed File System, Elastic Logstash Search, Kafka/RabbitMQ, Consul/Apollo**, etc. |
There was a problem hiding this comment.
The term Elastic Logstash Search is a bit confusing. Logstash is typically used for data processing pipelines, while Elasticsearch is for search and analytics. It seems you mean Elasticsearch here, as also indicated by the description on line 107. Suggest changing to Elasticsearch for clarity.
| - IDMP External Dependencies (Full): **Redis, MySQL, Distributed File System, Elastic Logstash Search, Kafka/RabbitMQ, Consul/Apollo**, etc. | |
| - IDMP External Dependencies (Full): **Redis, MySQL, Distributed File System, Elasticsearch, Kafka/RabbitMQ, Consul/Apollo**, etc. |
| - **Redis**: Cache and short-term state (also commonly used for distributed locks, counters, etc.). | ||
| - **MySQL**: Relational metadata (users and permissions, configurations, task definitions, business meta-information, etc.). | ||
| - **Distributed File System**: File/object persistence (metadata, uploaded graphics, import/export files, uploaded documents, equipment documents, etc.). | ||
| - **Elastic Search**: Centralized index management and search. |
There was a problem hiding this comment.
Elastic Search is conventionally written as a single word, Elasticsearch. It would be best to correct this for consistency.
| - **Elastic Search**: Centralized index management and search. | |
| - **Elasticsearch**: Centralized index management and search. |
| Redis/MySQL/DFS should be configured with backups and high availability; once MQ/Configuration Center/Search are enabled, they should be included in monitoring, alerting, and capacity planning. | ||
|
|
||
| 5. **Unified Observability** | ||
| It is recommended to collect logs and metrics uniformly for the gateway, IDMP, TSDB, and key dependencies, and establish alerting and tracing |
There was a problem hiding this comment.
This sentence appears to be incomplete. It ends abruptly with ...and establish alerting and tracing. Please complete the sentence to make it clear. For example: ...and establish alerting and tracing links to avoid difficulties in troubleshooting.
| It is recommended to collect logs and metrics uniformly for the gateway, IDMP, TSDB, and key dependencies, and establish alerting and tracing | |
| It is recommended to collect logs and metrics uniformly for the gateway, IDMP, TSDB, and key dependencies, and establish alerting and tracing links to avoid difficulties in troubleshooting. |
| 4. **High Availability for External Dependencies**: It is recommended to configure Redis/MySQL in master-slave or cluster mode in production environments to ensure high availability. | ||
| 5. **Reserve Expansion Space**: It is recommended to plan resources at 1.5 times the expected peak data points to allow for business growth. | ||
|
|
||
| > **Note**: The above configurations are reference suggestions. Actual resource requirements are also affected by factors such as collection frequency, query concurrency, and data retention periods. Please adjust according to actual business scenarios. For TDengine TSDB's resource planning, please refer to [TDengine System Requirements](https://docs.tdengine.com/operations-and-maintenance/system-requirements/)。 |
There was a problem hiding this comment.
There's a typo here. A Chinese period 。 is used at the end of the sentence. It should be an English period ..
| > **Note**: The above configurations are reference suggestions. Actual resource requirements are also affected by factors such as collection frequency, query concurrency, and data retention periods. Please adjust according to actual business scenarios. For TDengine TSDB's resource planning, please refer to [TDengine System Requirements](https://docs.tdengine.com/operations-and-maintenance/system-requirements/)。 | |
| > **Note**: The above configurations are reference suggestions. Actual resource requirements are also affected by factors such as collection frequency, query concurrency, and data retention periods. Please adjust according to actual business scenarios. For TDengine TSDB's resource planning, please refer to [TDengine System Requirements](https://docs.tdengine.com/operations-and-maintenance/system-requirements/). |
There was a problem hiding this comment.
Pull request overview
This pull request adds comprehensive deployment architecture documentation and resource requirement guidelines for TDengine IDMP. The changes introduce three deployment patterns (single instance, simplified HA, and complex HA) along with detailed resource planning recommendations based on data point scale.
Key changes:
- New deployment architecture documentation covering three deployment patterns
- Resource planning tables with CPU, memory, disk, and instance recommendations for different scales
- Addition of HTTPS port (6034) documentation
- Deployment diagrams (PNG files) illustrating the three architecture patterns
Reviewed changes
Copilot reviewed 4 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| i18n/en/docusaurus-plugin-content-docs/current/07-operation/00-deploy/00-deploy.md | New English deployment architecture documentation with component descriptions, port information, and deployment recommendations |
| i18n/en/docusaurus-plugin-content-docs/current/07-operation/00-deploy/images/single_deploy.png | Binary image file for single instance deployment diagram |
| i18n/en/docusaurus-plugin-content-docs/current/07-operation/01-planning.md | Updated English planning documentation with resource requirement tables and recommendations |
| docs/07-operation/00-deploy/00-deploy.md | New Chinese deployment architecture documentation (mirrors English version) |
| docs/07-operation/01-planning.md | Updated Chinese planning documentation with resource requirement tables and recommendations, plus new HTTPS port documentation |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| 5. **统一观测** | ||
| 建议对网关、IDMP、TSDB 与关键依赖统一采集日志与指标，建立告警与追踪链路，避免故障定位困难。 |
There was a problem hiding this comment.
Inconsistent punctuation: The Chinese text ends with a period after "避免故障定位困难。" (line 138), but the English equivalent at line 138 in the English file appears incomplete and lacks proper punctuation. Consider completing the sentence or adding a period for consistency.
| Redis/MySQL/DFS should be configured with backups and high availability; once MQ/Configuration Center/Search are enabled, they should be included in monitoring, alerting, and capacity planning. | ||
|
|
||
| 5. **Unified Observability** | ||
| It is recommended to collect logs and metrics uniformly for the gateway, IDMP, TSDB, and key dependencies, and establish alerting and tracing |
There was a problem hiding this comment.
Incomplete sentence: The sentence "It is recommended to collect logs and metrics uniformly for the gateway, IDMP, TSDB, and key dependencies, and establish alerting and tracing" appears to be cut off. The sentence should be completed with proper punctuation or additional content.
| It is recommended to collect logs and metrics uniformly for the gateway, IDMP, TSDB, and key dependencies, and establish alerting and tracing | |
| It is recommended to collect logs and metrics uniformly for the gateway, IDMP, TSDB, and key dependencies, and establish alerting and tracing. |
| - Web Browser →（穿越 Firewall）→ **API Gateway** → **TDengine IDMP（多实例）** | ||
| - taosX agent →（穿越 Firewall）→ **TDengine TSDB（6055）** | ||
| - **TDengine IDMP → TDengine TSDB（6041）** | ||
| - IDMP 外部依赖（完整展示）：**Redis、MySQL、Distributed File System、Elastic Logdash Search、Kafka/RabbitMQ、Consule/Apollo** 等 |
There was a problem hiding this comment.
Spelling error: "Logstash" is misspelled as "Logdash". The correct spelling should be "Logstash" when referring to the Elastic Stack component.
| - IDMP 外部依赖（完整展示）：**Redis、MySQL、Distributed File System、Elastic Logdash Search、Kafka/RabbitMQ、Consule/Apollo** 等 | |
| - IDMP 外部依赖（完整展示）：**Redis、MySQL、Distributed File System、Elastic Logstash Search、Kafka/RabbitMQ、Consule/Apollo** 等 |
| - **Distributed File System**：文件/对象持久化（元数据、上传图元、导入导出文件、上传文件、设备文档等）。 | ||
| - **Elastic Search**：集中索引管理和搜索。 | ||
| - **Kafka/RabbitMQ**：异步消息与事件总线（解耦、削峰、任务编排、通知）。 | ||
| - **Consule/Apollo**：服务治理/配置中心（服务发现、动态配置、配置版本管理）。 |
There was a problem hiding this comment.
Spelling error: "Consul" is misspelled as "Consule". The correct spelling should be "Consul" when referring to HashiCorp Consul.
| - **Consule/Apollo**：服务治理/配置中心（服务发现、动态配置、配置版本管理）。 | |
| - **Consul/Apollo**：服务治理/配置中心（服务发现、动态配置、配置版本管理）。 |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| | 1 万 ~ 10 万 | 8 核 | 16 GB | 100 GB | 中小型生产环境 | | ||
| | 10 万 ~ 50 万 | 16 核 | 32 GB | 200 GB | 中型生产环境 | | ||
| | 50 万 ~ 100 万 | 32 核 | 64 GB | 500 GB | 大型生产环境 | | ||
| | 100 万以上 | 64 核+ | 128 GB+ | 1 TB+ | 超大规模生产环境 | |
There was a problem hiding this comment.
🚫 [AutoCorrect Lint] <AutoCorrect> reported by reviewdog 🐶
| | 100 万以上 | 64 核+ | 128 GB+ | 1 TB+ | 超大规模生产环境 | | |
| | 100 万以上 | 64 核 + | 128 GB+ | 1 TB+ | 超大规模生产环境 | |
Description
Issue(s)
Checklist
Please check the items in the checklist if applicable.