Proposal: Package manager migration from `poetry` to `uv`

[bowenliang123]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Proposal: Package manager migration from poetry to uv

Background

Dify’s situation

Dify has started to swiching package manager from pip to poetry in #4513 since Jun 2024.

poetry brought many importance benefits to Dify comparing to pip at the time, as lockfile support, dependency grouping, package caching, isntallation performance etc.

And with Plugin System introduced and sepeating plugin daemon from the main repo, Dify is using different Python package managers.

• Poetry: For Python depdencies of api and worker services in Dify main repo (https://github.com/langgenius/dify)
• uv: For installing Python depdencies of plugins , ( https://github.com/langgenius/dify-plugin-daemon )

However, poetry manager has exposed more shortage and problems, which limits agile delivery, community contribution and daily use.

1. Poor performance in updating or syncing the lockfile, which cost ~15 sec with usual operations
2. lockfile confilicts from time to time with the version hashes
3. unable to set Pip mirror URL at runtime, once the lockfile is set
4. lack of support for easier pip usage support for new comers

Other projects

Many morden popular LLM / API projects have switiching to use uv as package manager, from services, SDKs to Multi-Agent frameworks.

• OpenAI's Agents SDK : https://github.com/openai/openai-agents-python
• Microsoft's Autogen : https://github.com/microsoft/autogen/tree/main/python
• Google's A2A: https://github.com/google/A2A

Objectives

uv is an extremely fast Python package and project manager, written in Rust.

Bringing uv into Dify's main repo, for the following purposes:

1. speed up the runtime performance in package installation and lockfile syncing
2. prevent single file hash lock in the lockfile, for easier collaboration and community contribution
3. bing in uv pip commands for easier pip operations

Migration Steps

1. migrate poetry to uv as package manager tool, updating [build-system] in pyprojects.toml
2. migrate poetry's dynamic dependencies declaration in [tool.poetry.dependencies] into python standard [dependency-groups] section in pyprojects.toml
3. construct CI tests and checking the validation and availability of the generated uv lockfile
4. update all the workflows' actions usage related to poetry
5. replace poetry to uv Dockerfile in api folder for both api and worker services

PR

• build: introduce uv as Python package manager #16317

2. Additional context or comments

No response

3. Can you help us with this feature?

• I am interested in contributing to this feature.

[bowenliang123]

cc @takatost @JohnJyong @laipz8200 @Yeuoly @MatriQ

[yihong0618]

Cool

[QuantumGhost]

Thank you for the proposal. I have one key concern regarding the transition:

How can we ensure consistency between uv and Poetry's lockfiles?

From what I understand, uv and Poetry use different package resolution algorithms, which could lead to discrepancies in the generated lockfiles. My primary concern is ensuring that the versions selected by uv are fully compatible with dify-api and maintain internal consistency across all dependencies.

Could you elaborate on how we might address this potential issue during the migration process?

[bowenliang123]

Hi @QuantumGhost , thanks for raising your concern in the lockfiles.

I would say consistency in the lockfiles is NOT our goal. As all the dependency manager does, it focus in direct dependencies declaration. And the lockfile could have been updated for any reasons while maintaining stable.

We does guarantee the dependency declaration aligned in Poetry and UV one by one mapping with the same specifier operation and versions, and the lockfiles for Poetry and UV are properly generated and pass all the CI tests on either environement prepared by each dependency manager.