`poetry build` gets slower the more files are in the project directory

[DanielHabenicht]

Poetry
Version: 1.4.0
Python:  3.10.4

Virtualenv
Python:         3.10.4
Implementation: CPython
Path:           /home/codespace/.cache/pypoetry/virtualenvs/example-p6HmNsBj-py3.10
Executable:     /home/codespace/.cache/pypoetry/virtualenvs/example-p6HmNsBj-py3.10/bin/python
Valid:          True

System
Platform:   linux
OS:         posix
Python:     3.10.4
Path:       /usr/local/python/3.10.4
Executable: /usr/local/python/3.10.4/bin/python3.10

• pyproject.toml: poetry new defaults

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

With an increasing amount of files inside a project folder (let's say 50 000+) poetry build takes increasingly more time, e.g.:

Building example (0.1.0)
  - Building sdist
<Waiting for a few minutes>
  - Built example -0.1.0.tar.gz
  - Building wheel
  - Built example -0.1.0-py3-none-any.whl

I also found an issue from somebody on Stackoverflow, they reference a real world scenario of using node_modules somewhere in the project dir.

I know it is a real edge-case but it would still be nice if exluded files would not be considered at all.

Here is a short reproduction:

pip install poetry
poetry new example
cd example
mkdir folder
cd folder
for i in {1..70000}; do
  mkdir s"$i"
  touch s"$i"/file1.file
  touch s"$i"/file2.file
  touch s"$i"/file3.file
done

It just shows a small delay on my screenshot, but I have a real project where the delay is much more pronounced.

Using exclude = ["folder"] takes no effect on the speed of the build.

Just for my reference: https://github.com/DanielHabenicht/bug.poetry

[dimbleby]

related python-poetry/poetry-core#499

(though I'd suggest that having 70,000210,000 files in your python project that actually aren't part of your python project might be best avoided!)

[DanielHabenicht]

related python-poetry/poetry-core#499

(though I'd suggest that having 70,000210,000 files in your python project that actually aren't part of your python project might be best avoided!)

Yes, I can only second that. But sometimes one is generating a lot of test data and things accumulate™.
Nice that there is already a PR for this.

[rmarquis]

Yes, I can only second that. But sometimes one is generating a lot of test data and things accumulate™.

This is actually our use case. This would be more of a convenient fix than something absolutely critical, but it would be nice to see the related PR reviewed.

[TxJson]

I also found an issue from somebody on Stackoverflow, they reference a real world scenario of using node_modules somewhere in the project dir.

I have currently come across the issue outlined here. We are in the process of building a python package used for an internal tool. This includes a python wrapped npm and webpack compiled javascript bit, with node_modules. We have excluded the whole js/ directory when building the package since we only care about the bundle produced by webpack. However, we have found poetry to still traverse the files within this excluded directory, including all the node_modules within. This results in a build that normally should take just a few seconds, ending up taking 30-40+ minutes. It is worth noting that this is on a relatively small project. This really should not be happening.

Unless I have missed some build step that resolves this, I can imagine that this issue is a massive blocker in allowing poetry to be used within hobby projects, or even larger scale projects.

I can see that there has not been any progress on this issue in quite some time. Do we have any update at all? I find poetry quite convenient so would hate to have to abandon it over this issue.

[radoering]

As already mentioned in previous comments, there is python-poetry/poetry-core#499, which however is quite old and has conflicts.

If you want to help, you can check if this PR solves your issue and ask the author if he is still interested and wants to rebase it and if not adopt the PR and do the rebasing yourself.