feat: remove input-hash for PyPI dependencies #5174

nichmor · 2025-12-19T17:30:18Z

Replace hash-based validation for path-based PyPI packages with metadata-based validation. Instead of computing and storing a hash of pyproject.toml, setup.py, and setup.cfg, we now:

Don't store any hash in the lock file for path-based packages
During satisfiability checks, parse pyproject.toml and compare the name and version with the locked metadata

This approach is similar to how PR #5011 removed input_hash for conda packages. The metadata comparison is more explicit about what changed (name vs version) rather than just indicating "something changed via hash mismatch".

Changes:

pypi.rs: Stop computing and storing hash for SourceDist::Path and SourceDist::Directory
satisfiability/mod.rs: Replace PypiSourceTreeHashable hash comparison with metadata parsing and comparison
Add PypiSourceMetadataMismatch and PypiSourceMetadataError types
Remove SourceTreeHashMismatch type and related imports

Description

Fixes #{issue}

How Has This Been Tested?

AI Disclosure

This PR contains AI-generated content.
- I have tested any AI-generated content in my PR.
- I take responsibility for any AI-generated content in my PR.

Tools: {e.g., Claude, Codex, GitHub Copilot, ChatGPT, etc.}

Checklist:

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added sufficient tests to cover my changes.
I have verified that changes that would impact the JSON schema have been made in schema/model.py.

baszalmstra

Nice! This is a good start but its not enough, we need to also compare the requirements, required python version optional dependencies, etc. Furthermore, some fields can be dynamic which means we need to invoke python, and we might also have to deal with setup.py only sources.

tdejager · 2025-12-19T17:46:18Z

Nice! This is a good start but its not enough, we need to also compare the requirements, required python version optional dependencies, etc. Furthermore, some fields can be dynamic which means we need to invoke python, and we might also have to deal with setup.py only sources.

Indeed! It might be good to let Claude explore the uv codebase as well for inspiration, although if I recall their satisfiability check is less extensive, they regenerate more often.

nichmor · 2026-01-06T15:33:25Z

Attaching benchmarks here:

Dynamic Metadata (minimal-project with hatch_build.py, setup-project with setup.py)

Main branch:

Command	Mean [s]	Min [s]	Max [s]	Relative
`/tmp/pixi-main install --locked`	1.070 ± 0.017	1.042	1.089	1.00

This branch:

Command	Mean [s]	Min [s]	Max [s]	Relative
`/tmp/pixi-branch install --locked`	1.140 ± 0.048	1.076	1.199	1.00

Static Metadata (minimal-project without hatch_build.py)

Main branch:

Command	Mean [s]	Min [s]	Max [s]	Relative
`/tmp/pixi-main install --locked`	1.196 ± 0.134	1.074	1.365	1.00

This branch:

Command	Mean [s]	Min [s]	Max [s]	Relative
`/tmp/pixi-branch install --locked`	1.309 ± 0.242	1.085	1.668	1.00

tdejager · 2026-01-07T11:00:12Z

crates/pixi_core/src/lock_file/outdated.rs

+pub struct EnvironmentBuildCache {
+    /// Lazily initialized build dispatch dependencies (interpreter, env, etc.)
+    pub lazy_build_dispatch_deps: LazyBuildDispatchDependencies,
+    /// Optional conda prefix updater (created during satisfiability checking)
+    pub conda_prefix_updater: OnceCell<CondaPrefixUpdater>,
+    /// Cached registry client (with Online connectivity)
+    pub registry_client: OnceCell<Arc<RegistryClient>>,
+    /// Cached index locations
+    pub index_locations: OnceCell<IndexLocations>,
+    /// Cached build options
+    pub build_options: OnceCell<BuildOptions>,
+    /// Cached flat index (populated lazily when needed, async initialization)
+    pub flat_index: AsyncOnceCell<FlatIndex>,
+    /// Cached dependency metadata
+    pub dependency_metadata: OnceCell<DependencyMetadata>,
+}


I'm not sure we need to cache most of these structs, as the most interesting things are chached to disk. I think there are non in-memory caches, only the InFlight which is not here and maybe in the RegistryClient. I'd rather not introduce another struct containing too many OnceCell's. Maybe we can figure out what we need to keep? I think the LazyBuildDispatch is something we should cache?

removed all other fields except conda_prefix_updater. I think it's ok to keep it, because it contains an in-memory result ( OnceCell field) of installing a prefix environment.

crates/pixi_install_pypi/src/lib.rs

nichmor · 2026-01-07T11:10:16Z

pyproject.toml

+  "**/*.ipynb",
+  "**/docs_hooks.py",
+  "scripts/test_native_certs.py",
+  "examples/dynamic-source-deps/minimal-project/hatch_build.py",


this will be removed when we remove dynamic-source-deps

nichmor · 2026-01-07T11:10:47Z

examples/dynamic-source-deps/pixi.toml

@@ -0,0 +1,19 @@
+[workspace]


It's added just to show how to test it

ruben-arts · 2026-01-07T13:41:21Z

Test plan:

Run all examples pixi run run-all-examples
Run all the pypi path examples:
- with a clean env
- not rebuilding on a second install
- rebuilding on a file change
- not rebuilding on editable mode file change, unless pyproject.toml/pixi.toml change
Try with real world project like py-rattler or boltons etc.
Start with an environment build by a previous version of pixi and check if the rebuild logic still works on that "dirty" environment.

If all of these cases succeed next to the integration and unit tests I would say it is ready to merge, I did a similar thing for the conda input-hash removal.

nichmor · 2026-01-07T16:28:18Z

added some stuff to cache reading static metadata, these are the new results ( with cache clean on every run )

Dynamic Metadata (minimal-project with hatch_build.py, setup-project with setup.py)

Main branch:

Command	Mean [s]	Min [s]	Max [s]	Relative
`/tmp/pixi-main install --locked`	5.646 ± 0.469	5.184	6.173	1.00

This branch:

Command	Mean [s]	Min [s]	Max [s]	Relative
`/tmp/pixi-branch install --locked`	6.260 ± 0.192	6.019	6.544	1.00

Static Metadata (minimal-project without hatch_build.py)

Main branch:

Command	Mean [s]	Min [s]	Max [s]	Relative
`/tmp/pixi-main install --locked`	5.468 ± 0.249	5.234	5.805	1.00

This branch:

Command	Mean [s]	Min [s]	Max [s]	Relative
`/tmp/pixi-branch install --locked`	6.523 ± 0.884	5.987	8.082	1.00

tdejager · 2026-01-08T12:42:26Z

crates/pixi_core/src/lock_file/satisfiability/pypi_metadata.rs

+    if !current.is_version_dynamic
+        && let Some(current_version) = &current.version
+        && &locked.version != current_version
+    {
+        return Some(MetadataMismatch::Version {
+            locked: locked.version.clone(),
+            current: current_version.clone(),
+        });
+    }


This needs some extra logic to work correctly when switching between dynamic and non-dynamic versions.

tdejager · 2026-01-08T12:44:44Z

crates/pixi_core/src/lock_file/outdated.rs

+/// Cache for build-related resources that can be shared between
+/// satisfiability checking and PyPI resolution.
+#[derive(Default)]
+pub struct EnvironmentBuildCache {


Maybe name it something with PyPI to differentiate from pixi build?

tdejager · 2026-01-08T13:50:55Z

Might be good to user-test if hatch-vcs is working as expected :)

baszalmstra requested changes Dec 19, 2025

View reviewed changes

lucascolley added enhancement New features pypi Issue related to PyPI dependencies lock file labels Dec 31, 2025

This was referenced Jan 2, 2026

Leave out development package out of lock file #2512

Open

Lock incorrectly says it updates lock file when a dependency has an extra that does not exist #5050

Open

nichmor force-pushed the claude/remove-pypi-input-hash-9zwnN branch from 76d5042 to 2b7191e Compare January 3, 2026 17:34

misc: move to oncecell to reuse the same structs

08b5efe

nichmor force-pushed the claude/remove-pypi-input-hash-9zwnN branch from 2b7191e to 08b5efe Compare January 5, 2026 15:55

misc: share the cache between runs

dd92288

nichmor added 6 commits January 6, 2026 18:19

misc: collapse multiple arguments in a ctx struct

c1fe577

misc: fix tests conversion

95ca43f

misc: remove unused code

830ccd6

misc: add temporary basedpyright ignore

ac772a2

misc: add dynamic-source-deps example

612e7a4

misc: remove rayon comment

67dba34

nichmor added the test:extra_slow Run the extra slow tests label Jan 7, 2026

nichmor added 2 commits January 7, 2026 13:06

misc: add some tests for dynamic metadata

218653a

Merge branch 'main' into claude/remove-pypi-input-hash-9zwnN

ed8c619

nichmor marked this pull request as ready for review January 7, 2026 11:07

tdejager requested changes Jan 7, 2026

View reviewed changes

nichmor commented Jan 7, 2026

View reviewed changes

nichmor added 2 commits January 7, 2026 17:39

misc: simplify caching struct

95990e2

misc: add caching for static metadata

e88f96c

nichmor added 2 commits January 8, 2026 12:26

misc: treat dynamic version as an indicator to build dynamic metadata

090c2a6

misc: cache static metadata

edd3df7

nichmor added 3 commits January 8, 2026 13:01

misc: add ribasim example

7463ad9

misc: when dynamic, build requirements

33df41b

misc: update pixi.lock file

eb50586

tdejager requested changes Jan 8, 2026

View reviewed changes

ruben-arts mentioned this pull request Jan 8, 2026

Pixi lockfile v7 bump #5248

Open

feat: remove input-hash for PyPI dependencies #5174

Are you sure you want to change the base?

feat: remove input-hash for PyPI dependencies #5174

Conversation

nichmor commented Dec 19, 2025

Description

How Has This Been Tested?

AI Disclosure

Checklist:

Uh oh!

baszalmstra left a comment

Choose a reason for hiding this comment

Uh oh!

tdejager commented Dec 19, 2025

Uh oh!

nichmor commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dynamic Metadata (minimal-project with hatch_build.py, setup-project with setup.py)

Static Metadata (minimal-project without hatch_build.py)

Uh oh!

tdejager Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

nichmor Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nichmor Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

nichmor Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

ruben-arts commented Jan 7, 2026

Uh oh!

nichmor commented Jan 7, 2026

Dynamic Metadata (minimal-project with hatch_build.py, setup-project with setup.py)

Static Metadata (minimal-project without hatch_build.py)

Uh oh!

tdejager Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

tdejager Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

tdejager commented Jan 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nichmor commented Jan 6, 2026 •

edited

Loading