-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Add virtual information on DAG edges #35258
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add virtual information on DAG edges #35258
Conversation
b88123f to
562afeb
Compare
9875a5c to
84d1d48
Compare
84d1d48 to
8471718
Compare
8471718 to
d9bf846
Compare
d9bf846 to
d8ae02c
Compare
| self.parent = parent | ||
| self.spec = spec | ||
| self.deptypes = dp.canonical_deptype(deptypes) | ||
| self.parameters = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it really necessary to have a dict here? Personally I would keep redundancy & the number of indirections small so that we don't hit even more Python slowness. (One of the bottlenecks is creating the edges when deserializing the db, so if a normal prop works that's likely better than a string and hashmap lookup -- ok, to be fair, we do go from json string -> dict -> props, so in a sense it could also be faster)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using properties is a step towards adding custom attributes on edges. This is needed for "use variants" as proposed in spack/seps#2 and can be implemented on top of #35322, so 2 PRs from here.
lib/spack/spack/spec.py
Outdated
| """ | ||
| possible_virtuals = [x for x in spec.package.dependencies if Spec(x).virtual] | ||
| for vspec in possible_virtuals: | ||
| if vspec in spec: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you want to restrict to direct deps here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That should be done already, since the loop below selects only outgoing edges from the root.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, but in case vspec is not a direct dep you walk the whole dag? And what if it is a dependency of a dependency?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Turned out the reconstruction was incomplete. See 3f1205a
d8ae02c to
6ad4286
Compare
826f5ac to
a8521cd
Compare
6ad4286 to
f5eb6ac
Compare
This works for both the new and the old concretizer. Also, added type hints to involved functions.
3f1205a to
83e5c45
Compare
`spec_clauses()` attempts to look up package information for concrete specs in order to determine which virtuals they may provide. This fails for renamed/deleted dependencies of buildcaches and installed packages. This will eventually be fixed by #35258, which adds virtual information on edges, but we need a workaround to make older buildcaches usable. - [x] make an exception for renamed packages and omit their virtual constraints - [x] add a note that this will be solved by adding virtuals to edges
`spec_clauses()` attempts to look up package information for concrete specs in order to determine which virtuals they may provide. This fails for renamed/deleted dependencies of buildcaches and installed packages. This will eventually be fixed by #35258, which adds virtual information on edges, but we need a workaround to make older buildcaches usable. - [x] make an exception for renamed packages and omit their virtual constraints - [x] add a note that this will be solved by adding virtuals to edges
`spec_clauses()` attempts to look up package information for concrete specs in order to determine which virtuals they may provide. This fails for renamed/deleted dependencies of buildcaches and installed packages. This will eventually be fixed by #35258, which adds virtual information on edges, but we need a workaround to make older buildcaches usable. - [x] make an exception for renamed packages and omit their virtual constraints - [x] add a note that this will be solved by adding virtuals to edges
`spec_clauses()` attempts to look up package information for concrete specs in order to determine which virtuals they may provide. This fails for renamed/deleted dependencies of buildcaches and installed packages. This will eventually be fixed by #35258, which adds virtual information on edges, but we need a workaround to make older buildcaches usable. - [x] make an exception for renamed packages and omit their virtual constraints - [x] add a note that this will be solved by adding virtuals to edges
`spec_clauses()` attempts to look up package information for concrete specs in order to determine which virtuals they may provide. This fails for renamed/deleted dependencies of buildcaches and installed packages. This will eventually be fixed by spack#35258, which adds virtual information on edges, but we need a workaround to make older buildcaches usable. - [x] make an exception for renamed packages and omit their virtual constraints - [x] add a note that this will be solved by adding virtuals to edges
`spec_clauses()` attempts to look up package information for concrete specs in order to determine which virtuals they may provide. This fails for renamed/deleted dependencies of buildcaches and installed packages. This will eventually be fixed by #35258, which adds virtual information on edges, but we need a workaround to make older buildcaches usable. - [x] make an exception for renamed packages and omit their virtual constraints - [x] add a note that this will be solved by adding virtuals to edges
see #34821
fixes #34886
Currently virtual dependencies are used only during concretization to select a provider. No annotation is kept on edges to reflect that a leaf node is providing a virtual dependency. This PR adds virtual information on edges, so we can keep track of which spec is providing what virtual both in memory and in the JSON / YAML representation.
The way edge attributes are added is generic, so to reuse the same representation later when the
use_variantwill be implemented, see spack/seps#2. A sample representation is:Modifications:
parametersattribute toDependencySpec, which can hold arbitrary attributesSpec.to_node_dictto output information on virtuals in the `"dependencies" sectionDependencySpecto hold an arbitrary number of attributes (98b9806)