feat(compartment-mapper)!: new hooks & type overhaul #2988

boneskull · 2025-10-09T23:48:39Z

This PR supersedes #2929 which should now be considered abandoned.

This PR targets #2915, if you didn't notice.

Description

This PR is "complete" and ready for review. It makes a significant overhaul to types and adds new options to several public APIs.

Description of changes to Types

The types around CompartmentMapDescriptor have been vastly expanded to reflect the different "flavors" of CompartmentMapDescriptor, CompartmentDescriptor, ModuleDescriptor (now ModuleDescriptorConfiguration to differentiate it from ses' ModuleDescriptor), and the formatting of the compartment names (keys).
CompartmentDescriptor.label is now a canonical name.
CompartmentDescriptor.path is removed
CompartmentDescriptor.compartments is removed
Added many type guards for validation of these differing types
- Validation of different types of compartment map
- Stricter validation
- Validation of all types of compartment descriptors, module descriptor configurations, etc.

New Default Behavior in `mapNodeModules()`

Currently, mapNodeModules() avoids adding a ModuleDescriptor (ModuleDescriptorConfiguration) to a nascent CompartmentDescriptor's modules prop if policy disallows it. This PR changes the behavior to consider policy & excise the dependency from the Node itself before the (dependency) Graph is translated into a CompartmentDescriptor. This has implications around what sort of errors are thrown when Compartment A attempts to load a Compartment B to which it has no access (see note about policy tests below).

Other Notes

Since CompartmentDescriptor.compartments has been removed, that means implicitly allowing dynamic requires of parent/ancestor Compartments from a given Compartment via absolute path is no longer supported. That's fine, since:
- This should be explicit in policy anyway
- @lavamoat/node was the only user of this feature and no longer needs it
The entry compartment now has a canonical name of $root$ . It is valid within PackagePolicy; i.e. some other package can be allowed to access the $root$ compartment.
Many tests in test/policy.test.ts needed to change because of how mapNodeModules' packageDependencies hook works. Instead of rejecting module descriptors (ModuleDescriptorConfigurations) based on policy, we reject entire dependencies before they can be "digested" into module descriptors. This results in ScopeDescriptors not being populated, in addition to ModuleDescriptorConfigurations, and causes downstream effects. Different exceptions are thrown at different times, but the intent of these tests doesn't deviate from "ensure this naughty behavior is not allowed."

Security Considerations

None that I'm aware of.

Scaling Considerations

Any consumer providing a resource-intensive blocking hook to @endo/compartment-mapper gets the performance they deserve.

Documentation Considerations

This is a breaking change to the contents of the archives as well as a breaking change to the types. It is not a breaking change to the archive format, but it will contain different values (instead of packageName-v<version> the compartment descriptors will be keyed on the canonical name).

Testing Considerations

There are some unit tests but we need more integration tests.

Compatibility Considerations

The breaking changes mentioned above, but it also solves like ten different problems @lavamoat/node was having trying to get things to work. It significantly improves ecosystem compatibility and will improve the performance of @lavamoat/node.

Upgrade Considerations

Include *BREAKING*: in the commit message with migration instructions for any breaking change.
Update NEWS.md for user-facing changes.

This commit introduces a comprehensive universal hook system for @endo/compartment-mapper,
enabling extensible customization of module mapping, bundling, and policy enforcement.

BREAKING CHANGES: CompartmentDescriptor objects no longer have a path property and the label property is now a canonical name. Types have changed dramatically. Enhanced validation of CompartmentMapDescriptor objects.

New Options

`mapNodeModules()`

packageDataHook: Called once before translateGraph with data about all packages found while crawling node_modules. Receives a read-only Map<CanonicalName, PackageData> where each entry contains:
- name: Package name
- packageDescriptor: The package.json contents
- location: File URL to the package
- canonicalName: The canonical name used in policy
packageDependenciesHook: Called for each package's dependencies during graph translation, allowing dynamic modification of the dependency graph. Can add or remove dependencies from packages based on policy or other criteria.
unknownCanonicalNameHook: Called when policy references canonical names that don't exist in the dependency graph, with optional suggestions for typos/similar names.

`makeImportHookMaker()`

moduleSourceHook(): Called when module source objects are created (either from local files containing bytes, exit modules, or "error-type" sources)

`captureFromMap()`

packageConnectionsHook: Called during digest; surfaces "connections" between compartment descriptors

Type System Overhaul (`src/types/`)

Major Breaking Type Changes in `compartment-map-schema.ts`:

CompartmentDescriptor Interface Restructure:

BREAKING: Removed path property - compartment descriptors no longer track dependency paths
BREAKING: label property type changed from string to CanonicalName<U> - labels are now type-safe canonical names
BREAKING: Made CompartmentDescriptor generic with <T extends ModuleDescriptorConfiguration, U extends string> for better type safety
Added location: string as a required property for all compartment descriptors
Added optional sourceDirname as found in the sources

CompartmentMapDescriptor Genericization (Is That A Word? No):

BREAKING: CompartmentMapDescriptor is now generic: CompartmentMapDescriptor<T, Name, EntryName>
BREAKING: compartments property changed from Record<string, CompartmentDescriptor> to CompartmentDescriptors<T, Name>
BREAKING: entry property type changed from EntryDescriptor to EntryDescriptor<EntryName>
New specialized types: PackageCompartmentMapDescriptor, FileCompartmentMapDescriptor, and DigestedCompartmentMapDescriptor

Enhanced Module Descriptor System:

ModuleDescriptor is now ModuleDescriptorConfiguration to differentiate between it & the ModuleDescriptor from ses
Added ModuleDescriptorConfigurationCreator enum tracking module creation source ('link' | 'transform' | 'import-hook' | 'digest' | 'node-modules')
Enhanced BaseModuleDescriptorConfiguration with __createdBy property for debugging
Added ErrorModuleDescriptorConfiguration for deferred error handling
Improved type discrimination with ModuleDescriptorConfigurationKindToType utility type

New Type Infrastructure:

`canonical-name.ts` - Canonical Name Type System:

NEW: Type-level canonical name validation using template literal types
CanonicalName<S> - validates npm package name chains separated by '>' (e.g., "foo>bar", "@scope/pkg>dep")
ScopedPackageName and UnscopedPackageName for npm package name validation
SplitOnGt<S> and AllValidPackageNames<Parts> for compile-time canonical name parsing
IsCanonicalName<S> predicate type for conditional type logic

Enhanced Type Safety Features:

Compartment Descriptor Validation:

DigestedCompartmentDescriptor - restricted descriptor for archived compartment maps
Properties marked as never for digested descriptors: path, retained, scopes, parsers, types, __createdBy, sourceDirname
CompartmentDescriptorWithPolicy<T> - enforces policy presence where required

Module Configuration Type Safety:

ModuleDescriptorConfigurationKind union for module type discrimination
Type-safe module configuration creators with __createdBy tracking

Policy Integration:

Enhanced policy types in PackageCompartmentDescriptor with canonical name constraints
LiteralUnion usage for special canonical names (ATTENUATORS_COMPARTMENT, ENTRY_COMPARTMENT)
Policy-aware compartment descriptor types with enhanced validation

Type System Utilities:

`typescript.ts` Enhancements:

Enhanced LiteralUnion<L, B> for better literal type preservation
Type utilities supporting the new generic compartment map architecture
Moved Simplify from tests here (because it's useful dammit)

Enhanced Policy Validation

Policy Validation:

Unknown canonical name detection with suggestion system
Cross-reference policy resources against actual compartment contents
Detailed error reporting with path information for policy issues
Hook integration for custom policy validation logic

boneskull · 2025-10-10T03:10:10Z

Fix:

Error: ../compartment-mapper/src/policy-format.d.ts(9,76): error TS2304: Cannot find name 'WildcardPolicy'.
Error: ../compartment-mapper/src/policy-format.d.ts(16,81): error TS2304: Cannot find name 'SomePolicy'.

boneskull · 2025-10-16T22:21:23Z

policy.test.ts needs further fixes since it breaks prior behaviors.

The `PolicyItem` type was having compatibility issues with `SomePackagePolicy`; this changes `PolicyItem` to a non-distributive conditional such that a type argument of `void` defaults to the "base" type. End-users can still provide type arguments for custom `PolicyItem`s.

- Adds `src/guards.js` which will be used elsewhere (not just tests) - Adds `test/snapshot-utilities.js` which provides utilities for sanitizing snapshots - Update `ProjectFixture` to have an optional `entrypoint` property

This fixes `@endo/env-options` so that other packages can access its shipped type declarations.

…compartment-mapper

This commit introduces a comprehensive universal hook system for `@endo/compartment-mapper`, enabling extensible customization of module mapping, bundling, and policy enforcement. BREAKING CHANGES: `CompartmentDescriptor` objects no longer have a `path` property and the `label` property is now a canonical name. Types have changed dramatically. Enhanced validation of `CompartmentMapDescriptor` objects. ## Core Hook Infrastructure ### New Hook System Components: - `src/hooks.js`: Universal hook executor and configuration management - `src/types/hooks.ts`: Comprehensive TypeScript definitions for all hook types - Hook definitions for: `mapNodeModules`, `importLocation`, `link`, `loadLocation`, `digestCompartmentMap`, `captureFromMap`, & `loadFromMap` (many of these are shared) ### Hook Executor Features: - Support for single hooks, hook arrays (pipelines), and phased hooks (pre/post) - Type-safe hook parameter validation and return type checking - Error handling with wrapped exceptions for debugging - Default hook configuration merging and application - Logging integration for hook execution tracing ## Hook Integration Points ### `mapNodeModules` Hooks: - `packageDescriptor`: Called after parsing each package.json with package metadata - `packageDependencies`: Called for each package with its dependency list (supports filtering/modification) - `unknownCanonicalName`: Called when policy references unknown packages for validation/reporting ### `link` Hooks (available where `link()` is called): - `moduleSource`: Called when module source objects are created (either from local files containing `bytes`, exit modules, or "error-type" sources) ### `captureFromMap` Hooks - `packageConnections`: Called during digest; surfaces "connections" between compartment descriptors ## Type System Overhaul (`src/types/`) ### Major Breaking Changes in `compartment-map-schema.ts`: #### CompartmentDescriptor Interface Restructure: - **BREAKING**: Removed `path` property - compartment descriptors no longer track dependency paths - **BREAKING**: `label` property type changed from `string` to `CanonicalName<U>` - labels are now type-safe canonical names - **BREAKING**: Made `CompartmentDescriptor` generic with `<T extends ModuleDescriptorConfiguration, U extends string>` for better type safety - Added `location: string` as a required property for all compartment descriptors - Added optional `sourceDirname` as found in the sources #### CompartmentMapDescriptor Genericization: - **BREAKING**: `CompartmentMapDescriptor` is now generic: `CompartmentMapDescriptor<T, Name, EntryName>` - **BREAKING**: `compartments` property changed from `Record<string, CompartmentDescriptor>` to `CompartmentDescriptors<T, Name>` - **BREAKING**: `entry` property type changed from `EntryDescriptor` to `EntryDescriptor<EntryName>` - New specialized types: `PackageCompartmentMapDescriptor`, `FileCompartmentMapDescriptor`, and `DigestedCompartmentMapDescriptor` #### Enhanced Module Descriptor System: - `ModuleDescriptor` is now `ModuleDescriptorConfiguration` to differentiate between it & the `ModuleDescriptor` from `ses` - Added `ModuleDescriptorConfigurationCreator` enum tracking module creation source (`'link' | 'transform' | 'import-hook' | 'digest' | 'node-modules'`) - Enhanced `BaseModuleDescriptorConfiguration` with `__createdBy` property for debugging - Added `ErrorModuleDescriptorConfiguration` for deferred error handling - Improved type discrimination with `ModuleDescriptorConfigurationKindToType` utility type ### New Type Infrastructure: #### `canonical-name.ts` - Canonical Name Type System: - **NEW**: Type-level canonical name validation using template literal types - `CanonicalName<S>` - validates npm package name chains separated by '>' (e.g., "foo>bar", "@scope/pkg>dep") - `ScopedPackageName` and `UnscopedPackageName` for npm package name validation - `SplitOnGt<S>` and `AllValidPackageNames<Parts>` for compile-time canonical name parsing - `IsCanonicalName<S>` predicate type for conditional type logic #### `hooks.ts` - Universal Hook Type System: - `HookFn<TInput>` - generic hook function with input/output type safety - `HookConfiguration<T>` - utility type making all hook properties optional with pipeline support - `HookExecutorFn<T>` - type-safe hook executor supporting both regular and phased hooks - `PhasedHookDefinition` - pre/post hook phase support with type safety - Comprehensive hook definitions for all integration points: - `MapNodeModulesHooks` - package discovery and dependency processing - `MakeImportHookMakerHooks` - module source processing - `LinkHooks` - module linking operations - `LoadLocationHooks` - location loading operations - `DigestCompartmentMapHooks` - compartment map digestion - `CaptureFromMapHooks` - compartment map capture operations ### Enhanced Type Safety Features: #### Compartment Descriptor Validation: - `DigestedCompartmentDescriptor` - restricted descriptor for archived compartment maps - Properties marked as `never` for digested descriptors: `path`, `retained`, `scopes`, `parsers`, `types`, `__createdBy`, `sourceDirname` - `CompartmentDescriptorWithPolicy<T>` - enforces policy presence where required #### Module Configuration Type Safety: - `ModuleDescriptorConfigurationKind` union for module type discrimination - Type-safe module configuration creators with `__createdBy` tracking #### Policy Integration: - Enhanced policy types in `PackageCompartmentDescriptor` with canonical name constraints - `LiteralUnion` usage for special canonical names (`ATTENUATORS_COMPARTMENT`, `ENTRY_COMPARTMENT`) - Policy-aware compartment descriptor types with enhanced validation ### Type System Utilities: #### `typescript.ts` Enhancements: - Enhanced `LiteralUnion<L, B>` for better literal type preservation - Type utilities supporting the new generic compartment map architecture - Moved `Simplify` from tests here (because it's useful dammit) #### `external.ts` Hook Integration: - Re-exported all hook types from `hooks.ts` for public API - Enhanced options types with hook configuration support ## Comprehensive Test Coverage ### New Test Files: - `test/hook-executor.hooks.test.js`: Core hook executor functionality - `test/map-node-modules.hooks.test.js`: `mapNodeModules` hook integration - `test/make-import-hook-maker.hooks.test.js`: `makeImportHookMaker` hook integration - `test/apply-hook-defaults.hooks.test.js`: Hook configuration merging and defaults - [ ] TODO: `test/capture-from-map.hooks.test.ts`: `captureFromMap` hook integration ## Enhanced Policy Validation ### Policy Validation: - Unknown canonical name detection with suggestion system - Cross-reference policy resources against actual compartment contents - Detailed error reporting with path information for policy issues - Hook integration for custom policy validation logic ### Improved Error Handling: - Wrapped hook errors with context information - Detailed path reporting for policy validation failures - Suggestion system for likely intended canonical names

- This hook just dumps all known canonical names. - The `unknownCanonicalName` hook has changed its parameters.

I don't think we're going to need this; removing it for now since it just adds more complexity.

naugtur · 2025-10-29T17:16:45Z

packages/compartment-mapper/src/policy.js

+  label,
+  { policy, packagePolicy = undefined } = {},
+) => {
+  if (packagePolicy !== undefined) {


how would a package policy be already known but still go through this function? What is the point of doing it here instead of checking at call location?

naugtur · 2025-10-29T17:19:20Z

packages/compartment-mapper/src/policy.js

  } else {
-    defaultAttenuator =
-      compartmentDescriptors[ATTENUATORS_COMPARTMENT].policy.defaultAttenuator;
+    // TODO: the attenuators compartment must always have a non-empty policy; maybe throw if it doesn't


sounds good

naugtur · 2025-10-29T17:30:38Z

packages/compartment-mapper/src/node-modules.js

+  message,
+  log,
+}) => {
+  log(`WARN: Invalid resource ${q(canonicalName)} in policy: ${message}`);


This kind of things, at least in my experience with webpack, worked better if aggregated and displayed as a single warning. In a debug mode of sorts they could be used to generate proposed policy fixes.

naugtur · 2025-10-30T19:12:41Z

packages/compartment-mapper/src/link.js

+const DefaultCompartment = Compartment;

 /**
+ * TODO: can we just use `Object.hasOwn` instead?


It's a new addition to SES AFAIR

naugtur · 2025-10-31T12:28:49Z

packages/compartment-mapper/src/node-modules.js

  }
+
+  // #region packageDescriptor hook
+  if (packageDescriptorHook) {


I'm not sure this is useful, but it could be useful if it allowed altering the descriptor (eg. in case it contain something that was not supported or needed a fix.

naugtur · 2025-10-31T12:31:46Z

packages/compartment-mapper/src/capture-lite.js

    Compartment: CompartmentOption = DefaultCompartment,
    log = noop,
    preload = [],
+    packageConnectionsHook,


packageDependenciesHook and packageConnectionsHook are similar

I looked at both of these hooks and I think packageConnectionsHook could be replaced with packageDependenciesHook and the main difference is that one of them allows returning a mutated set.
If we could offer the same in digest, we'd have a single hook for both.

…riptorHook into a single packageData hook

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from 6c51178 to d2e827c Compare October 10, 2025 00:21

boneskull mentioned this pull request Oct 10, 2025

feat(compartment-mapper)!: create Compartment Map Transforms #2929

Closed

boneskull self-assigned this Oct 10, 2025

boneskull added enhancement New feature or request ecosystem-compatibility Tracks a compatibility issue in a third-party package or packages. labels Oct 10, 2025

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from d2e827c to ee56c40 Compare October 10, 2025 00:35

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from ee56c40 to 5e78f40 Compare October 16, 2025 22:18

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch 4 times, most recently from c37edbe to 1d864b5 Compare October 17, 2025 21:39

boneskull force-pushed the boneskull/force-load branch 2 times, most recently from 8b5ac3e to 38e98bf Compare October 21, 2025 21:05

boneskull added 10 commits October 21, 2025 14:26

fix(env-options): do not deny access to generated declarations

f3dbb03

This fixes `@endo/env-options` so that other packages can access its shipped type declarations.

chore(compartment-mapper): add @endo/env-options as dev dep to @endo/…

e5e59f0

…compartment-mapper

feat(compartment-mapper): add MapNodeModulesHooks.canonicalNames

ad9d81a

- This hook just dumps all known canonical names. - The `unknownCanonicalName` hook has changed its parameters.

chore(compartment-mapper): remove phased hooks

0a76c12

I don't think we're going to need this; removing it for now since it just adds more complexity.

chore(compartment-mapper): reformat

594bd26

chore(compartment-mapper): update snapshots

41733d8

chore(compartment-mapper): reformat

983ad2e

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from b21326c to 983ad2e Compare October 21, 2025 21:40

boneskull marked this pull request as ready for review October 22, 2025 18:11

naugtur added the lavamoat label Oct 23, 2025

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from a978be1 to 7916a39 Compare October 24, 2025 20:34

chore(compartment-mapper): refactor to plain old options

d73f3d9

boneskull force-pushed the boneskull/compartment-mapper-hook-system branch from 7916a39 to d73f3d9 Compare October 24, 2025 20:41

boneskull added a commit to LavaMoat/LavaMoat that referenced this pull request Oct 27, 2025

fix(node): update code to use latest version of endojs/endo#2988

66ccc47

naugtur reviewed Oct 29, 2025

View reviewed changes

naugtur reviewed Oct 30, 2025

View reviewed changes

naugtur reviewed Oct 31, 2025

View reviewed changes

boneskull changed the title ~~feat(compartment-mapper)!: universal hooks implementation~~ feat(compartment-mapper)!: new hooks Oct 31, 2025

boneskull changed the title ~~feat(compartment-mapper)!: new hooks~~ feat(compartment-mapper)!: new hooks & type overhaul Oct 31, 2025

boneskull mentioned this pull request Oct 31, 2025

chore(node): add support for webpack LavaMoat/LavaMoat#1593

Draft

chore(compartment-mapper): refactor canonicalNameHook and packageDesc…

36892b9

…riptorHook into a single packageData hook

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(compartment-mapper)!: new hooks & type overhaul #2988

feat(compartment-mapper)!: new hooks & type overhaul #2988

boneskull commented Oct 9, 2025 •

edited

Loading

Uh oh!

boneskull commented Oct 10, 2025 •

edited

Loading

Uh oh!

boneskull commented Oct 16, 2025

Uh oh!

naugtur Oct 29, 2025

Uh oh!

naugtur Oct 29, 2025

Uh oh!

naugtur Oct 29, 2025

Uh oh!

naugtur Oct 30, 2025

Uh oh!

naugtur Oct 31, 2025

Uh oh!

naugtur Oct 31, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat(compartment-mapper)!: new hooks & type overhaul #2988

Are you sure you want to change the base?

feat(compartment-mapper)!: new hooks & type overhaul #2988

Conversation

boneskull commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Description of changes to Types

New Default Behavior in mapNodeModules()

Other Notes

Security Considerations

Scaling Considerations

Documentation Considerations

Testing Considerations

Compatibility Considerations

Upgrade Considerations

New Options

mapNodeModules()

makeImportHookMaker()

captureFromMap()

Type System Overhaul (src/types/)

Major Breaking Type Changes in compartment-map-schema.ts:

CompartmentDescriptor Interface Restructure:

CompartmentMapDescriptor Genericization (Is That A Word? No):

Enhanced Module Descriptor System:

New Type Infrastructure:

canonical-name.ts - Canonical Name Type System:

Enhanced Type Safety Features:

Compartment Descriptor Validation:

Module Configuration Type Safety:

Policy Integration:

Type System Utilities:

typescript.ts Enhancements:

Enhanced Policy Validation

Policy Validation:

Uh oh!

boneskull commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boneskull commented Oct 16, 2025

Uh oh!

naugtur Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

naugtur Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

naugtur Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

naugtur Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

naugtur Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

naugtur Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

boneskull commented Oct 9, 2025 •

edited

Loading

New Default Behavior in `mapNodeModules()`

`mapNodeModules()`

`makeImportHookMaker()`

`captureFromMap()`

Type System Overhaul (`src/types/`)

Major Breaking Type Changes in `compartment-map-schema.ts`:

`canonical-name.ts` - Canonical Name Type System:

`typescript.ts` Enhancements:

boneskull commented Oct 10, 2025 •

edited

Loading

naugtur Oct 31, 2025 •

edited

Loading