Registry - Potential OCI Artifacts user experience

[majastrz]

Goals

This is intended to provide an approximation of the Bicep registry experience IF we chose OCI Artifacts as the package manager and implement the integration. The existence of this issue DOES NOT indicate that OCI was selected as the implementation of the Bicep Registry. (See #2128 for details about other candidates.)

Gallery Experience

TBD

CLI

bicep build

Since Bicep modules can contain references to modules published to an artifact registry, the artifact contents may not exist on the local system. If the Bicep file contains references to external modules, bicep build pulls the referenced artifacts before type checking and code generation and stores them on the local file system. (In the compiler pipeline, the pull step occurs after parsing but before type checking is done.)

By default, the artifact cache is located at %USERPROFILE%\.bicep\artifacts on Windowsand ~/.bicep/artifacts on Linux and Mac. The location can be customized via the BICEP_ARTIFACTS environment variable. (If the language server and CLI tools are pointed to different artifact cache paths, they will each download a separate copy of each referenced artifact.)

Note: This is a custom mechanism that we will have to build ourselves. It will have to account for possibility of concurrent pull operations.

Open Question: Do we need to expose a setting for the artifact cache path in bicepconfig.json like NuGet does?

bicep pull

In certain usecases (Docker and some CI systems), the restore and build operations need to be separated. This can be accomplished as follows:

1. bicep pull
2. bicep build --no-pull main.bicep

Open Question: What verb do we use for this command? (NPM and Go use install. .Net/MSBuild use restore. Docker/OCI use pull.)

Open Question: Docker has been aggressively throttling anonymous artifact/image pull requests since late 2020. Do we need to support authentication even for public registries as a result.

Reference module from an artifact

To reference a module from a package, the opens a new or existing Bicep file in VS code and types one or more declarations like the following:

module mod 'majastrzoci.azurecr.io/bicep/modules/test:0.1-alpha' = {
  ...
}

OCI does not have a concept of a package feed like NuGet does. Instead, references are made directly via a URI. The URI has the following components:

• majastrzoci.azurecr.io - URI of the registry
• bicep/modules/test - Identifies the repository. A repository is a collection of artifacts with the same name but different tags.
• 0.1-alpha - Identifies the artifact tag, which is similar to a package version in NuGet. (However, the same image/artifact content may have multiple tags or may be untagged.)

See https://docs.microsoft.com/en-us/azure/container-registry/container-registry-concepts for more details.

Open Question: This example covers the most minimal syntax we could have. The syntax proposals for referencing external modules are tracked in #3186.

OCI also allows referencing artifacts by their SHA256 digest. The syntax for this would look like the following:

module mod 'majastrzoci.azurecr.io/bicep/modules/test@sha256:07601524fed07e648d40018070ea8927acb9d2bc695e9ebd3566ac113b98dda9' = {
  ...
}

Open Question: Should we even support references by digest? In otherwords, should we allow references to untagged artifacts?

Open Question: Even if we decide to support digest references, do we need to offer completions for digests?

No mechanism exists to enumerate all possible registry URIs, which means we will NOT be able to provide completions for the registry URI portion of the module type string. Once the registry URI is typed in (and auth is setup), we will be able to provide completions for the image name and tags.

Open Question: The lack of full completion experience suggests that we should develop a custom mechanism to configure the list of registries separately from the .bicep file and reference them by alias.

If the current module has any external module references, the language server queues up a pull action in the background. The background process will check the package cache for the artifact name and tag. If missing, the package will be downloaded from the registry.

The background pull operation does not happen instantly. Until the artifact is in the local artifact cache, accurate type information is not available:

• The language server falls back to the widest possible type for the given context. (any or object)
• A warning is shown on the module ref indicating that the artifact is not yet available.
• Property name and property access completions involving this particular module are unavailable.
• The language server remains otherwise functional.

If/when restore fails:

• An error is shown on the module ref explaing the type of failure.
• The language server remains otherwise functional.

Once the background pull operation is finished successfully, the language server recompiles the module to make use of the downloaded type info (we can reuse the parse tree). Property name and property access completions are now available for the module.

Open Question: Where do we expose the OCI logs? In VS code "Output" pane under "Bicep Package Manager"?

Setup private registry

Several registries support OCI artifacts. A good list is available at https://oras.land/implementors/. An Azure Container Registry can be easily created via the Azure Portal or a .bicep file.

Open Question: How will the auth work for registries? Is there a standard extensibility model or do we need to custom build it?

Setup local registry

Using a local docker installation, you can run the registry image:

docker run -it --rm -p 5000:5000 registry

This makes a non-persistent registry available at localhost:5000.

Create package

OCI artifacts are either pushed from local file system to a registry or pulled from the registry down to the local file system. (It is also possible to transfer artifacts between registries, but it's less relevant here.) There is no "package" concept that is a single-file representation of all the layers in an OCI artifact that would be equivalent to a .nupkg in NuGet or similar in other package managers.

Push artifact to registry

An artifact can be pushed to the registry via the bicep push <file> command. The command performs the following operations:

• Pull referenced artifacts to populate the local cache (can opt-out with --no-pull)
• Validate module (including configured linter rules)
• Prepare artifact content for publishing in a temp location
• Push to the registry

The artifact contents will be as follows:

• The main module.
• The local modules referenced by the main module.
• External modules referenced by the main and local modules become dependencies of the artifact.

A more detailed discussion about package contents is tracked in #3266. The discussion of module metadata lives in #3187.

Open Question: How do we express the dependency on other artifacts? Do we implement our own custom metadata to annotate dependencies or do we lean into the layering capabilities of OCI? (The former has implications to artifact pull latency.)

Open Question: OCI is completely generic and requires only artifact name and tag. What other metadata do we need?

Open Question: How to encode min/max Bicep version in the artifact? Can we add our own custom annotations to the manifest?

Sign artifact

TBD

[majastrz]

This will be updated in the future after discussions with OCI SMEs.

[stan-sz]

OCI does not have a concept of a package feed like NuGet does. Instead, references are made directly via a URI. The URI has the following components:

Imagine an issue of importing a bicep module from one ACR to another (e.g. airgapped). The problem is that if the references to the dependent modules are hardcoded then simply migrating is not enough, as also the contents need to be updated. We are facing a similar problem with helm charts and the problem is well described at https://stevelasker.blog/2020/10/21/is-it-time-to-change-default-registry-references/ and a discussion is at opencontainers/artifacts#29. @majastrz - can you think of a way to make the ACR hostname configurable (through command line or env variable) so bicep modules are transferrable between registries?

Bonus question: how one can transfer a bicep module with all dependent modules from one registry to another without the need to discover all dependent modules?

[majastrz]

Imagine an issue of importing a bicep module from one ACR to another (e.g. airgapped). The problem is that if the references to the dependent modules are hardcoded then simply migrating is not enough, as also the contents need to be updated. We are facing a similar problem with helm charts and the problem is well described at https://stevelasker.blog/2020/10/21/is-it-time-to-change-default-registry-references/ and a discussion is at opencontainers/artifacts#29. @majastrz Marcin Jastrzebski FTE - can you think of a way to make the ACR hostname configurable (through command line or env variable) so bicep modules are transferrable between registries?

This is a great point. To support these scenarios, we would need to separate the registry URI from the Bicep files themselves. This is similar to how NuGet configures sources via the NuGet.config file. If we proceed with OCI as the registry implementation, then we could implement a similar mechanism. We already have a bicepconfig.json that could be extended for this purpose or we could introduce a new file. Regardless, it would also require the ability to override via command line or env vars (could work like the nuget sources command).

My preference is for a config file based approach (with cmd overrides) instead of a purely cmd-based solution to make it possible for the language server to consume this information and provide a good completion experience for external modules.

Question: Do you always point to a single ACR? Or are there ever cases where you're pulling images/artifacts from multiple ACRs?

Bonus question: how one can transfer a bicep module with all dependent modules from one registry to another without the need to discover all dependent modules?

The idiomatic OCI method of dealing with dependencies appears to be to build them into the artifact at bicep push time as separate file layers. (It's a variation on option 2 in #3266.) This way you could just transfer the artifact to a different registry and wouldn't need to worry about dependencies. Similarly bicep pull latency would likely improve because we'd only need to pull 1 artifact per external module reference without having to do secondary pulls to obtain the closures of all the dependencies.

[stan-sz]

Question: Do you always point to a single ACR? Or are there ever cases where you're pulling images/artifacts from multiple ACRs?

To avoid disruption or uncontrolled variation of the dependencies (e.g. tag update), we pull all dependencies into a test ACR first and deploy to test environments from there. Later in the release pipeline import the exact same versions to production ACR and run deploy to production environments from there. The ultimate test if all dependencies have been correctly captured and imported to production ACR is an isolated environment, where nothing can be pulled from outside.

[majastrz]

If most users would end up with a single registry, then maybe we should have a concept of a "default" registry. Then, the references could become my/amazing/artifact:myTag instead of myregistry.azurecr.io/my/amazing/artifact:myTag or myRegistryAlias:my/amazing/artifact:myTag.

[stan-sz]

Fair point, the "default" registry could be the current registry, while still have the ability to pull other modules from public sources within the same template.
I suggest reaching out to @SteveLasker for guidance on designing OCI artifact handling.

[majastrz]

Yeah, we'd support the default and additional registries and ensure the references in the bicep file are/can be registry uri agnostic.

[rouke-broersma]

I think there is a danger in default registries in that where the image comes from is abstracted away so it is not immediately obvious. What if you forget to set your default registry? Suddenly you might be downloading your packages from 'a public default' registry instead of your private registry. This helps create supply chain attacks which we've seen a lot of reports on lately.

If the option of a default registry gets added there should not be any 'default for the default' imo. And I think it should still be allowed to be explicit about your registry location if you wish.

[majastrz]

@SteveLasker given your article at https://stevelasker.blog/2020/10/21/is-it-time-to-change-default-registry-references/, I'd love to hear/see your thoughts on how we should approach referencing Bicep modules from an OCI registry 😊